If pub_timestamp could possibly match I'd suggest making it a timeuuid type instead. With the above schema it's not a failure or data loss if the timestamp is duplicated - your writes all probably made it - the duplicates just got overwritten.
On Tue, Jun 14, 2016 at 9:40 PM, linbo liao <llbg...@gmail.com> wrote: > I am not sure, but looks it will cause the update other than insert. If it > is true, the only way is request includes IF NOT EXISTS, inform the > client it failed? > > Thanks, > Linbo > > 2016-06-15 10:59 GMT+08:00 Ben Slater <ben.sla...@instaclustr.com>: > >> Is it possible that your pub_timestamp values are colliding (which would >> result in an update rather than an insert)? >> >> On Wed, 15 Jun 2016 at 12:55 linbo liao <llbg...@gmail.com> wrote: >> >>> Hi, >>> >>> I use Erlang driver to send data to Cassandra, do testing at local >>> environment meet data lost issue. I have no idea what step is wrong. >>> >>> *Environment:* >>> >>> 1. Ubuntu 12.04 LTS x64bit >>> 2. Cassandra 3.5 single instance, not a cluster, installed via the >>> offical installation document, and didn't change any configuration except >>> enable the authenticator and authorizer. >>> 3. Cassandra binary protocol v4 >>> 3. Latest Erlang driver https://github.com/matehat/cqerl >>> 4. Erlang OTP 18.3 >>> >>> *Schema:* >>> >>> > DESCRIBE TABLE message.history >>>> >>>> CREATE TABLE message.history ( >>>> appkey text, >>>> pub_date text, >>>> pub_timestamp timestamp, >>>> apns text, >>>> message blob, >>>> message_id text, >>>> pub_method smallint, >>>> qos smallint, >>>> recv_type smallint, >>>> topic text, >>>> PRIMARY KEY ((appkey, pub_date), pub_timestamp) >>>> ) WITH CLUSTERING ORDER BY (pub_timestamp ASC) >>>> AND bloom_filter_fp_chance = 0.01 >>>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} >>>> AND comment = '' >>>> AND compaction = {'class': >>>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', >>>> 'max_threshold': '32', 'min_threshold': '4', >>>> 'unchecked_tombstone_compaction': 'true'} >>>> AND compression = {'chunk_length_in_kb': '64', 'class': >>>> 'org.apache.cassandra.io.compress.LZ4Compressor'} >>>> AND crc_check_chance = 1.0 >>>> AND dclocal_read_repair_chance = 0.1 >>>> AND default_time_to_live = 0 >>>> AND gc_grace_seconds = 3600 >>>> AND max_index_interval = 2048 >>>> AND memtable_flush_period_in_ms = 0 >>>> AND min_index_interval = 128 >>>> AND read_repair_chance = 0.0 >>>> AND speculative_retry = '99PERCENTILE'; >>>> >>> >>> >>> *Issue:* >>> >>> The client send 6 insert request to Server (ttl is 1000s), check the >>> TCP package every thing works fine, but data will miss random at Server >>> side(select insert data was missing). >>> >>> >>> > select * from history; >>>> >>>> appkey | pub_date | >>>> pub_timestamp | >>>> apns | message >>>> | message_id | pub_method | >>>> qos | recv_type | topic >>>> >>>> --------------------------+------------+---------------------------------+--------------------------------------------------------------+-------------------------------------------- >>>> >>>> ----------------------------------+----------------------+------------+-----+-----------+----------------- >>>> 52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 >>>> 09:16:24.700000+0000 >>>> | | >>>> 0x68656c6c6f20746f20616c6961732066726f6d207 >>>> 075626c697368325f746f5f616c696173 | 589020307122032641 | 2 >>>> | 1 | 1 | alias_mqttc_sub >>>> 52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 >>>> 09:16:24.817000+0000 | >>>> {"aps":{"sound":"bingbong.aiff","badge":3,"alert":"douban"}} >>>> | 0x7b22612 >>>> 23a2066726f6d207075626c697368327d | 11833652203486491113 | 2 >>>> | 1 | 0 | t2thi >>>> 52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 >>>> 09:16:24.818000+0000 >>>> | | >>>> 0x66726f6d20707974686f6e | 589020307579211776 | 2 >>>> | 1 | 0 | testtopic2 >>>> 52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 >>>> 09:16:24.890000+0000 | >>>> {"aps":{"sound":"bingbong.aiff","badge":3,"alert":"douban"}} | >>>> 0x66726f6d207075626c69736832 | 589020307814092800 | 2 >>>> | 1 | 0 | testtopic2 >>>> 52fcc04c4dc903d66d6f8f92 | 2016-06-13 | 2016-06-13 >>>> 09:16:25.024000+0000 >>>> | | >>>> 0x68656c6c6f20746f20616c696173 | 589020307818287105 | 2 >>>> | 1 | 1 | mytestalias1 >>>> >>> >>> >>> *TCP package inserted succeed flow:* >>> >>> 17:16:24.818210 IP localhost.38918 > localhost.9042: Flags [P.], seq >>>> 1953472577:1953472814, ack 1963420469, win 530, options [nop,nop,TS val >>>> 72982868 ecr 72940042], length 237 >>>> 0x0000: 4500 0121 e5b1 4000 4006 5623 7f00 0001 E..!..@.@.V#.... >>>> 0x0010: 7f00 0001 9806 2352 746f a041 7507 6b35 ......#Rto.Au.k5 >>>> 0x0020: 8018 0212 ff15 0000 0101 080a 0459 a154 .............Y.T >>>> 0x0030: 0458 fa0a *0400 0000 0a00 0000 e400 1005* >>>> .X.............. >>>> 0x0040: 6e06 c1fc 222c 813f 6228 61c5 7364 6500 n...",.?b(a.sde. >>>> 0x0050: 0105 000b 0000 0018 3532 6663 6330 3463 ........52fcc04c >>>> 0x0060: 3464 6339 3033 6436 3664 3666 3866 3932 4dc903d66d6f8f92 >>>> 0x0070: 0000 000a 3230 3136 2d30 362d 3133 0000 ....2016-06-13.. >>>> 0x0080: 0008 0000 0155 490c 3571 0000 0002 0001 .....UI.5q...... >>>> 0x0090: 0000 0002 0002 0000 0014 3131 3833 3336 ..........118336 >>>> 0x00a0: 3532 3230 3334 3836 3439 3131 3133 0000 52203486491113.. >>>> 0x00b0: 0014 7b22 6122 3a20 6672 6f6d 2070 7562 ..{"a":.from.pub >>>> 0x00c0: 6c69 7368 327d 0000 003c 7b22 6170 7322 lish2}...<{"aps" >>>> 0x00d0: 3a7b 2273 6f75 6e64 223a 2262 696e 6762 :{"sound":"bingb >>>> 0x00e0: 6f6e 672e 6169 6666 222c 2262 6164 6765 ong.aiff","badge >>>> 0x00f0: 223a 332c 2261 6c65 7274 223a 2264 6f75 ":3,"alert":"dou >>>> 0x0100: 6261 6e22 7d7d 0000 0005 7432 7468 6900 ban"}}....t2thi. >>>> 0x0110: 0000 0200 0000 0000 0400 0003 e800 0000 ................ >>>> 0x0120: 64 d >>>> 17:16:24.819204 IP localhost.9042 > localhost.38918: Flags [P.], seq >>>> 1:14, ack 237, win 630, options [nop,nop,TS val 72982868 ecr 72982868], >>>> length 13 >>>> 0x0000: 4500 0041 1733 4000 4006 2582 7f00 0001 E..A.3@.@.%..... >>>> 0x0010: 7f00 0001 2352 9806 7507 6b35 746f a12e ....#R..u.k5to.. >>>> 0x0020: 8018 0276 fe35 0000 0101 080a 0459 a154 ...v.5.......Y.T >>>> 0x0030: 0459 a154 *8400 0000 0800 0000 0400 0000* >>>> .Y.T............ >>>> 0x0040: * 01 * . >>>> 17:16:24.819238 IP localhost.38918 > localhost.9042: Flags [.], ack 14, >>>> win 530, options [nop,nop,TS val 72982868 ecr 72982868], length 0 >>>> 0x0000: 4500 0034 e5b2 4000 4006 570f 7f00 0001 E..4..@.@.W..... >>>> 0x0010: 7f00 0001 9806 2352 746f a12e 7507 6b42 ......#Rto..u.kB >>>> 0x0020: 8010 0212 fe28 0000 0101 080a 0459 a154 .....(.......Y.T >>>> 0x0030: 0459 a154 .Y.T >>>> >>> >>> >>> >>> >>> *The missing TCP flow:* >>>> >>>> 17:16:24.820392 IP localhost.49128 > localhost.9042: Flags [P.], seq >>>> 202:396, ack 14, win 530, options [nop,nop,TS val 72982868 ecr 72982840], >>>> length 194 >>>> 0x0000: 4500 00f6 370c 4000 4006 04f4 7f00 0001 E...7.@.@....... >>>> 0x0010: 7f00 0001 bfe8 2352 83eb 4b09 bfe6 87a4 ......#R..K..... >>>> 0x0020: 8018 0212 feea 0000 0101 080a 0459 a154 .............Y.T >>>> 0x0030: 0459 a138 *0400 0000 0a00 0000 b900 1005* >>>> .Y.8............ >>>> 0x0040: 6e06 c1fc 222c 813f 6228 61c5 7364 6500 n...",.?b(a.sde. >>>> 0x0050: 0105 000b 0000 0018 3532 6663 6330 3463 ........52fcc04c >>>> 0x0060: 3464 6339 3033 6436 3664 3666 3866 3932 4dc903d66d6f8f92 >>>> 0x0070: 0000 000a 3230 3136 2d30 362d 3133 0000 ....2016-06-13.. >>>> 0x0080: 0008 0000 0155 490c 3572 0000 0002 0001 .....UI.5r...... >>>> 0x0090: 0000 0002 0002 0000 0014 3131 3833 3336 ..........118336 >>>> 0x00a0: 3532 3230 3334 3836 3439 3131 3132 0000 52203486491112.. >>>> 0x00b0: 0020 6672 6f6d 2070 7974 686f 6e20 7769 ..from.python.wi >>>> 0x00c0: 7468 2067 6976 656e 206d 6573 7361 6765 th.given.message >>>> 0x00d0: 4964 0000 0000 0000 000a 7465 7374 746f Id........testto >>>> 0x00e0: 7069 6332 0000 0002 0000 0000 0004 0000 pic2............ >>>> 0x00f0: 03e8 0000 0064 .....d >>>> 17:16:24.834414 IP localhost.9042 > localhost.49128: Flags [P.], seq >>>> 14:27, ack 396, win 579, options [nop,nop,TS val 72982872 ecr 72982868], >>>> length 13 >>>> 0x0000: 4500 0041 f907 4000 4006 43ad 7f00 0001 E..A..@.@.C..... >>>> 0x0010: 7f00 0001 2352 bfe8 bfe6 87a4 83eb 4bcb ....#R........K. >>>> 0x0020: 8018 0243 fe35 0000 0101 080a 0459 a158 ...C.5.......Y.X >>>> 0x0030: 0459 a154 *8400 0000 0800 0000 0400 0000* >>>> .Y.T............ >>>> 0x0040: *01* . >>>> 17:16:24.834452 IP localhost.49128 > localhost.9042: Flags [.], ack 27, >>>> win 530, options [nop,nop,TS val 72982872 ecr 72982872], length 0 >>>> 0x0000: 4500 0034 370d 4000 4006 05b5 7f00 0001 E..47.@.@....... >>>> 0x0010: 7f00 0001 bfe8 2352 83eb 4bcb bfe6 87b1 ......#R..K..... >>>> 0x0020: 8010 0212 fe28 0000 0101 080a 0459 a158 .....(.......Y.X >>>> 0x0030: 0459 a158 .Y.X >>>> >>> >>> >>> Check the TCP flow, the data send and ack succeed, but not sure why the >>> data doesn't insert in Server side successful. Any tool or method can I >>> debug the Cassandra Server, or am I missing something? >>> >>> Thanks, >>> Linbo >>> >> -- >> ———————— >> Ben Slater >> Chief Product Officer >> Instaclustr: Cassandra + Spark - Managed | Consulting | Support >> +61 437 929 798 >> > >
