> > read_repair_chance=1.000000 AND There’s your problem.
When read repair is active for a read request the coordinator will over read to all UP replicas. Your client request will only block waiting for the one request (the data request), the rest of the repair will happen in the background. Setting this to 1.0 will mean it’s active across the entire cluster for each read. Change read_repair_chance to 0 and set dclocal_read_repair_chance to 0.1 so that read repair will only happen local to the DC you are connected to. Hope that helps. A ----------------- Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 12/05/2014, at 5:37 pm, DuyHai Doan <doanduy...@gmail.com> wrote: > Ins't read repair supposed to be done asynchronously in background ? > > > On Mon, May 12, 2014 at 2:07 AM, graham sanderson <gra...@vast.com> wrote: > You have a read_repair_chance of 1.0 which is probably why your query is > hitting all data centers. > > On May 11, 2014, at 3:44 PM, Mark Farnan <devm...@petrolink.com> wrote: > > > Im trying to understand READ load in Cassandra across a multi-datacenter > > cluster. (Specifically why it seems to be hitting more than one DC) and > > hope someone can help. > > > > From what Iím seeing here, a READ, with Consistency LOCAL_ONE, seems to > > be hitting All 3 datacenters, rather than just the one Iím connected to. > > I see 'Read 101 live and 0 tombstoned cells' from EACH of the 3 DC"s in > > the trace, which seems, wrong. > > I have tried every Consistency level, same result. This also is same > > from my C# code via the DataStax driver, (where I first noticed the issue). > > > > Can someone please shed some light on what is occurring ? Specifically I > > dont' want a query on one DC, going anywhere near the other 2 as a rule, as > > in production, these DC's will be accross slower links. > > > > > > Query: (NOTE: Whilst this uses a kairosdb table, i'm just playing with > > queries against it as it has 100k columns in this key for testing). > > > > cqlsh:kairosdb> consistency local_one > > Consistency level set to LOCAL_ONE. > > > > cqlsh:kairosdb> select * from data_points where key = > > 0x6d61726c796e2e746573742e74656d70340000000145b514a400726f6f6d3d6f66666963653a > > limit 1000; > > > > ... Some return data rows listed here which I've removed .... > > > > <CassandraQuery.txt> > > Query Respose Trace: > > > > activity > > | timestamp > > | source | source_elapsed > > ------------------------------------------------------------------------------------------------------------------------------------------+--------------+----------------+---------------- > > > > execute_cql3_query | 07:18:12,692 > > | 192.168.25.111 | 0 > > > > Message received from /192.168.25.111 | 07:18:00,706 > > | 192.168.25.131 | 50 > > > > Executing single-partition query on data_points | 07:18:00,707 > > | 192.168.25.131 | 760 > > > > Acquiring sstable references | 07:18:00,707 > > | 192.168.25.131 | 814 > > > > Merging memtable tombstones | 07:18:00,707 > > | 192.168.25.131 | 924 > > > > Bloom filter allows skipping sstable 191 | 07:18:00,707 > > | 192.168.25.131 | 1050 > > > > Bloom filter allows skipping sstable 190 | 07:18:00,707 > > | 192.168.25.131 | 1166 > > > > Key cache hit for sstable 189 | 07:18:00,707 > > | 192.168.25.131 | 1275 > > > > Seeking to partition beginning in data file | 07:18:00,707 > > | 192.168.25.131 | 1293 > > Skipped 0/3 > > non-slice-intersecting sstables, included 0 due to tombstones | > > 07:18:00,708 | 192.168.25.131 | 2173 > > > > Merging data from memtables and 1 sstables | 07:18:00,708 > > | 192.168.25.131 | 2195 > > > > Read 1001 live and 0 tombstoned cells | 07:18:00,709 > > | 192.168.25.131 | 3259 > > > > Enqueuing response to /192.168.25.111 | 07:18:00,710 > > | 192.168.25.131 | 4006 > > > > Sending message to /192.168.25.111 | 07:18:00,710 > > | 192.168.25.131 | 4210 > > Parsing select * from data_points where key = > > 0x6d61726c796e2e746573742e74656d70340000000145b514a400726f6f6d3d6f66666963653a > > limit 1000; | 07:18:12,692 | 192.168.25.111 | 52 > > > > Preparing statement | 07:18:12,692 > > | 192.168.25.111 | 257 > > > > Sending message to /192.168.25.121 | 07:18:12,693 > > | 192.168.25.111 | 1099 > > > > Sending message to /192.168.25.131 | 07:18:12,693 > > | 192.168.25.111 | 1254 > > > > Executing single-partition query on data_points | 07:18:12,693 > > | 192.168.25.111 | 1269 > > > > Acquiring sstable references | 07:18:12,693 > > | 192.168.25.111 | 1284 > > > > Merging memtable tombstones | 07:18:12,694 > > | 192.168.25.111 | 1315 > > > > Key cache hit for sstable 205 | 07:18:12,694 > > | 192.168.25.111 | 1592 > > > > Seeking to partition beginning in data file | 07:18:12,694 > > | 192.168.25.111 | 1606 > > Skipped 0/1 > > non-slice-intersecting sstables, included 0 due to tombstones | > > 07:18:12,695 | 192.168.25.111 | 2423 > > > > Merging data from memtables and 1 sstables | 07:18:12,695 > > | 192.168.25.111 | 2498 > > > > Read 1001 live and 0 tombstoned cells | 07:18:12,695 > > | 192.168.25.111 | 3167 > > > > Message received from /192.168.25.121 | 07:18:12,697 > > | 192.168.25.111 | null > > > > Processing response from /192.168.25.121 | 07:18:12,697 > > | 192.168.25.111 | null > > > > Message received from /192.168.25.131 | 07:18:12,699 > > | 192.168.25.111 | null > > > > Processing response from /192.168.25.131 | 07:18:12,699 > > | 192.168.25.111 | null > > > > Message received from /192.168.25.111 | 07:19:49,432 > > | 192.168.25.121 | 68 > > > > Executing single-partition query on data_points | 07:19:49,433 > > | 192.168.25.121 | 824 > > > > Acquiring sstable references | 07:19:49,433 > > | 192.168.25.121 | 840 > > > > Merging memtable tombstones | 07:19:49,433 > > | 192.168.25.121 | 898 > > > > Bloom filter allows skipping sstable 193 | 07:19:49,433 > > | 192.168.25.121 | 983 > > > > Key cache hit for sstable 192 | 07:19:49,433 > > | 192.168.25.121 | 1055 > > > > Seeking to partition beginning in data file | 07:19:49,433 > > | 192.168.25.121 | 1073 > > Skipped 0/2 > > non-slice-intersecting sstables, included 0 due to tombstones | > > 07:19:49,434 | 192.168.25.121 | 1803 > > > > Merging data from memtables and 1 sstables | 07:19:49,434 > > | 192.168.25.121 | 1839 > > > > Read 1001 live and 0 tombstoned cells | 07:19:49,434 > > | 192.168.25.121 | 2518 > > > > Enqueuing response to /192.168.25.111 | 07:19:49,435 > > | 192.168.25.121 | 3026 > > > > Sending message to /192.168.25.111 | 07:19:49,435 > > | 192.168.25.121 | 3128 > > > > Request complete | 07:18:12,696 > > | 192.168.25.111 | 4387 > > > > > > Other Stats about the cluster: > > > > [root@cdev101 conf]# nodetool status > > Datacenter: DC3 > > =============== > > Status=Up/Down > > |/ State=Normal/Leaving/Joining/Moving > > -- Address Load Tokens Owns Host ID > > Rack > > UN 192.168.25.131 80.67 MB 256 34.2% > > 6ec61643-17d4-4a2e-8c44-57e08687a957 RAC1 > > Datacenter: DC2 > > =============== > > Status=Up/Down > > |/ State=Normal/Leaving/Joining/Moving > > -- Address Load Tokens Owns Host ID > > Rack > > UN 192.168.25.121 79.46 MB 256 30.6% > > 976626fb-ea80-405b-abb0-eae703b0074d RAC1 > > Datacenter: DC1 > > =============== > > Status=Up/Down > > |/ State=Normal/Leaving/Joining/Moving > > -- Address Load Tokens Owns Host ID > > Rack > > UN 192.168.25.111 61.82 MB 256 35.2% > > 9475e2da-d926-42d0-83fb-0188d0f8f438 RAC1 > > > > > > cqlsh> describe keyspace kairosdb > > > > CREATE KEYSPACE kairosdb WITH replication = { > > 'class': 'NetworkTopologyStrategy', > > 'DC2': '1', > > 'DC3': '1', > > 'DC1': '1' > > }; > > > > USE kairosdb; > > > > CREATE TABLE data_points ( > > key blob, > > column1 blob, > > value blob, > > PRIMARY KEY (key, column1) > > ) WITH COMPACT STORAGE AND > > bloom_filter_fp_chance=0.010000 AND > > caching='KEYS_ONLY' AND > > comment='' AND > > dclocal_read_repair_chance=0.000000 AND > > gc_grace_seconds=864000 AND > > index_interval=128 AND > > read_repair_chance=1.000000 AND > > replicate_on_write='true' AND > > populate_io_cache_on_flush='false' AND > > default_time_to_live=0 AND > > speculative_retry='NONE' AND > > memtable_flush_period_in_ms=0 AND > > compaction={'class': 'SizeTieredCompactionStrategy'} AND > > compression={'sstable_compression': 'LZ4Compressor'}; > > > > > > > >
