Re: [Marketing Mail] Re: [Marketing Mail] Re: nodetool status high load info
Actually there is no delete right now. Only inserts. I use twcs and not much compaction occurs. It just miscalculates sstable sizes. On 12-04-2017 14:58, anuja jain wrote: Do you perform a lot of deletes or updates on your database? On restart, it performs major compaction which can reduce the load on your node by removing stale data. Try configuring compaction in you conf to perform minor compaction i.e. compactions at a regular interval. Thanks, Anuja On Wed, Apr 12, 2017 at 3:02 PM, Osman YOZGATLIOGLU <osman.yozgatlio...@krontech.com<mailto:osman.yozgatlio...@krontech.com>> wrote: Hello, Here is the problem loads, first node shows 206TB data. After cassandra restart it shows 51TB, like df shows. Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN x.x.x.1 206 TB 256 50.6% xx rack1 UN x.x.x.2 190.77 TB 256 49.9% yy rack1 .. -- Address Load Tokens Owns (effective) Host ID Rack UN x.x.x.1 51.01 TB 256 50.6% xx rack1 UN x.x.x.2 49.84 TB 256 49.9% yy rack1 .. nodetool tpstats; Pool NameActive Pending Completed Blocked All time blocked MutationStage 2 175536494778 0 0 ViewMutationStage 0 0 0 0 0 ReadStage 0 0 41402 0 0 RequestResponseStage 0 035515109625 0 0 ReadRepairStage 0 0 3 0 0 CounterMutationStage 0 0 0 0 0 MiscStage 0 0 0 0 0 CompactionExecutor5 5 732161 0 0 MemtableReclaimMemory 0 0 198602 0 0 PendingRangeCalculator0 0 11 0 0 GossipStage 0 03854373 0 0 SecondaryIndexManagement 0 0 0 0 0 HintsDispatcher 1 7 6 0 0 MigrationStage0 0 6 0 0 MemtablePostFlush 0 0 200265 0 0 ValidationExecutor0 0 0 0 0 Sampler 0 0 0 0 0 MemtableFlushWriter 0 0 198602 0 0 InternalResponseStage 0 05209219 0 0 AntiEntropyStage 0 0 0 0 0 CacheCleanupExecutor 0 0 0 0 0 Native-Transport-Requests 0 015910719923 0 192131887 Message type Dropped READ 0 RANGE_SLICE 0 _TRACE 0 HINT 0 MUTATION 185 COUNTER_MUTATION 0 BATCH_STORE 0 BATCH_REMOVE 0 REQUEST_RESPONSE 0 PAGED_RANGE 0 READ_REPAIR 0 sar values; 05:10:01CPU %user %nice %system %iowait%steal %idle 05:20:01all 26.96 16.09 3.73 2.23 0.00 50.99 05:30:02all 26.99 16.83 3.82 2.86 0.00 49.50 05:40:01all 27.17 18.19 3.83 0.89 0.00 49.91 05:50:01all 27.16 18.74 3.80 0.28 0.00 50.02 06:00:01all 26.30 19.88 3.88 0.29 0.00 49.64 06:10:01all 28.02 21.11 3.91 0.28 0.00 46.68 06:20:01all 28.37 19.64 3.98 0.40 0.00 47.61 06:30:01all 29.56 19.51 4.08 0.45 0.00 46.40 06:40:01all 29.28 20.56 4.08 0.34 0.00 45.74 06:50:01all 29.46 19.15 3.99 0.19 0.00 47.20 07:00:01all 29.45 21.09 4.07 0.26 0.00 45.13 07:10:01all 29.23 21.59 4.18 0.29 0.00 44.71 07:20:01all 30.78 21.24 4.09 0.48 0.00 43.40 07:30:01all 29.06 21.63 4.09 0.27 0.00 44.94 07:40:01all 28.84 21.85 4.13 1.76 0.00 43.41 07:50:01all 29.22 21.
Re: [Marketing Mail] Re: nodetool status high load info
all 28.85 21.74 4.16 2.46 0.00 42.79 09:20:01all 30.15 20.79 4.31 2.44 0.00 42.31 Average:all 22.78 15.21 3.34 0.79 0.00 57.88 Regards, Osman On 12-04-2017 11:53, Bhuvan Rawal wrote: Try nodetool tpstats - it can lead you to where your threads are stuck. There could be various reasons for load factor to go high like disk/cpu getting choked, you'll probably need to check dstat & iostat output along with Cassandra Threadpool stats to get a decent idea. On Wed, Apr 12, 2017 at 1:48 PM, Osman YOZGATLIOGLU <osman.yozgatlio...@krontech.com<mailto:osman.yozgatlio...@krontech.com>> wrote: Hello, Nodetool status shows much more than actual data size. When I restart node, it shows normal a while and increase load in time. Where should I look? Cassandra 3.0.8, jdk 1.8.121 Regards, Osman This e-mail message, including any attachments, is for the sole use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. KRON makes no warranty that this e-mail is error or virus free. This e-mail message, including any attachments, is for the sole use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. KRON makes no warranty that this e-mail is error or virus free.
nodetool status high load info
Hello, Nodetool status shows much more than actual data size. When I restart node, it shows normal a while and increase load in time. Where should I look? Cassandra 3.0.8, jdk 1.8.121 Regards, Osman This e-mail message, including any attachments, is for the sole use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. KRON makes no warranty that this e-mail is error or virus free.
Re: [Marketing Mail] Re: [Marketing Mail] Re: sstableloader question
Hello, It's about 2500 sstables worth 25TB of data. -t parameter doesn't change -t 1000 and -t 1 Most probably I face some limitation at target cluster. I'm preparing to split sstables and run up to ten parallel sstableloader sessions. Regards, Osman On 11-10-2016 21:46, Rajath Subramanyam wrote: How many sstables are you trying to load ? Running sstableloaders in parallel will help. Did you try setting the "-t" parameter and see if you are getting the expected throughput ? - Rajath Rajath Subramanyam On Mon, Oct 10, 2016 at 2:02 PM, Osman YOZGATLIOGLU <osman.yozgatlio...@krontech.com<mailto:osman.yozgatlio...@krontech.com>> wrote: Hello, Thank you Adam and Rajath. I'll split input sstables and run parallel jobs for each. I tested this approach and run 3 parallel sstableloader job without -t parameter. I raised stream_throughput_outbound_megabits_per_sec parameter from 200 to 600 Mbit/sec at all of target nodes. But each job runs about 10MB/sec only and generates about 100Mbit'sec network traffic. At total this can be much more. Source and target servers has plenty of unused cpu, io and network resource. Do you have any idea how can I increase speed of sstableloader job? Regards, Osman On 10-10-2016 22:05, Rajath Subramanyam wrote: Hi Osman, You cannot restart the streaming only to the failed nodes specifically. You can restart the sstableloader job itself. Compaction will eventually take care of the redundant rows. - Rajath Rajath Subramanyam On Sun, Oct 9, 2016 at 7:38 PM, Adam Hutson <a...@datascale.io<mailto:a...@datascale.io><mailto:a...@datascale.io<mailto:a...@datascale.io>>> wrote: It'll start over from the beginning. On Sunday, October 9, 2016, Osman YOZGATLIOGLU <osman.yozgatlio...@krontech.com<mailto:osman.yozgatlio...@krontech.com><mailto:osman.yozgatlio...@krontech.com<mailto:osman.yozgatlio...@krontech.com>>> wrote: Hello, I have running a sstableloader job. Unfortunately some of nodes restarted since beginnig streaming. I see streaming stop for those nodes. Can I restart those streaming somehow? Or if I restart sstableloader job, will it start from beginning? Regards, Osman This e-mail message, including any attachments, is for the sole use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. KRON makes no warranty that this e-mail is error or virus free. -- Adam Hutson Data Architect | DataScale +1 (417) 224-5212<tel:%2B1%20%28417%29%20224-5212><tel:%2B1%20%28417%29%20224-5212> a...@datascale.io<mailto:a...@datascale.io><mailto:a...@datascale.io<mailto:a...@datascale.io>> This e-mail message, including any attachments, is for the sole use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. KRON makes no warranty that this e-mail is error or virus free. This e-mail message, including any attachments, is for the sole use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. KRON makes no warranty that this e-mail is error or virus free.
Re: [Marketing Mail] Re: sstableloader question
Hello, Thank you Adam and Rajath. I'll split input sstables and run parallel jobs for each. I tested this approach and run 3 parallel sstableloader job without -t parameter. I raised stream_throughput_outbound_megabits_per_sec parameter from 200 to 600 Mbit/sec at all of target nodes. But each job runs about 10MB/sec only and generates about 100Mbit'sec network traffic. At total this can be much more. Source and target servers has plenty of unused cpu, io and network resource. Do you have any idea how can I increase speed of sstableloader job? Regards, Osman On 10-10-2016 22:05, Rajath Subramanyam wrote: Hi Osman, You cannot restart the streaming only to the failed nodes specifically. You can restart the sstableloader job itself. Compaction will eventually take care of the redundant rows. - Rajath Rajath Subramanyam On Sun, Oct 9, 2016 at 7:38 PM, Adam Hutson <a...@datascale.io<mailto:a...@datascale.io>> wrote: It'll start over from the beginning. On Sunday, October 9, 2016, Osman YOZGATLIOGLU <osman.yozgatlio...@krontech.com<mailto:osman.yozgatlio...@krontech.com>> wrote: Hello, I have running a sstableloader job. Unfortunately some of nodes restarted since beginnig streaming. I see streaming stop for those nodes. Can I restart those streaming somehow? Or if I restart sstableloader job, will it start from beginning? Regards, Osman This e-mail message, including any attachments, is for the sole use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. KRON makes no warranty that this e-mail is error or virus free. -- Adam Hutson Data Architect | DataScale +1 (417) 224-5212<tel:%2B1%20%28417%29%20224-5212> a...@datascale.io<mailto:a...@datascale.io> This e-mail message, including any attachments, is for the sole use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. KRON makes no warranty that this e-mail is error or virus free.
sstableloader question
Hello, I have running a sstableloader job. Unfortunately some of nodes restarted since beginnig streaming. I see streaming stop for those nodes. Can I restart those streaming somehow? Or if I restart sstableloader job, will it start from beginning? Regards, Osman This e-mail message, including any attachments, is for the sole use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. KRON makes no warranty that this e-mail is error or virus free.
