Re: Bootstrap performance.
On Mon, Apr 20, 2015 at 8:09 PM, Dikang Gu dikan...@gmail.com wrote: Why do you say steaming is single threaded? I see a lot of background streaming threads running, for example: Imprecise : Each stream is a single thread. As I said, first place to look is throttles... but I would not be surprised if the overall number of threads available to streaming is a meaningful bound. =Rob
Bootstrap performance.
Hi guys, We have a 100+ nodes cluster, each node has about 400G data, and is running on a flash disk. We are running 2.1.2. When I bring in a new node into the cluster, it introduces significant load to the cluster. For the new node, the cpu usage is 100%, but disk write io is only around 50MB/s, while we have 10G network. Does it sound normal to you? Here are some iostat and vmstat metrics: iostat avg-cpu: %user %nice %system %iowait %steal %idle 88.523.994.110.000.003.38 Device:tpsMB_read/sMB_wrtn/sMB_readMB_wrtn sda 1.00 0.00 0.04 0 0 sdb 156.50 0.0055.62 01 vmstat = 138 0 0 86781912 438780 10152336800 0 31893 264496 247316 95 4 1 0 0 2015-04-21 01:04:01 UTC 147 0 0 86562400 438780 10160724800 0 90510 456635 245849 91 5 4 0 0 2015-04-21 01:04:03 UTC 143 0 0 86341168 438780 10169222400 0 32392 284495 273656 92 4 4 0 0 2015-04-21 01:04:05 UTC Thanks. -- Dikang
bootstrap performance.
Re: Bootstrap performance.
On Mon, Apr 20, 2015 at 6:08 PM, Dikang Gu dikan...@gmail.com wrote: When I bring in a new node into the cluster, it introduces significant load to the cluster. For the new node, the cpu usage is 100%, but disk write io is only around 50MB/s, while we have 10G network. Does it sound normal to you? Have you unthrottled both compaction and streaming via JMX/nodetool? Streaming is single threaded and can (?) be CPU bound, I would not be surprised if JIRA contains a ticket on the upper bounds of streaming performance in current implementation. =Rob
Re: Bootstrap performance.
Hi Rob, Why do you say steaming is single threaded? I see a lot of background streaming threads running, for example: STREAM-IN-/10.210.165.49 daemon prio=10 tid=0x7f81fc001000 nid=0x107075 runnable [0x7f836b256000] STREAM-IN-/10.213.51.57 daemon prio=10 tid=0x7f81f0002000 nid=0x107073 runnable [0x7f836b1d4000] STREAM-IN-/10.213.51.61 daemon prio=10 tid=0x7f81e8001000 nid=0x107070 runnable [0x7f836b11] STREAM-IN-/10.213.51.63 daemon prio=10 tid=0x7f81dc001800 nid=0x10706f runnable [0x7f836b0cf000] Thanks Dikang. On Mon, Apr 20, 2015 at 6:48 PM, Robert Coli rc...@eventbrite.com wrote: On Mon, Apr 20, 2015 at 6:08 PM, Dikang Gu dikan...@gmail.com wrote: When I bring in a new node into the cluster, it introduces significant load to the cluster. For the new node, the cpu usage is 100%, but disk write io is only around 50MB/s, while we have 10G network. Does it sound normal to you? Have you unthrottled both compaction and streaming via JMX/nodetool? Streaming is single threaded and can (?) be CPU bound, I would not be surprised if JIRA contains a ticket on the upper bounds of streaming performance in current implementation. =Rob -- Dikang
