[
https://issues.apache.org/jira/browse/CASSANDRA-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798727#comment-15798727
]
Corentin Chary edited comment on CASSANDRA-4663 at 1/5/17 6:38 AM:
-------------------------------------------------------------------
The thing is that as far as I understand StreamSession#getOrCreateNextSession()
will create multiple sessions:
{code}
INFO [main] 2017-01-04 16:42:41,003 StorageService.java:1438 - JOINING:
Starting to bootstrap...
INFO [main] 2017-01-04 16:42:41,532 StreamResultFuture.java:90 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Executing streaming plan for Bootstrap
INFO [StreamConnectionEstablisher:1] 2017-01-04 16:42:41,538
StreamSession.java:266 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Starting streaming to /10.50.4.115
INFO [StreamConnectionEstablisher:1] 2017-01-04 16:42:41,541
StreamCoordinator.java:264 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b,
ID#0] Beginning stream session with /10.50.4.115
INFO [STREAM-IN-/10.50.4.115:7000] 2017-01-04 16:42:41,562
StreamResultFuture.java:187 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Session with /10.50.4.115 is complete
INFO [StreamConnectionEstablisher:2] 2017-01-04 16:42:41,574
StreamSession.java:266 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Starting streaming to /10.50.4.115
INFO [StreamConnectionEstablisher:2] 2017-01-04 16:42:41,576
StreamCoordinator.java:264 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b,
ID#1] Beginning stream session with /10.50.4.115
INFO [STREAM-IN-/10.50.4.115:7000] 2017-01-04 16:42:41,594
StreamResultFuture.java:187 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Session with /10.50.4.115 is complete
INFO [StreamConnectionEstablisher:3] 2017-01-04 16:42:41,601
StreamSession.java:266 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Starting streaming to /10.50.4.115
INFO [StreamConnectionEstablisher:3] 2017-01-04 16:42:41,602
StreamCoordinator.java:264 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b,
ID#2] Beginning stream session with /10.50.4.115
INFO [STREAM-IN-/10.50.4.115:7000] 2017-01-04 16:42:41,607
StreamResultFuture.java:187 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Session with /10.50.4.115 is complete
INFO [StreamConnectionEstablisher:4] 2017-01-04 16:42:41,609
StreamSession.java:266 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Starting streaming to /10.50.4.126
INFO [StreamConnectionEstablisher:4] 2017-01-04 16:42:41,610
StreamCoordinator.java:264 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b,
ID#0] Beginning stream session with /10.50.4.126
INFO [STREAM-IN-/10.50.4.126:7000] 2017-01-04 16:42:41,676
StreamResultFuture.java:173 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b
ID#0] Prepare completed. Receiving 7 files(1.181GiB), sending 0 files(0.000KiB)
INFO [StreamConnectionEstablisher:5] 2017-01-04 16:42:41,679
StreamSession.java:266 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Starting streaming to /10.50.4.126
INFO [StreamConnectionEstablisher:5] 2017-01-04 16:42:41,682
StreamCoordinator.java:264 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b,
ID#1] Beginning stream session with /10.50.4.126
INFO [STREAM-IN-/10.50.4.126:7000] 2017-01-04 16:42:41,718
StreamResultFuture.java:173 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b
ID#1] Prepare completed. Receiving 8 files(772.249MiB), sending 0
files(0.000KiB)
INFO [StreamConnectionEstablisher:6] 2017-01-04 16:42:41,719
StreamSession.java:266 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Starting streaming to /10.50.4.126
INFO [StreamConnectionEstablisher:6] 2017-01-04 16:42:41,720
StreamCoordinator.java:264 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b,
ID#2] Beginning stream session with /10.50.4.126
INFO [STREAM-IN-/10.50.4.126:7000] 2017-01-04 16:42:41,726
StreamResultFuture.java:187 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Session with /10.50.4.126 is complete
{code}
Because of fetchAsync() iterates on keyspaces to call requestRanges() this will
spread the keyspaces across multiple sessions (it would be even better to
spread the ranges).
Since the bottleneck is the CPU bound STREAM-IN thread (IncomingMessageHandler)
it will improve the performances by creating multiple of these threads.
With three keyspaces and three connections I was able to double the throughput
and hopefully that is not just placebo effect.
was (Author: iksaif):
The thing is that as far as I understand
{code}StreamSession#getOrCreateNextSession(){code} will create multiple
sessions:
{code}
INFO [main] 2017-01-04 16:42:41,003 StorageService.java:1438 - JOINING:
Starting to bootstrap...
INFO [main] 2017-01-04 16:42:41,532 StreamResultFuture.java:90 - [Stream
#cef1c0e0-d29c-11e6-b978-d922683f145b] Executing streaming plan for Bootstrap
INFO [StreamConnectionEstablisher:1] 2017-01-04 16:42:41,538
StreamSession.java:266 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Starting streaming to /10.50.4.115
INFO [StreamConnectionEstablisher:1] 2017-01-04 16:42:41,541
StreamCoordinator.java:264 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b,
ID#0] Beginning stream session with /10.50.4.115
INFO [STREAM-IN-/10.50.4.115:7000] 2017-01-04 16:42:41,562
StreamResultFuture.java:187 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Session with /10.50.4.115 is complete
INFO [StreamConnectionEstablisher:2] 2017-01-04 16:42:41,574
StreamSession.java:266 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Starting streaming to /10.50.4.115
INFO [StreamConnectionEstablisher:2] 2017-01-04 16:42:41,576
StreamCoordinator.java:264 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b,
ID#1] Beginning stream session with /10.50.4.115
INFO [STREAM-IN-/10.50.4.115:7000] 2017-01-04 16:42:41,594
StreamResultFuture.java:187 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Session with /10.50.4.115 is complete
INFO [StreamConnectionEstablisher:3] 2017-01-04 16:42:41,601
StreamSession.java:266 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Starting streaming to /10.50.4.115
INFO [StreamConnectionEstablisher:3] 2017-01-04 16:42:41,602
StreamCoordinator.java:264 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b,
ID#2] Beginning stream session with /10.50.4.115
INFO [STREAM-IN-/10.50.4.115:7000] 2017-01-04 16:42:41,607
StreamResultFuture.java:187 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Session with /10.50.4.115 is complete
INFO [StreamConnectionEstablisher:4] 2017-01-04 16:42:41,609
StreamSession.java:266 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Starting streaming to /10.50.4.126
INFO [StreamConnectionEstablisher:4] 2017-01-04 16:42:41,610
StreamCoordinator.java:264 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b,
ID#0] Beginning stream session with /10.50.4.126
INFO [STREAM-IN-/10.50.4.126:7000] 2017-01-04 16:42:41,676
StreamResultFuture.java:173 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b
ID#0] Prepare completed. Receiving 7 files(1.181GiB), sending 0 files(0.000KiB)
INFO [StreamConnectionEstablisher:5] 2017-01-04 16:42:41,679
StreamSession.java:266 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Starting streaming to /10.50.4.126
INFO [StreamConnectionEstablisher:5] 2017-01-04 16:42:41,682
StreamCoordinator.java:264 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b,
ID#1] Beginning stream session with /10.50.4.126
INFO [STREAM-IN-/10.50.4.126:7000] 2017-01-04 16:42:41,718
StreamResultFuture.java:173 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b
ID#1] Prepare completed. Receiving 8 files(772.249MiB), sending 0
files(0.000KiB)
INFO [StreamConnectionEstablisher:6] 2017-01-04 16:42:41,719
StreamSession.java:266 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Starting streaming to /10.50.4.126
INFO [StreamConnectionEstablisher:6] 2017-01-04 16:42:41,720
StreamCoordinator.java:264 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b,
ID#2] Beginning stream session with /10.50.4.126
INFO [STREAM-IN-/10.50.4.126:7000] 2017-01-04 16:42:41,726
StreamResultFuture.java:187 - [Stream #cef1c0e0-d29c-11e6-b978-d922683f145b]
Session with /10.50.4.126 is complete
{code}
And because of {code}fetchAsync(){code} iterates on keyspaces to call
{code}requestRanges(){code} this will spread the keyspace accross multiple
sessions (it would be even better to spread the ranges).
And because the bottleneck is the STREAM-IN thread (IncomingMessageHandler) it
will improve the performances.
With three keyspaces and three connections I was able to double the throughput
and hopefully that is not just placebo effect.
> Streaming sends one file at a time serially.
> ---------------------------------------------
>
> Key: CASSANDRA-4663
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4663
> Project: Cassandra
> Issue Type: Improvement
> Reporter: sankalp kohli
> Priority: Minor
> Fix For: 3.x
>
> Attachments:
> 0001-streaming-add-a-way-to-configure-the-number-of-conne.patch
>
>
> This is not fast enough when someone is using SSD and may be 10G link. We
> should try to create multiple connections and send multiple files in
> parallel.
> Current approach under utilize the link(even 1G).
> This change will improve the bootstrapping time of a node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
