[
https://issues.apache.org/jira/browse/CASSANDRA-12229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959536#comment-15959536
]
Jason Brown edited comment on CASSANDRA-12229 at 4/6/17 6:56 PM:
-----------------------------------------------------------------
Had an offline conversation with [~cnlwsu], and he explained to me that my
current patch breaks external tools, like BulkLoader. This is because if the
BulkLoader is running on the same box as a c* node, the BulkLoader won't be
able to bind to an address to listen to callbacks from the c* process (amongst
other things). Thus, I'm gonna have to rethink that part - it shouldn't affect
things too much. Stay tuned.
was (Author: jasobrown):
Had an offline conversation with [~cnlwsu], and he explained to me that this
code breaks external tools, like BulkLoader. This is because if the BulkLoader
is running on the same box as a c* node, the BulkLoader won't be able to bind
to an address to listen to callbacks from the c* process (amongst other
things). Thus, I'm gonna have to rethink that part - it shouldn't affect things
too much. Stay tuned.
> Move streaming to non-blocking IO and netty (streaming 2.1)
> -----------------------------------------------------------
>
> Key: CASSANDRA-12229
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12229
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
> Reporter: Jason Brown
> Assignee: Jason Brown
> Fix For: 4.0
>
>
> As followup work to CASSANDRA-8457, we need to move streaming to use netty.
> Streaming 2.0 (CASSANDRA-5286) brought many good improvements to how files
> are transferred between nodes in a cluster. However, the low-level details of
> the current streaming implementation does not line up nicely with a
> non-blocking model, so I think this is a good time to review some of those
> details and add in additional goodness. The current implementation assumes a
> sequential or "single threaded" approach to the sending of stream messages as
> well as the transfer of files. In short, after several iterative prototypes,
> I propose the following:
> 1) use a single bi-diredtional connection (instead of requiring to two
> sockets & two threads)
> 2) send the "non-file" {{StreamMessage}} s (basically anything not
> {{OutboundFileMessage}}) via the normal internode messaging. This will
> require a slight bit more management of the session (the ability to look up a
> {{StreamSession}} from a static function on {{StreamManager}}, but we have
> have most of the pieces we need for this already.
> 3) switch to a non-blocking IO model (facilitated via netty)
> 4) Allow files to be streamed in parallel (CASSANDRA-4663) - this should just
> be a thing already
> 5) If the entire sstable is to streamed, in addition to the DATA component,
> transfer all the components of the sstable (primary index, bloom filter,
> stats, and so on). This way we can avoid the CPU and GC pressure from
> deserializing the stream into objects. File streaming then amounts to a
> block-level transfer.
> Note: The progress/results of CASSANDRA-11303 will need to be reflected here,
> as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
