Jason Brown created CASSANDRA-12229:
---------------------------------------
Summary: Move streaming to non-blocking IO and netty (streaming
2.1)
Key: CASSANDRA-12229
URL: https://issues.apache.org/jira/browse/CASSANDRA-12229
Project: Cassandra
Issue Type: Improvement
Components: Streaming and Messaging
Reporter: Jason Brown
Assignee: Jason Brown
Fix For: 4.0
As followup work to CASSANDRA-8457, we need to move streaming to use netty.
Streaming 2.0 (CASSANDRA-5286) brought many good improvements to how files are
transferred between nodes in a cluster. However, the low-level details of the
current streaming implementation does not line up nicely with a non-blocking
model, so I think this is a good time to review some of those details and add
in additional goodness. The current implementation assumes a sequential or
"single threaded" approach to the sending of stream messages as well as the
transfer of files. In short, after several iterative prototypes, I propose the
following:
1) use a single bi-diredtional connection (instead of requiring to two sockets
& two threads)
2) send the "non-file" {{StreamMessage}}s (basically anything not
{{OutboundFileMessage}}) via the normal internode messaging. This will require
a slight bit more management of the session (the ability to look up a
{{StreamSession}} from a static function on {{StreamManager}}, but we have have
most of the pieces we need for this already.
3) Allow files to be streamed in parallel (CASSANDRA-4663) - this should just
be a thing already
4) If the entire sstable is to streamed, in addition to the DATA component,
transfer all the components of the sstable (primary index, bloom filter, stats,
and so on). This way we can avoid the CPU and GC pressure from deserializing
the stream into objects. File streaming then amounts to a block-level transfer.
Note: The progress/results of CASSANDRA-11303 will need to be reflected here,
as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
