[
https://issues.apache.org/jira/browse/CASSANDRA-579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829985#action_12829985
]
Stu Hood commented on CASSANDRA-579:
------------------------------------
I think we should delay making too many changes to streaming until we've
finalized the SSTable versioning/interface changes proposed on #674. All of the
possible approaches to optimizing this depend on the file format. For instance,
sending only portions of the file depends on a splittable format, and
performing the compaction on the sending side and then writing a new SSTable on
the receiving side depends on the format of the serialized CompactedRows that
would be sent across.
> Add support to io.Streaming API for sending Streams
> ---------------------------------------------------
>
> Key: CASSANDRA-579
> URL: https://issues.apache.org/jira/browse/CASSANDRA-579
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Stu Hood
> Fix For: 0.7
>
>
> The io.Streaming API currently requires a file on disk to stream, which means
> that bootstrap and repairs need to perform an anti-compaction that writes a
> bunch of data to disk, only to have it be deleted after the streaming has
> finished.
> Ideally, the Streaming API should allow for streaming from an InputStream (or
> any other class we think we need to design to make the streaming as efficient
> as possible). That way, anti-compaction for repair/bootstrap does not perform
> any writing: it simply streams the relevant portion of the file to the
> neighbor.
> Additionally, this opens up interesting possibilities, such as providing the
> Streaming API as a (Java only?) client API. One use case would be for a
> Hadoop OutputFormat: rather than writing BinaryMemtables, the OutputFormat
> could literally write an SSTable to the stream. This might require better
> integration with gossip, to ensure that you aren't writing to the completely
> wrong node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
