[
https://issues.apache.org/jira/browse/CASSANDRA-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082251#comment-13082251
]
Sylvain Lebresne commented on CASSANDRA-3003:
---------------------------------------------
bq. In order to use EchoedRow, we'd have to move where we do CounterContext
cleanup
I really think it is not very hard to do 'inline'. We really just want to
deserialize, cleanup, reserialize. It should be super easy to add some
"CounterCleanedRow" that does that.
bq. at it could be done at read time if we added "fromRemote" as a field in the
metadata of an SSTable
Yes, but it does sound a bit complicated to me compared to doing the cleanup
right away during streaming. It would also be less efficient, because until we
have compacted the streamed sstable, each read will have to call the cleanup
over and over, while we really only care to have it done twice (unless we
completely change where we do cleanup).
bq. Perhaps we can use this as an opportunity to switch to using only
PrecompactedRow (for narrow rows which might go to cache) or EchoedRow (for
wide rows, which go directly to disk)?
I agree in that there is no point in doing manual deserialization there. About
the PrecompactedRow for narrow rows which might go to cache, I'll just precise
that it is worth using PrecompactedRow only if 1) we are doing AES streaming
and 2) the row is in cache in the first place (which we can know since we
always at least deserialize the row key).
> Trunk single-pass streaming doesn't handle large row correctly
> --------------------------------------------------------------
>
> Key: CASSANDRA-3003
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3003
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Sylvain Lebresne
> Assignee: Yuki Morishita
> Priority: Critical
> Labels: streaming
>
> For normal column family, trunk streaming always buffer the whole row into
> memory. In uses
> {noformat}
> ColumnFamily.serializer().deserializeColumns(in, cf, true, true);
> {noformat}
> on the input bytes.
> We must avoid this for rows that don't fit in the inMemoryLimit.
> Note that for regular column families, for a given row, there is actually no
> need to even recreate the bloom filter of column index, nor to deserialize
> the columns. It is enough to filter the key and row size to feed the index
> writer, but then simply dump the rest on disk directly. This would make
> streaming more efficient, avoid a lot of object creation and avoid the
> pitfall of big rows.
> Counters column family are unfortunately trickier, because each column needs
> to be deserialized (to mark them as 'fromRemote'). However, we don't need to
> do the double pass of LazilyCompactedRow for that. We can simply use a
> SSTableIdentityIterator and deserialize/reserialize input as it comes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
