[
https://issues.apache.org/jira/browse/CASSANDRA-15299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111564#comment-17111564
]
Olivier Michallat commented on CASSANDRA-15299:
-----------------------------------------------
OK, not much to add on the field sizes, I hope the future will prove you right.
{quote}The only real difference between having this in the header and nominally
being part of the body is that the integrity of the header is protected
{quote}
The integrity of the payload is protected as well. Is it that the guarantee is
stronger for the header because it is shorter?
Speaking of the header, I think there is an issue in the way it's currently
encoded. It's probably a bit early code for reviews, but I want to make sure I
understand this correctly. If I use {{FrameEncoderCrc}} for a self-contained,
uncompressed outer frame with a 4-bytes payload:
{code:java}
ByteBuffer buffer = ByteBuffer.allocate(6);
FrameEncoderCrc.writeHeader(buffer, true, 4);
System.out.println(Bytes.toHexString(buffer)); {code}
The buffer ends up with {{0x04000201f9f2}}. In binary that's
{{000001000000000000000010000000011111100111110010}}. If I align the first 24
bytes in the spec's diagram, the length is in a weird order, and the flag is
not in the right position:
{code:java}
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0|0|0 0 0 0 1 0| CRC24...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ {code}
That's because we do a byte-by-byte reversal, but our fields are not aligned on
byte boundaries.
I think the simplest fix would be to build the header data in network order,
and only reverse it when we compute the CRC:
{code:java}
int header3b = dataLength;
header3b <<= 1;
if (isSelfContained)
header3b |= 1;
header3b <<= 6; // padding
int crc = crc24(Integer.reverseBytes(header3b)>>8, 3);
// Same as put3b, but big-endian
frame.put(0, (byte)(header3b >>> 16));
frame.put(1, (byte)(header3b >>> 8) );
frame.put(2, (byte) header3b );
put3b(frame, 3, crc);
{code}
This produces {{0x000240979ee2}}, or in binary:
{code:java}
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0|1|0 0 0 0 0 0| CRC24...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
{code}
> CASSANDRA-13304 follow-up: improve checksumming and compression in protocol
> v5-beta
> -----------------------------------------------------------------------------------
>
> Key: CASSANDRA-15299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15299
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
> Reporter: Aleksey Yeschenko
> Assignee: Sam Tunnicliffe
> Priority: Normal
> Labels: protocolv5
> Fix For: 4.0-alpha
>
>
> CASSANDRA-13304 made an important improvement to our native protocol: it
> introduced checksumming/CRC32 to request and response bodies. It’s an
> important step forward, but it doesn’t cover the entire stream. In
> particular, the message header is not covered by a checksum or a crc, which
> poses a correctness issue if, for example, {{streamId}} gets corrupted.
> Additionally, we aren’t quite using CRC32 correctly, in two ways:
> 1. We are calculating the CRC32 of the *decompressed* value instead of
> computing the CRC32 on the bytes written on the wire - losing the properties
> of the CRC32. In some cases, due to this sequencing, attempting to decompress
> a corrupt stream can cause a segfault by LZ4.
> 2. When using CRC32, the CRC32 value is written in the incorrect byte order,
> also losing some of the protections.
> See https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf for
> explanation for the two points above.
> Separately, there are some long-standing issues with the protocol - since
> *way* before CASSANDRA-13304. Importantly, both checksumming and compression
> operate on individual message bodies rather than frames of multiple complete
> messages. In reality, this has several important additional downsides. To
> name a couple:
> # For compression, we are getting poor compression ratios for smaller
> messages - when operating on tiny sequences of bytes. In reality, for most
> small requests and responses we are discarding the compressed value as it’d
> be smaller than the uncompressed one - incurring both redundant allocations
> and compressions.
> # For checksumming and CRC32 we pay a high overhead price for small messages.
> 4 bytes extra is *a lot* for an empty write response, for example.
> To address the correctness issue of {{streamId}} not being covered by the
> checksum/CRC32 and the inefficiency in compression and checksumming/CRC32, we
> should switch to a framing protocol with multiple messages in a single frame.
> I suggest we reuse the framing protocol recently implemented for internode
> messaging in CASSANDRA-15066 to the extent that its logic can be borrowed,
> and that we do it before native protocol v5 graduates from beta. See
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderCrc.java
> and
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/FrameDecoderLZ4.java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
