Re: Implement checksum against malformed packets
Il Gio 9 Apr 2020, 05:36 Michael Han ha scritto: > >> Why? > > A packet can be captured, altered its payload, and recomputed its > checksum(s) while transmitting, that's why encryption is needed. This > example sounds pedantic but serving the purpose of the argument. > With encryption the data integrity is ensured during transmission at least, > but data can still be corrupted (after being decrypted and put in a payload > buffer and then bit flip happens). This is really an end to end problem and > the only way to guarantee integrity is to figure out what the "ends" are > and checksum accordingly. I feel there is no need to over work on this in > our case given the probabilities of this happening in practice and the > impact. > I think that a zk level checksum can detect bugs in our code or in user code. Practical example: A buggy application may reuse the byte array in setData an partially overwrite the contents before we send it to the network. I saw this kind of bugs in Bookkeeper where the API lets you pass Netty ByteBufs to BK client write APIs. If the application reuses the buffer due to errors in recounting and the buffer is still in use by BK the result is that we are sending corrupted data to the network. BK detects this problem because it creates a checksum of data immediately (inside the 'write' API) then all of the processing is async Enrico > On Wed, Apr 8, 2020 at 1:49 AM Ted Dunning wrote: > > > The only way to corrupt a message transmitted on an encrypted link is to > > break the encryption. There is a tiny argument to be made that messages > > should be checksummed for protection even without the encryption, but > these > > links should always be encrypted. > > > > The chance of a 160 bit checksum being corrupted transparently by > accident > > is simply not worth considering. The TCP checksum is definitely small > > enough to see accidental transparent corruption, but we are talking about > > very different situation with much longer checks. > > > > On Wed, Apr 8, 2020, 1:40 AM Andor Molnar wrote: > > > > > Hi Michael, > > > > > > "Checksum itself (even cryptographic checksum) can’t guarantee data > > > integrity as packet in flight can be altered along with the new > > checksum.” > > > > > > Why? > > > > > > If either checksum or the payload was altered, the packet will be > > > retransmitted. Furthermore we could implement error correction codes > like > > > Reed-Solomon in order to avoid retransmission, but I don’t have > practical > > > experience with that, so might be a crazy idea. :) > > > > > > Andor > > > > > > > > > > > > > On 2020. Apr 8., at 3:17, Michael Han wrote: > > > > > > > >>> but not bulletproof? > > > > > > > > tcp checksum is weak and can't detect all corrupt packet; and bit > flip > > > can > > > > happen anywhere within the stack, and can be caused by malfunction of > > > ram, > > > > or say, cosmic rays. > > > > Statistically a corrupt packet survives all layers of check is low, > and > > > > even a packet is corrupted, app layer behavior might still be ok. > > > > > > > >>> wire encryption probably resolves it for free. > > > > > > > > This would be my recommendation here as well to ensure data integrity > > > > during transmission. Checksum itself (even cryptographic checksum) > > can't > > > > guarantee data integrity as packet in flight can be altered along > with > > > the > > > > new checksum. > > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 1:37 AM Andor Molnar > wrote: > > > > > > > >> Hi Zk community, > > > >> > > > >> I’ve been looking at the following ticket for a while and made me > > > thinking: > > > >> ZOOKEEPER-2175 Checksum validation for malformed packets needs to > > > handle. > > > >> https://issues.apache.org/jira/browse/ZOOKEEPER-2175 > > > >> > > > >> It has been found in 3.4.6 originally and not resolved since then > > saying > > > >> that wire encryption probably resolves it for free. I have doubts > > > against > > > >> the issue could actually happen at all, but the evidence in the jira > > is > > > >> quite convincing. > > > >> > > > >> 1) But what if somebody don’t want the overhead of using encryption? > > > >> 2) How often this issue could ever happen given that TCP packets > have > > > >> their own checksum which also protects the payload, but not > > bulletproof? > > > >> 3) Found a suggestion in a comment to send session id in the payload > > of > > > >> Ping packets, but malformed packets could happen in any other > > > >> communication, not just session ids. > > > >> 4) Does GRPC have a built-in mechanism to protect the communication? > > > Maybe > > > >> this would be the best way forward. > > > >> > > > >> What do you think? > > > >> > > > >> Andor > > > >> > > > >> > > > >> > > > > > > > > >
Re: Implement checksum against malformed packets
>> Why? A packet can be captured, altered its payload, and recomputed its checksum(s) while transmitting, that's why encryption is needed. This example sounds pedantic but serving the purpose of the argument. With encryption the data integrity is ensured during transmission at least, but data can still be corrupted (after being decrypted and put in a payload buffer and then bit flip happens). This is really an end to end problem and the only way to guarantee integrity is to figure out what the "ends" are and checksum accordingly. I feel there is no need to over work on this in our case given the probabilities of this happening in practice and the impact. On Wed, Apr 8, 2020 at 1:49 AM Ted Dunning wrote: > The only way to corrupt a message transmitted on an encrypted link is to > break the encryption. There is a tiny argument to be made that messages > should be checksummed for protection even without the encryption, but these > links should always be encrypted. > > The chance of a 160 bit checksum being corrupted transparently by accident > is simply not worth considering. The TCP checksum is definitely small > enough to see accidental transparent corruption, but we are talking about > very different situation with much longer checks. > > On Wed, Apr 8, 2020, 1:40 AM Andor Molnar wrote: > > > Hi Michael, > > > > "Checksum itself (even cryptographic checksum) can’t guarantee data > > integrity as packet in flight can be altered along with the new > checksum.” > > > > Why? > > > > If either checksum or the payload was altered, the packet will be > > retransmitted. Furthermore we could implement error correction codes like > > Reed-Solomon in order to avoid retransmission, but I don’t have practical > > experience with that, so might be a crazy idea. :) > > > > Andor > > > > > > > > > On 2020. Apr 8., at 3:17, Michael Han wrote: > > > > > >>> but not bulletproof? > > > > > > tcp checksum is weak and can't detect all corrupt packet; and bit flip > > can > > > happen anywhere within the stack, and can be caused by malfunction of > > ram, > > > or say, cosmic rays. > > > Statistically a corrupt packet survives all layers of check is low, and > > > even a packet is corrupted, app layer behavior might still be ok. > > > > > >>> wire encryption probably resolves it for free. > > > > > > This would be my recommendation here as well to ensure data integrity > > > during transmission. Checksum itself (even cryptographic checksum) > can't > > > guarantee data integrity as packet in flight can be altered along with > > the > > > new checksum. > > > > > > > > > > > > On Tue, Apr 7, 2020 at 1:37 AM Andor Molnar wrote: > > > > > >> Hi Zk community, > > >> > > >> I’ve been looking at the following ticket for a while and made me > > thinking: > > >> ZOOKEEPER-2175 Checksum validation for malformed packets needs to > > handle. > > >> https://issues.apache.org/jira/browse/ZOOKEEPER-2175 > > >> > > >> It has been found in 3.4.6 originally and not resolved since then > saying > > >> that wire encryption probably resolves it for free. I have doubts > > against > > >> the issue could actually happen at all, but the evidence in the jira > is > > >> quite convincing. > > >> > > >> 1) But what if somebody don’t want the overhead of using encryption? > > >> 2) How often this issue could ever happen given that TCP packets have > > >> their own checksum which also protects the payload, but not > bulletproof? > > >> 3) Found a suggestion in a comment to send session id in the payload > of > > >> Ping packets, but malformed packets could happen in any other > > >> communication, not just session ids. > > >> 4) Does GRPC have a built-in mechanism to protect the communication? > > Maybe > > >> this would be the best way forward. > > >> > > >> What do you think? > > >> > > >> Andor > > >> > > >> > > >> > > > > >
Re: Implement checksum against malformed packets
The only way to corrupt a message transmitted on an encrypted link is to break the encryption. There is a tiny argument to be made that messages should be checksummed for protection even without the encryption, but these links should always be encrypted. The chance of a 160 bit checksum being corrupted transparently by accident is simply not worth considering. The TCP checksum is definitely small enough to see accidental transparent corruption, but we are talking about very different situation with much longer checks. On Wed, Apr 8, 2020, 1:40 AM Andor Molnar wrote: > Hi Michael, > > "Checksum itself (even cryptographic checksum) can’t guarantee data > integrity as packet in flight can be altered along with the new checksum.” > > Why? > > If either checksum or the payload was altered, the packet will be > retransmitted. Furthermore we could implement error correction codes like > Reed-Solomon in order to avoid retransmission, but I don’t have practical > experience with that, so might be a crazy idea. :) > > Andor > > > > > On 2020. Apr 8., at 3:17, Michael Han wrote: > > > >>> but not bulletproof? > > > > tcp checksum is weak and can't detect all corrupt packet; and bit flip > can > > happen anywhere within the stack, and can be caused by malfunction of > ram, > > or say, cosmic rays. > > Statistically a corrupt packet survives all layers of check is low, and > > even a packet is corrupted, app layer behavior might still be ok. > > > >>> wire encryption probably resolves it for free. > > > > This would be my recommendation here as well to ensure data integrity > > during transmission. Checksum itself (even cryptographic checksum) can't > > guarantee data integrity as packet in flight can be altered along with > the > > new checksum. > > > > > > > > On Tue, Apr 7, 2020 at 1:37 AM Andor Molnar wrote: > > > >> Hi Zk community, > >> > >> I’ve been looking at the following ticket for a while and made me > thinking: > >> ZOOKEEPER-2175 Checksum validation for malformed packets needs to > handle. > >> https://issues.apache.org/jira/browse/ZOOKEEPER-2175 > >> > >> It has been found in 3.4.6 originally and not resolved since then saying > >> that wire encryption probably resolves it for free. I have doubts > against > >> the issue could actually happen at all, but the evidence in the jira is > >> quite convincing. > >> > >> 1) But what if somebody don’t want the overhead of using encryption? > >> 2) How often this issue could ever happen given that TCP packets have > >> their own checksum which also protects the payload, but not bulletproof? > >> 3) Found a suggestion in a comment to send session id in the payload of > >> Ping packets, but malformed packets could happen in any other > >> communication, not just session ids. > >> 4) Does GRPC have a built-in mechanism to protect the communication? > Maybe > >> this would be the best way forward. > >> > >> What do you think? > >> > >> Andor > >> > >> > >> > >
Re: Implement checksum against malformed packets
Hi Michael, "Checksum itself (even cryptographic checksum) can’t guarantee data integrity as packet in flight can be altered along with the new checksum.” Why? If either checksum or the payload was altered, the packet will be retransmitted. Furthermore we could implement error correction codes like Reed-Solomon in order to avoid retransmission, but I don’t have practical experience with that, so might be a crazy idea. :) Andor > On 2020. Apr 8., at 3:17, Michael Han wrote: > >>> but not bulletproof? > > tcp checksum is weak and can't detect all corrupt packet; and bit flip can > happen anywhere within the stack, and can be caused by malfunction of ram, > or say, cosmic rays. > Statistically a corrupt packet survives all layers of check is low, and > even a packet is corrupted, app layer behavior might still be ok. > >>> wire encryption probably resolves it for free. > > This would be my recommendation here as well to ensure data integrity > during transmission. Checksum itself (even cryptographic checksum) can't > guarantee data integrity as packet in flight can be altered along with the > new checksum. > > > > On Tue, Apr 7, 2020 at 1:37 AM Andor Molnar wrote: > >> Hi Zk community, >> >> I’ve been looking at the following ticket for a while and made me thinking: >> ZOOKEEPER-2175 Checksum validation for malformed packets needs to handle. >> https://issues.apache.org/jira/browse/ZOOKEEPER-2175 >> >> It has been found in 3.4.6 originally and not resolved since then saying >> that wire encryption probably resolves it for free. I have doubts against >> the issue could actually happen at all, but the evidence in the jira is >> quite convincing. >> >> 1) But what if somebody don’t want the overhead of using encryption? >> 2) How often this issue could ever happen given that TCP packets have >> their own checksum which also protects the payload, but not bulletproof? >> 3) Found a suggestion in a comment to send session id in the payload of >> Ping packets, but malformed packets could happen in any other >> communication, not just session ids. >> 4) Does GRPC have a built-in mechanism to protect the communication? Maybe >> this would be the best way forward. >> >> What do you think? >> >> Andor >> >> >>
Re: Implement checksum against malformed packets
>> but not bulletproof? tcp checksum is weak and can't detect all corrupt packet; and bit flip can happen anywhere within the stack, and can be caused by malfunction of ram, or say, cosmic rays. Statistically a corrupt packet survives all layers of check is low, and even a packet is corrupted, app layer behavior might still be ok. >> wire encryption probably resolves it for free. This would be my recommendation here as well to ensure data integrity during transmission. Checksum itself (even cryptographic checksum) can't guarantee data integrity as packet in flight can be altered along with the new checksum. On Tue, Apr 7, 2020 at 1:37 AM Andor Molnar wrote: > Hi Zk community, > > I’ve been looking at the following ticket for a while and made me thinking: > ZOOKEEPER-2175 Checksum validation for malformed packets needs to handle. > https://issues.apache.org/jira/browse/ZOOKEEPER-2175 > > It has been found in 3.4.6 originally and not resolved since then saying > that wire encryption probably resolves it for free. I have doubts against > the issue could actually happen at all, but the evidence in the jira is > quite convincing. > > 1) But what if somebody don’t want the overhead of using encryption? > 2) How often this issue could ever happen given that TCP packets have > their own checksum which also protects the payload, but not bulletproof? > 3) Found a suggestion in a comment to send session id in the payload of > Ping packets, but malformed packets could happen in any other > communication, not just session ids. > 4) Does GRPC have a built-in mechanism to protect the communication? Maybe > this would be the best way forward. > > What do you think? > > Andor > > >
Implement checksum against malformed packets
Hi Zk community, I’ve been looking at the following ticket for a while and made me thinking: ZOOKEEPER-2175 Checksum validation for malformed packets needs to handle. https://issues.apache.org/jira/browse/ZOOKEEPER-2175 It has been found in 3.4.6 originally and not resolved since then saying that wire encryption probably resolves it for free. I have doubts against the issue could actually happen at all, but the evidence in the jira is quite convincing. 1) But what if somebody don’t want the overhead of using encryption? 2) How often this issue could ever happen given that TCP packets have their own checksum which also protects the payload, but not bulletproof? 3) Found a suggestion in a comment to send session id in the payload of Ping packets, but malformed packets could happen in any other communication, not just session ids. 4) Does GRPC have a built-in mechanism to protect the communication? Maybe this would be the best way forward. What do you think? Andor
