> On Sept. 21, 2012, 3:20 p.m., Brock Noland wrote: > > Each event is fairly small. Since CRC is faster than MD5, have you > > considered using CRC? > > > > If we put the checksum on the event footer we could verify every event when > > reading it. Any thoughts on that?
MD5 is just a place holder. We can change that to anything else. Putting it in the header or footer shouldn't matter, right? When reading it we would read the corresponding "put" out, which would contain the checksum in its header, which we can verify. The reason I didn't put it in the footer was that the checksum would now be an optional field in all record types - puts, takes, commits, rollbacks - and it does not make sense for the other three. So more of a software engineering reason + I didn't know whether there is an overhead for an optional field that does not exist. Also, I felt verifying the checksum during replay was the only thing required - assuming the corruption occurs outside the channel. It is not difficult to do this during every take. - Hari ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7141/#review11775 ----------------------------------------------------------- On Sept. 18, 2012, 7:12 a.m., Hari Shreedharan wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7141/ > ----------------------------------------------------------- > > (Updated Sept. 18, 2012, 7:12 a.m.) > > > Review request for Flume. > > > Description > ------- > > Submitting this for early review. There are empty blocks of code which say > "Do something etc." Those will be filled in before this is in any commit-able > shape. This is being submitted now to get some early feedback on the logic. > Also, I didn't include the checksum in TxnEventFooter, since that will > unnecessarily be included(even if marked as optional, I wasn't sure whether > protos will generate something for it - also, including it in only in Puts > helps avoid confusion). As of this patch, the checksum is only on the event > body. This can be expanded later if the need arises. > > > This addresses bug FLUME-1586. > https://issues.apache.org/jira/browse/FLUME-1586 > > > Diffs > ----- > > flume-ng-channels/flume-file-channel/pom.xml 658c06f > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CheckpointRebuilder.java > 748f49a > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java > c2e904b > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java > be2f633 > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java > e36eafb > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java > d47b1c8 > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java > 81f6172 > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java > e6d4957 > flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto > 3a4e828 > flume-ng-channels/flume-file-channel/src/main/resources/ASL20 PRE-CREATION > > Diff: https://reviews.apache.org/r/7141/diff/ > > > Testing > ------- > > None yet. > > > Thanks, > > Hari Shreedharan > >
