[
https://issues.apache.org/jira/browse/FLUME-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630098#comment-13630098
]
Brock Noland commented on FLUME-1968:
-------------------------------------
Hari,
With this new format, we wouldn't have to read the whole log file if we stopped
storing the offsets. We'd have the last writer order id stored in the
checkpoint and then we'd simply do a binary search to that position the log. I
also think we'd want to write sync points much more often than every
checkpoint. I think we'd want to write them every event or every few events.
They are quite small in comparison with our events so the overhead is nothing
to worry about.
Brock
> FileChannel new format while being backwards compatible
> -------------------------------------------------------
>
> Key: FLUME-1968
> URL: https://issues.apache.org/jira/browse/FLUME-1968
> Project: Flume
> Issue Type: Bug
> Components: Channel, File Channel
> Reporter: Brock Noland
>
> There are a couple issues with the current format:
> 1) We have to track the offset at checkpoint time and write the offset to a
> special location so we can seek to that offset during replay. In FLUME-1516
> we are tracking two offsets.
> 2) We have no way to detect partial writes FLUME-1967
> 3) We can only checksum the body of the event, not the entire record
> FLUME-1485 and therefore cannot detect corruption outside an event body.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
