[
https://issues.apache.org/jira/browse/FLUME-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628210#comment-13628210
]
Brock Noland commented on FLUME-1968:
-------------------------------------
Hi Mike,
There are two ways to use avro:
1) as you mentioned, use the file format directly
2) Similar to what we do today with protobufs, serialize avro objects to bytes
and store in our own file format
My concerns with approach number 1 are:
1) I don't see the current readers supporting random read of specific events.
It does look like we might be able to work around this by calling
DataFileWriter.sync() after each event and thereby having a sync marker after
each event so that we could then use DataFileReader.seek() to find the specific
event. We'd want to know much overhead there is going to be on the read side.
2) Is there a checksum? Doesn't look like it
http://avro.apache.org/docs/current/spec.html#Object+Container+Files if not,
then we have to handle that ourselves. Beyond that, we'd need to have an API to
interrogate a corrupt file in order to resync without resorting to the next
sync marker which could lose data if there are non-corrupt events between the
corruption and the next sync marker. Setting the sync marker per event should
help with this.
3) Can we pre-allocate the file? We pre-allocate the file to ensure we have
disk space to write. The NN edit logs saw tons of issues without pre-allocation
and indeed all the write ahead logs I am aware of do pre-allocation.
I am not against using the avro container format, I just think we'd need to
thoroughly investigate these open questions.
> FileChannel new format while being backwards compatible
> -------------------------------------------------------
>
> Key: FLUME-1968
> URL: https://issues.apache.org/jira/browse/FLUME-1968
> Project: Flume
> Issue Type: Bug
> Components: Channel, File Channel
> Reporter: Brock Noland
>
> There are a couple issues with the current format:
> 1) We have to track the offset at checkpoint time and write the offset to a
> special location so we can seek to that offset during replay. In FLUME-1516
> we are tracking two offsets.
> 2) We have no way to detect partial writes FLUME-1967
> 3) We can only checksum the body of the event, not the entire record
> FLUME-1485 and therefore cannot detect corruption outside an event body.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
