[
https://issues.apache.org/jira/browse/AVRO-820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034903#comment-13034903
]
Scott Carey commented on AVRO-820:
----------------------------------
Lets take a step back.
This patch allows the following things that were previously impossible:
If certain exception types occur when calling append(datum), the file will not
become corrupt if:
* append(datum) is called again
* close() is called
* flush() is called
What the user chooses to do after the error is not so important.
In my case, it is trivial to tell the difference. NullPointerException is not
an IOException. The problem is that this isn't so much an EncoderException as
a DatumWriterException. I think making a DatumWriter throw something like a
InvalidDatumException would be useful, but should be in a different ticket.
Regardless of the cause, if the file writer gets an unexpected error while
invoking the DatumWriter, it needs to rewind and consider any data written to
the block during the call to the DatumWriter invalid.
Whether the user should continue to write or close the file is besides the
point. Neither option works without this patch.
Let me also clarify my requirements:
It is not about proving or guaranteeing that it is not possible to corrupt a
file. It is about the worthiness of fixing any bugs found that can corrupt a
file (within reason).
My use case is high volume asynchronous logging. My schema is 6K of json so
the header is huge and opening a new file is very expensive. If one in 10000
calls to append(datum) fail because of a bug introduced into the system that
failed to initialize a non-nullable field, it is not acceptable to close the
file and open a new one for every error. The logging queue would back up and
data may be lost, as the total logging throughput might dip too low or the
buffer overrun. Currently, if an IOException is thrown, or two other
Exceptions are thrown in a row, an error is logged, an attempt is made to close
the file, and a new one is opened. If a single lone exception occurs an error
is logged and it moves on in the same file. In production, the former case has
never occurred. The latter recently happened, and resulted in corrupt files.
Closing the file would have helped very little since it would write an invalid
block (though no valid data would be written past the corrupted spot, the Avro
file reader and downstream readers fail).
> Java: Exceptions thrown while encoding a record while writing an Avro Data
> file will produce a corrupt file.
> -------------------------------------------------------------------------------------------------------------
>
> Key: AVRO-820
> URL: https://issues.apache.org/jira/browse/AVRO-820
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.4.1, 1.5.0, 1.5.1
> Reporter: Scott Carey
> Assignee: Scott Carey
> Priority: Critical
> Attachments: AVRO-820.patch
>
>
> If an exception is thrown while serializing a record in
> DataFileWriter<D>.append(D) partial contents of that serialization will end
> up in the file. This corrupts the block.
> DataFileWriter should ensure that the buffer is rewound to the state prior to
> the record write in the case of an exception thrown during serialization to
> prevent creating a corrupt file.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
