[
https://issues.apache.org/jira/browse/AVRO-724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168536#comment-13168536
]
Doug Cutting commented on AVRO-724:
-----------------------------------
> The ideal solution would be to have fixed length block header fields but that
> would require a change to the spec.
This makes sense. To do this we'd probably want to increment the file format's
magic number, i.e., from {'O','b','j',1} to {'O','b','j',2}. And it would be
best to update all implementations to read the new format before making it the
default for any implementation.
> C implementation does not write datum values that are larger than the memory
> write buffer (currently 16K)
> ---------------------------------------------------------------------------------------------------------
>
> Key: AVRO-724
> URL: https://issues.apache.org/jira/browse/AVRO-724
> Project: Avro
> Issue Type: Bug
> Components: c
> Affects Versions: 1.4.1
> Reporter: Jeremy Hinegardner
>
> The current C implementation does not allow for datum values greater than 16K.
> The {{avro_file_writer_append}} flushes blocks to disk over time, but does
> not deal with the single case of a single datum being larger than
> {{avro_file_writer_t.datum_buffer}}. This is noted in the source code:
> {code:title=datafile.c:294-313}
> int avro_file_writer_append(avro_file_writer_t w, avro_datum_t datum)
> {
> int rval;
> if (!w || !datum) {
> return EINVAL;
> }
> rval = avro_write_data(w->datum_writer, w->writers_schema, datum);
> if (rval) {
> check(rval, file_write_block(w));
> rval =
> avro_write_data(w->datum_writer, w->writers_schema, datum);
> if (rval) {
> /* TODO: if the datum encoder larger than our buffer,
> just write a single large datum */
> return rval;
> }
> }
> w->block_count++;
> return 0;
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
