[
https://issues.apache.org/jira/browse/AVRO-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187227#comment-13187227
]
Doug Cutting commented on AVRO-991:
-----------------------------------
For the record, the thinking behind the varied sync marker is that it makes
collisions less likely. In theory this is not true, but in practice my concern
was that, once a value was fixed and known, there'd be a significantly higher
probability that someone would include it in some data. Perhaps that's not
correct, though.
As for expanding the spec, as I mentioned above, we can do that at present,
since the file's magic number can never be the start of a valid block. So if a
block ever starts with the magic number then a reader could assume that it's an
appended file. It's perhaps not the way one would design an appendable format
from scratch, but I think it's workable.
> Allow combining multiple Avro files within a stream. (no files on disk)
> -----------------------------------------------------------------------
>
> Key: AVRO-991
> URL: https://issues.apache.org/jira/browse/AVRO-991
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.6.1
> Reporter: Frank Grimes
>
> It would be nice to be able to do as follows:
> cat file1.avro file2.avro | java -jar avro-tools.jar streamcombine >
> combined-file.avro
> or similarly
>
> hadoop dfs -cat hdfs://hadoop/file1.avro hdfs://hadoop/file2.avro | java
> -jar avro-tools.jar streamcombine | hdfs -put -
> hdfs://hadoop/combined-file.avro
> See the following thread for details:
> http://mail-archives.apache.org/mod_mbox/avro-user/201201.mbox/%[email protected]%3e
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
