[
https://issues.apache.org/jira/browse/AVRO-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185815#comment-13185815
]
Tom White commented on AVRO-991:
--------------------------------
Is it worth considering expanding the spec to support concatenated Avro data
files, just like gzip allows
(http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage )? Then you'd
be able to do
{noformat}
cat file1.avro file2.avro > combined-file.avro
{noformat}
and read combined-file.avro as a regular Avro file.
However, I'm not sure how splitting would work since the sync markers would be
different in the original Avro files. (If only Avro files had a fixed sync
marker! Like bzip2 does. This also tripped me up recently for another reason:
two Avro files with the same contents do not have the same binary
representation, so you can't just checksum them to see if they are the same.)
> Allow combining multiple Avro files within a stream. (no files on disk)
> -----------------------------------------------------------------------
>
> Key: AVRO-991
> URL: https://issues.apache.org/jira/browse/AVRO-991
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.6.1
> Reporter: Frank Grimes
>
> It would be nice to be able to do as follows:
> cat file1.avro file2.avro | java -jar avro-tools.jar streamcombine >
> combined-file.avro
> or similarly
>
> hadoop dfs -cat hdfs://hadoop/file1.avro hdfs://hadoop/file2.avro | java
> -jar avro-tools.jar streamcombine | hdfs -put -
> hdfs://hadoop/combined-file.avro
> See the following thread for details:
> http://mail-archives.apache.org/mod_mbox/avro-user/201201.mbox/%[email protected]%3e
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
