[
https://issues.apache.org/jira/browse/AVRO-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760356#action_12760356
]
Doug Cutting commented on AVRO-127:
-----------------------------------
In Avro 1.1 we added DataFileWriter#addSchema(Schema):
http://hadoop.apache.org/avro/docs/current/api/java/org/apache/avro/file/DataFileWriter.html#addSchema%28org.apache.avro.Schema%29
This only works if the top-level schema is a union. It lets you add clauses to
that union as you write the file. So, for example, you might do something like:
{noformat}
DataFileWriter<Object> out =
new DataFileWriter<Object>(Schema.createUnion(new ArrayList()[])),
new FileOutputStream("foo"), new GenericDatumWriter());
...
out.addSchema(x.getSchema());
out.write(x);
...
out.addSchema(y.getSchema());
out.write(y)
...
{noformat}
Does that achieve what you want? This should work with specific or reflect
data too.
Note that you should only add a given schema once. Also, if you have a large
number of schemas (>10?) per file then performance may lag a bit, since it
scans the union linearly each time an entry is appended. But that could easily
be optimized if it became a problem.
I am tempted to make this the default for DataFileWriter: that we add a
constructor that does not include a schema, in which case an empty union is
used. Also, we could then automatically add schemas to the union if they're
not already present. Would that be useful?
> Avro should support multiple schemas from the same AVRO file
> --------------------------------------------------------------
>
> Key: AVRO-127
> URL: https://issues.apache.org/jira/browse/AVRO-127
> Project: Avro
> Issue Type: New Feature
> Environment: all systems
> Reporter: Yang Yang
>
> in our application, we often have to merge together all the day's data into
> one daily file,
> data file schemas must be changing within the day, so we have different
> schemas within the same file.
> AVRO should support multiple schemas within the same file, ideally different
> schemas can be parsed out
> to different children classes of the same schema class.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.