[
https://issues.apache.org/jira/browse/BEAM-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16192568#comment-16192568
]
Etienne Chauchot commented on BEAM-2993:
----------------------------------------
thanks [~jkff] for your points:
* yes it works with the side input example above. What I propose is an
improvement of the AvroIO even if we can workaround using the side Input and
the {{DynamicAvroDestiantions}}
* in the PR that I'm about to send, it indeed choses the schema of the "first"
(but PCollection is not ordered) element of the PCollection. So, the schema
needs to be the same for all elements of the PCollection. This is the case in
our use case. But the current implementation {{write(SCHEMA)}},
{{write(class)}} or {{writeGenericRecords(SCHEMA)}} also needs all the elements
of the PCollection to have {{SCHEMA}} as a schema because this schema is passed
to the {{TypedWrite}} then to the {{ConstantAvroDestination}}. Or am I missing
something?
*As PCollection elements have the same schema in our use case, there is no
point of grouping per schema. And moreover, if we have the ability to do
{{AvroIO.write()}} I guess most of the interests of having a network schema
registry become null, except maybe for the lazy avro coder to avoid doing an
{{element.getSchema()}} each time we {{encode}} or {{decode}} an element
PS: please note that I used {{GenericRecord}} rather than parent
{{IndexedRecord}} to describe our use case in the previous comments to stick to
the generic object chosen in AvroIO :)
> AvroIO.write without specifying a schema
> ----------------------------------------
>
> Key: BEAM-2993
> URL: https://issues.apache.org/jira/browse/BEAM-2993
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-extensions
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
>
> Similarly to https://issues.apache.org/jira/browse/BEAM-2677, we should be
> able to write to avro files using {{AvroIO}} without specifying a schema at
> build time. Consider the following use case: a user has a
> {{PCollection<GenericRecord>}} but the schema is only known while running
> the pipeline. {{AvroIO.writeGenericRecords}} needs the schema, but the
> schema is already available in {{GenericRecord}}. We should be able to call
> {{AvroIO.writeGenericRecords()}} with no schema.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
