[
https://issues.apache.org/jira/browse/BEAM-8953?focusedWorklogId=359311&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-359311
]
ASF GitHub Bot logged work on BEAM-8953:
----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Dec/19 11:01
Start Date: 13/Dec/19 11:01
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #10360: [BEAM-8953]
Extend ParquetIO read builders for AvroParquetReader
URL: https://github.com/apache/beam/pull/10360#discussion_r357588539
##########
File path:
sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
##########
@@ -126,6 +127,15 @@ public static Read read(Schema schema) {
return new AutoValue_ParquetIO_Read.Builder().setSchema(schema).build();
}
+ /**
+ * Like {@link #read(Schema)}, with an additional {@link
org.apache.avro.generic.GenericData}
+ * argument, allowing users to signal to the underlying AvroParquetReader
the type of model to
+ * associate with the wrapped {@link
org.apache.parquet.avro.AvroReadSupport} instance.
+ */
+ public static Read read(Schema schema, GenericData model) {
Review comment:
Instead of creating a new constructor, could we simply create a
`withDataModel(GenericData model)` method? The "model" parameter seems
optional.
Users could then chain the methods while using the IO like this:
```
ParquetIO.read(schema).withDataModel(model) ...
ParquetIO.readFiles(schema).withDataModel(model) ...
```
Too look for inspiration see
[withDelimiter()](https://github.com/apache/beam/blob/f3c823c65555c188c3e4ff7213dc3c810d89ab13/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L381)
method, or other similar examples that you can find in other IOs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 359311)
Time Spent: 1h (was: 50m)
> Extend ParquetIO.Read/ReadFiles.Builder to support Avro GenericData model
> -------------------------------------------------------------------------
>
> Key: BEAM-8953
> URL: https://issues.apache.org/jira/browse/BEAM-8953
> Project: Beam
> Issue Type: Improvement
> Components: examples-java
> Affects Versions: 2.16.0
> Reporter: Ryan Berti
> Assignee: Ryan Berti
> Priority: Minor
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When utilizing ParquetIO to deserialize objects into case classes in Scala,
> we'd like to utilize a downstream converter which takes GenericRecords and
> converts them to instances of our case classes, rather than relying on
> ParquetIO to deserialize into the case class via reflection + implementing
> the IndexedRecord interface.
> The ParquetIO.Read / ParquetIO.ReadFiles Builders currently support a
> filepattern + schema / schema arguments respectively. When using the Read /
> ReadFiles Builders with these arguments, the underlying AvroParquetReader
> object that gets created in the ParquetIO.ReadFiles.ReadFn method defaults to
> utilizing an AvroReadSupport instance whose GenericData model gets set to
> SpecificData. We'd like to have the the underlying AvroReadSupport utilize
> the GenericData model, but there's currently no way to force this to happen
> via the existing ParquetIO Read / ReadFiles builders.
> I'd like to extend the ParquetIO Read / ReadFiles builders to support a new
> method allowing users to define a GenericData model, which will then be
> passed into the AvroParquetReader builder. I've tested and validated that
> this method allows ParquetIO to generate GenericRecord instances without
> requiring that the users classes can be reflectively instantiated and
> initialized via the IndexedRecord interface.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
