damccorm opened a new issue, #20770:
URL: https://github.com/apache/beam/issues/20770
There are multiple issues wrong with the API of withProjection:
1. The current API requires an extra encoderSchema that is not needed when
projecting data in Parquet. The simplest way to get this with the Parquet API
is by passing the projectionSchema like this:
>
{color:#000000}AvroReadSupport{color}.setAvroReadSchema({color:#871094}conf{color},
{color:#871094}projectionSchema{color});
>
olor:#000000}AvroReadSupport{color}.setRequestedProjection({color:#871094}conf{color},
{color:#871094}projectionSchema{color});
> We can offer an alternative method `withProjection(Configuration conf,
List<String\> fields)` so users don't have to build their own projection
Schema, but historically we have let users to rely on the upstream connector
API. If we follow this we can better document in ParquetIO how to project
fields by relying in the Parquet APIs and avoid maintaining this extra code in
the Beam side.
Imported from Jira
[BEAM-11908](https://issues.apache.org/jira/browse/BEAM-11908). Original Jira
may contain additional context.
Reported by: iemejia.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
