[
https://issues.apache.org/jira/browse/BEAM-11913?focusedWorklogId=563141&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-563141
]
ASF GitHub Bot logged work on BEAM-11913:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 09/Mar/21 15:38
Start Date: 09/Mar/21 15:38
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #14171:
URL: https://github.com/apache/beam/pull/14171#discussion_r590479989
##########
File path:
sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
##########
@@ -641,11 +661,14 @@ public ReadFiles withProjection(Schema projectionSchema,
Schema encoderSchema) {
/** Specify Hadoop configuration for ParquetReader. */
public ReadFiles withConfiguration(Map<String, String> configuration) {
+ checkArgument(configuration != null, "configuration can not be null");
return
toBuilder().setConfiguration(SerializableConfiguration.fromMap(configuration)).build();
}
- public ReadFiles withConfiguration(SerializableConfiguration
configuration) {
Review comment:
This was added in the ongoing version and it has not been released so we
don't need to deprecate it, that's the reason why I wanted this fixed before
the branch cut.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 563141)
Time Spent: 40m (was: 0.5h)
> Add support for Hadoop configuration on ParquetIO
> -------------------------------------------------
>
> Key: BEAM-11913
> URL: https://issues.apache.org/jira/browse/BEAM-11913
> Project: Beam
> Issue Type: Improvement
> Components: io-java-parquet
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: P2
> Fix For: 2.29.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This is a common request from users and we did not do it in the past because
> we tried to avoid Hadoop objects in ParquetIO's public API. However there are
> valid reasons to do it:
> 1. Many functionalities of Parquet are configurable via public helper methods
> on Parquet that prepare data inside of Hadoop's Configuration object, e.g.
> Column Projection via
> `{color:#000000}AvroReadSupport{color}.setRequestedProjection({color:#871094}conf{color},
> {color:#871094}projectionSchema{color});` or Predicate Filters via
> `P{color:#000000}arquetInputFormat{color}.setFilterPredicate({color:#871094}sc{color}.hadoopConfiguration(),
> {color:#871094}filterPredicate{color});`. Giving access to those would allow
> power users to do advanced stuff without any maintenance on the IO side.
> 2. The main reason to avoid the Hadoop Configuration object was to align with
> future non Hadoop required APIs on Parquet see PARQUET-1126 for details but
> this does not seem that will happen soon.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
