[
https://issues.apache.org/jira/browse/BEAM-11527?focusedWorklogId=529018&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529018
]
ASF GitHub Bot logged work on BEAM-11527:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Dec/20 04:14
Start Date: 29/Dec/20 04:14
Worklog Time Spent: 10m
Work Description: anantdamle commented on a change in pull request #13619:
URL: https://github.com/apache/beam/pull/13619#discussion_r549561660
##########
File path:
sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
##########
@@ -835,13 +906,18 @@ public void processElement(ProcessContext processContext)
throws Exception {
SeekableByteChannel seekableByteChannel = file.openSeekable();
- AvroParquetReader.Builder builder =
- AvroParquetReader.<GenericRecord>builder(new
BeamParquetInputFile(seekableByteChannel));
+ AvroParquetReader.Builder<GenericRecord> builder =
+ AvroParquetReader.builder(new
BeamParquetInputFile(seekableByteChannel));
if (modelClass != null) {
// all GenericData implementations have a static get method
builder = builder.withDataModel((GenericData)
modelClass.getMethod("get").invoke(null));
}
+ if (hadoopBaseConfig != null) {
Review comment:
looking at the
[`SerializableConfiguration#91`](https://github.com/apache/beam/blob/3bb232fb098700de408f574585dfe74bbaff7230/sdks/java/io/hadoop-common/src/main/java/org/apache/beam/sdk/io/hadoop/SerializableConfiguration.java#L91)
, It seems `null` is the expected value for building a default configuration.
I've also replaced all `configuration != null` checks with
`SerializableConfiguration.newConfiguration(configuration)` for consistency and
to avoid `NPE`. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 529018)
Time Spent: 0.5h (was: 20m)
> Support user configurable Hadoop Configuration flags for ParquetIO
> ------------------------------------------------------------------
>
> Key: BEAM-11527
> URL: https://issues.apache.org/jira/browse/BEAM-11527
> Project: Beam
> Issue Type: Improvement
> Components: io-java-parquet
> Reporter: Anant Damle
> Assignee: Anant Damle
> Priority: P2
> Labels: parquet
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Provide an user configurable interface to provide/input Hadoop configuration
> for ParquetIO.
> Current behaviour supports only
> \{code}AvroReadSupport.AVRO_COMPATIBILITY\{code} flag.
>
> There are now more options supported for reader as introduced through
> [PARQUET-1928|https://issues.apache.org/jira/browse/PARQUET-1928] and
> [PR/831|https://github.com/apache/parquet-mr/pull/831]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
