peterphitran commented on code in PR #18238:
URL: https://github.com/apache/iceberg/pull/18238#discussion_r4147670218
##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -1528,16 +1531,29 @@ public <D> CloseableIterable<D> build() {
if (batchedReaderFunc != null
|| batchedReaderFuncWithSchema != null
|| readerFunction != null) {
+ // Hadoop's Configuration rejects null values
+ Map<String, String> readProperties =
+ Maps.filterEntries(
+ properties,
+ entry ->
+ entry.getValue() != null
+ &&
!READ_PROPERTIES_TO_REMOVE.contains(entry.getKey()));
+
ParquetReadOptions.Builder optionsBuilder;
if (file instanceof HadoopConfigurable) {
// remove read properties already set that may conflict with this
read
Configuration conf = new Configuration(((HadoopConfigurable)
file).getConf());
for (String property : READ_PROPERTIES_TO_REMOVE) {
conf.unset(property);
}
+ // must happen before builder(conf) below, which parses some fields
from conf once
+ readProperties.forEach(conf::set);
+ applyVectoredIoDefault(new HadoopParquetConfiguration(conf));
Review Comment:
could use Hadoop's Configuration.setIfUnset and drop the wrapper, if that's
preferred. PlainParquetConfiguration doesn't have setIfUnset, so would still
need its own check, something like:
```
conf.setIfUnset(ParquetInputFormat.HADOOP_VECTORED_IO_ENABLED, "true");
optionsBuilder = HadoopReadOptions.builder(conf);
} else {
PlainParquetConfiguration conf = new
PlainParquetConfiguration(readProperties);
if (conf.get(ParquetInputFormat.HADOOP_VECTORED_IO_ENABLED) == null) {
conf.setBoolean(ParquetInputFormat.HADOOP_VECTORED_IO_ENABLED, true);
}
optionsBuilder = ParquetReadOptions.builder(conf);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
