vdiravka commented on a change in pull request #2351:
URL: https://github.com/apache/drill/pull/2351#discussion_r753617750
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java
##########
@@ -263,20 +269,29 @@ private void newSchema() throws IOException {
// We don't want this number to be too small either. Ideally, slightly
bigger than the page size,
// but not bigger than the block buffer
int initialPageBufferSize = max(MINIMUM_BUFFER_SIZE, min(pageSize +
pageSize / 10, initialBlockBufferSize));
+ ValuesWriterFactory valWriterFactory = writerVersion ==
WriterVersion.PARQUET_1_0
+ ? new DefaultV1ValuesWriterFactory()
+ : new DefaultV2ValuesWriterFactory();
+
ParquetProperties parquetProperties = ParquetProperties.builder()
.withPageSize(pageSize)
.withDictionaryEncoding(enableDictionary)
.withDictionaryPageSize(initialPageBufferSize)
- .withWriterVersion(writerVersion)
.withAllocator(new ParquetDirectByteBufferAllocator(oContext))
- .withValuesWriterFactory(new DefaultV1ValuesWriterFactory())
+ .withValuesWriterFactory(valWriterFactory)
+ .withWriterVersion(writerVersion)
.build();
+
// TODO: Replace ParquetColumnChunkPageWriteStore with
ColumnChunkPageWriteStore from parquet library
// once DRILL-7906 (PARQUET-1006) will be resolved
pageStore = new
ParquetColumnChunkPageWriteStore(codecFactory.getCompressor(codec), schema,
parquetProperties.getInitialSlabSize(), pageSize,
parquetProperties.getAllocator(),
parquetProperties.getColumnIndexTruncateLength(),
parquetProperties.getPageWriteChecksumEnabled());
- store = new ColumnWriteStoreV1(pageStore, parquetProperties);
+
+ store = writerVersion == WriterVersion.PARQUET_1_0
Review comment:
oh yes, but starting from `java14`. Sorry.
```
store = switch(writerVersion) {
case WriterVersion.PARQUET_1_0 -> new ColumnWriteStoreV1(schema,
pageStore, parquetProperties);
case WriterVersion.PARQUET_2_0 -> new ColumnWriteStoreV2(schema,
pageStore, parquetProperties);
};
```
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetReaderConfig.java
##########
@@ -186,11 +187,13 @@ public ParquetReaderConfig build() {
readerConfig.enableTimeReadCounter =
conf.getBoolean(ENABLE_TIME_READ_COUNTER, readerConfig.enableTimeReadCounter);
}
- // last assign values from session options, session options have higher
priority than other configurations
+ // last assign values from session or query scoped options which have
higher priority than other configurations
if (options != null) {
- String option =
options.getOption(ExecConstants.PARQUET_READER_STRINGS_SIGNED_MIN_MAX_VALIDATOR);
- if (!option.isEmpty()) {
- readerConfig.enableStringsSignedMinMax = Boolean.valueOf(option);
+ String optVal = (String) options.getOption(
+ ExecConstants.PARQUET_READER_STRINGS_SIGNED_MIN_MAX
+ ).getValueMinScope(OptionValue.OptionScope.SESSION);
Review comment:
I thought about other direction :)
Not sure I understood the purpose for this method fully. If session and
query scope options are absent, the system and boot scope options will not be
set to `optVal`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
