>From Preetham Poluparthi <[email protected]>: Preetham Poluparthi has submitted this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20788?usp=email )
Change subject: [ASTERIXDB-3392][EXT] Fix false warnings while querying parquet ...................................................................... [ASTERIXDB-3392][EXT] Fix false warnings while querying parquet - user model changes: no - storage format changes: no - interface changes: no Details: When querying Parquet files, we were seeing false warning counts, which this patch fixes. It also corrects the Parquet file naming format. Previously, files were named .parquet.zstd; they are now correctly named .zstd.parquet Ext-ref: MB-70108 Change-Id: Id2dc25a30ea1bf7012f945803befc2751f33b86a Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20788 Tested-by: Jenkins <[email protected]> Reviewed-by: Hussain Towaileb <[email protected]> Integration-Tests: Jenkins <[email protected]> Reviewed-by: Preetham Poluparthi <[email protected]> --- M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/provider/ExternalWriterProvider.java 2 files changed, 7 insertions(+), 3 deletions(-) Approvals: Anon. E. Moose #1000171: Preetham Poluparthi: Looks good to me, but someone else must approve Hussain Towaileb: Looks good to me, approved Jenkins: Verified; Verified diff --git a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java index b820147..82653c2 100644 --- a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java +++ b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java @@ -321,9 +321,8 @@ } restoreConfig(ctx); JobConf readerConf = conf; - if (ctx.getWarningCollector().shouldWarn() - && configuration.get(ExternalDataConstants.KEY_INPUT_FORMAT.trim()) - .equals(ExternalDataConstants.INPUT_FORMAT_PARQUET)) { + if (configuration.get(ExternalDataConstants.KEY_INPUT_FORMAT.trim()) + .equals(ExternalDataConstants.INPUT_FORMAT_PARQUET)) { /* * JobConf is used to pass warnings from the ParquetReadSupport to ParquetReader. As multiple * partitions can issue different warnings, we might have a race condition on JobConf. Thus, we diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/provider/ExternalWriterProvider.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/provider/ExternalWriterProvider.java index bdfffa0..763a7a1 100644 --- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/provider/ExternalWriterProvider.java +++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/provider/ExternalWriterProvider.java @@ -99,6 +99,11 @@ Map<String, String> configuration = sink.getConfiguration(); String format = getFormat(configuration); String compression = getCompression(configuration); + if (format.equalsIgnoreCase(ExternalDataConstants.FORMAT_PARQUET)) { + // Parquet file extension format is like .snappy.parquet + return (compression.isEmpty() ? "" : compression.toLowerCase() + ".") + + ExternalDataConstants.FORMAT_PARQUET; + } return format + (compression.isEmpty() ? "" : "." + compression); } -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20788?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Id2dc25a30ea1bf7012f945803befc2751f33b86a Gerrit-Change-Number: 20788 Gerrit-PatchSet: 2 Gerrit-Owner: Preetham Poluparthi <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hussain Towaileb <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Preetham Poluparthi <[email protected]>
