Yanjia Gary Li has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/14711 )
Change subject: IMPALA-8778: Support Apache Hudi Read Optimized Table ...................................................................... IMPALA-8778: Support Apache Hudi Read Optimized Table Hudi Read Optimized Table contains multiple versions of parquet files, in order to load the correct table, Impala needs to recognize Hudi Read Optimized Table as a HdfsTable and load the latest version of the file using HoodieROTablePathFilter. Tests - Unit test for Hudi in FileMetadataLoader - Query tests in create-table.test - Query tests in hudiparquet.test Change-Id: I65e146b347714df32fe968409ef2dde1f6a25cdf --- M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node.cc M bin/impala-config.sh M bin/rat_exclude_files.txt M common/thrift/CatalogObjects.thrift M fe/pom.xml M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java A fe/src/main/java/org/apache/impala/util/HudiUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M impala-parent/pom.xml M testdata/bin/create-load-data.sh M testdata/bin/generate-schema-statements.py A testdata/data/hudicow/.hoodie/20200112194517.clean A testdata/data/hudicow/.hoodie/20200112194517.clean.inflight A testdata/data/hudicow/.hoodie/20200112194517.clean.requested A testdata/data/hudicow/.hoodie/20200112194517.commit A testdata/data/hudicow/.hoodie/20200112194517.commit.requested A testdata/data/hudicow/.hoodie/20200112194517.inflight A testdata/data/hudicow/.hoodie/20200112194529.clean A testdata/data/hudicow/.hoodie/20200112194529.clean.inflight A testdata/data/hudicow/.hoodie/20200112194529.clean.requested A testdata/data/hudicow/.hoodie/20200112194529.commit A testdata/data/hudicow/.hoodie/20200112194529.commit.requested A testdata/data/hudicow/.hoodie/20200112194529.inflight A testdata/data/hudicow/.hoodie/hoodie.properties A testdata/data/hudicow/year=2015/month=03/day=16/.hoodie_partition_metadata A testdata/data/hudicow/year=2015/month=03/day=16/ca51fa17-681b-4497-85b7-4f68e7a63ee7-0_1-38-282_20200112194529.parquet A testdata/data/hudicow/year=2015/month=03/day=16/ca51fa17-681b-4497-85b7-4f68e7a63ee7-0_1-5-10_20200112194517.parquet A testdata/data/hudicow/year=2015/month=03/day=17/.hoodie_partition_metadata A testdata/data/hudicow/year=2015/month=03/day=17/45c9fa97-e514-41e8-91d2-6098e5995cdb-0_0-38-281_20200112194529.parquet A testdata/data/hudicow/year=2015/month=03/day=17/45c9fa97-e514-41e8-91d2-6098e5995cdb-0_0-5-9_20200112194517.parquet A testdata/data/hudicow/year=2016/month=03/day=15/.hoodie_partition_metadata A testdata/data/hudicow/year=2016/month=03/day=15/17dda230-e48a-4110-8c29-c613a3ac0b70-0_2-38-283_20200112194529.parquet A testdata/data/hudicow/year=2016/month=03/day=15/17dda230-e48a-4110-8c29-c613a3ac0b70-0_2-5-11_20200112194517.parquet M testdata/workloads/functional-query/functional-query_core.csv M testdata/workloads/functional-query/functional-query_dimensions.csv M testdata/workloads/functional-query/functional-query_exhaustive.csv M testdata/workloads/functional-query/functional-query_pairwise.csv M testdata/workloads/functional-query/queries/QueryTest/create-table.test A testdata/workloads/functional-query/queries/QueryTest/hudiparquet.test M tests/common/test_dimensions.py M tests/query_test/test_scanners.py 50 files changed, 582 insertions(+), 48 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/14711/11 -- To view, visit http://gerrit.cloudera.org:8080/14711 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I65e146b347714df32fe968409ef2dde1f6a25cdf Gerrit-Change-Number: 14711 Gerrit-PatchSet: 11 Gerrit-Owner: Yanjia Gary Li <yanjia.gary...@gmail.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Norbert Luksa <norbert.lu...@cloudera.com> Gerrit-Reviewer: Yanjia Gary Li <yanjia.gary...@gmail.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
