Yanjia Gary Li has uploaded a new patch set (#13). ( 
http://gerrit.cloudera.org:8080/14711 )

Change subject: IMPALA-8778: Support Apache Hudi Read Optimized Table
......................................................................

IMPALA-8778: Support Apache Hudi Read Optimized Table

Hudi Read Optimized Table contains multiple versions of parquet files,
in order to load the correct table, Impala needs to recognize Hudi Read
Optimized Table as a HdfsTable and load the latest version of the file
using HoodieROTablePathFilter.

Tests
 - Unit test for Hudi in FileMetadataLoader
 - Query tests in create-table.test
 - Query tests in hudiparquet.test

Change-Id: I65e146b347714df32fe968409ef2dde1f6a25cdf
---
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node.cc
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java
M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/util/HudiUtil.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M impala-parent/pom.xml
M testdata/bin/create-load-data.sh
M testdata/bin/generate-schema-statements.py
A testdata/data/hudicow/.hoodie/20200112194517.clean
A testdata/data/hudicow/.hoodie/20200112194517.clean.inflight
A testdata/data/hudicow/.hoodie/20200112194517.clean.requested
A testdata/data/hudicow/.hoodie/20200112194517.commit
A testdata/data/hudicow/.hoodie/20200112194517.commit.requested
A testdata/data/hudicow/.hoodie/20200112194517.inflight
A testdata/data/hudicow/.hoodie/20200112194529.clean
A testdata/data/hudicow/.hoodie/20200112194529.clean.inflight
A testdata/data/hudicow/.hoodie/20200112194529.clean.requested
A testdata/data/hudicow/.hoodie/20200112194529.commit
A testdata/data/hudicow/.hoodie/20200112194529.commit.requested
A testdata/data/hudicow/.hoodie/20200112194529.inflight
A testdata/data/hudicow/.hoodie/hoodie.properties
A testdata/data/hudicow/year=2015/month=03/day=16/.hoodie_partition_metadata
A 
testdata/data/hudicow/year=2015/month=03/day=16/ca51fa17-681b-4497-85b7-4f68e7a63ee7-0_1-38-282_20200112194529.parquet
A 
testdata/data/hudicow/year=2015/month=03/day=16/ca51fa17-681b-4497-85b7-4f68e7a63ee7-0_1-5-10_20200112194517.parquet
A testdata/data/hudicow/year=2015/month=03/day=17/.hoodie_partition_metadata
A 
testdata/data/hudicow/year=2015/month=03/day=17/45c9fa97-e514-41e8-91d2-6098e5995cdb-0_0-38-281_20200112194529.parquet
A 
testdata/data/hudicow/year=2015/month=03/day=17/45c9fa97-e514-41e8-91d2-6098e5995cdb-0_0-5-9_20200112194517.parquet
A testdata/data/hudicow/year=2016/month=03/day=15/.hoodie_partition_metadata
A 
testdata/data/hudicow/year=2016/month=03/day=15/17dda230-e48a-4110-8c29-c613a3ac0b70-0_2-38-283_20200112194529.parquet
A 
testdata/data/hudicow/year=2016/month=03/day=15/17dda230-e48a-4110-8c29-c613a3ac0b70-0_2-5-11_20200112194517.parquet
M testdata/workloads/functional-query/functional-query_core.csv
M testdata/workloads/functional-query/functional-query_dimensions.csv
M testdata/workloads/functional-query/functional-query_exhaustive.csv
M testdata/workloads/functional-query/functional-query_pairwise.csv
M testdata/workloads/functional-query/queries/QueryTest/create-table.test
A testdata/workloads/functional-query/queries/QueryTest/hudiparquet.test
M tests/common/test_dimensions.py
M tests/query_test/test_scanners.py
50 files changed, 582 insertions(+), 48 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/14711/13
--
To view, visit http://gerrit.cloudera.org:8080/14711
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I65e146b347714df32fe968409ef2dde1f6a25cdf
Gerrit-Change-Number: 14711
Gerrit-PatchSet: 13
Gerrit-Owner: Yanjia Gary Li <yanjia.gary...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <norbert.lu...@cloudera.com>
Gerrit-Reviewer: Yanjia Gary Li <yanjia.gary...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>

Reply via email to