ji chen has uploaded a new patch set (#6). (
http://gerrit.cloudera.org:8080/22914 )
Change subject: IMPALA-14081: Support create/drop paimon table for impala
......................................................................
IMPALA-14081: Support create/drop paimon table for impala
this patch mainly implement the creation/drop of paimon table through impala.
syntax for creating paimon table:
CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
(
[col_name data_type ,...]
[PRIMARY KEY (col1,col2)]
)
[PARTITIONED BY (col_name data_type [COMMENT 'col_comment'], ...)]
STORED AS PAIMON
[LOCATION 'hdfs_path']
[TBLPROPERTIES (
'primary-key'='col1,col2',
'file.format' = 'orc/parquet',
'bucket' = '2',
'bucket-key' = 'col3',
];
two types of paimon catalogs are supported.
(1) create table with hive catalog:
CREATE TABLE paimon_hive_cat(userid INT,movieId INT)
STORED AS PAIMON;
(2) create table with hadoop catalog:
CREATE EXTERNAL TABLE paimon_hadoop_cat
STORED AS PAIMON
TBLPROPERTIES('paimon.catalog'='hadoop',
'paimon.catalog_location'='/path/to/paimon_hadoop_catalog',
'paimon.table_identifier'='paimondb.paimontable');
SHOW TABLE STAT/SHOW COLUMN STAT/SHOW PARTITIONS/SHOW FILES statements are also
supported.
Testing:
Add unit test for paimon impala type conversion.
Update default_file_format TestEnumCase in
be/src/service/query-options-test.cc.
Update test case in
testdata/workloads/functional-query/queries/QueryTest/set.test.
Add test cases in metadata/test_show_create_table.py.
Add custom cluster test test_paimon.py.
Change-Id: I57e77f28151e4a91353ef77050f9f0cd7d9d05ef
---
M be/src/service/query-options-test.cc
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M docs/impala.ditamap
M docs/impala_keydefs.ditamap
A docs/topics/impala_paimon.xml
M docs/topics/impala_reserved_words.xml
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AnalysisUtils.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
A fe/src/main/java/org/apache/impala/analysis/paimon/PaimonAnalyzer.java
A
fe/src/main/java/org/apache/impala/analysis/paimon/PaimonCatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
A fe/src/main/java/org/apache/impala/catalog/local/LocalPaimonTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
A fe/src/main/java/org/apache/impala/catalog/paimon/FePaimonTable.java
A fe/src/main/java/org/apache/impala/catalog/paimon/FeShowFileStmtSupport.java
A fe/src/main/java/org/apache/impala/catalog/paimon/ImpalaTypeUtils.java
A fe/src/main/java/org/apache/impala/catalog/paimon/PaimonTable.java
A
fe/src/main/java/org/apache/impala/catalog/paimon/PaimonTableLoadingException.java
A fe/src/main/java/org/apache/impala/catalog/paimon/PaimonUtil.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
A fe/src/test/java/org/apache/impala/catalog/paimon/ImpalaTypeUtilsTest.java
M java/pom.xml
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/bucket-0/data-c8f2e95a-29ab-4c48-81f0-759337368f9b-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/index/index-dad92a93-a433-437f-9676-49c4b72e0261-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/index-manifest-a3e48eaa-babc-43cf-9481-3ead5024875d-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-761aa062-e0f9-49f7-b042-5d67f27dc463-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-list-063d626f-0ba6-4e26-98d4-1aab432b8c93-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-list-063d626f-0ba6-4e26-98d4-1aab432b8c93-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-list-c8a4c812-4b0f-4d47-ae19-a9752cf5b380-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-list-c8a4c812-4b0f-4d47-ae19-a9752cf5b380-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/schema/schema-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/snapshot/snapshot-2
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-1db1d7b8-9b21-4241-a0c6-8189c368a0bc-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-list-0d476a95-8f7e-4440-a444-c95d6f5994fc-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-list-0d476a95-8f7e-4440-a444-c95d6f5994fc-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-list-7d4be95d-a1af-4e9e-be9c-80cf5834a0a9-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-list-7d4be95d-a1af-4e9e-be9c-80cf5834a0a9-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-list-bdc200c7-5434-48bd-a6a8-dfad46747221-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-list-bdc200c7-5434-48bd-a6a8-dfad46747221-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-2
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-3
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-4
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-5
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/snapshot/snapshot-2
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/snapshot/snapshot-3
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/statistics/stat-aa9d7545-ac10-47c7-9d5b-3a50fc5d50c2-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/statistics/stat-ddd0adaa-6188-4f52-9aef-b56f1a5a83a2-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/userId=1/bucket-0/data-964618f9-57d1-4bb2-a253-543ad9c5883e-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/userId=2/bucket-0/data-d4624994-754e-4b53-a0d6-4e77d278df34-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/userId=3/bucket-0/data-6dc75745-8879-4741-b516-ea5188448a82-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/userId=4/bucket-0/data-19d31044-cf9d-4ee2-9235-c28dab1a491d-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/userId=5/bucket-0/data-8691b1e1-bbbd-4046-b772-948732f93478-0.parquet
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/paimon-ddl-negative.test
A testdata/workloads/functional-query/queries/QueryTest/paimon-ddl.test
A testdata/workloads/functional-query/queries/QueryTest/paimon-show-stats.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
A
testdata/workloads/functional-query/queries/QueryTest/show-create-table-paimon.test
A tests/custom_cluster/test_paimon.py
M tests/metadata/test_show_create_table.py
84 files changed, 3,967 insertions(+), 182 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/22914/6
--
To view, visit http://gerrit.cloudera.org:8080/22914
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I57e77f28151e4a91353ef77050f9f0cd7d9d05ef
Gerrit-Change-Number: 22914
Gerrit-PatchSet: 6
Gerrit-Owner: ji chen <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
