ji chen has uploaded a new patch set (#7). (
http://gerrit.cloudera.org:8080/22914 )
Change subject: IMPALA-14081: Support create/drop paimon table for impala
......................................................................
IMPALA-14081: Support create/drop paimon table for impala
this patch mainly implement the creation/drop of paimon table through impala.
syntax for creating paimon table:
CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
(
[col_name data_type ,...]
[PRIMARY KEY (col1,col2)]
)
[PARTITIONED BY (col_name data_type [COMMENT 'col_comment'], ...)]
STORED AS PAIMON
[LOCATION 'hdfs_path']
[TBLPROPERTIES (
'primary-key'='col1,col2',
'file.format' = 'orc/parquet',
'bucket' = '2',
'bucket-key' = 'col3',
];
two types of paimon catalogs are supported.
(1) create table with hive catalog:
CREATE TABLE paimon_hive_cat(userid INT,movieId INT)
STORED AS PAIMON;
(2) create table with hadoop catalog:
CREATE EXTERNAL TABLE paimon_hadoop_cat
STORED AS PAIMON
TBLPROPERTIES('paimon.catalog'='hadoop',
'paimon.catalog_location'='/path/to/paimon_hadoop_catalog',
'paimon.table_identifier'='paimondb.paimontable');
SHOW TABLE STAT/SHOW COLUMN STAT/SHOW PARTITIONS/SHOW FILES statements are also
supported.
Testing:
Add unit test for paimon impala type conversion.
Update default_file_format TestEnumCase in
be/src/service/query-options-test.cc.
Update test case in
testdata/workloads/functional-query/queries/QueryTest/set.test.
Add test cases in metadata/test_show_create_table.py.
Add custom cluster test test_paimon.py.
Change-Id: I57e77f28151e4a91353ef77050f9f0cd7d9d05ef
---
M be/src/service/query-options-test.cc
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M docs/impala.ditamap
M docs/impala_keydefs.ditamap
A docs/topics/impala_paimon.xml
M docs/topics/impala_reserved_words.xml
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AnalysisUtils.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
A fe/src/main/java/org/apache/impala/analysis/paimon/PaimonAnalyzer.java
A
fe/src/main/java/org/apache/impala/analysis/paimon/PaimonCatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
A fe/src/main/java/org/apache/impala/catalog/local/LocalPaimonTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
A fe/src/main/java/org/apache/impala/catalog/paimon/FePaimonTable.java
A fe/src/main/java/org/apache/impala/catalog/paimon/FeShowFileStmtSupport.java
A fe/src/main/java/org/apache/impala/catalog/paimon/ImpalaTypeUtils.java
A fe/src/main/java/org/apache/impala/catalog/paimon/PaimonTable.java
A
fe/src/main/java/org/apache/impala/catalog/paimon/PaimonTableLoadingException.java
A fe/src/main/java/org/apache/impala/catalog/paimon/PaimonUtil.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
A fe/src/test/java/org/apache/impala/catalog/paimon/ImpalaTypeUtilsTest.java
M java/pom.xml
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/schema/schema-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-1db1d7b8-9b21-4241-a0c6-8189c368a0bc-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-2
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-3
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-4
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-5
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/snapshot/LATEST
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/paimon-ddl-negative.test
A testdata/workloads/functional-query/queries/QueryTest/paimon-ddl.test
A testdata/workloads/functional-query/queries/QueryTest/paimon-show-stats.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
A
testdata/workloads/functional-query/queries/QueryTest/show-create-table-paimon.test
A tests/custom_cluster/test_paimon.py
M tests/metadata/test_show_create_table.py
58 files changed, 3,746 insertions(+), 90 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/22914/7
--
To view, visit http://gerrit.cloudera.org:8080/22914
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I57e77f28151e4a91353ef77050f9f0cd7d9d05ef
Gerrit-Change-Number: 22914
Gerrit-PatchSet: 7
Gerrit-Owner: ji chen <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
