wangsheng has uploaded a new patch set (#7). (
http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Supported query Icebreg table by impala
......................................................................
IMPALA-9741: Supported query Icebreg table by impala
This patch mainly realizes the query of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't identity this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When query iceberg table, we pushdown
partition column predicates to iceberg to decided which data files
need to be scanned, and then transformed these information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in custom cluster test test_iceberg.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A
testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A
testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/queries/QueryTest/iceberg_create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg_query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
M tests/custom_cluster/test_iceberg.py
82 files changed, 1,304 insertions(+), 159 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/7
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 7
Gerrit-Owner: wangsheng <sky...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
Gerrit-Reviewer: wangsheng <sky...@163.com>
