wangsheng has uploaded a new patch set (#15). (
http://gerrit.cloudera.org:8080/16143 )
Change subject: IMPALA-9741: Support querying Iceberg table by impala
......................................................................
IMPALA-9741: Support querying Iceberg table by impala
This patch mainly realizes the querying of iceberg table through impala,
we can use the following sql to create an external iceberg table:
CREATE EXTERNAL TABLE default.iceberg_test (
level string,
event_time timestamp,
message string,
)
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
Or just including table name and location like this:
CREATE EXTERNAL TABLE default.iceberg_test
STORED AS ICEBERG
LOCATION 'hdfs://xxx'
TBLPROPERTIES ('iceberg_file_format'='parquet');
'iceberg_file_format' is the file format in iceberg, currently only
support PARQUET, other format would be supported in the future. And
if you don't specify this property in your SQL, default file format
is PARQUET.
We achieved this function by treating the iceberg table as normal
unpartitioned hdfs table. When querying iceberg table, we pushdown
partition column predicates to iceberg to decide which data files
need to be scanned, and then transfer this information to BE to
do the real scan operation.
Testing:
- Unit test for Iceberg in FileMetadataLoaderTest
- Create table tests in functional_schema_template.sql
- Iceberg table query test in test_scanners.py
Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
---
M be/src/runtime/descriptors.cc
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M testdata/data/README
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00001-100-e1a80ed6-1064-494d-9cdd-c4a30c1ab8dc-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00003-102-511427f2-85f0-43ae-9b39-a456f8dc57b6-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00004-103-00fc55e1-6ef7-4241-ace2-6d075b9737fc-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00006-105-ef9e76d5-c060-4040-8aa1-b7c275610daa-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00007-106-c09c9c8d-9478-44f9-8501-f85f53112bc3-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00009-108-3b4f06ac-dca3-4f4e-be60-bf42d9927b5b-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00011-110-1e653ccf-0963-4fb0-941c-32c9de13268b-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00012-111-dfa70658-eb4b-4fa0-9ffa-b892cf90d6ac-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00014-113-2d16e751-e2a4-4856-ab89-145996e3815e-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00015-114-0f710621-cbbf-4509-a93d-b58808978e2e-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00017-116-0b666c79-53df-4507-906c-542e65a83443-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00019-118-1bc6bc6e-e061-4da3-9d1e-a427a306c471-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00020-119-ae7b2c67-1538-4429-8246-4998960e3817-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00022-121-8db0f1e1-d88c-4aad-a8b3-24fd07329cdb-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00023-122-de57b6b0-f54b-40ac-85cd-e783505094b6-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00025-124-abc7db70-869b-4b31-bd08-af6a65ef6454-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00027-126-b6f00d23-18c8-4986-a3bf-470423740bca-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00028-127-1d3d230e-44f7-40d6-b187-e846e1f9fd55-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00030-129-563ad56e-ce2e-4fd1-99f2-ec430fbc683a-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/data/00031-130-a08e8345-71dc-4eab-8ac2-d945dd9651a7-00000.parquet
A
testdata/data/iceberg_test/iceberg_non_partitioned/metadata/31d5b60b-cf6b-43e4-bcca-5d1845fd4c69-m0.avro
A
testdata/data/iceberg_test/iceberg_non_partitioned/metadata/snap-3073203018179936330-1-31d5b60b-cf6b-43e4-bcca-5d1845fd4c69.avro
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_non_partitioned/metadata/version-hint.text
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00001-67-03f4a155-5c92-4ad4-850e-ae0a3b7c194b-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00006-72-07b295fc-d263-421e-8c00-6959fac7a930-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00009-75-d7c8ce71-63c1-43ad-8728-6615ef0bdc31-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00017-83-8e011813-b7ae-41c3-aac5-399104b45a97-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00023-89-08d70289-11ff-477b-83ab-6cf63dd35250-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00027-93-1807aad2-214f-4b2f-a6c3-e920eb77219d-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00030-96-313966b3-5c55-48cb-9e23-3abb1c4f812c-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-08/action=view/00031-97-d13159bb-4533-4acb-98ef-579dcf3b3bf3-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00004-70-7603d78b-1b7b-4fb1-b2aa-665146a81112-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00014-80-bedd207e-4b5f-4d2b-957a-b583d0f83adb-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00015-81-5c91e152-e1de-4fe5-b18f-0126a48a2fb3-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00019-85-302f9f1e-059a-4d12-a761-633a4752f531-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00020-86-b0c0b05c-09da-4743-8903-3ec48266c50f-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-09/action=click/00028-94-ee10232e-4e1d-4259-853a-74be1d19028a-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00003-69-3728f5a4-8be2-4574-b422-c1c7a7a63367-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00007-73-0fcb2338-895f-4147-90e3-dee95433bb03-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00011-77-e4bfb177-120f-4d97-be49-ecddd1e3de7d-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00012-78-ee6b490b-5544-4676-9a43-c38d0dcd4102-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00022-88-8f638a54-34b9-4f2f-855c-0b95b2b48094-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/data/event_time_hour=2020-01-01-10/action=download/00025-91-8b7ebe94-1c15-43ac-bcec-dc61a85c889c-00000.parquet
A
testdata/data/iceberg_test/iceberg_partitioned/metadata/a02a7aba-35c1-437c-905d-845d0ca9f19f-m0.avro
A
testdata/data/iceberg_test/iceberg_partitioned/metadata/snap-3643999835850110095-1-a02a7aba-35c1-437c-905d-845d0ca9f19f.avro
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_partitioned/metadata/version-hint.text
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
R testdata/workloads/functional-query/queries/QueryTest/iceberg-create.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-profile.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test
M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test
R tests/query_test/test_iceberg.py
M tests/query_test/test_scanners.py
88 files changed, 1,510 insertions(+), 196 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/16143/15
--
To view, visit http://gerrit.cloudera.org:8080/16143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I856cfee4f3397d1a89cf17650e8d4fbfe1f2b006
Gerrit-Change-Number: 16143
Gerrit-PatchSet: 15
Gerrit-Owner: wangsheng <sky...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>
Gerrit-Reviewer: wangsheng <sky...@163.com>
