Zoltan Borok-Nagy has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/23141 )
Change subject: IMPALA-14018: Adding utility scripts to run Lakekeeper in
Impala dev envinroment
......................................................................
IMPALA-14018: Adding utility scripts to run Lakekeeper in Impala dev envinroment
This patch adds utility scripts to run Lakekeeper (an open source
Iceberg REST Catalog) in Impala's dev environment. Lakekeeper's HDFS
support is in preview phase, so we are using a preview docker image
for now.
IcebergRESTCatalog's config setup is also refactored, and now we don't
always set "credentials" in the SessionContext, only if they are
provided.
Usage
To start Lakekeeper:
testdata/bin/run-lakekeeper.sh
To stop Lakekeeper:
testdata/bin/stop-lakekeeper.sh
Now you can create schemas and tables via Trino (need to rebuild the
Trino image for this, TODO: use docker compose for this):
docker stop impala-minicluster-trino
docker rm impala-minicluster-trino
./testdata/bin/build-trino-docker-image.sh
./testdata/bin/run-trino.sh
Then via Trino CLI:
testdata/bin/trino-cli.sh
show catalogs;
create schema iceberg_lakekeeper.trino_db;
use iceberg_lakekeeper.trino_db;
create table trino_t (i int);
insert into trino_t values (35);
After this, you should be able to query the table via Impala:
mkdir /tmp/iceberg_lakekeeper
cp testdata/bin/minicluster_trino/iceberg_lakekeeper.properties
/tmp/iceberg_lakekeeper
bin/start-impala-cluster.py --no_catalogd \
--impalad_args="--catalogd_deployed=false --use_local_catalog=true \
--catalog_config_dir=/tmp/iceberg_lakekeeper/"
bin/impala-shell.sh
Change-Id: I610f5859f92b2ff82e310f46356e3f118e986b2c
Reviewed-on: http://gerrit.cloudera.org:8080/23141
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
---
M bin/rat_exclude_files.txt
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
A testdata/bin/minicluster_lakekeeper/README.md
A testdata/bin/minicluster_lakekeeper/create-default-warehouse.json
A testdata/bin/minicluster_lakekeeper/docker-compose.yaml
M testdata/bin/minicluster_trino/Dockerfile
A testdata/bin/minicluster_trino/iceberg_lakekeeper.properties
A testdata/bin/run-lakekeeper.sh
A testdata/bin/stop-lakekeeper.sh
9 files changed, 332 insertions(+), 31 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/23141
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I610f5859f92b2ff82e310f46356e3f118e986b2c
Gerrit-Change-Number: 23141
Gerrit-PatchSet: 8
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Daniel Becker <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
