[
https://issues.apache.org/jira/browse/IMPALA-14018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18004633#comment-18004633
]
ASF subversion and git services commented on IMPALA-14018:
----------------------------------------------------------
Commit 9c12ef66cc9c4c54388d07b638fdc86548e392c2 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=9c12ef66c ]
IMPALA-14018: Adding utility scripts to run Lakekeeper in Impala dev envinroment
This patch adds utility scripts to run Lakekeeper (an open source
Iceberg REST Catalog) in Impala's dev environment. Lakekeeper's HDFS
support is in preview phase, so we are using a preview docker image
for now.
IcebergRESTCatalog's config setup is also refactored, and now we don't
always set "credentials" in the SessionContext, only if they are
provided.
Usage
To start Lakekeeper:
testdata/bin/run-lakekeeper.sh
To stop Lakekeeper:
testdata/bin/stop-lakekeeper.sh
Now you can create schemas and tables via Trino (need to rebuild the
Trino image for this, TODO: use docker compose for this):
docker stop impala-minicluster-trino
docker rm impala-minicluster-trino
./testdata/bin/build-trino-docker-image.sh
./testdata/bin/run-trino.sh
Then via Trino CLI:
testdata/bin/trino-cli.sh
show catalogs;
create schema iceberg_lakekeeper.trino_db;
use iceberg_lakekeeper.trino_db;
create table trino_t (i int);
insert into trino_t values (35);
After this, you should be able to query the table via Impala:
mkdir /tmp/iceberg_lakekeeper
cp testdata/bin/minicluster_trino/iceberg_lakekeeper.properties
/tmp/iceberg_lakekeeper
bin/start-impala-cluster.py --no_catalogd \
--impalad_args="--catalogd_deployed=false --use_local_catalog=true \
--catalog_config_dir=/tmp/iceberg_lakekeeper/"
bin/impala-shell.sh
Change-Id: I610f5859f92b2ff82e310f46356e3f118e986b2c
Reviewed-on: http://gerrit.cloudera.org:8080/23141
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Test Impala against Lakekeeper
> ------------------------------
>
> Key: IMPALA-14018
> URL: https://issues.apache.org/jira/browse/IMPALA-14018
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure, Test
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> Now that Lakekeeper supports HDFS, we could integrate it into our test
> environment, so we could have tests against a real Iceberg REST server, not
> just against our stub.
> https://github.com/lakekeeper/lakekeeper/blob/ct/hdfs/docs/docs/storage.md#hdfs-storage
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
