yuqi1129 commented on code in PR #5243: URL: https://github.com/apache/gravitino/pull/5243#discussion_r1821005120
########## docs/iceberg-rest-service.md: ########## @@ -388,13 +388,28 @@ SELECT * FROM dml.test; You could run Gravitino Iceberg REST server though docker container: ```shell -docker run -d -p 9001:9001 apache/gravitino-iceberg-rest:0.6.0 +docker run -d -p 9001:9001 apache/gravitino-iceberg-rest:0.7.0-incubating ``` -Or build it manually to add custom logics: +Gravitino Iceberg REST server in docker image could access local storage, you could change the configuration by environment variables to access S3 or GCS storage, please refer to storage section for more details. Review Comment: Please add a link to the `storage section`. ########## dev/docker/iceberg-rest-server/start-iceberg-rest-server.sh: ########## @@ -0,0 +1,29 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +set -ex +bin_dir="$(dirname "${BASH_SOURCE-$0}")" +iceberg_rest_server_dir="$(cd "${bin_dir}/../">/dev/null; pwd)" + +cd ${iceberg_rest_server_dir} + +python bin/rewrite_config.py Review Comment: Is the python command installed by default? ########## dev/docker/iceberg-rest-server/iceberg-rest-server-dependency.sh: ########## @@ -34,6 +34,34 @@ cd distribution tar xfz gravitino-iceberg-rest-server-*.tar.gz cp -r gravitino-iceberg-rest-server*-bin ${iceberg_rest_server_dir}/packages/gravitino-iceberg-rest-server +cd ${gravitino_home} +./gradlew :bundles:gcp-bundle:jar +./gradlew :bundles:aws-bundle:jar + +# prepare bundle jar +cd ${iceberg_rest_server_dir} +mkdir -p bundles +cp ${gravitino_home}/bundles/gcp-bundle/build/libs/gravitino-gcp-bundle-*.jar bundles/ +cp ${gravitino_home}/bundles/aws-bundle/build/libs/gravitino-aws-bundle-*.jar bundles/ + +iceberg_gcp_bundle="iceberg-gcp-bundle-1.5.2.jar" +if [ ! -f "bundles/${iceberg_gcp_bundle}" ]; then + wget -P bundles https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-gcp-bundle/1.5.2/${iceberg_gcp_bundle} +fi + +iceberg_aws_bundle="iceberg-aws-bundle-1.5.2.jar" +if [ ! -f "bundles/${iceberg_aws_bundle}" ]; then + wget -P bundles https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws-bundle/1.5.2/${iceberg_aws_bundle} Review Comment: Also, Is `wget` installed? ########## docs/iceberg-rest-service.md: ########## @@ -388,13 +388,28 @@ SELECT * FROM dml.test; You could run Gravitino Iceberg REST server though docker container: ```shell -docker run -d -p 9001:9001 apache/gravitino-iceberg-rest:0.6.0 +docker run -d -p 9001:9001 apache/gravitino-iceberg-rest:0.7.0-incubating ``` -Or build it manually to add custom logics: +Gravitino Iceberg REST server in docker image could access local storage, you could change the configuration by environment variables to access S3 or GCS storage, please refer to storage section for more details. Review Comment: Gravitino Iceberg REST server in docker image could access local storage by default, however, you should add the following variables the storage is cloud/remote storage like S3, xxxx,xxxxx ########## dev/docker/iceberg-rest-server/rewrite_config.py: ########## @@ -0,0 +1,73 @@ +#!/usr/bin/env +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import os + +env_map = { + "GRAVITINO_IO_IMPL" : "io-impl", + "GRAVITINO_URI" : "uri", + "GRAVITINO_WAREHOUSE" : "warehouse", + "GRAVITINO_CREDENTIAL_PROVIDER_TYPE" : "credential-provider-type", + "GRAVITINO_GCS_CREDENTIAL_FILE_PATH" : "gcs-credential-file-path", + "GRAVITINO_S3_ACCESS_KEY" : "s3-access-key-id", + "GRAVITINO_S3_SECRET_KEY" : "s3-secret-access-key", + "GRAVITINO_S3_REGION" : "s3-region", + "GRAVITINO_S3_ROLE_ARN" : "s3-role-arn", + "GRAVITINO_S3_EXTERNAL_ID" : "s3-external-id" +} + + +def parse_config_file(file_path): + config_map = {} + with open(file_path, 'r') as file: + for line in file: + stripped_line = line.strip() + if stripped_line and not stripped_line.startswith('#'): + key, value = stripped_line.split('=') + key = key.strip() + value = value.strip() + config_map[key] = value + return config_map + +config_prefix = "gravitino.iceberg-rest." + +def update_config(config, key, value): + config[config_prefix + key] = value + +config_file_path = 'conf/gravitino-iceberg-rest-server.conf' +config_map = parse_config_file(config_file_path) + +update_config(config_map, "catalog-backend", "jdbc") +update_config(config_map, "jdbc-driver", "org.sqlite.JDBC") +update_config(config_map, "uri", "jdbc:sqlite::memory:") +update_config(config_map, "jdbc-user", "iceberg") +update_config(config_map, "jdbc-password", "iceberg") +update_config(config_map, "jdbc-initialize", "true") +update_config(config_map, "jdbc.schema-version", "V1") Review Comment: I believe we can merge those methods into one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
