kevinjqliu commented on code in PR #3928:
URL: https://github.com/apache/iceberg-python/pull/3928#discussion_r3983541558
##########
dev/run-s3.sh:
##########
@@ -20,14 +20,9 @@
set -ex
-if [ $(docker ps -q --filter "name=pyiceberg-minio" --filter "status=running"
) ]; then
- echo "Minio backend running"
+if [ $(docker ps -q --filter "name=pyiceberg-rustfs" --filter "health=healthy"
) ]; then
Review Comment:
we like health checks!
##########
dev/docker-compose-integration.yml:
##########
@@ -59,41 +59,51 @@ services:
- AWS_REGION=us-east-1
- CATALOG_WAREHOUSE=s3://warehouse/
- CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
- - CATALOG_S3_ENDPOINT=http://minio:9000
+ - CATALOG_S3_ENDPOINT=http://rustfs:9000
- CATALOG_JDBC_STRICT__MODE=true
- minio:
- image: minio/minio
- container_name: pyiceberg-minio
+ rustfs:
Review Comment:
maybe we can just use "object_store" or something generic so we can make
switching easier the next time 😄
##########
dev/docker-compose-integration.yml:
##########
@@ -59,41 +59,51 @@ services:
- AWS_REGION=us-east-1
- CATALOG_WAREHOUSE=s3://warehouse/
- CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
- - CATALOG_S3_ENDPOINT=http://minio:9000
+ - CATALOG_S3_ENDPOINT=http://rustfs:9000
- CATALOG_JDBC_STRICT__MODE=true
- minio:
- image: minio/minio
- container_name: pyiceberg-minio
+ rustfs:
+ image: rustfs/rustfs:1.0.0-rc.5
+ container_name: pyiceberg-rustfs
networks:
iceberg_net:
aliases:
- - warehouse.minio
+ - warehouse.rustfs
ports:
- 9001:9001
- 9000:9000
environment:
- - MINIO_ROOT_USER=admin
- - MINIO_ROOT_PASSWORD=password
- - MINIO_DOMAIN=minio
- command: ["server", "/data", "--console-address", ":9001"]
- mc:
- image: minio/mc
- container_name: pyiceberg-mc
+ - RUSTFS_ACCESS_KEY=admin
+ - RUSTFS_SECRET_KEY=password
+ - RUSTFS_VOLUMES=/data
+ - RUSTFS_ADDRESS=:9000
+ - RUSTFS_CONSOLE_ENABLE=true
+ - RUSTFS_CONSOLE_ADDRESS=:9001
+ # Resolves virtual-hosted-style `warehouse.rustfs` to the warehouse
bucket.
+ # The port is required until rustfs/rustfs#7051 ships.
+ - RUSTFS_SERVER_DOMAINS=rustfs:9000
Review Comment:
can we keep the env vars aligned between this file and `docker-compose.yml`?
##########
dev/docker-compose-integration.yml:
##########
@@ -59,41 +59,51 @@ services:
- AWS_REGION=us-east-1
- CATALOG_WAREHOUSE=s3://warehouse/
- CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
- - CATALOG_S3_ENDPOINT=http://minio:9000
+ - CATALOG_S3_ENDPOINT=http://rustfs:9000
- CATALOG_JDBC_STRICT__MODE=true
- minio:
- image: minio/minio
- container_name: pyiceberg-minio
+ rustfs:
+ image: rustfs/rustfs:1.0.0-rc.5
+ container_name: pyiceberg-rustfs
networks:
iceberg_net:
aliases:
- - warehouse.minio
+ - warehouse.rustfs
ports:
- 9001:9001
- 9000:9000
environment:
- - MINIO_ROOT_USER=admin
- - MINIO_ROOT_PASSWORD=password
- - MINIO_DOMAIN=minio
- command: ["server", "/data", "--console-address", ":9001"]
- mc:
- image: minio/mc
- container_name: pyiceberg-mc
+ - RUSTFS_ACCESS_KEY=admin
+ - RUSTFS_SECRET_KEY=password
+ - RUSTFS_VOLUMES=/data
+ - RUSTFS_ADDRESS=:9000
+ - RUSTFS_CONSOLE_ENABLE=true
+ - RUSTFS_CONSOLE_ADDRESS=:9001
+ # Resolves virtual-hosted-style `warehouse.rustfs` to the warehouse
bucket.
+ # The port is required until rustfs/rustfs#7051 ships.
+ - RUSTFS_SERVER_DOMAINS=rustfs:9000
+ healthcheck:
+ test: ["CMD-SHELL", "curl --fail http://127.0.0.1:9000/health";]
+ interval: 5s
+ timeout: 5s
+ retries: 10
+ start_period: 5s
+ aws-cli:
+ image: amazon/aws-cli:2.36.40
+ container_name: pyiceberg-aws-cli
networks:
iceberg_net:
depends_on:
- - minio
+ rustfs:
+ condition: service_healthy
environment:
- AWS_ACCESS_KEY_ID=admin
- AWS_SECRET_ACCESS_KEY=password
- AWS_REGION=us-east-1
- entrypoint: >
- /bin/sh -c "
- until (/usr/bin/mc alias set minio http://minio:9000 admin password) do
echo '...waiting...' && sleep 1; done;
- /usr/bin/mc mb minio/warehouse;
- /usr/bin/mc policy set public minio/warehouse;
- tail -f /dev/null
- "
+ - AWS_ENDPOINT_URL=http://rustfs:9000
+ entrypoint: /bin/sh
+ command:
+ - -c
+ - "aws s3 mb s3://warehouse"
Review Comment:
same here, the other one uses
```
- "aws s3 rb s3://warehouse --force 2>/dev/null; aws s3 mb
s3://warehouse"
```
should we keep them aligned?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
