ang6300 commented on issue #32:
URL: https://github.com/apache/polaris/issues/32#issuecomment-2547173623
Hi @lefebsy
I git clone the latest branch and modified
./regtests/run_spark_sql_s3compatible.sh to test with my on-premise S3 storage.
It works and able to create the tables, metadata and data successfully.
Though I added the role-arn in the run_spark_sql_s3compatible.sh. From the sts
and s3 object storage log, I did not see the AssumeRole being used.
curl -s -i -X PUT -H "Authorization: Bearer ${SPARK_BEARER_TOKEN}" \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
http://${POLARIS_HOST:-localhost}:8181/api/management/v1/catalogs/manual_spark \
-d "{
\"currentEntityVersion\":1,
\"properties\": {
\"default-base-location\": \"${S3_LOCATION}\"
},
\"storageConfigInfo\": {
\"storageType\": \"S3_COMPATIBLE\",
\"allowedLocations\":
[\"${S3_LOCATION}/\",\"${S3_LOCATION_2}/\"],
\"s3.endpoint\": \"https://sgdemo.example.com\";,
\"s3.pathStyleAccess\": true,
\"s3.credentials.catalog.accessKeyId\": \"CATALOG_ID\",
\"s3.credentials.catalog.secretAccessKey\": \"CATALOG_SECRET\",
**\"s3.roleArn\":
\"arn:aws:iam::06103531234567:role/polaris-s3-role\"**
}
}"
I have this environment variable to use on premise STS endpoint for the
above roleArn.
export AWS_ENDPOINT_URL_STS="https://sts.sgdemo.example.com";
I expect to see POST command from Polaris to request temporary S3 credential
and STS token but cannot find such request. I am not familiar with Polaris,
please excuse me if my expectation is incorrect.
Below is the list of parquet and metadata objects created by the test
script.
aws s3 ls --recursive s3://polaris-sg
2024-12-16 23:28:59 599
db1/ns1/table1/data/00000-0-4c469ac4-7549-4415-abe5-33e71ee1824f-0-00001.parquet
2024-12-16 23:29:00 599
db1/ns1/table1/data/00000-1-6b8e9cba-86c0-420e-94ba-3da0a70a015c-0-00001.parquet
2024-12-16 23:29:01 599
db1/ns1/table1/data/00000-2-9a140927-8285-45bb-b4c5-406325ec7ecc-0-00001.parquet
2024-12-16 23:28:56 1068
db1/ns1/table1/metadata/00000-aaa77f85-b980-4f09-8bbf-4f9eff867841.metadata.json
2024-12-16 23:29:00 2118
db1/ns1/table1/metadata/00001-7abef047-3216-48c3-b8db-d5b585883a3b.metadata.json
2024-12-16 23:29:01 3119
db1/ns1/table1/metadata/00002-8373053f-9eda-4612-b32b-c60207f0f6fa.metadata.json
2024-12-16 23:29:01 4119
db1/ns1/table1/metadata/00003-b181b656-87e1-4e19-9a9b-49119166904a.metadata.json
2024-12-16 23:28:59 6657
db1/ns1/table1/metadata/471ff32d-0730-47ea-b227-1812509c132f-m0.avro
2024-12-16 23:29:01 6657
db1/ns1/table1/metadata/777c11a6-8268-4874-8321-27ae35081954-m0.avro
2024-12-16 23:29:00 6656
db1/ns1/table1/metadata/ecb30eb7-15f8-4885-adc3-02c2dd01e2e7-m0.avro
2024-12-16 23:29:00 4291
db1/ns1/table1/metadata/snap-1262924453040864421-1-ecb30eb7-15f8-4885-adc3-02c2dd01e2e7.avro
2024-12-16 23:29:01 4341
db1/ns1/table1/metadata/snap-6670614743443502061-1-777c11a6-8268-4874-8321-27ae35081954.avro
2024-12-16 23:28:59 4221
db1/ns1/table1/metadata/snap-879933852356086450-1-471ff32d-0730-47ea-b227-1812509c132f.avro
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
