netapp-acheng commented on issue #3440:
URL: https://github.com/apache/polaris/issues/3440#issuecomment-3762414875
I gave the same AWS user with the AWS credential set in the environment
variables with all S3 permission to this s3 path (i.e. allow the user to
perform all s3 actions within the subfolder table2 only.
"Resource": "arn:aws:s3:::sts1-polaris/ns1/table2/*"
From the spark-shell, create the table2 under ns1.
Polaris using the temporary credential + token to PUT this:
PUT
/ns1/table2/metadata/00000-70be4d0b-4132-4b98-aed4-59b74a13ee7c.metadata.json
Polaris using the credential from environment variables to perform these s3
requests. Because I gave the S3 permission to the user (without using
AssumeRole) under subfolder table2, the table2 is created successfully.
PUT
/ns1/table2/data/00001-4-a7768eac-d5a7-4f89-9058-822c3c2e5197-0-00001.parquet
PUT
/ns1/table2/data/00000-3-a7768eac-d5a7-4f89-9058-822c3c2e5197-0-00001.parquet
PUT
/ns1/table2/data/00002-5-a7768eac-d5a7-4f89-9058-822c3c2e5197-0-00001.parquet
PUT /ns1/table2/metadata/9562e1bf-1734-41c7-a005-110af905b287-m0.avro
PUT
/ns1/table2/metadata/snap-7024417234542457332-1-9562e1bf-1734-41c7-a005-110af905b287.avro
HEAD
/ns1/table2/metadata/snap-7024417234542457332-1-9562e1bf-1734-41c7-a005-110af905b287.avro
GET
/ns1/table2/metadata/snap-7024417234542457332-1-9562e1bf-1734-41c7-a005-110af905b287.avro
Within the same spark-shell session, create table3 in ns1. Table created
successfully. But again fail when insert data to table3 (because I did not give
S3 permission to table3 subfolder to this AWS user). This user gets the
permission only through the AssumeRole.
scala> spark.sql("""
| CREATE TABLE IF NOT EXISTS sts1_catalog.ns1.table3 (
| id INT,
| data STRING
| )
| USING iceberg
| TBLPROPERTIES ('format-version'='2')
| """)
res14: org.apache.spark.sql.DataFrame = []
scala> spark.sql("""
| INSERT INTO sts1_catalog.ns1.table3 VALUES
| (1, 'alpha'),
| (2, 'beta'),
| (3, 'gamma')
| """)
26/01/16 19:52:13 ERROR Utils: Aborting task
java.io.UncheckedIOException: Failed to close current writer
This proves that pr-3445 is not using sts token + temporary credentials when
inserting data into the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
