MonkeyCanCode opened a new issue, #390: URL: https://github.com/apache/polaris/issues/390
### Is this a possible security vulnerability? - [X] This is NOT a possible security vulnerability ### Describe the bug With proper settings, spark can fail on credential vending when performing `remove_orphan_files`. This seems to be related to https://github.com/apache/iceberg/pull/7914. Here is the slack thread for this discussion: https://apache-iceberg.slack.com/archives/C07HADW86HZ/p1729527019849439 Also, this operation works when using Trino with vended-credentials-enabled enabled. ### To Reproduce 1. Setup proper settings for Spark to work with Polaris 2. Ensure spark client can perform DDL/DML and couple data ops on Iceberg table based on settings from step 1 3. Perform remove_orphan_files within spark ### Actual Behavior Spark will first reported `No FileSystem for scheme "s3"` as it is trying to use HadoopFileIO. After manually set the spark.hadoop.fs.s3.impl to S3A. If the client has S3 credential with needed access, it will work. However, if through credential vending from Polars, it can fail (in this case, client doesn't have S3 credential). ### Expected Behavior Spark should use credential vending then perform remove_orphan_files and no manual specify of `spark.hadoop.fs.s3.impl` should be needed. ### Additional context _No response_ ### System information Polaris version: latest Iceberg version: 1.6.1 Spark version: 3.5.2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
