guangyu-yang-rokt opened a new issue, #12701: URL: https://github.com/apache/gluten/issues/12701
### Backend VL (Velox) ### Bug description When a Spark job reads an Apache Iceberg table through a REST catalog that uses **credential vending** (`X-Iceberg-Access-Delegation: vended-credentials`, e.g. Apache Polaris), Gluten's native scan path cannot read the table's data files: every native `TableScan` fails with S3 403 while the same query on vanilla Spark succeeds. ``` VeloxRuntimeError ... S3ReadFile.cpp:163 preadInternal: Failed to get S3 object due to: 'Access denied' ``` The failure is deterministic (task retries exhaust → job abort) whenever the compute's own IAM identity has no direct grant on the warehouse bucket — which is the *point* of credential vending: the catalog returns per-table, short-TTL, prefix-scoped STS credentials on `loadTable`, and only the JVM `S3FileIO` ever sees them. **Mechanism** - JVM path (works): Iceberg REST `loadTable` puts `s3.access-key-id` / `s3.secret-access-key` / `s3.session-token` on the table's FileIO properties; `S3FileIO` reads data files with them. - Native path (fails): `GlutenIcebergSourceUtil.genSplitInfo` extracts only file paths/starts/lengths/deletes into the `LocalFiles` split; Velox's S3 filesystem builds its client from the global `hive.s3.*` config + the AWS default credential chain. The vended credentials never leave the JVM. There is no per-table (or even session-token) credential channel: velox `S3Config` has no session-token key at all, so an STS triple is not even expressible statically. Reproduced on Gluten v1.6.0 (Spark 3.5.2, iceberg-spark-runtime 1.6.0, Apache Polaris with STS vending). Code on `main` has the same gap: the iceberg module still reads no FileIO properties, and the `fs.s3a.*` → `hive.s3.*` static mapping has no session-token entry. Related but distinct: #10113 / #8170 (session-level / custom credential providers not reaching native) are session-global; vended credentials are **per-table**, so two tables in one bucket can carry different credentials in one query — a bucket- or session-scoped channel cannot express this. **Prior art:** apache/datafusion-comet hit exactly this and fixed it in two stages — static extraction of the vended credentials from the table's FileIO properties into the native scan payload (comet#3523), then a pluggable per-path credential provider for refresh (comet#4309). ### How we fixed it (patch set available, happy to upstream) We run Gluten v1.6.0 with three bundle patches that make native scans of vended-credential tables work end-to-end, validated against Apache Polaris: 1. **JVM extract + carry:** at split planning, read the scan table's `FileIO.properties()`; when a vended `s3.access-key-id`/`s3.secret-access-key` pair is present, attach it (+ session token/expiry/endpoint/region) with the table location to a new table-scoped `ReadRel.LocalFiles.read_properties` map (one `LocalFiles` is single-table by construction). Config-gated with a kill switch whose *off* branch fails native validation → clean vanilla fallback instead of 403s. 2. **Native install:** parse the map onto gluten's `SplitInfo`; union all scans' entries into a `filesystems::TokenProvider` implementation (longest-prefix match of file path against normalized table locations → per-table resolution, multi-table-same-bucket safe) passed to `QueryCtx::create`. The velox line Gluten pins (IBM/velox `dft-*` branches) already threads `fsTokenProvider()` through `FileHandleKey` (credential identity keys the handle cache) into `openFileForRead` for both data and positional-delete files. 3. **Velox S3 consumption:** `S3FileSystem::openFileForRead/Write` consult `FileOptions::tokenProvider` (new `S3AccessTokenKey`/`S3AccessToken` types) and serve the open from a per-credential-fingerprint client cache; `S3Config` gains `hive.s3.session-token` (per-bucket capable) so static STS credentials are expressible too. Known limitation (same as comet#3523's explicit non-goal): credentials are snapshotted at planning; refresh-on-open is a follow-up. If maintainers are interested I can open PRs for the gluten side (and the velox piece to the pinned velox line). Adoption of REST-catalog credential vending is growing quickly (Polaris, Unity, Glue IRC all vend), so I'd expect this class to hit more users as `use_gluten`-style rollouts meet lakehouse catalogs. ### Spark version Spark-3.5.x ### Spark configurations spark.plugins=org.apache.gluten.GlutenPlugin spark.sql.catalog.<cat>=org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.<cat>.type=rest spark.sql.catalog.<cat>.header.X-Iceberg-Access-Delegation=vended-credentials ### System information Gluten v1.6.0 (velox backend, IBM/velox dft-2026_02_06), Spark 3.5.2, JDK 17, iceberg-spark-runtime-3.5_2.12:1.6.0, aws-sdk-cpp 1.11.655, arm64 + x86_64 ### Relevant logs ``` VeloxRuntimeError: ... S3ReadFile.cpp:163 preadInternal: Failed to get S3 object due to: 'Access denied' (HTTP 403 GetObject on the REST-catalog warehouse bucket; IAM: the pod role, not the vended credentials) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
