mbutrovich commented on code in PR #4309:
URL: https://github.com/apache/datafusion-comet/pull/4309#discussion_r3291356377
##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -367,12 +367,13 @@ case class CometScanRule(session: SparkSession)
val hadoopDerivedProperties =
CometIcebergNativeScan.hadoopToIcebergS3Properties(hadoopS3Options)
- // Extract vended credentials from FileIO (REST catalog credential
vending).
- // FileIO properties take precedence over Hadoop-derived
properties because
- // they contain per-table credentials vended by the REST catalog.
+ // Forward the full FileIO property bag (including
credentials.uri, OAuth tokens,
Review Comment:
On the wire: this rides the same channel that already carries Hadoop
delegation tokens, S3A vended credentials, and Iceberg REST credentials from
driver to executors via `SparkSession` / Hadoop conf, so the property bag here
isn't a new exposure relative to that baseline. Deployments that need wire
encryption already have `spark.network.crypto.enabled`.
On expiry: the properties forwarded in the proto are the catalog bootstrap
identity (REST URI, OAuth client config), not the live credential.
`getCredentialsForPath` is called per request and is the refresh contract,
which is why the SPI is shaped this way rather than serializing a one-shot
credential into the plan.
Were you flagging a specific provider where the bootstrap bag itself carries
a short-lived bearer token?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
