dwsmith1983 commented on code in PR #6753:
URL: https://github.com/apache/datafusion-comet/pull/6753#discussion_r4218556581
##########
spark/src/main/scala/org/apache/comet/objectstore/NativeConfig.scala:
##########
@@ -197,6 +253,46 @@ object NativeConfig {
* The result feeds object_store's parse_url_opts natively.
*/
def extractObjectStoreOptions(hadoopConf: Configuration, uri: URI):
Map[String, String] = {
+ val (options, aliasOptions) =
objectStoreOptionsAndAliasOptions(hadoopConf, uri)
+ options ++ aliasOptions
+ }
+
+ /**
+ * The object store options for a scan of `uris`:
`extractObjectStoreOptions` of one URI per
+ * scheme, and of one per bucket for an opted-in alias. Each scheme forwards
keys under its own
+ * prefixes. An alias bucket keeps only the settings translated for that
bucket, applied after
+ * every scheme's own keys so that they win over a raw
`fs.s3a.bucket.<bucket>.*` key.
+ */
+ def extractObjectStoreOptions(
+ hadoopConf: Configuration,
+ uris: Iterable[URI]): Map[String, String] = {
+ val s3CompliantSchemes = resolveS3CompliantSchemes(hadoopConf)
+ val representatives = scala.collection.mutable.LinkedHashMap[(String,
Option[String]), URI]()
+ uris.foreach { uri =>
+ val scheme = lowerScheme(uri).getOrElse("file")
+ val aliasBucket =
+ if (s3CompliantSchemes.contains(scheme)) bucketForUri(uri,
s3CompliantSchemes) else None
+ representatives.getOrElseUpdate((scheme, aliasBucket), uri)
+ }
+ val parts = representatives.values.toSeq.map { uri =>
+ val (options, aliasOptions) =
objectStoreOptionsAndAliasOptions(hadoopConf, uri)
Review Comment:
> Please translate alias options only when that scheme actually routes to
native S3.
Fixed. A file whose scheme routes to libhdfs now contributes no translated
alias settings to the scan's options, using the same `ObjectStoreKey.isLibhdfs`
rule the fallback check and the native planner apply. With `blob` in both
scheme lists, a scan over `s3a://bucket/...` and `blob://bucket/...` now
forwards the same options as the `s3a` file alone, so the S3 store keeps
`fs.s3a.endpoint`. `NativeConfigSuite` covers your case in both file orders, a
libhdfs-routed alias on its own, and a native-S3 alias next to `s3a`, which
still gets its translated endpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
