andygrove commented on code in PR #7221:
URL: https://github.com/apache/datafusion-comet/pull/7221#discussion_r4379189034
##########
spark/src/main/scala/org/apache/comet/objectstore/AbfsAuthResolver.scala:
##########
@@ -467,6 +469,42 @@ private[comet] object AbfsAuthResolver extends Logging {
Markers.ERROR_MESSAGE -> message)
}
+ /**
+ * Why the native store cannot read `uri` under `outcome`, or None when it
can: a mechanism
+ * Hadoop named that runs Java code per request, a built-in OAuth provider
with no
+ * `object_store` counterpart, or a resolution that failed. All are known on
the driver, so the
+ * scan falls back to Spark's own reader, which authenticates through
hadoop-azure, instead of
+ * planning natively and failing every task.
+ */
+ def nativeFallbackReason(uri: URI, outcome: Outcome): Option[String] = {
+ // The container and host only: a password in the user info stays out of
the reason.
+ val authority = Option(uri.getRawAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val safeAuthority =
+ if (at < 0) authority
+ else authority.substring(0, at).takeWhile(_ != ':') +
authority.substring(at)
+ val location = s"${uri.getScheme}://$safeAuthority"
+ def unsupported(mechanism: String): String =
+ s"Native scan does not support ABFS authentication $mechanism for
$location"
+ outcome match {
+ case Declined(authType, providerClass, assertionProviderClass) =>
+ val classes = providerClass.map(c => s" with provider
$c").getOrElse("") +
+ assertionProviderClass.map(c => s" and client assertion provider
$c").getOrElse("")
+ Some(unsupported(s"auth type $authType$classes"))
+ case Resolved(
+ AuthTypes.OAUTH,
+ Some(c @ (Providers.REFRESH_TOKEN | Providers.USER_PASSWORD)),
+ _) =>
Review Comment:
I traced the Workload Identity authority case I left out above, and it fails
the same way. With an `http://` `fs.azure.account.oauth2.msi.authority`, the
store builds and then every token request fails with `reqwest ... BadScheme`,
because `object_store` builds its client with `https_only` unless `allow_http`
is set. I saw this in a throwaway `azure.rs` test, not just from reading the
code, and the `http://` client credentials endpoint failed identically. Hadoop
posts to that authority over `http`, so Spark reads the account.
`nativeFallbackReason` now declines a `Resolved`
`WorkloadIdentityTokenProvider` whose authority is not an `https` URL with a
host. The reason leaves the authority out, as it does for the client
credentials endpoint. The MSI provider is unaffected: `object_store` does not
use the authority on that path. The new tests cover the fallback, a password in
the authority staying out of the reason, and an `https` authority with a path
staying native, and `datasources.md` and the `azure.rs` doc line now include
it. Pushed in 5ee970e6c5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
