andygrove opened a new pull request, #7221:
URL: https://github.com/apache/datafusion-comet/pull/7221
## Which issue does this PR close?
Closes #7202.
Part of #7204.
## Rationale for this change
Since #6059, the native Parquet and CSV scans resolve ABFS authentication
through Hadoop on the driver. When that resolution is something the native
store cannot use, the scan still planned natively, and every task then failed
with the native "unsupported" or driver error. 1.1.0 read these scans natively
through the ambient identity, so this is a regression for 1.2.0.
All of these outcomes are known on the driver at planning time, so the scan
can fall back to Spark's own reader, which authenticates through hadoop-azure,
the same way it already does for scans that span more than one container or
account.
## What changes are included in this PR?
- `AbfsAuthResolver.nativeFallbackReason` returns a fallback reason for:
- a mechanism Hadoop declines: `Custom`, `UserboundSASWithOAuth`, a custom
SAS token provider, and Workload Identity with a custom client assertion
provider;
- OAuth with `RefreshTokenBasedTokenProvider` or
`UserPasswordTokenProvider`, which Hadoop resolves but `object_store` has no
counterpart for;
- a resolution that fails on the driver, including a hadoop-azure that
Comet cannot load or reflect into (missing, or a vendor fork). This also covers
the environment-only configurations in the issue, since the handles load before
the unconfigured check.
- `NativeConfig.extractScanObjectStoreOptions` resolves once and returns
that reason as a `Left`, which `CometNativeScan` and `CometCsvNativeScanExec`
already turn into a fallback. Unconfigured accounts still read natively from
the environment, and supported mechanisms still read natively with the same
options as before.
- The reason names the auth type and provider class, or carries Hadoop's
already-redacted error. It names the URL as scheme, container and host, so a
password in the URL stays out of it.
- `datasources.md` now says these cases fall back instead of failing.
- The native `declined` and `error` arms stay as a safety net. Native writes
to ABFS (off by default) still fail with them.
#7127 touches the same files. Whichever lands second will need a small
rebase.
## How are these changes tested?
New `NativeConfigSuite` tests cover each case: `Custom`, a custom SAS
provider, RefreshToken and UserPassword, an unknown auth type, a driver that
cannot load hadoop-azure, and a URL password kept out of the reason. One more
test checks that unconfigured and SharedKey scans still return native options.
`NativeConfigSuite`, `AbfsAuthResolverSuite` and `AbfsAuthParitySuite` pass on
the default Spark 4.1 profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
