andygrove opened a new pull request, #7285: URL: https://github.com/apache/datafusion-comet/pull/7285
## Which issue does this PR close? Backport of #7221 to `branch-1.2`. #7221 closes #7202 on `main`. Part of #7204. ## Rationale for this change Since #6059, the native Parquet and CSV scans resolve ABFS authentication through Hadoop on the driver. When that resolution is something the native store cannot use, the scan still plans natively and every task then fails. 1.1.0 read these scans natively through the ambient identity, so this is a regression in 1.2.0. #7221 makes the scan fall back to Spark's own reader instead. `branch-1.2` has the regression, so it needs the fix before 1.2.0 ships. 1.1.0 does not have it, so there is no `branch-1.1` backport. ## What changes are included in this PR? `git cherry-pick -x 61fbbe2e14` of the squash commit of #7221, with no adaptations. The cherry-pick applied without conflicts, and the resulting tree is identical to the pick I tested before the source PR merged. It adds `AbfsAuthResolver.nativeFallbackReason` and a new JNI function, `NativeBase.abfsStoreError`, which runs the native store construction on the driver. It also makes `NativeConfig.extractScanObjectStoreOptions` return the fallback reason, tightens the `azure.rs` endpoint checks, and updates `datasources.md`. See #7221 for the full description. ## How are these changes tested? On this branch, with a `libcomet` built from the pick: - `cargo check --all-targets` in `native/` passes. - `AbfsScanFallbackSuite` and `NativeConfigSuite` pass, 59 tests. I did not run the other suites that #7221 ran, and I did not confirm that the new tests fail on `branch-1.2` without the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
