dwsmith1983 commented on PR #6059: URL: https://github.com/apache/datafusion-comet/pull/6059#issuecomment-6022858940
> The container-scoped OAuth concern already tracked in #6605 is materially worsened by provider-specific filtering and remains a P2 blocker. Each round so far fixed one Hadoop rule the Rust lookup had missed, and the next round found another, so this revision stops reimplementing the ABFS driver and asks it instead. `AbfsAuthResolver` (0fb805ec2) builds the classpath's `AbfsConfiguration` the way `AzureBlobFileSystemStore` does and resolves in `initializeClient` order with Hadoop's public getters: `getAuthType`, `getStorageAccountKey`, `getTokenProviderClass` for both provider types, `getTokenProvider` and `getSASTokenProvider` for Hadoop's own validation, and `getPasswordString` for the values. Container-scoped keys, the auth-type guard, trimming, blank values, credential providers and the per-version differences are then whatever hadoop-azure does, on 3.3.4, 3.4.1, 3.4.2 and 3.5.0. The driver forwards the outcome as `comet.azure.*` markers plus the resolved values under the global key names, and `azure.rs` (d4f49d95d) only maps them: the environment is read only when no auth-bearing key is set for the account, a mech anism the native scan cannot build fails naming the class, and any Hadoop exception or reflection failure fails the scan. The description has the tradeoff: parity by construction against a dependency on hadoop-azure being on the driver classpath and reflection on its constructors, with the native side on a Comet-defined contract. Your container-scoped endpoint configuration is one of 137 cases in `AbfsAuthParitySuite` (5538c381a). Each runs through the resolver and, offline, through `FileSystem.newInstance`; the outcome class and root cause must agree, and they do on all four hadoop-azure versions. The expectation per version is recorded with hand-written checks for the cases where versions differ or Hadoop's rules surprise (global auth type parsed before the account one, empty mandatory values, MSI tenant mandatory only on 3.3.4, host with port, `SimpleKeyProvider`'s container-less lookup, both guards, Base64 validation). Writing the cross-check found one more gap, `fs.azure.account.key.null.<account>`, fixed in the same commit. Regeneration is described in the contributor guide. The user guide's Azure section (5fb4bdd88) now describes this model and the changes from 1.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
