morningman opened a new issue, #68103: URL: https://github.com/apache/doris/issues/68103
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description #67470 (`[fix](iceberg) Route Azure data files through native credentials`) moves Azure Iceberg data access off the Hadoop ABFS compatibility path: - FE preserves Azure account/container/object URIs instead of rewriting them to `s3://container/path`, and emits provider-owned native Azure credentials (`AZURE_AUTH_TYPE`, `AZURE_ENDPOINT`, `AZURE_ACCOUNT_NAME`, `AZURE_SAS_TOKEN`, ...) to BE. - BE routes `FILE_S3` ranges with `provider=azure` through the native Azure Blob client (SharedKey, SAS with expiry validation, OAuth2 client secret), with a separate bounded client cache and SAS-redacted diagnostics. - Iceberg REST vended SAS credentials replace the static authentication group per request, for both the native data path (BE) and the Iceberg FileIO metadata path (FE, `ADLSFileIO`). - Serialized metadata-table tasks carry a fixed credential expiry so the BE JNI scanner fails before opening an expired FileIO. That PR touches 135 files (+13179 / -946) across BE, the FE filesystem SPI, fe-core, the Iceberg connector and the BE Java metadata scanner, which is too large to review as one change. It is being split into a series of independently reviewable PRs. This issue tracks the series; every PR in the series references this issue. ### Split plan and merge order Rules for the series: - Consumers land before producers. BE accepts both the legacy `AWS_*` SharedKey map and the new `AZURE_*` map before FE starts emitting the new one. - All SPI additions land in one PR with default implementations, so the plugin API version is bumped once (filesystem `1.0 -> 2.0`, connector `9.0 -> 10.0`); later PRs do not touch the frozen surface. - Each PR compiles and passes its own unit tests on its own; no PR leaves half-wired behavior on master. | # | PR | Scope | Depends on | |---|---|---|---| | BE-1 | `[feature](be) Native Azure credentials, SAS expiry and client lifetime in S3ClientFactory` | `AzureCredentialOptions` (SharedKey/SAS/OAuth2), `AzureAuthFactory` validation, `S3ClientFactory` native `AZURE_*` protocol + legacy `AWS_*` compatibility, separate expiring Azure client cache, `S3URI` for `abfs[s]://` / `wasb[s]://` / Azure HTTP(S), SAS redaction in logs | - | | BE-2 | `[feature](be) Validate Azure credentials at reader/writer open and keep Azure display paths` | access-time SAS validation in `S3FileReader` / `S3FileSystem`, Azure URI container/account checks in `get_key`, original URI in diagnostics | BE-1 | | FE-1 | `[feature](fe) Storage/connector SPI: access-time validation, backend view selection and request-local storage access` | `StorageProperties` / `FileSystemProperties` / `FileSystemProvider` / `ConnectorStorageContext` / `ConnectorScanRange` default methods, `ConnectorStorageAccess`, `ConnectorStorageAccessResolver`, API version bump | - | | FE-2 | `[fix](fe) Record the synthetic default HDFS binding at creation instead of re-guessing it` | `bindDefault` / `isSyntheticDefault`, `StorageAdapter.isExplicitlyConfigured`, Iceberg `selectEffectiveStorages` drops the synthetic HDFS binding for object-storage-only catalogs | FE-1 | | FE-3 | `[fix](fe) Preserve Azure account authority in AzureUri and fix Azure listing pagination` | `AzureAccountHost`, `AzureUri` literal ABFS object names + original authority, `AzureFileSystem` pagination fixes | - | | FE-4 | `[feature](fe) Emit native Azure credentials (SharedKey/SAS/OAuth2) and route Azure locations by provider` | `AzureAuthType`, `AzureSasToken`, `AzureFileSystemProperties` native `AZURE_*` backend map + `NATIVE` kind + URI preservation, fe-core `StorageAdapter` / `LocationPath` provider routing, `StorageUriUtils` Azure normalization removed, `ObjectInfoAdapter` SAS, `S3SourceOffsetProvider` object keys | FE-1, FE-3 (end-to-end: BE-1, BE-2) | | FE-5 | `[feature](fe) Request-local storage bindings: vended credentials replace the static authentication group` | `AzureVendedSas`, `AzureFileIOSharedKey`, `bindVended`, Iceberg FileIO views in the Azure provider; `FileSystemPluginManager.bindVended/bindAllExcept`, `StorageAdapter.ofVended/ofAllExcept`, `DefaultConnectorContext.newStorageAccessResolver/resolveStorageProperties`, `PluginDrivenSplit` backend file type | FE-1, FE-2, FE-4 | | ICE-1 | `[feature](iceberg) Select provider-owned Azure FileIO credentials in the REST catalog load path` | `IcebergRestFileIOClient` / `IcebergRestFileIOProperties`, `IcebergPostSuccessFileIOInitializationException`, `iceberg-azure` dependency (JDK HTTP transport) | FE-1, FE-5 | | ICE-2 | `[feature](iceberg) Route Iceberg scan and write data access through request-local storage bindings` | `IcebergScanPlanProvider` `ReadStorageAccess`, `IcebergWritePlanProvider` single resolved access per sink, `IcebergStatementScope.readStorageAccess`, `IcebergScanRange.backendFileType` | FE-5, ICE-1 | | ICE-3 | `[feature](iceberg) Carry a fixed FileIO credential expiry with serialized metadata tasks` | `TIcebergFileDesc.file_io_expiry_ms` (optional), both BE JNI readers, `IcebergSysTableJniScanner` expiry check, scanner plugin Azure dependencies, `IcebergMetadataTaskProperties` | ICE-1 (ICE-2 first to avoid conflicts) | | REG | `[test](regression) Native Azure Iceberg acceptance suite` | opt-in `external_table_p2` suite against a pre-provisioned Azure REST catalog | all | Suggested serial order: BE-1, BE-2, FE-1, FE-3, FE-2, FE-4, FE-5, ICE-1, ICE-2, ICE-3, REG. BE-1, FE-1 and FE-3 are independent and can be opened in parallel. ### Progress - [ ] BE-1 - [ ] BE-2 - [ ] FE-1 - [ ] FE-2 - [ ] FE-3 - [ ] FE-4 - [ ] FE-5 - [ ] ICE-1 - [ ] ICE-2 - [ ] ICE-3 - [ ] REG - [ ] Close #67470 once the series is merged ### Compatibility and rolling upgrade - SharedKey: a new FE emits both the `AZURE_*` fields and the matching legacy `AWS_*` group, so an old BE keeps working during a rolling upgrade; a new BE verifies that the two groups agree and rejects conflicting values. - SAS and OAuth2 have no legacy wire form. Upgrade all BEs before enabling native SAS/OAuth2 data access from a new FE. To roll back, stop native SAS/OAuth2 use first, then roll back the FE/BE pair. - Genuine Fabric OneLake locations keep their explicit Hadoop (`FILE_HDFS`) path. ### Carved out of the series - The `$files` metadata-table projection fix (keep `file_size` when `split_offsets` is requested) that was bundled into #67470 is unrelated to Azure and will be submitted as a standalone fix. - The two design notes under `docs/` in #67470 are not merged; the acceptance fixture and rolling-upgrade notes move into the PR descriptions. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
