djouallah opened a new issue, #6058:
URL: https://github.com/apache/datafusion-comet/issues/6058
### What is the problem the feature request solves?
The native Iceberg scan declines every table whose data files live on Azure
Data Lake Storage Gen2, so Comet accelerates nothing for Iceberg on Azure.
`CometScanRule.icebergReadableSchemes` is `Set("file", "s3", "s3a", "gs",
"oss")`. An Iceberg table at
`abfss://<container>@<account>.dfs.core.windows.net/...` (or OneLake's
`abfss://<workspace>@onelake.dfs.fabric.microsoft.com/...`) is rejected at
planning time and the scan runs on the JVM exactly as it would without the
plugin. The comment above the set says the omission is deliberate because
iceberg-rust's OpenDAL storage factory could not build the scheme.
That no longer looks true. At the iceberg-rust rev Comet pins (`665c64e`):
- `native/Cargo.toml` already enables the `opendal-azdls` feature on
`iceberg-storage-opendal`.
- `OpenDalStorageFactory::Azdls` exists and builds an
`OpenDalStorage::Azdls` from `adls.*` properties (`adls.account-name`,
`adls.account-key`, `adls.sas-token`, `adls.tenant-id`, `adls.client-id`,
`adls.client-secret`, `adls.authority-host`, `adls.connection-string`).
- On the JVM side, `NativeConfig.scala` already maps `abfs` and `abfss` to
the `fs.azure.*` / `fs.abfs.*` / `fs.abfss.*` Hadoop config surface for the
Parquet native scan.
So the backend is compiled in and the config plumbing exists for the other
scan path. What is missing is the Iceberg storage factory arm and the key
translation.
Observed with Spark 4.1.3,
`org.apache.datafusion:comet-spark-spark4.1_2.13:1.0.0`, Iceberg 1.11.0
(`iceberg-spark-runtime-4.1_2.13`), a REST catalog (Microsoft OneLake),
`spark.comet.scan.icebergNative.enabled=true`. Plan falls back with the scheme
message; `spark.comet.enabled=false` reads the same table fine through
`HadoopFileIO` + `hadoop-azure`.
### Describe the potential solution
1.
`native/core/src/execution/operators/iceberg_common.rs::storage_factory_for`:
add `"abfs" | "abfss" => Ok(Arc::new(OpenDalStorageFactory::Azdls))`.
2. Translate the Hadoop ABFS keys into iceberg-rust's `adls.*` keys when
building the catalog/FileIO properties, the way `s3.*` is derived from
`fs.s3a.*` today:
- `fs.azure.account.key.<host>` -> `adls.account-key` (+
`adls.account-name` from the URI host's first label)
- `fs.azure.account.oauth2.client.id` / `.client.secret` /
`.client.endpoint` -> `adls.client-id` / `adls.client-secret` /
`adls.tenant-id` (tenant is the path segment of the endpoint URL) and
`adls.authority-host`
- `fs.azure.sas.fixed.token` -> `adls.sas-token`
3. Add `abfs` and `abfss` to `icebergReadableSchemes` (or, per #5541, derive
that set from the native factory so the two cannot drift again).
Two things worth getting right while there:
- **Custom token providers.** A lot of Azure deployments set
`fs.azure.account.oauth.provider.type` to a class (`MsiTokenProvider`,
`WorkloadIdentityTokenProvider`, `CustomTokenProviderAdapter`, ...) rather than
a client secret. Those cannot be reproduced natively from `fs.azure.*` keys
alone. The S3 path already has a `customized_credential_load` hook built on the
JVM side; the equivalent here would be to obtain the bearer token via the
configured Hadoop provider on the JVM and pass it through, or at minimum honour
OpenDAL/reqsign's workload-identity federation from `AZURE_CLIENT_ID` /
`AZURE_TENANT_ID` / `AZURE_FEDERATED_TOKEN_FILE`. #4747 was the same gap on the
Parquet path.
- **Non-`core.windows.net` hosts.** The account should come from the URI's
first host label and the endpoint from the full host, so OneLake
(`onelake.dfs.fabric.microsoft.com`), sovereign clouds and Azurite work rather
than only `<account>.dfs.core.windows.net`.
### Additional context
Related: #5541 (the three scheme lists disagree, and `abfss` is one of the
schemes that currently passes one check and fails another), #4747 (workload
identity ignored on the native Parquet scan).
Happy to test a branch against OneLake with an OIDC-federated identity; we
have a CI benchmark that runs Iceberg-on-OneLake through Spark 4.1 + Iceberg
1.11 REST and can report timings with and without Comet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
