xinnyuli opened a new pull request, #12726: URL: https://github.com/apache/seatunnel/pull/12726
### Purpose of this pull request Close #12691 Adds an opt-in `storage.type: azure` for Zeta checkpoint and IMap storage, backed by hadoop-azure (WASB), following the configuration contract agreed in #12691 and the GCS pattern from #12551. With the existing `hdfs` type, IMap state on Azure Blob Storage is not durable: `HdfsWriter` relies on hflush, but a WASB block blob is only committed on close, so a new instance read 0 of 20 entries in the probe described in #12691. IMap now uses a dedicated `AzureWriter` (a `CloudWriter`), which rewrites and closes the current object on every WAL update. ### Changes - `AzureConfiguration` for checkpoint and IMap. `azure.container` is required and must be a `wasb://` or `wasbs://` container URI with a `<container>@<account>` authority. `abfs://` and `abfss://` are rejected with a message that ADLS Gen2 is out of scope. The value becomes `fs.defaultFS`. - Sets `fs.wasb.impl` / `fs.wasbs.impl`. Checkpoint honours `disable.cache` like the other checkpoint types; IMap leaves the FileSystem cache at the Hadoop default, like IMap S3/OSS/GCS. - All `fs.azure.*` keys are passed through unchanged (storage account key or container SAS token). No configuration values are logged. - `AzureWriter extends CloudWriter`, registered in `DiscoveryWalFileFactory`. - No change to any existing storage type or default. ### Runtime dependencies The Azure jars are user-supplied in `lib/`, as with GCS. Nothing is bundled or shaded, and SeaTunnel's LICENSE/NOTICE are unchanged. `hadoop-azure` and `testcontainers` appear in the two module POMs with `test` scope only, for the tests below. The docs list the exact runtime set, confirmed with `mvn dependency:tree`: - hadoop-azure-3.1.4.jar (Apache-2.0), matching the shipped Hadoop version - azure-storage-7.0.0.jar (MIT) - azure-keyvault-core-1.0.0.jar (MIT) - jetty-util-ajax-9.4.20.v20190813.jar and jetty-util-9.4.20.v20190813.jar (Apache-2.0 / EPL-1.0) ### Does this PR introduce any user-facing change? Yes, a new opt-in `storage.type: azure` for checkpoint and IMap storage. Docs updated in en and zh (`checkpoint-storage.md`, `hybrid-cluster-deployment.md`, `separated-cluster-deployment.md`), including the container requirement and the IMap trade-off that each update rewrites the current object. ### How was this patch tested? Unit tests (no emulator): - `AzureConfigurationTest` (checkpoint and IMap): required key, accepted wasb/wasbs URIs, rejected abfs/abfss and non-WASB URIs, missing container or account, `fs.azure.*` pass-through, impl keys, `disable.cache`, `block.size`, and `azure` writer/reader selection. Azurite tests: - `AzureFileCheckpointTest`: checkpoint write, list, restore and delete. - `IMapFileAzureStorageTest`: load from a new instance after the writer closes; load from a new instance while the writer is still open (crash-equivalent); latest update and delete visible to a new instance. hadoop-azure's emulator mode only connects to `127.0.0.1:10000`, so the Azurite container binds that host port for the lifetime of each test class only, and fails with an explicit message if the port is unavailable. The tests use the public development storage account only, with no real credential or endpoint. They are skipped when Docker is not available. Locally on JDK 17: both modules pass (`checkpoint-storage-hdfs` and `imap-storage-file`, 0 failures), and `spotless:check` passes. ### Check list - [x] No new jar binary package; Azure jars stay user-supplied - [x] Documentation updated (en + zh) - [x] No incompatible change; existing storage types and defaults unchanged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
