xinnyuli opened a new issue, #12691: URL: https://github.com/apache/seatunnel/issues/12691
### Search before asking - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description Zeta can keep checkpoint and IMap state on HDFS, S3, OSS, COS and (since #12551) GCS, but there is no Azure option. I wanted to know whether Azure Blob Storage already works through the existing `hdfs` storage type before proposing anything, so I ran a local probe. **Setup:** dev at 3ac7003fb, hadoop-azure 3.1.4 (matching the Hadoop version Zeta ships), Azurite (blob service) as the emulator, `storage.type: hdfs` with `fs.defaultFS: wasb://seatunnel@storageemulator/`, using only the existing `seatunnel.hadoop.*` pass-through. No code changes. **Results:** | | Result | |---|---| | Checkpoint (`HdfsStorage`, the 4 tests in `AbstractFileCheckPointTest`) | Pass | | IMap, same instance stores 20 entries then `loadAll()` | 0 of 20 | | IMap, writer still open, new instance `loadAll()` (restart after a crash) | 0 of 20 | | IMap, same as above with `fs.azure.page.blob.dir` set for the IMap path | 7 of 20, no error | Checkpoints work because each checkpoint file is written and closed. IMap goes through `HdfsWriter`, which keeps one output stream open and relies on `hflush`/`hsync` for visibility. A WASB block blob is only committed on `close()`, so nothing written is visible until the writer closes, and a crash before that loses the IMap data. Page blobs make some of the data visible but not all of it, silently, so that is not a safe workaround either. This is the same reason S3, OSS, COS and GCS use `CloudWriter` (rewrite the current segment on each write, roll over at `blockSize`) instead of `HdfsWriter`. ### Usage Scenario Users who run Zeta on Azure (for example on AKS) and want checkpoint and IMap state in Azure Blob Storage, the same way S3 and GCS users can today. ### Related issues #12551 (GCS checkpoint and IMap storage), #12293 (ADLS connector) ### Proposal Follow the GCS change (#12551): - Checkpoint: add an `azure` storage type to `FileConfiguration` with an Azure configuration class that maps the account, key and container options to hadoop-azure settings. - IMap: add `AzureWriter extends CloudWriter`, and register the `azure` type in the IMap file storage configuration, so IMap state does not depend on `hflush`. - Keep the Azure jars `provided` (users add them to `lib/`), like the other cloud types. #12293 also brings in hadoop-azure, so the two could share the dependency setup. - Tests against Azurite, plus docs (en + zh) with a configuration example. ### Not verified yet - **ADLS Gen2 (`abfs://`):** Azurite does not support it, so I have not tested it. ABFS supports flush, so the existing `hdfs` type may already work there. I can test it on a real storage account if that would help decide the scope. - The page blob result comes from Azurite and may differ on real Azure. It does not change the conclusion, though. Two questions before I start: 1. Should the `azure` type cover Blob only (`wasb://`) first, or Blob and ADLS Gen2 together? 2. Since checkpoints already work through `hdfs`, should the new type still cover checkpoint storage for consistency with the other cloud types, or only IMap? ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
