sudeephazra opened a new pull request, #12293:
URL: https://github.com/apache/seatunnel/pull/12293
### Purpose of this pull request
This pull request adds an Azure Data Lake Storage Gen2 file connector
(`ADLSFile`) for SeaTunnel Connector V2.
The connector:
- Adds ADLS Gen2 source and sink implementations using the secure Hadoop
ABFS (`abfss://`) filesystem.
- Supports Shared Key and Microsoft Entra OAuth client-credentials
authentication.
- Validates storage account, container, endpoint, credentials, and advanced
Hadoop properties.
- Adds an ADLS runtime compatibility module based on `hadoop-azure`.
- Reuses the existing file connector framework and supported file formats.
- Adds source and sink documentation in English and Chinese.
- Adds unit tests for ADLS configuration, authentication, runtime
configuration, and source-file metadata.
- Adds an opt-in integration test for the ADLS file lifecycle against a real
HNS-enabled storage account.
### Does this PR introduce _any_ user-facing change?
Yes.
Previously, SeaTunnel did not provide a dedicated Connector V2 source and
sink for Azure Data Lake Storage Gen2.
This PR introduces the `ADLSFile` source and sink. Users can read from and
write to ADLS Gen2 through secure `abfss://` endpoints using either:
- `SHARED_KEY` authentication with `account_key`
- `OAUTH_CLIENT_CREDENTIALS` authentication with `tenant_id`, `client_id`,
and `client_secret`
Example:
```hocon
source {
ADLSFile {
path = "/landing/input"
account_name = "mystorageaccount"
container = "data"
auth_type = "SHARED_KEY"
account_key = ${?ADLS_ACCOUNT_KEY}
file_format_type = "csv"
skip_header_row_number = 1
schema {
fields {
id = bigint
name = string
amount = double
}
}
}
}
### How was this patch tested?
The following tests were added:
- ADLSConfigValidatorTest
- Rejects credentials belonging to the wrong authentication mode.
- Rejects attempts to override connector-owned Hadoop properties.
- Accepts non-conflicting advanced Hadoop properties.
- ADLSHadoopConfTest
- Verifies secure ABFS URI generation.
- Verifies Shared Key configuration.
- Verifies OAuth client-credentials configuration.
- ADLSRuntimeCompatibilityTest
- Verifies availability of the secure ABFS implementation.
- Verifies base Hadoop configuration and both authentication modes.
- Verifies rejection of invalid account names and missing credentials.
- ADLSRuntimeCompatibilityIT
- Provides an opt-in integration test that creates, writes, reads,
renames, and deletes a file in a real ADLS Gen2 filesystem.
- SourceFileNameCollectorTest and PreserveSourceFilenameTest
- Cover source filename metadata, multi-file routing, duplicate basenames,
missing metadata, and conflicting sink options.
### Check list
* [] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [x ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
