goutamadwant opened a new pull request, #11852:
URL: https://github.com/apache/seatunnel/pull/11852
### Purpose of this pull request
Fixes #11309.
This adds a sink-only Deep Lake Connector-V2 implementation using the
managed Deep Lake REST SQL API.
The connector:
- writes rows through the parameterized batch query endpoint
- supports scalar, text, binary, vector, and array fields
- supports create-if-missing, validate-existing, and ignore schema modes
- maps upstream fields and primary keys without generating synthetic IDs
- rejects update and delete row kinds because the first version is
append-only
- provides at-least-once delivery without automatically retrying ambiguous
failed writes
- registers the connector in plugin discovery and the distribution
- adds English and Chinese documentation
No Python, native library, or Deep Lake client dependency is required on
SeaTunnel workers.
### Does this PR introduce _any_ user-facing change?
Yes.
Users can configure `DeepLake` as a sink for batch or streaming pipelines
targeting a managed Deep Lake workspace. Existing connectors and behavior are
unchanged.
### How was this patch tested?
Added focused tests covering:
- SQL generation, identifier quoting, nullability, primary keys, and type
mapping
- unsupported vector type rejection
- REST paths and authentication headers
- schema creation and parameterized batch writes
- binary, vector, and nested array conversion
- failed-batch retention
- rejection of non-insert row kinds
Verified with JDK 11 using:
`./mvnw -pl seatunnel-connectors-v2/connector-deeplake package`
Result: 7 tests passed.
The REST contract tests use a local HTTP server. An authenticated write to a
managed Deep Lake workspace was not run because credentials were not available.
### Check list
* [x] Reviewed dependency licensing. Apache HttpClient is already used by
SeaTunnel, so no new License Notice entry is required.
* [x] Added English and Chinese connector documentation.
* [x] No incompatible behavior is introduced.
* [x] Reviewed the connector integration files:
1. Updated `plugin-mapping.properties`
2. Updated `seatunnel-dist/pom.xml`
3. Added the connector labeler rule
4. Added deterministic REST contract tests instead of a
credential-dependent E2E test
5. Updated `config/plugin_config`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
