davidzollo opened a new issue, #11309: URL: https://github.com/apache/seatunnel/issues/11309
### Search before asking - [x] I had searched the existing feature and RAG-related issues and did not find a dedicated DeepLake connector request. - [x] I confirmed that the latest `dev` branch already contains Lance sink support and multimodal `Embedding` support, but no `deeplake` connector module, plugin registration, or documentation. ### Description SeaTunnel already has several building blocks for multimodal knowledge-sync style pipelines: - multimodal `Embedding` transform support for text, image, and video inputs - `connector-lance` for writing data into Lance datasets However, there is still no DeepLake connector in the latest `dev` branch. Please add a `Connector-V2` DeepLake sink so SeaTunnel pipelines can write multimodal records, embeddings, and related metadata into DeepLake datasets. A minimal first step would be: - add a `connector-deeplake` module - register it in plugin discovery - support writing basic scalar fields, vector fields, and binary/text metadata commonly used in multimodal ingestion - document the connector with a runnable example ### Usage Scenario A common target pipeline is: 1. read text, image, or video metadata and content from upstream sources 2. optionally generate embeddings through SeaTunnel `Embedding` 3. write raw payload metadata plus vectors into DeepLake Typical use cases include: - multimodal knowledge base ingestion - dataset preparation for retrieval or training workflows - syncing document, image, and video assets into a DeepLake-backed store ### Related issues - #9570 and PR #9894 already cover Lance dataset writing - #9713 tracks broader RAG-oriented connector preparation - #10889 tracks umbrella Knowledge Sync work This issue focuses only on the missing DeepLake dataset sink capability. ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's Code of Conduct -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
