chovy-3012 opened a new issue, #13262:
URL: https://github.com/apache/seatunnel/issues/13262

   ### Search before asking
   
   - [x] I had searched in the 
[feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22)
 and found no similar feature requirement.
   
   ### Description
   
   The Lance connector supports writing only. As the docs state, "It provides a 
sink only; there is no Lance source connector."
   
   SeaTunnel jobs can write rows into a Lance dataset but can never read them 
back: data written by the Lance sink cannot be read for validation, migration 
or downstream processing, and Lance datasets produced by other systems cannot 
be ingested into SeaTunnel pipelines at all. The Lance format ecosystem already 
covers reading elsewhere — the official Spark and Trino integrations can both 
read Lance datasets.
   
   A Lance source connector — batch reading from a Lance dataset, from the same 
kinds of locations the sink writes to (local filesystem or object storage such 
as S3/MinIO) — would close this gap and make SeaTunnel's Lance support 
symmetric.
   
   ### Usage Scenario
   
   RAG and vector pipelines. Lance is designed for vector and multimodal AI 
data: embeddings and metadata are written into Lance first, then read back for 
retrieval, training or serving. Today SeaTunnel can only cover the write half 
of that loop; a Lance source would let SeaTunnel read the data back and 
continue the pipeline — e.g. Lance source → transforms → a downstream vector 
database or another store — without leaving SeaTunnel.
   
   It also serves general lake ETL: moving Lance datasets written by 
Spark/Trino jobs into other SeaTunnel-supported storages.
   
   ### Related issues
   
   - #9570 — the feature request that delivered the Lance sink; this issue is 
its read-side counterpart.
   
   ### Are you willing to submit a PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to