chovy-3012 commented on issue #13262:
URL: https://github.com/apache/seatunnel/issues/13262#issuecomment-6857821831

   Thanks @SEZ9 for the detailed scope questions! Here is the planned scope for 
the first version:
   
   1. **Batch-only**: the first version is a bounded batch source. The dataset 
will be split into one split per fragment and assigned round-robin across 
readers; pending splits stay in checkpoint state, so a failover only reassigns 
the unread fragments.
   
   2. **Column projection and dataset version selection**: neither is planned 
for v1 — each split scans all columns of its fragment, and the dataset is 
always opened at its latest version. lance-core 0.33.0 already exposes the 
knobs (a `ScanOptions` column subset, a `ReadOptions` version), so both are 
natural follow-ups.
   
   3. **Filter push-down**: out of scope for v1, for the same reason.
   
   4. **SDK version**: lance-core 0.33.0, the version the existing sink already 
ships — source and sink share one client and no dependency change is needed.
   
   5. **Type mapping** (the reverse of the sink's `SchemaUtils`): scalar types 
map 1:1 — int(32)/int(64) → int/bigint, float → float, double → double, 
Utf8/LargeUtf8 → string, Bool → boolean, Date → date, Time → time, Decimal → 
decimal with the dataset's precision and scale, Binary/LargeBinary → bytes. 
Timestamp maps to timestamp, and timezone-aware columns are converted through 
the timezone declared in the dataset schema. On the types you specifically 
called out: list → `ARRAY` with the element type derived from the list's child 
field (a list without an element field is rejected with a clear error); vector 
(Arrow `FixedSizeBinary`) has no faithful SeaTunnel counterpart, so it is 
rejected up front with an error naming the offending field — fail fast rather 
than silently coercing to bytes; blob is not specially handled in v1 (a column 
that surfaces as plain binary reads as bytes, anything else falls into the same 
fail-fast rejection) — happy to make that an e
 xplicit map-or-reject decision if you prefer.
   
   The PR I'm preparing will follow the Connector-V2 source conventions and 
reuse the sink's storage/credential options so both ends share one 
configuration:
   
   - **Config consistency**: the source will add `lance.read.storage.options` 
with the same keys as the sink's `lance.write.storage.options` 
(`aws_endpoint_url`, `aws_access_key_id`, `aws_secret_access_key`, 
`allow_http`, ...). Every `Dataset.open` in the source will pass these via 
`ReadOptions`, so `s3://` datasets can authenticate purely from the job config, 
with no `AWS_*` environment variables.
   
   - **Docs**: a new source page in EN and ZH with the options table, type 
mapping and examples, and the sink page's "there is no Lance source connector" 
sentence will link to the new source page.
   
   One finding from verifying the existing sink against MinIO: 
`LanceSinkWriter` passes storage options only to `Dataset.create` — its 
`Dataset.open` calls (initial open, post-create and post-append reopen) and the 
`Fragment.create` inside `FragmentConverter` never receive them, so `s3://` 
targets fail with `CredentialsNotLoaded` even when 
`lance.write.storage.options` is set, and writing to object storage currently 
only works through `AWS_*` environment variables. Since the source shares the 
same credential design, I plan to fix these paths in the same PR — happy to 
split them into a separate fix PR if you prefer.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to