SEZ9 commented on issue #13262:
URL: https://github.com/apache/seatunnel/issues/13262#issuecomment-6870114711

   @chovy-3012 thanks for laying out the v1 scope so clearly.
   
   Quick reactions, point by point:
   
   1. **Batch-only, one split per fragment, pending splits in checkpoint 
state** — sounds good; that is the right granularity for a bounded source and 
keeps failover simple.
   2./3. **No column projection, version selection or filter push-down in v1** 
— fine with me. Please make sure the source doc states these limitations 
explicitly so users aren't surprised, and note them as follow-ups in the PR 
description.
   4. **lance-core 0.33.0 shared with the sink** — good, one client version for 
both ends is what we want.
   5. **Type mapping** — the scalar table looks correct. Mapping list to 
`ARRAY` with the element type from the child field is right, and I agree with 
fail-fast on `FixedSizeBinary` vectors rather than coercing to bytes. For blob, 
yes, please make it an explicit decision rather than whatever falls through: 
either map the plain-binary case to bytes and document it, or reject with the 
same field-naming error. Either is acceptable for v1 as long as the behavior is 
deterministic, documented, and the error names the offending column.
   
   On config consistency: mirroring the sink's `lance.write.storage.options` 
keys as `lance.read.storage.options` and passing them via `ReadOptions` on 
every `Dataset.open` is exactly the approach I'd like, so `s3://` datasets work 
from job config alone without `AWS_*` environment variables.
   
   Your comment appears to have been cut off at the docs section ("a new source 
page in EN"). Could you post the rest? In particular, please confirm the source 
documentation will cover both EN and ZH and include the unsupported-types / 
no-projection / latest-version-only notes above.
   
   Please link the PR here once it's open so we can continue the review there. 
Looking forward to it.
   
   <!-- streview-comment:2330 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to