DanielLeens commented on issue #12437: URL: https://github.com/apache/seatunnel/issues/12437#issuecomment-6217828717
Classification: D / Connector-V2 file-source architecture and recovery semantics. Thank you for decomposing the problem. I checked current dev and found existing byte/row-delimiter and Parquet row-group split strategies, while BaseFileSourceReader still consumes an entire queued split in one poll and snapshots only queued splits. That confirms a resumable reader changes the reader/state contract, not only split planning. I found no open PR covering this complete scope. The proposed work is too broad to reserve as one implementation lane. Please start with a focused child issue or design note for one format-correct split contract, including the exact reader/parser boundary, malformed-record behaviour, checkpoint state compatibility, and deterministic tests. CSV quote and escape handling must use the same parser semantics as the reader; a byte boundary alone is not sufficient. Do not add media ingestion, generic resumability, or public provider APIs to that first change. Once the smallest contract and its recovery tests are agreed, it can become a separate contribution. No assignment or help-wanted decision is made in this round. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
