AdityRajGupta commented on PR #54059: URL: https://github.com/apache/arrow/pull/54059#issuecomment-6352804588
> dmitry-chirkov-dremio yeahh you are totally right soo the fundamental problem was the iterator that had stripped off `\r` from the end of the block and appended it to the next one. If the read length was short enough, a resulting block did not have a terminator and got rejected by the chunker. Now I have modified the iterator such that a block ending on `\r` is kept whole and appended to the next block in order to keep the stream consistent and fix the issue of block boundaries, not breaking anything in terms of existing cases of CRLF/header. And I also added the requested coverage: - Your `MixedLineEndingsStraddling` case across all 3 reader modes and multiple chunk splits. - `SmallInputsAcrossBlockSizes`: 222 combinations covering CR/LF/CRLF, missing final terminators, quoted fields, embedded newlines, and block sizes 4–16. The full CSV suite passes (292 tests). The matrix did find some cases that also fail on the parent implementation (rows spanning more than two blocks ), so those are documented and excluded. Everything else matches the parent, plus the GH-51368 quoted-CRLF corruption cases are now fixed :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
