srbiswal opened a new pull request, #6680: URL: https://github.com/apache/hive/pull/6680
For tables with skip.header.line.count/skip.footer.line.count, SkippingTextInputFormat called readLine() then getPos() on the same FSDataInputStream (in getCachedStartIndex and getCachedEndIndex). When the data contains a lone \r (not followed by \n), DataInputStream.readLine() pushes back its look-ahead byte by wrapping the stream in a non-Seekable PushbackInputStream. The following getPos() then throws ClassCastException during Tez split generation. LF, CRLF, and \r-at-EOF were unaffected. Replace readLine()+getPos() in both methods with a ByteCountingLineReader that handles \n, \r\n, and lone \r, computing offsets from bytes consumed, so getPos() is never called on a mutated stream. Offset behavior is unchanged. As a safety net, makeSplitInternal now turns any unexpected error from header/footer detection into a clear, file-specific message instead of a raw ClassCastException ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added unit tests in `TestSkippingTextInputFormat`: - **`testSkipFileSplitsLoneCR`** — reproduces the `ClassCastException` on a lone-CR file with `skip.header.line.count=1` (fails on master, passes with this fix). - **`testSkipHeaderSplitOffsetsAcrossLineEndings`** — writes the same content with LF, lone-CR, and CRLF terminators and asserts exact split `getStart()`/`getLength()`, proving lone-CR matches LF and there is no boundary regression. - **`testSkipFileSplitsLoneCRHeaderFooter`** — exercises the footer path (`getCachedEndIndex`) with lone-CR and skip header/footer, verifying the header and footer rows are skipped. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
