parthchandra commented on PR #6784: URL: https://github.com/apache/datafusion-comet/pull/6784#issuecomment-6372117090
> Thanks @parthchandra. Splitting first and then stripping the BOM from the first record matches Hadoop for every file I tried, and the `BOM a|b`, `BOM|x` and BOM-only tests pin it. > > sunchao's casing finding at `CometScanRule.scala:483` holds up when I read it. The eligibility check builds its Hadoop conf from `scan.options.asScala`, which lowercases the keys, while `TextScan.createReaderFactory` and your serializer use `asCaseSensitiveMap`. A differently cased read option can then hide a finite `line.maxlength` from the check, and the native reader returns lines Spark skips. Could that call use `scan.options.asCaseSensitiveMap.asScala.toMap`, with a test for sunchao's case? That's the last thing I see, and I'll approve once it's in. Done. The check now builds its Hadoop conf from scan.options.asCaseSensitiveMap.asScala.toMap, like TextScan.createReaderFactory and the serializer. Added a test for sunchao's case: line.maxlength=5 in the Hadoop conf plus a differently cased ...line.maxLength=2147483647 read option now falls back and matches Spark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
