sunchao commented on PR #5365: URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5399703210
### [P2] Preserve the unsigned-small-integer safety fallback At `6d20ad69`, the [Delta claim path](https://github.com/apache/datafusion-comet/blob/6d20ad69fd39f52c35230d2fa5373aa215e7ed5e/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanContrib.scala#L79-L83) bypasses `CometScanTypeChecker`, including the default-on `spark.comet.scan.unsignedSmallIntSafetyCheck`. This exposes the result mismatch that the existing core gate protects against. I verified a catalog Parquet table with an `INT32` column annotated `UINT_8`, containing `[-1, 0, 127, 255, 256]`, converted successfully using `CONVERT TO DELTA`: - Spark 4.0.2 / Delta 4.0.0 infers `ShortType` and reads `[-1, 0, 127, 255, 256]`. - The current core schema checker rejects the scan with the safety setting enabled, but Delta's `declineReason` returns `None` and the production `CometScanContrib.firstClaimFrom` path returns `CometDeltaNativeScanExec`. - Parquet 58.4 decodes the same file as `UInt8`, producing `[255, 0, 127, 255, 0]`; widening to `Int16` preserves those changed values. The production DataFusion/Comet source path decodes the physical type first and casts afterward, rather than requesting `Int16` during decoding. This is specifically about **out-of-range/malformed `UINT_8` payloads**, not valid-range unsigned values. Verification used current admission code, a real converted Delta table, and the locked-version decoder/cast; it was not a full Comet/JNI query run. Could we preserve the existing unsigned-small-integer fallback before claiming the scan, and add a converted-Parquet regression with the safety setting enabled? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
