MaxGekk commented on PR #55326: URL: https://github.com/apache/spark/pull/55326#issuecomment-4661723439
**Read-path guard is stricter than the original → ON/OFF behavior difference** `TimeTypeParquetOps.requireCompatibleParquetType` requires `INT64 && unit == MICROS && !isAdjustedToUTC`. The original `ParquetRowConverter` guard it replaces only checked `TimeLogicalTypeAnnotation && unit == MICROS` — it did **not** check `isAdjustedToUTC`. So the new guard is strictly tighter, even though the code comment states it "Mirrors the inline guard that existed in `ParquetRowConverter` before the framework dispatch". This produces a difference between framework ON and OFF when reading a `TIME(MICROS, isAdjustedToUTC=true)` column as `TimeType` (reachable via an explicit read schema, since schema inference already maps `isAdjustedToUTC=true` to `illegalType()`). I verified it empirically: - framework **OFF** → succeeds (returns `23:59:59.123456`) - framework **ON** → fails with `FAILED_READ_FILE` / `cannotCreateParquetConverterForDataTypeError` This contradicts the "behavior is identical in both cases" statement in the PR description. Since this is an edge case (only via explicit read schema) and orthogonal to the framework wiring in this PR, I'd suggest handling it in a **separate follow-up** rather than blocking this one. The follow-up should decide intent and either: - mirror the original guard exactly (drop the `!isAdjustedToUTC` / `INT64` checks) to preserve ON/OFF equivalence, or - if the stricter check is intentional, apply the same tightening to the `*Default` path too and add a test documenting the (now intended) change, plus update the comment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
