andygrove opened a new issue, #5021: URL: https://github.com/apache/datafusion-comet/issues/5021
## Background While reviewing #4976 (interval codegen dispatch for nested values and native shuffle), it became clear that Comet now has several independent data-type support predicates that each had to learn about `YearMonthIntervalType` and `DayTimeIntervalType` separately: - `supportedSinkDataType` in `spark/src/main/scala/org/apache/comet/serde/operator/CometSink.scala` - `supportedSerializableDataType` in `spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala` - `isTypeSupported` in `spark/src/main/scala/org/apache/spark/sql/comet/CometLocalTableScanExec.scala` - `supportedDataType` in `spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala` In #4976, `supportedSinkDataType` is effectively a copy of `supportedDataType(_, allowComplex = true)` with intervals added at each nesting level. That was the least invasive option for that PR, but it means these predicates can drift out of sync as new types are added. ## Proposal Consider consolidating so that adding support for a new data type only requires touching one place. Options to explore: - Give `supportedDataType` an additional flag (for example `allowIntervals`) rather than forking the traversal in `CometSink`. - Or converge the sink / shuffle / local-scan predicates onto a shared helper with explicit, named capability flags for the differences that genuinely need to vary per boundary (for example, interval file scans staying unsupported while LocalTableScan supports them). ## Notes This is a maintainability cleanup, not a correctness bug. The predicates are correct today. Filed as follow-up from the #4976 review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
