andygrove commented on code in PR #6726:
URL: https://github.com/apache/datafusion-comet/pull/6726#discussion_r4456344699
##########
spark/src/main/scala/org/apache/spark/sql/comet/CometSparkToColumnarExec.scala:
##########
@@ -138,13 +138,20 @@ object CometSparkToColumnarExec extends
CometSink[SparkPlan] with DataTypeSuppor
CometScanWrapper(nativeOp, CometSparkToColumnarExec(op))
}
+ /**
+ * Declines calendar intervals at any depth. Arrow holds the time part of an
interval in
+ * nanoseconds, so `IntervalMonthDayNanoWriter` overflows on one of more
than about 292 years,
+ * which Spark accepts (#5279).
+ */
override def isTypeSupported(
dt: DataType,
name: String,
fallbackReasons: ListBuffer[String]): Boolean = dt match {
- case ArrayType(StringType, _) => true
- case MapType(StringType, StringType, _) => true
- case _: ArrayType | _: MapType => false
+ case CalendarIntervalType =>
Review Comment:
#6005 and #6548 have both merged, so native shuffle now hashes a wide
decimal as Spark does, and #3079 is closed. I rebased on main (merge commit
`0520d7d88`) and added the regression test in `26429c793`.
`a converted wide-decimal shuffle joins the JVM columnar shuffle with xs`
and `... with m` build the setup from the review: `DECIMAL(38,0)` keys `0..99`,
10 shuffle partitions, broadcast joins off. The left input is an RDD with an
`array<int>` or `map<int,int>` column, which this PR converts to a native
shuffle. The right input is an RDD with a `NullType` column and stays on the
JVM columnar shuffle. Each case runs with AQE on and off, asserts one converted
native shuffle and one columnar shuffle, and compares against Spark's answer.
Both pass with all 100 rows.
I re-ran `CometShuffleInputConversionSuite`, `CometTypedDatasetSuite`,
`CometNativeShuffleSuite` and the `SparkToColumnar` tests in `CometExecSuite`
on Spark 4.1 against the merged tree. All pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
