0lai0 commented on code in PR #5050:
URL: https://github.com/apache/datafusion-comet/pull/5050#discussion_r4307690783
##########
spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegenInput.scala:
##########
@@ -245,6 +245,18 @@ private[codegen] object CometBatchKernelCodegenInput
extends CometTypeShim {
}
}
+ /**
+ * True when a decimal column can be read through the unscaled-long fast
path.
+ *
+ * `Decimal.createUnsafe` produces a long-backed `Decimal` whose
`decimalVal` is null, and
+ * Spark's conversions then index `Decimal.POW_10(scale)`. That throws for a
negative scale,
+ * which `spark.sql.legacy.allowNegativeScaleOfDecimal=true` allows, so a
`try_cast` would
+ * silently return NULL and a plain cast would fail. The `BigDecimal`-backed
reader has no such
+ * assumption, so negative scales take it regardless of precision.
+ */
+ private def canReadDecimalAsUnscaledLong(dt: DecimalType): Boolean =
+ dt.precision <= Decimal.MAX_LONG_DIGITS && dt.scale >= 0
Review Comment:
Fixed, and your query is now a test.
I confirmed it has teeth by reverting just the scale check and re-running:
`ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 19`, the
same error you saw. `Decimal(20,-5)` is in the test too, to pin that the
`BigDecimal` branch stays unaffected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
