comphead commented on code in PR #7505:
URL: https://github.com/apache/datafusion-comet/pull/7505#discussion_r4485577208
##########
spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala:
##########
@@ -112,22 +112,27 @@ case class EliminateRedundantTransitions(session:
SparkSession)
case ColumnarToRowExec(range: CometRangeExec) =>
range.originalPlan
case c @ ColumnarToRowExec(child) if hasCometNativeChild(child,
containsCometPlanMemo) =>
- val op = createColumnarToRowExec(child)
- if (c.logicalLink.isEmpty) {
- op.unsetTagValue(SparkPlan.LOGICAL_PLAN_TAG)
- op.unsetTagValue(SparkPlan.LOGICAL_PLAN_INHERITED_TAG)
+ if (containsVariantType(child.schema)) {
+ // TODO: Remove this fallback once Comet columnar-to-row conversion
supports Variant
+ // getters and Spark's Variant UnsafeRow encoding.
+ // https://github.com/apache/datafusion-comet/issues/5436
+ withFallbackReason(
+ c,
+ "Comet columnar-to-row conversion does not support type
VariantType")
Review Comment:
This branch now tags the original `ColumnarToRowExec` instead of building a
new one, and the PR also removes the two Variant assertions that referenced the
native converter. I could not find a test that checks the Variant
`ColumnarToRowExec` carries this reason.
`CometVariantProjectionSuite.checkNative` only excludes `ColumnarToRowExec`
from the Comet-operator check. Would it be worth asserting in `checkNative`
that `new ExtendedExplainInfo().getFallbackReasons(plan)` contains `Comet
columnar-to-row conversion does not support type VariantType`? I have not run
it, so I do not know whether another node already contributes a matching reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
