Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/17087
@marmbrus thank you for your comments.
For feedback mechanism, I imagine that you are talking about
[this](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala#L357-L364).
When the whole-stage codegen is enabled, this fallback works.
In this case, the whole-stage codegen is disabled since the number of
fields are big and [these two
variables](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala#L463-L466)
will have `false`.
I agree that we should prepare more generic approach that can be applicable
to more cases. However, it is not easy to implement for now. It would require
some refactoring.
As one of simple refactoring, if an exception occurs, `newPredicate`
returns `null` or `none`. Then, caller of `newPredicate` always checks return
value for handling of code generation failure. It make caller code simpler.
While more complex refactoring could be applicable, another PR should
address it.
What do you think?
FYI: I have just noticed that this problem may occur at
[CartesianProductExec](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/CartesianProductExec.scala#L104-L110)
and
[InMemoryTableScanExec](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala#L136).
I have to do the same thing in these two places.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
