Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18656
Btw, can you also add a test for this? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18656
And please also add SQL tag to the PR title, e.g., [SPARK-21441][SQL].
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user DonnyZone commented on the issue:
https://github.com/apache/spark/pull/18656
I have validated both cases with and without CodegenFallback expressions
for `SortMergeJoinExec`.
The fix works well.
---
If your project is set up for it, you can reply to this email and
Github user DonnyZone commented on the issue:
https://github.com/apache/spark/pull/18656
Great! I'm also considering to disable codegen for `SortMergeJoinExec` with
CodegenFallback expressions.
Thanks for your advise. I will work on it and validate in our environment.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18656
I think the check for `SortMergeJoinExec` in `insertInputAdapter` should be
corrected to:
private def insertInputAdapter(plan: SparkPlan): SparkPlan = plan match
{
case p if
Github user DonnyZone commented on the issue:
https://github.com/apache/spark/pull/18656
I notice that the CollapseCodegenStages rule will still enable codegen for
SortMergeJoinExec without checking CodegenFallback expressions.
The logic in `insertInputAdapter` seems to skip
Github user DonnyZone commented on the issue:
https://github.com/apache/spark/pull/18656
That's interesting, I will take a look at why the codegen is enabled
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18656
No. I meant if there's a CodegenFallback expression, wholestage codegen
will not be enabled.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user DonnyZone commented on the issue:
https://github.com/apache/spark/pull/18656
Yeah, CodegenFallback just provide a fallback mode.
However, in such case, SortMergeJoinExec passes incomplete row as input to
hiveUDF that implements CodegenFallback.
---
If your project
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/18656
Will CodegenFallback be used in wholestage codegen? I think it's not
supported.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user DonnyZone commented on the issue:
https://github.com/apache/spark/pull/18656
Hi, @cloud-fan, @vanzin , could you help to take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18656
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
12 matches
Mail list logo
