MaxGekk commented on PR #59225: URL: https://github.com/apache/spark/pull/59225#issuecomment-6044748477
The follow-ups of this PR, each with its own JIRA: | JIRA | What | Why it is not in this PR | |---|---|---| | [SPARK-60021](https://issues.apache.org/jira/browse/SPARK-60021) | Label `ExpandExec`'s outputs held in mutable state as global fields. | A bug fix of its own: an UNPIVOT's large common subexpression fails to compile on master (and 4.1.3, 4.2.0, 4.3.0). This PR works around it with the compacted-slot rule, which the fix makes redundant; it is removed in the same change if it is on master by then. | | [SPARK-60046](https://issues.apache.org/jira/browse/SPARK-60046) | Sort-merge join keys that split their code trip SPARK-22668's testing assertion on the streamed row. | The assertion runs only under testing, with a low `methodSplitThreshold`, and it happens on master too. | | [SPARK-60047](https://issues.apache.org/jira/browse/SPARK-60047) | Split large expressions other than CASE WHEN under whole-stage codegen: the other callers of `splitExpressionsWithCurrentInputs` (`Coalesce`, `In`, `Greatest`/`Least`, `CreateNamedStruct`, the hashes and others) pass their sources, and `Expression.reduceCodeSize` handles whole-stage codegen. | Both are skipped inside a stage by the same gate. This PR adds the input collector, the sources-aware split and the stage gate for CASE WHEN only. | | [SPARK-60048](https://issues.apache.org/jira/browse/SPARK-60048) | Remember the split decision by a key that ignores literal values. | A literal that changes on every run, such as `current_timestamp()`, makes a stage decide afresh and compile twice each run. A literal-insensitive key needs care near the method limit. | | [SPARK-60049](https://issues.apache.org/jira/browse/SPARK-60049) | Keep each method's bytecode size in the compile cache, so that the trial compile does not compile twice on a cache hit. | The trial's held reports would have to be rebuilt from the cache. | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
