slfan1989 opened a new pull request, #1990:
URL: https://github.com/apache/auron/pull/1990
<!--
Thanks for sending a pull request! Please keep the following tips in mind:
- Start the PR title with the related issue ID, e.g. '[AURON #XXXX] Short
summary...'.
- Make your PR title clear and descriptive, summarizing what this PR
changes.
- Provide a concise example to reproduce the issue, if possible.
- Keep the PR description up to date with all changes.
-->
### Which issue does this PR close?
Closes #1404.
### Rationale for this change
[AURON#1404] Support for Spark 4.0.1 Compatibility in Auron.
### What changes are included in this PR?
To support **Spark 4**, Auron needs to be adapted accordingly. Currently,
`Celeborn` already supports **Spark 4.0**, and `Iceberg` has also supported
**Spark 4.0** for some time. The Iceberg community has already voted to
deprecate support for **Spark 3.4**, and it will be removed soon.
For this PR, I have made the following changes:
- Three changes encountered during compilation:
- `NativeShuffleExchangeExec#ShuffleWriteProcessor`: Due to SPARK-44605
restructuring the write method in the API, I refactored the partition and rdd
handling here to retrieve them from dependencies for compatibility with other
interfaces. In the future, we should switch to the new interface and make
further changes to nativeRssShuffleWrite / nativeShuffleWrite.
- `NativeBroadcastExchangeBase#getBroadcastTimeout`: In Spark 4.0,
getBroadcastTimeout needs to be fetched from getActiveSession.
- `NativeBroadcastExchangeBase#getRelationFuture`: In Spark 4.0, the type
of SparkSession has changed to org.apache.spark.sql.classic.SparkSession, so I
made the necessary adjustments to the way it is accessed.
### Are there any user-facing changes?
No.
### How was this patch tested?
CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
