Copilot commented on code in PR #1958:
URL: https://github.com/apache/auron/pull/1958#discussion_r2730590438
##########
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/shuffle/AuronShuffleDependency.scala:
##########
@@ -41,7 +43,19 @@ class AuronShuffleDependency[K: ClassTag, V: ClassTag, C:
ClassTag](
keyOrdering,
aggregator,
mapSideCombine,
- shuffleWriterProcessor) {}
+ shuffleWriterProcessor) {
+
+ // Serialize _rdd
+ val inputRdd: RDD[_ <: Product2[K, V]] = getInputRdd
+
+ // Spark 3+: Not required
+ @sparkver("3.0 / 3.1 / 3.2 / 3.3 / 3.4 / 3.5")
+ def getInputRdd: RDD[_ <: Product2[K, V]] = null
Review Comment:
`inputRdd` is always `null` for Spark 3.x builds (`getInputRdd` returns
null) but it is a public `val`, which is easy to misuse and can cause NPEs if
referenced accidentally. Consider defining `inputRdd` only for Spark 4.1 via
`@sparkver("4.1")` (or making it an `Option`/private accessor) so the Spark 3.x
artifact doesn’t expose a null field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
