cloud-fan commented on code in PR #50230:
URL: https://github.com/apache/spark/pull/50230#discussion_r2321244413
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -1650,6 +1650,21 @@ package object config {
s"The buffer size must be greater than 0 and less than or equal to
${Int.MaxValue}.")
.createWithDefault(4096)
+ private[spark] val SHUFFLE_ORDER_INDEPENDENT_CHECKSUM_ENABLED =
Review Comment:
My understanding is: this PR adds the framework in core and provides
internal APIs (`ShuffleDependency`) to specify the checksum calculator. The
actual place to define the checksum calculator is `ShuffleExchangeExec`, which
is in sql.
For non-SQL shuffle, we need to serialize the key-values to calculate the
checksum (`UnsafeRow` in sql is already in binary format), which means
significant overhead. But if we really want to, we can add new RDD APIs to
specify the checksum calculator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
