LantaoJin commented on a change in pull request #27828: [SPARK-31068][SQL]
Avoid IllegalArgumentException in broadcast exchange
URL: https://github.com/apache/spark/pull/27828#discussion_r391457078
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala
##########
@@ -181,6 +182,13 @@ case class BroadcastExchangeExec(
}
object BroadcastExchangeExec {
+ // Since the maximum number of keys that BytesToBytesMap supports is 1 << 29,
+ // and only 70% of the slots can be used before growing in HashedRelation,
+ // here the limitation should not be over 341 million.
+ val MAX_BROADCAST_TABLE_ROWS = ((1L << 29) / 1.5).toLong
Review comment:
`val MAX_BROADCAST_TABLE_ROWS = (BytesToBytesMap.MAX_CAPACITY / 1.5).toLong`
Symbol MAX_CAPACITY is inaccessible from this place
Since MAX_CAPACITY misses `public` restriction.
But I think it's ok to add a `public` on it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
