linehrr commented on a change in pull request #23666: [SPARK-26718][SS] Fixed
integer overflow in SS kafka rateLimit calculation
URL: https://github.com/apache/spark/pull/23666#discussion_r251486061
##########
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala
##########
@@ -231,7 +231,11 @@ private[kafka010] class KafkaMicroBatchStream(
val begin = from.get(tp).getOrElse(fromNew(tp))
val prorate = limit * (size / total)
// Don't completely starve small topicpartitions
- val off = begin + (if (prorate < 1) Math.ceil(prorate) else
Math.floor(prorate)).toLong
+ val prorateLong = (if (prorate < 1) Math.ceil(prorate) else
Math.floor(prorate)).toLong
+ // need to be careful of integer overflow
+ // therefore added canary checks where to see if off variable
could be overflowed
+ // refer to [https://issues.apache.org/jira/browse/SPARK-26718]
+ val off = if (prorateLong > Long.MaxValue - begin) Long.MaxValue
else begin + prorateLong
Review comment:
will fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
