HeartSaVioR commented on a change in pull request #23666:
[SPARK-26718][SS][Master] Fixed integer overflow in SS kafka rateLimit
calculation
URL: https://github.com/apache/spark/pull/23666#discussion_r251280936
##########
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala
##########
@@ -231,7 +231,11 @@ private[kafka010] class KafkaMicroBatchStream(
val begin = from.get(tp).getOrElse(fromNew(tp))
val prorate = limit * (size / total)
// Don't completely starve small topicpartitions
- val off = begin + (if (prorate < 1) Math.ceil(prorate) else
Math.floor(prorate)).toLong
+ val prorateLong = (if (prorate < 1) Math.ceil(prorate) else
Math.floor(prorate)).toLong
+ // need to be careful of integer overflow
+ // therefore added canary checks where to see if off variable
could be overflowed
+ // refer to [https://issues.apache.org/jira/browse/SPARK-26718]
+ val off = if (prorateLong > Long.MaxValue - begin) Long.MaxValue
else begin + prorateLong
Review comment:
Line length is limited to 100 chars and this line exceeds it.
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101744/console
You are encouraged to run `dev/scalastyle` and fix all style issues before
submitting additional commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
