linehrr commented on issue #23666: [SPARK-26718][SS] Fixed integer overflow in SS kafka rateLimit calculation URL: https://github.com/apache/spark/pull/23666#issuecomment-458635644 @dongjoon-hyun after some quick investigation, I found out that it's ` testUtils.sendMessages(topic, (0 until 5000).map(_.toString).toArray, Some(0))` that's slowing down the whole test case, it can sometimes take up to 6 seconds to generate those 5000 messages(kinda make sense). I put this 5000 magic number there to make sure it can trigger the overflow due to `begin + prorateLong`. but then I also found out that merely 5 messages could achieve the same purpose with no problem. and this brought the message generation time from 5-6 seconds to only about 0.1 second(make sense with only 0.1% volume compare to before). based on above, I changed the number from 5000 to 5. and also verified that it triggers error from both V1Datasource and V2Datasource. therefore I consider it as a valid test case still.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
