uncleGen edited a comment on issue #24283: [SPARK-27355][SS] Make query execution more sensitive to epoch message late or lost URL: https://github.com/apache/spark/pull/24283#issuecomment-480770205 @gaborgsomogyi As you said, different queues are filled up with different speed, so we can not set a proper value for this config. If we set it too small, the queue will be filled up quickly when the number of partition is large enough. But if we set it too large, we may wait for many epochs before failure if partition number is small enough, like 1. So can we merge these two configs, just keep the epoch threshold? Then there is no need to check `partitionCommits` and `partitionOffsets` again. At one level, this makes sense. These queues would not be unbounded and consume up all the memory easily.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
