aminebag commented on PR #41423: URL: https://github.com/apache/spark/pull/41423#issuecomment-1575726835
I don't think the answer is more resources. If we had more memory or CPU power we would just delay the issue we wouldn't prevent it. Also, if we had more CPU power the listener would become faster at consuming events, but the producer as well could become faster at producing them. Besides, increasing resources for the driver is vertical scaling which, I believe, is against the spirit of Spark and distributed computing. One solution would be to make the events queue blocking and never drop events (block until there's space in the queue to put the event). But this would mean that the scheduler (the event producer) would be impacted by any slowness of the listener. I don't think that it's a good idea to have a secondary feature (application monitoriong) impact the performance of a main feature (task scheduling). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
