SaurabhChawla100 commented on pull request #29413: URL: https://github.com/apache/spark/pull/29413#issuecomment-672939687
> I am wondering if we don't want to drop any events, why don't we just set the capacity to `Integer.MAX_VALUE`? The `LinkedBlockingQueue` doesn't really allocate that much memory at the beginning. > > And I find there's concern about it: > > https://github.com/apache/spark/blob/643cd876e4cfc7faa307db9a2d1dd1b5ca0881f1/core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala#L47-L48 > > But I doubt it now. Because we never throw such explicit error but only drop events when `offer` returns `false`, right? > > And even if we want to throw the error, OOM is still possible when users set a huge capacity, right? > > Another thought is, the default value(10000) of `spark.scheduler.listenerbus.eventqueue.capacity` is probably too small. **I am wondering if we don't want to drop any events, why don't we just set the capacity to `Integer.MAX_VALUE`? The `LinkedBlockingQueue` doesn't really allocate that much memory at the beginning.** - This will cause Driver to OOM for long Running Jobs if there are large number of events came at point of time, since these event takes driver memory when present on queue. **But I doubt it now. Because we never throw such explicit error but only drop events when offer returns false, right?** - yes thats right **And even if we want to throw the error, OOM is still possible when users set a huge capacity, right?** - Generally on running production spark jobs. when we set the huge capacity of the queue, at that point of higher Driver memory is also set compared to what was there previously . **Another thought is, the default value(10000) of `spark.scheduler.listenerbus.eventqueue.capacity` is probably too small.** - yes this is very small, with this size seen lots event drop. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
