andygrove commented on code in PR #6401: URL: https://github.com/apache/datafusion-comet/pull/6401#discussion_r4146965137
########## docs/source/user-guide/latest/tuning/memory.md: ########## @@ -199,6 +203,64 @@ The executor logs one more line after its last native plan finishes, and an `all `spark.comet.memory.logInterval` is read when an executor starts its first Comet native plan, so set it when the application is submitted. Set it to `0` to turn the log off. +[Reading the Memory Usage Log from the Event Log]: #reading-the-memory-usage-log-from-the-event-log + +### Reading the Memory Usage Log from the Event Log + +When `spark.eventLog.enabled` is `true` and the application runs the Comet plugin +(`spark.plugins=org.apache.spark.CometPlugin`), each executor also sends its samples to the driver, +which writes a summary of them to the application's event log. The event log keeps every executor's +samples in one place, next to the jobs and stages they ran alongside, while executor logs are spread +over the cluster and often go away with their containers. + +The driver writes, and flushes, each event of its event log as it goes, so it does not write every +sample. For each executor, it takes the samples a minute at a time and writes two of them: the one +with the most untracked memory, and the last, which in the minute an executor goes idle is the one +after its last native plan finishes. It writes them at the executor's first heartbeat a minute or +more after the first of them arrived, which comes every `spark.executor.heartbeatInterval`, 10 +seconds by default, busy or not, and when the executor is removed, such as when the cluster manager +kills it, or the application stops. So each executor adds about two events a minute while it runs +native plans, however short the interval, and the event log has every sample within about a minute Review Comment: The event log doesn't get every sample, only each minute's peak and last, as the start of this paragraph says. Could this say that each minute's samples reach the event log within about a minute of the first of them arriving? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
