dwsmith1983 commented on PR #5613: URL: https://github.com/apache/datafusion-comet/pull/5613#issuecomment-5813821814
> Could you rerun the 2g off-heap pressure pair from your TPC-H comment on this head? Rerun on the merged head (`88d881d04`) against main at `8166586fa`, same SF10 data, same 2g off-heap engine file (`memoryLimitPerTask=256 MB` in every executor log), base, head, head, base at three iterations each over the 22 queries, then q18 and q21 at five iterations alternated. | workload | metric | base | head | |---|---|---|---| | 22 queries x3, per arm | disk bytes spilled | 22,361,276,112 | 22,361,276,112 | | 22 queries x3, per arm | spills | 360 | 360 | | 22 queries x3, per arm | failed tasks / executors lost | 0 / 0 | 0 / 0 | | q18 x5, per arm | spills / bytes | 80 / 7,170,972,000 | 80 / 7,170,972,000 | | q21 x5, per arm | spills / bytes | 150 / 4,758,706,960 | 150 / 4,758,706,960 | Same numbers as the earlier run on both sides. Every query result hashes the same as before. Executor stderr on every arm has zero lines for `fair limit`, `only got`, `overcommit`, `dropped CometFairMemoryPool` or OutOfMemory, and the head library does contain those strings, so the grep is meaningful. No overcommit was recorded anywhere. Two warnings from main are new since the last run and appear in the same counts on both sides: Spark's short grant message from `CometTaskMemoryManager` and the close warning, which the anchor thread covers. I am not quoting wall times from this rerun. Other builds were running on the same machine for part of it and the per-arm sums drift with them. The earlier pass, where the machine was quiet, stands at a head to base ratio of 0.99. > Could you bring it in line with the current head? Description rewritten against the head. The registry paragraph is gone, the fairness sentence now describes the two windows the guide documents and what still holds, the JNI trait is named as main's `SparkMemoryManager` with the two items this PR adds to `SparkMemory`, the anchor paragraph matches the code including the accounting from the thread above, and the testing section describes coverage without counts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
