dwsmith1983 commented on PR #5613: URL: https://github.com/apache/datafusion-comet/pull/5613#issuecomment-5603789243
@sunchao the microbenchmarks from your first pass, against a real Spark task memory manager rather than the stub: the harness launches a JVM in-process through the jni crate's invocation feature, builds a Spark 4.1.3 `TaskMemoryManager` over an off-heap `UnifiedMemoryManager` with a 1 GiB pool, wraps it in `CometTaskMemoryManager`, and drives base (main's `fair_pool.rs` compiled as a sibling module) and head through the real JNI bridge in one binary. Three recorded rounds after warmup, 20,000 operations per thread. Latencies are mean / p50 / p99 in microseconds. Fair-limit rejects and other errors were zero on every row; Rust `used` and the Spark balance netted to zero after quiesce on every row, with head holding its one anchor byte until drop and zero after. | scenario | threads | ops/s base to head | grow base to head | release base to head | short grants base / head | |---|---|---|---|---|---| | full grants | 1 | 1.52M to 1.92M (1.26x) | 0.7/0.7/0.9 to 0.5/0.5/0.7 | 0.4/0.4/0.5 to 0.4/0.3/0.5 | 0 / 0 | | partial grants | 1 | 1.15M to 1.57M (1.36x) | 0.8/0.7/7.0 to 0.6/0.5/3.0 | 0.4/0.4/1.2 to 0.4/0.4/0.5 | 1941 / 1941 | | registration churn | 1 | 1.89M to 1.94M (1.03x) | 0.5/0.5/1.5 to 0.5/0.5/1.2 | 0.4/0.4/1.1 to 0.4/0.4/0.6 | 0 / 0 | | full grants | 4 | 0.92M to 1.37M (1.50x) | 4.4/1.3/32.4 to 3.6/1.2/26.5 | 4.2/1.0/31.8 to 0.8/0.6/3.7 | 0 / 0 | | partial grants | 4 | 0.72M to 1.34M (1.86x) | 5.2/1.5/43.9 to 3.1/1.1/40.3 | 7.5/1.4/67.4 to 0.8/0.7/5.8 | 29314 / 23544 | | registration churn | 4 | 0.97M to 1.60M (1.65x) | 4.1/0.7/37.3 to 3.0/1.1/26.2 | 4.0/0.5/37.3 to 0.7/0.6/1.5 | 0 / 0 | | full grants | 8 | 0.61M to 1.40M (2.27x) | 12.9/3.7/83.4 to 7.2/1.2/93.1 | 12.9/4.0/83.3 to 0.8/0.6/2.5 | 0 / 0 | | partial grants | 8 | 0.49M to 1.23M (2.49x) | 16.5/12.4/93.5 to 7.1/1.2/129.5 | 13.4/1.8/89.2 to 0.7/0.7/1.8 | 110853 / 78614 | | registration churn | 8 | 0.62M to 1.40M (2.27x) | 12.9/2.6/92.1 to 7.2/1.1/91.9 | 12.8/2.5/92.0 to 0.8/0.6/3.0 | 0 / 0 | The release path is where the change shows: at 4 and 8 threads its p99 drops from tens of microseconds to single digits because a release no longer waits behind an in-flight acquire. Grow p99 is unchanged, since that is Spark's own work under the call, while mean grow latency and throughput improve 1.5x to 2.5x under contention. Head sees fewer short grants in the partial scenarios because it cycles past the release faster. Registration churn counts only depend on run length. The harness needs a JDK and a full Spark classpath so it is not a criterion bench in tree; the source and raw output are available if wanted. macOS arm64, single machine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
