imbajin opened a new issue, #3346: URL: https://github.com/apache/hugegraph/issues/3346
## Goal Track long-term, evidence-driven performance work across HugeGraph traversal and its shared execution/collection paths. Start with Kout as a reproducible case, then extend to other representative operations when measurements reveal shared bottlenecks. Optimize end-to-end efficiency under realistic workloads, rather than greedily improving one method, microbenchmark, or configuration at the expense of correctness, memory, latency, or concurrent throughput. ## Starting evidence [🔗 Server #3342](https://github.com/apache/hugegraph/pull/3342) fixes lost Kout results by serializing shared result-set updates and distinct-limit admission. Preserve this correctness guarantee. [Bounded measurements and limitations](https://github.com/apache/hugegraph/pull/3342#pullrequestreview-5514009398) show workload-dependent trade-offs between internal parallelism and concurrent requests. They do not establish a production regression caused by the fix, isolate monitor cost, or establish production-wide performance acceptance. No narrowed-lock optimization has been benchmarked. This is a long-term task, not an additional 1.8 release gate. Candidate optimizations below are hypotheses to evaluate, not commitments to implement. ## TODO - [ ] Establish a reusable, correctness-checked baseline and retain reproducible commands, data generation, raw results, environment details, and exact tested revisions. - [ ] Profile representative end-to-end workloads before prioritizing changes. - [ ] Rank opportunities by measured user impact, resource cost, implementation risk, and maintenance burden. - [ ] Implement justified optimizations separately, with before/after comparisons against a correct baseline. - [ ] Add stable regression coverage and document deployment/concurrency trade-offs. - [ ] Expand coverage to related traversals and shared infrastructure as evidence warrants. ### Candidate investigations - [ ] Internal worker counts, executor reuse/admission, and resource budgets across concurrent requests. - [ ] Bounded/lazy edge-query and iterator preparation to avoid unnecessary work after limits are satisfied. - [ ] Primitive IdSet bulk union to reduce numeric-ID wrapping during layer merges. - [ ] Empty-frontier termination and avoidable allocation/cancellation overhead. - [ ] Critical-section scope or aggregation strategies only where profiling justifies them; preserve exact distinct limits, deduplication, visibility, resource cleanup, and off-heap ID safety. ## Measurement and acceptance - [ ] Cover single and concurrent requests; several worker settings including the actual default; numeric/string IDs; unique/duplicate/convergent graphs; finite/unlimited limits; nearest modes; and representative graph sizes. - [ ] Include relevant backends and full API/service paths, not only isolated consumers. Distinguish warmed caches from genuinely cold storage. - [ ] Record throughput, latency distributions, CPU, allocation/GC, memory, and lock/scheduling observations. Use sufficient independent runs, warm-up, sustained durations, and variability reporting; short repeated samples are not independent experiments or production tail-latency proof. - [ ] Verify complete results where deterministic, exact counts and legal results for bounded queries, plus cancellation, capacity and resource-lifecycle behavior. - [ ] Agree workload-specific acceptance budgets before claiming success. Check for displaced costs and regressions in other supported workloads. - [ ] Change one causal factor at a time where practical. Report inconclusive results honestly; do not use unsafe old concurrent behavior as a performance target. Keep findings and follow-up PRs linked here so improvements accumulate into a coherent performance baseline rather than isolated tuning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
