rohityadav1993 commented on PR #19120: URL: https://github.com/apache/pinot/pull/19120#issuecomment-6495284773
@gortiz thanks. All three cases are in the sweep now. The setup, tables and limits are in the benchmark PR description (https://github.com/rohityadav1993/pinot/pull/1); summary: - LIMIT Integer.MAX_VALUE: this is the real leaf shape for a sorted merge join input (visitSort lowers the leaf limit only when the sort has a fetch). With ORDER BY tsCol, ON is 10 to 12x faster than OFF (PARTIAL overlap, each key in about 10 segments: 807 vs 8107 ms; DISJOINT, segments do not overlap: 501 vs 6063 ms). At FULL overlap (all 100 segments cover the same key range) it needs about 4x less heap (207-223 vs 924-956 MB). - Low-cardinality leading column: with one column, ON stays faster and lower on heap at every K (K = rows per tsCol value, 1 to 50,000). With two columns and one run per segment (K=50,000, so each segment has one tsCol value; FULL overlap), ON is 3x slower (3473 vs 1151 ms) and needs no less heap than OFF (788-820 vs 694-788 MB). - Large consuming segment (1M rows): at LIMIT 1M and FULL overlap, ON is 7x faster (1077 vs 7582 ms) with about half the heap (602-633 vs 1261-1324 MB). At LIMIT 1000 and DISJOINT overlap, ON is slower (23.6 vs 0.39 ms). One correction: my earlier two-column numbers ran with valCol sorted inside each segment, so they measured the single-column plan. With valCol unsorted and unique leading values (K=1, one row per tsCol value), ON is up to 27x slower (6601 vs 245 ms at LIMIT 1M, DISJOINT) and allocates up to 5.8x more (14.3 vs 2.5 GB per query at LIMIT Integer.MAX_VALUE, FULL). This is #19923 A1 (short runs refetch the non-ORDER BY columns), now measured. I will keep it deferred to #19923; the Known gaps in the description now list it. On AUTO: it is marked experimental in the enum javadoc, and I will do the same in the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
