gruuya commented on PR #7180: URL: https://github.com/apache/arrow-datafusion/pull/7180#issuecomment-1667281048
> How about K slightly smaller than the batch size? Say K=8000 when batch size=8192. Sorry didn't see this comment in time, maybe I could add that case later on. Anyway I've got some preliminary results and they're not very promising (if there's a change it's non-insignificant slowdown) ```text ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓ ┃ Query ┃ main ┃ top-k-eager-sorting ┃ Change ┃ ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩ │ Qsort utf8 │ 35484.89ms │ 37195.91ms │ no change │ │ with fetch │ │ │ │ │ None │ │ │ │ │ Qsort int │ 46779.63ms │ 46817.47ms │ no change │ │ with fetch │ │ │ │ │ None │ │ │ │ │ Qsort │ 36037.43ms │ 36471.88ms │ no change │ │ decimal with │ │ │ │ │ fetch None │ │ │ │ │ Qsort │ 51767.49ms │ 52308.29ms │ no change │ │ integer │ │ │ │ │ tuple with │ │ │ │ │ fetch None │ │ │ │ │ Qsort utf8 │ 35248.66ms │ 35866.21ms │ no change │ │ tuple with │ │ │ │ │ fetch None │ │ │ │ │ Qsort mixed │ 41124.15ms │ 42406.47ms │ no change │ │ tuple with │ │ │ │ │ fetch None │ │ │ │ │ Qsort utf8 │ 1894.58ms │ 2081.23ms │ 1.10x slower │ │ with fetch │ │ │ │ │ Some(1) │ │ │ │ │ Qsort int │ 1995.16ms │ 1992.60ms │ no change │ │ with fetch │ │ │ │ │ Some(1) │ │ │ │ │ Qsort │ 1811.48ms │ 1924.61ms │ 1.06x slower │ │ decimal with │ │ │ │ │ fetch │ │ │ │ │ Some(1) │ │ │ │ │ Qsort │ 1873.19ms │ 2138.09ms │ 1.14x slower │ │ integer │ │ │ │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(1) │ │ │ │ │ Qsort utf8 │ 2106.81ms │ 2584.92ms │ 1.23x slower │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(1) │ │ │ │ │ Qsort mixed │ 1948.55ms │ 2304.72ms │ 1.18x slower │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(1) │ │ │ │ │ Qsort utf8 │ 1841.42ms │ 2084.12ms │ 1.13x slower │ │ with fetch │ │ │ │ │ Some(10) │ │ │ │ │ Qsort int │ 1967.58ms │ 1994.95ms │ no change │ │ with fetch │ │ │ │ │ Some(10) │ │ │ │ │ Qsort │ 1811.81ms │ 1906.41ms │ 1.05x slower │ │ decimal with │ │ │ │ │ fetch │ │ │ │ │ Some(10) │ │ │ │ │ Qsort │ 1902.72ms │ 2107.02ms │ 1.11x slower │ │ integer │ │ │ │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(10) │ │ │ │ │ Qsort utf8 │ 2096.74ms │ 2567.12ms │ 1.22x slower │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(10) │ │ │ │ │ Qsort mixed │ 1983.58ms │ 2285.19ms │ 1.15x slower │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(10) │ │ │ │ │ Qsort utf8 │ 1860.15ms │ 2098.58ms │ 1.13x slower │ │ with fetch │ │ │ │ │ Some(100) │ │ │ │ │ Qsort int │ 2017.14ms │ 2018.33ms │ no change │ │ with fetch │ │ │ │ │ Some(100) │ │ │ │ │ Qsort │ 1831.45ms │ 1980.09ms │ 1.08x slower │ │ decimal with │ │ │ │ │ fetch │ │ │ │ │ Some(100) │ │ │ │ │ Qsort │ 1933.97ms │ 2171.31ms │ 1.12x slower │ │ integer │ │ │ │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(100) │ │ │ │ │ Qsort utf8 │ 2155.04ms │ 2657.61ms │ 1.23x slower │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(100) │ │ │ │ │ Qsort mixed │ 2028.39ms │ 2365.85ms │ 1.17x slower │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(100) │ │ │ │ │ Qsort utf8 │ 1970.11ms │ 2160.91ms │ 1.10x slower │ │ with fetch │ │ │ │ │ Some(1000) │ │ │ │ │ Qsort int │ 2142.49ms │ 2123.38ms │ no change │ │ with fetch │ │ │ │ │ Some(1000) │ │ │ │ │ Qsort │ 1891.53ms │ 2007.44ms │ 1.06x slower │ │ decimal with │ │ │ │ │ fetch │ │ │ │ │ Some(1000) │ │ │ │ │ Qsort │ 2089.55ms │ 2424.50ms │ 1.16x slower │ │ integer │ │ │ │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(1000) │ │ │ │ │ Qsort utf8 │ 2251.87ms │ 2860.59ms │ 1.27x slower │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(1000) │ │ │ │ │ Qsort mixed │ 2215.35ms │ 2739.82ms │ 1.24x slower │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(1000) │ │ │ │ │ Qsort utf8 │ 2273.12ms │ 2440.54ms │ 1.07x slower │ │ with fetch │ │ │ │ │ Some(10000) │ │ │ │ │ Qsort int │ 2352.27ms │ 2540.47ms │ 1.08x slower │ │ with fetch │ │ │ │ │ Some(10000) │ │ │ │ │ Qsort │ 2062.62ms │ 2255.48ms │ 1.09x slower │ │ decimal with │ │ │ │ │ fetch │ │ │ │ │ Some(10000) │ │ │ │ │ Qsort │ 2741.88ms │ 2937.61ms │ 1.07x slower │ │ integer │ │ │ │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(10000) │ │ │ │ │ Qsort utf8 │ 2554.86ms │ 2733.78ms │ 1.07x slower │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(10000) │ │ │ │ │ Qsort mixed │ 3408.55ms │ 3589.02ms │ 1.05x slower │ │ tuple with │ │ │ │ │ fetch │ │ │ │ │ Some(10000) │ │ │ │ └──────────────┴────────────┴─────────────────────┴──────────────┘ ``` Strangely I see slowdown even for base cases when eager sorting does not apply. I'll need to think about it a bit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
