andygrove commented on PR #2657:
URL:
https://github.com/apache/datafusion-ballista/pull/2657#issuecomment-6443057416
## SF1000 results after stacking on #2670 (5b57a9e89)
This run is on the current head, which includes #2670's review fixes:
- A remote pipelined stream now keeps its pooled connection leased while it
is read.
- Restricting a broadcast reader to some partitions keeps every location.
- A consumer that times out leaves no waiting slot in the registry.
- The pipelined timeout and queue-capacity settings are clamped to at least
1.
- The writer and registry were simplified.
Setup is the same as before: 2 executors x 8 vCPU, 48 GB pool, 1 TB spill
cap, 16 partitions, local data, one iteration.
All 22 queries pass in **2475s**. That is 1.01x the previous pipelined run
(2450s) and **0.73x** blocking mode (3374s).
| Query | Blocking (s) | Pipelined, 34a1151a (s) | Pipelined, 5b57a9e8 (s) |
vs previous | vs blocking |
|---|---:|---:|---:|---:|---:|
| 1 | 27.2 | 26.6 | 29.1 | 1.10x | 1.07x |
| 2 | 51.2 | 26.3 | 24.9 | 0.95x | 0.49x |
| 3 | 137.4 | 120.3 | 119.0 | 0.99x | 0.87x |
| 4 | 51.4 | 16.4 | 16.5 | 1.01x | 0.32x |
| 5 | 430.0 | 284.4 | 316.9 | 1.11x | 0.74x |
| 6 | 12.6 | 9.2 | 9.1 | 0.99x | 0.72x |
| 7 | 374.4 | 248.4 | 288.3 | 1.16x | 0.77x |
| 8 | 45.0 | 151.4 | 138.3 | 0.91x | 3.08x |
| 9 | 199.6 | 246.5 | 272.8 | 1.11x | 1.37x |
| 10 | 136.6 | 94.5 | 103.9 | 1.10x | 0.76x |
| 11 | 55.5 | 23.5 | 24.3 | 1.03x | 0.44x |
| 12 | 73.4 | 49.1 | 42.7 | 0.87x | 0.58x |
| 13 | 69.9 | 41.5 | 40.1 | 0.97x | 0.57x |
| 14 | 19.0 | 14.9 | 14.3 | 0.96x | 0.75x |
| 15 | 43.0 | 26.1 | 25.8 | 0.99x | 0.60x |
| 16 | 23.8 | 10.0 | 10.9 | 1.09x | 0.46x |
| 17 | 225.9 | 157.4 | 160.2 | 1.02x | 0.71x |
| 18 | 459.7 | 360.7 | 340.4 | 0.94x | 0.74x |
| 19 | 26.2 | 22.1 | 22.1 | 1.00x | 0.85x |
| 20 | 63.8 | 49.3 | 52.2 | 1.06x | 0.82x |
| 21 | 817.1 | 423.2 | 401.2 | 0.95x | 0.49x |
| 22 | 31.6 | 48.1 | 22.1 | 0.46x | 0.70x |
| **Total** | **3374** | **2450** | **2475** | **1.01x** | **0.73x** |
Notes:
- The totals are within noise. Q5, Q7, Q9 and Q10 were 10–16% slower than
the previous run, and Q8, Q12, Q18 and Q21 were faster. With one iteration
each, I can't separate noise from the effect of leasing a pooled connection to
each remote stream. Repeated runs of the join-heavy queries would settle it.
- Q22 is back to 22s, which confirms that the 48s in the previous run was
noise.
- Q8 is still the one large regression against blocking mode, for the reason
described above (the static size estimate of the filtered `part` side).
One gap showed up while running this. If a client process is killed
outright, as opposed to dropping its result stream, its pipelined job keeps
running on the scheduler and holds its vcores until it is cancelled through
`PATCH /api/job/{job_id}`. Cancel-on-drop only works while the client process
is alive, and the scheduler doesn't notice a client that has gone away.
Blocking jobs behave the same way, but a pipelined job's reservation blocks
every other pipelined job, so it matters more here. Possible fixes are a lease
or heartbeat from the client, or cancelling when the result fetch connections
all close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
