devmadhuu commented on PR #10549: URL: https://github.com/apache/ozone/pull/10549#issuecomment-5000167079
> @devmadhuu I have added a file based derby test to compare master with this PR with file based derby, here are the outcomes : > > **Test 7 (worst case: 1M rows rewritten, single transaction)** > In-memory vs File based Derby master (delete-all + insert) this PR (diff sync → all updates) > In-memory ~82 s ~18 s > File-based ~87 s >300 s → timed out > > **Full file-based comparison** > Test Operation master this PR > 1 INSERT 1M ~19 s ~17.6 s > 2 COUNT(*) 1M ~0.27 s ~0.24 s > 5 Paginated read 200K ~0.55 s ~0.53 s > 6 Full 1M read ~1.9 s ~1.9 s > 7 Rewrite 1M, one tx ~87 s >300 s (timeout) > > **In-memory:** no log flush, so 1M in-place UPDATEs (~18 s) beat master's bulk delete + insert (~82 s). > > **File-based:** the PR's Test 7 issues 1M single-row UPDATEs inside one transaction — each is an index seek + in-place modify + WAL entry, and nothing commits until the end. On disk that's dramatically more expensive than master's bulk delete + append-style insert (which Derby handles efficiently). Hence >300 s vs 87 s. > > But Test 7 is not reflects the production scenario IIUC. Production syncs ≤50k-container chunks (not 200k-in-one-transaction), and in steady state most rows are unchanged. The realistic measure is the 50k steady-state chunk: > > this PR: ~3.9 s (skips ~45k, writes ~5k) master would unconditionally delete + re-insert the entire working set every scan, regardless of churn. > > So IMO the PR is a clear win in the real steady-state pattern, and a regression only in the artificial all-churn-in-one-giant-transaction case. Thanks @umesh9794 for running the comparison. Please update the PR description / earlier comments. The original "82 s → 18 s, ~4.5× faster" line was an in-memory-only comparison and is misleading now that we have file-based numbers. Let's replace it with the same-backend results so no one reads the old figure. But it is worth to test with atleast 50K if not 200k and publish the results comparison. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
