umesh9794 commented on PR #10549: URL: https://github.com/apache/ozone/pull/10549#issuecomment-4988744500
@devmadhuu I have added a file based derby test to compare master with this PR with file based derby, here are the outcomes : **Test 7 (worst case: 1M rows rewritten, single transaction)** In-memory vs File based Derby | master (delete-all + insert) | this PR (diff sync → all updates) -- | -- | -- In-memory | ~82 s | ~18 s File-based | ~87 s | >300 s → timed out **Full file-based comparison** Test | Operation | master | this PR -- | -- | -- | -- 1 | INSERT 1M | ~19 s | ~17.6 s 2 | COUNT(*) 1M | ~0.27 s | ~0.24 s 5 | Paginated read 200K | ~0.55 s | ~0.53 s 6 | Full 1M read | ~1.9 s | ~1.9 s 7 | Rewrite 1M, one tx | ~87 s | >300 s (timeout) **In-memory:** no log flush, so 1M in-place UPDATEs (~18 s) beat master's bulk delete + insert (~82 s). **File-based:** the PR's Test 7 issues 1M single-row UPDATEs inside one transaction — each is an index seek + in-place modify + WAL entry, and nothing commits until the end. On disk that's dramatically more expensive than master's bulk delete + append-style insert (which Derby handles efficiently). Hence >300 s vs 87 s. But Test 7 is not reflects the production scenario IIUC. Production syncs ≤50k-container chunks (not 200k-in-one-transaction), and in steady state most rows are unchanged. The realistic measure is the 50k steady-state chunk: this PR: ~3.9 s (skips ~45k, writes ~5k) master would unconditionally delete + re-insert the entire working set every scan, regardless of churn. So IMO the PR is a clear win in the real steady-state pattern, and a regression only in the artificial all-churn-in-one-giant-transaction case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
