ankrgyl commented on PR #542: URL: https://github.com/apache/arrow-rs-object-store/pull/542#issuecomment-3513777978
> > It's true, but I figured the read:write ratio is SO high (basically, 100% reads within the TTL window), that I'd like to avoid the (relatively) higher cost of an RwLock for the majority case. Would it be useful to illustrate this with a benchmark, or are you categorically against adding new dependencies? > > I think a super read heavy workload will work well with `RWLock` > > > Would it be useful to illustrate this with a benchmark, or are you categorically against adding new dependencies? > > In my mind, the burden of evidence is much higher to add a new dependency. I know it sounds somewhat like a curmudgeon, but each new dependency adds some small (but real) additional maintenance overhead (and downstream work). Unless there is compelling demonstrated value to add a new dependency, we try and avoid it. > > Also, there are very few contributions _removing_ dependencies for some reason 😆 so once we add one we are typically stuck with them. That's totally fine with me! I am only writing in to contribute in a helpful manner. If I staunchly feel that arc_swap is the right solution, I can easily run a fork. So I'm aligned to doing whatever you feel like is the right thing for the repo. Here are the benchmark numbers run on my macbook pro. What I saw in production was dramatically more pronounced (mutex vs. arc swap, not arc swap vs. rwlock), but I don't have much more bandwidth to investigate that in depth. ``` === Results Summary === Median Latency (p50): Concurrency │ Mutex │ Arc-swap │ RwLock │ Arc vs Mutex │ RwLock vs Mutex ──────────────────────────────────────────────────────────────────────────────────────────── 100 │ 2.31ms │ 2.34ms │ 2.35ms │ -1.2% │ -1.8% 500 │ 2.57ms │ 2.62ms │ 2.65ms │ -1.8% │ -2.9% 5k │ 5.01ms │ 3.46ms │ 4.07ms │ 44.9% │ 23.2% 5k │ 5.15ms │ 3.50ms │ 4.15ms │ 46.9% │ 24.1% 10k │ 8.18ms │ 7.11ms │ 8.69ms │ 15.0% │ -5.9% 25k │ 21.94ms │ 19.04ms │ 23.18ms │ 15.3% │ -5.3% Tail Latency (p99): Concurrency │ Mutex │ Arc-swap │ RwLock │ Arc vs Mutex │ RwLock vs Mutex ──────────────────────────────────────────────────────────────────────────────────────────── 100 │ 5.98ms │ 3.54ms │ 3.57ms │ 68.9% │ 67.8% 500 │ 4.01ms │ 3.93ms │ 3.93ms │ 2.2% │ 2.1% 5k │ 6.60ms │ 5.36ms │ 5.94ms │ 23.2% │ 11.1% 5k │ 7.83ms │ 5.81ms │ 6.16ms │ 34.8% │ 27.0% 10k │ 11.59ms │ 12.67ms │ 15.97ms │ -8.5% │ -27.4% 25k │ 31.44ms │ 37.85ms │ 47.92ms │ -16.9% │ -34.4% ``` I'm happy to update the code however you'd like based on these findings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
