[jira] [Updated] (RATIS-2403) Improve linearizable follower read throughput instead of writes
[ https://issues.apache.org/jira/browse/RATIS-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated RATIS-2403: -- Attachment: 1362_review2.patch > Improve linearizable follower read throughput instead of writes > --- > > Key: RATIS-2403 > URL: https://issues.apache.org/jira/browse/RATIS-2403 > Project: Ratis > Issue Type: Improvement > Components: Linearizable Read >Reporter: Ivan Andika >Assignee: Ivan Andika >Priority: Major > Attachments: 1362_review.patch, 1362_review2.patch, > LAW_THEOREM_RATIS_ANALYSIS.md, leader-backpressure.patch, > leader-batch-write.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > > While benchmarking linearizable follower read, the observation is that the > more requests go to the followers instead of the leader, the better write > throughput becomes, we saw around 2-3x write throughput increase compared to > the leader-only write and read (most likely due to less leader resource > contention). However, the read throughput becomes worst than leader-only > write and read (some can be below 0.2x). Even with optimizations such as > RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, > the read throughput remains worse than leader-only write (it even improves > the write performance instead of the read performance). > I suspect that because write throughput increase, the read index increases at > a faster rate which causes follower linearizable read to wait longer. > The target is to improve read throughput by 1.5x - 2x of the leader-only > write and reads. Currently pure reads (no writes) performance improves read > throughput up to 1.7x, but total follower read throughput is way below this > target. > Currently my ideas are > * Sacrificing writes for reads: Can we limit the write QPS so that read QPS > can increase > ** From the benchmark, the read throughput only improves when write > throughput is lower > ** We can try to use backpressure mechanism so that writes do not advance so > quickly that read throughput suffer > *** Follower gap mechanisms (RATIS-1411), but this might cause leader to > stall if follower down for a while (e.g. restarted), which violates the > majority availability guarantee. It's also hard to know which value is > optimal for different workloads. > Raising this ticket for ideas. [~szetszwo] [~tanxinyu] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RATIS-2403) Improve linearizable follower read throughput instead of writes
[ https://issues.apache.org/jira/browse/RATIS-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated RATIS-2403: --- Attachment: LAW_THEOREM_RATIS_ANALYSIS.md > Improve linearizable follower read throughput instead of writes > --- > > Key: RATIS-2403 > URL: https://issues.apache.org/jira/browse/RATIS-2403 > Project: Ratis > Issue Type: Improvement > Components: Linearizable Read >Reporter: Ivan Andika >Assignee: Ivan Andika >Priority: Major > Attachments: 1362_review.patch, LAW_THEOREM_RATIS_ANALYSIS.md, > leader-backpressure.patch, leader-batch-write.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > While benchmarking linearizable follower read, the observation is that the > more requests go to the followers instead of the leader, the better write > throughput becomes, we saw around 2-3x write throughput increase compared to > the leader-only write and read (most likely due to less leader resource > contention). However, the read throughput becomes worst than leader-only > write and read (some can be below 0.2x). Even with optimizations such as > RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, > the read throughput remains worse than leader-only write (it even improves > the write performance instead of the read performance). > I suspect that because write throughput increase, the read index increases at > a faster rate which causes follower linearizable read to wait longer. > The target is to improve read throughput by 1.5x - 2x of the leader-only > write and reads. Currently pure reads (no writes) performance improves read > throughput up to 1.7x, but total follower read throughput is way below this > target. > Currently my ideas are > * Sacrificing writes for reads: Can we limit the write QPS so that read QPS > can increase > ** From the benchmark, the read throughput only improves when write > throughput is lower > ** We can try to use backpressure mechanism so that writes do not advance so > quickly that read throughput suffer > *** Follower gap mechanisms (RATIS-1411), but this might cause leader to > stall if follower down for a while (e.g. restarted), which violates the > majority availability guarantee. It's also hard to know which value is > optimal for different workloads. > Raising this ticket for ideas. [~szetszwo] [~tanxinyu] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RATIS-2403) Improve linearizable follower read throughput instead of writes
[ https://issues.apache.org/jira/browse/RATIS-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated RATIS-2403: -- Attachment: 1362_review.patch > Improve linearizable follower read throughput instead of writes > --- > > Key: RATIS-2403 > URL: https://issues.apache.org/jira/browse/RATIS-2403 > Project: Ratis > Issue Type: Improvement >Reporter: Ivan Andika >Assignee: Ivan Andika >Priority: Major > Attachments: 1362_review.patch, leader-backpressure.patch, > leader-batch-write.patch > > > While benchmarking linearizable follower read, the observation is that the > more requests go to the followers instead of the leader, the better write > throughput becomes, we saw around 2-3x write throughput increase compared to > the leader-only write and read (most likely due to less leader resource > contention). However, the read throughput becomes worst than leader-only > write and read (some can be below 0.2x). Even with optimizations such as > RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, > the read throughput remains worse than leader-only write (it even improves > the write performance instead of the read performance). > I suspect that because write throughput increase, the read index increases at > a faster rate which causes follower linearizable read to wait longer. > The target is to improve read throughput by 1.5x - 2x of the leader-only > write and reads. Currently pure reads (no writes) performance improves read > throughput up to 1.7x, but total follower read throughput is way below this > target. > Currently my ideas are > * Sacrificing writes for reads: Can we limit the write QPS so that read QPS > can increase > ** From the benchmark, the read throughput only improves when write > throughput is lower > ** We can try to use backpressure mechanism so that writes do not advance so > quickly that read throughput suffer > *** Follower gap mechanisms (RATIS-1411), but this might cause leader to > stall if follower down for a while (e.g. restarted), which violates the > majority availability guarantee. It's also hard to know which value is > optimal for different workloads. > Raising this ticket for ideas. [~szetszwo] [~tanxinyu] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RATIS-2403) Improve linearizable follower read throughput instead of writes
[ https://issues.apache.org/jira/browse/RATIS-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated RATIS-2403: -- Component/s: Linearizable Read > Improve linearizable follower read throughput instead of writes > --- > > Key: RATIS-2403 > URL: https://issues.apache.org/jira/browse/RATIS-2403 > Project: Ratis > Issue Type: Improvement > Components: Linearizable Read >Reporter: Ivan Andika >Assignee: Ivan Andika >Priority: Major > Attachments: 1362_review.patch, leader-backpressure.patch, > leader-batch-write.patch > > > While benchmarking linearizable follower read, the observation is that the > more requests go to the followers instead of the leader, the better write > throughput becomes, we saw around 2-3x write throughput increase compared to > the leader-only write and read (most likely due to less leader resource > contention). However, the read throughput becomes worst than leader-only > write and read (some can be below 0.2x). Even with optimizations such as > RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, > the read throughput remains worse than leader-only write (it even improves > the write performance instead of the read performance). > I suspect that because write throughput increase, the read index increases at > a faster rate which causes follower linearizable read to wait longer. > The target is to improve read throughput by 1.5x - 2x of the leader-only > write and reads. Currently pure reads (no writes) performance improves read > throughput up to 1.7x, but total follower read throughput is way below this > target. > Currently my ideas are > * Sacrificing writes for reads: Can we limit the write QPS so that read QPS > can increase > ** From the benchmark, the read throughput only improves when write > throughput is lower > ** We can try to use backpressure mechanism so that writes do not advance so > quickly that read throughput suffer > *** Follower gap mechanisms (RATIS-1411), but this might cause leader to > stall if follower down for a while (e.g. restarted), which violates the > majority availability guarantee. It's also hard to know which value is > optimal for different workloads. > Raising this ticket for ideas. [~szetszwo] [~tanxinyu] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RATIS-2403) Improve linearizable follower read throughput instead of writes
[ https://issues.apache.org/jira/browse/RATIS-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated RATIS-2403: --- Attachment: leader-batch-write.patch > Improve linearizable follower read throughput instead of writes > --- > > Key: RATIS-2403 > URL: https://issues.apache.org/jira/browse/RATIS-2403 > Project: Ratis > Issue Type: Improvement >Reporter: Ivan Andika >Priority: Major > Attachments: leader-backpressure.patch, leader-batch-write.patch > > > While benchmarking linearizable follower read, the observation is that the > more requests go to the followers instead of the leader, the better write > throughput becomes, we saw around 2-3x write throughput increase compared to > the leader-only write and read (most likely due to less leader resource > contention). However, the read throughput becomes worst than leader-only > write and read (some can be below 0.2x). Even with optimizations such as > RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, > the read throughput remains worse than leader-only write (it even improves > the write performance instead of the read performance). > I suspect that because write throughput increase, the read index increases at > a faster rate which causes follower linearizable read to wait longer. > The target is to improve read throughput by 1.5x - 2x of the leader-only > write and reads. Currently pure reads (no writes) performance improves read > throughput up to 1.7x, but total follower read throughput is way below this > target. > Currently my ideas are > * Sacrificing writes for reads: Can we limit the write QPS so that read QPS > can increase > ** From the benchmark, the read throughput only improves when write > throughput is lower > ** We can try to use backpressure mechanism so that writes do not advance so > quickly that read throughput suffer > *** Follower gap mechanisms (RATIS-1411), but this might cause leader to > stall if follower down for a while (e.g. restarted), which violates the > majority availability guarantee. It's also hard to know which value is > optimal for different workloads. > Raising this ticket for ideas. [~szetszwo] [~tanxinyu] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RATIS-2403) Improve linearizable follower read throughput instead of writes
[ https://issues.apache.org/jira/browse/RATIS-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated RATIS-2403: --- Attachment: leader-backpressure.patch > Improve linearizable follower read throughput instead of writes > --- > > Key: RATIS-2403 > URL: https://issues.apache.org/jira/browse/RATIS-2403 > Project: Ratis > Issue Type: Improvement >Reporter: Ivan Andika >Priority: Major > Attachments: leader-backpressure.patch > > > While benchmarking linearizable follower read, the observation is that the > more requests go to the followers instead of the leader, the better write > throughput becomes, we saw around 2-3x write throughput increase compared to > the leader-only write and read (most likely due to less leader resource > contention). However, the read throughput becomes worst than leader-only > write and read (some can be below 0.2x). Even with optimizations such as > RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, > the read throughput remains worse than leader-only write (it even improves > the write performance instead of the read performance). > I suspect that because write throughput increase, the read index increases at > a faster rate which causes follower linearizable read to wait longer. > The target is to improve read throughput by 1.5x - 2x of the leader-only > write and reads. Currently pure reads (no writes) performance improves read > throughput up to 1.7x, but total follower read throughput is way below this > target. > Currently my ideas are > * Sacrificing writes for reads: Can we limit the write QPS so that read QPS > can increase > ** From the benchmark, the read throughput only improves when write > throughput is lower > ** We can try to use backpressure mechanism so that writes do not advance so > quickly that read throughput suffer > *** Follower gap mechanisms (RATIS-1411), but this might cause leader to > stall if follower down for a while (e.g. restarted), which violates the > majority availability guarantee. It's also hard to know which value is > optimal for different workloads. > Raising this ticket for ideas. [~szetszwo] [~tanxinyu] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RATIS-2403) Improve linearizable follower read throughput instead of writes
[ https://issues.apache.org/jira/browse/RATIS-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated RATIS-2403: --- Description: While benchmarking linearizable follower read, the observation is that the more requests go to the followers instead of the leader, the better write throughput becomes, we saw around 2-3x write throughput increase compared to the leader-only write and read (most likely due to less leader resource contention). However, the read throughput becomes worst than leader-only write and read (some can be below 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput remains worse than leader-only write (it even improves the write performance instead of the read performance). I suspect that because write throughput increase, the read index increases at a faster rate which causes follower linearizable read to wait longer. The target is to improve read throughput by 1.5x - 2x of the leader-only write and reads. Currently pure reads (no writes) performance improves read throughput up to 1.7x, but total follower read throughput is way below this target. Currently my ideas are * Sacrificing writes for reads: Can we limit the write QPS so that read QPS can increase ** From the benchmark, the read throughput only improves when write throughput is lower ** We can try to use backpressure mechanism so that writes do not advance so quickly that read throughput suffer *** Follower gap mechanisms (RATIS-1411), but this might cause leader to stall if follower down for a while (e.g. restarted), which violates the majority availability guarantee. It's also hard to know which value is optimal for different workloads. Raising this ticket for ideas. [~szetszwo] [~tanxinyu] was: While benchmarking linearizable follower read, the observation is that the more requests go to the followers instead of the leader, the write throughput improves dramatically by around 2-3x compared to the leader-only write and read (most likely due to less leader resource contention). However, the read throughput becomes worst than leader-only write and read (some can be below 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput remains worse than leader-only write (it even improves the write performance instead of the read performance). I suspect that because write throughput increase, the read index increases at a faster rate which causes follower linearizable read to wait longer. The target is to improve read throughput by 1.5x - 2x of the leader-only write and reads. Currently pure reads (no writes) performance improves read throughput up to 1.7x. Currently my ideas are * Sacrificing writes for reads: Can we limit the write QPS so that read QPS can increase ** From the benchmark, the read throughput only improves when write throughput is lower ** We can try to use backpressure mechanism so that writes do not advance so quickly that read throughput suffer *** Follower gap mechanisms (RATIS-1411), but this might cause leader to stall if follower down for a while (e.g. restarted), which violates the majority availability guarantee. It's also hard to know which value is optimal for different workloads. Raising this ticket for ideas. [~szetszwo] [~tanxinyu] > Improve linearizable follower read throughput instead of writes > --- > > Key: RATIS-2403 > URL: https://issues.apache.org/jira/browse/RATIS-2403 > Project: Ratis > Issue Type: Improvement >Reporter: Ivan Andika >Priority: Major > > While benchmarking linearizable follower read, the observation is that the > more requests go to the followers instead of the leader, the better write > throughput becomes, we saw around 2-3x write throughput increase compared to > the leader-only write and read (most likely due to less leader resource > contention). However, the read throughput becomes worst than leader-only > write and read (some can be below 0.2x). Even with optimizations such as > RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, > the read throughput remains worse than leader-only write (it even improves > the write performance instead of the read performance). > I suspect that because write throughput increase, the read index increases at > a faster rate which causes follower linearizable read to wait longer. > The target is to improve read throughput by 1.5x - 2x of the leader-only > write and reads. Currently pure reads (no writes) performance improves read > throughput up to 1.7x, but total follower read throughput is way below this > target. > Currently my ideas are > * Sacrificing writes for reads:
[jira] [Updated] (RATIS-2403) Improve linearizable follower read throughput instead of writes
[ https://issues.apache.org/jira/browse/RATIS-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated RATIS-2403: --- Description: While benchmarking linearizable follower read, the observation is that the more requests go to the followers instead of the leader, the write throughput improves dramatically by around 2-3x compared to the leader-only write and read (most likely due to less leader resource contention). However, the read throughput becomes worst than leader-only write and read (some can be below 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput remains worse than leader-only write (it even improves the write performance instead of the read performance). I suspect that because write throughput increase, the read index increases at a faster rate which causes follower linearizable read to wait longer. The target is to improve read throughput by 1.5x - 2x of the leader-only write and reads. Currently pure reads (no writes) performance improves read throughput up to 1.7x. Currently my ideas are * Sacrificing writes for reads: Can we limit the write QPS so that read QPS can increase ** From the benchmark, the read throughput only improves when write throughput is lower ** We can try to use backpressure mechanism so that writes do not advance so quickly that read throughput suffer *** Follower gap mechanisms (RATIS-1411), but this might cause leader to stall if follower down for a while (e.g. restarted), which violates the majority availability guarantee. It's also hard to know which value is optimal for different workloads. Raising this ticket for ideas. [~szetszwo] [~tanxinyu] was: While benchmarking linearizable follower read, the observation is that the more requests go to the followers instead of the leader, the write throughput improves dramatically by around 2-3x compared to the leader-only write and read (most likely due to less leader resource contention). However, the read throughput becomes worst than leader-only write and read (some can be below 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput remains worse than leader-only write (it even improves the write performance instead of the read performance). I suspect that because write throughput increase, the read index increases at a faster rate which causes follower linearizable read to wait longer. The target is to improve read throughput by 1.5x - 2x of the leader-only write and reads. Currently my ideas are * Sacrificing writes for reads: Can we limit the write QPS so that read QPS can increase ** From the benchmark, the read throughput only improves when write throughput is lower ** We can try to use backpressure mechanism so that writes do not advance so quickly that read throughput suffer *** Follower gap mechanisms (RATIS-1411), but this might cause leader to stall if follower down for a while (e.g. restarted), which violates the majority availability guarantee. It's also hard to know which value is optimal for different workloads. Raising this ticket for ideas. [~szetszwo] [~tanxinyu] > Improve linearizable follower read throughput instead of writes > --- > > Key: RATIS-2403 > URL: https://issues.apache.org/jira/browse/RATIS-2403 > Project: Ratis > Issue Type: Improvement >Reporter: Ivan Andika >Priority: Major > > While benchmarking linearizable follower read, the observation is that the > more requests go to the followers instead of the leader, the write throughput > improves dramatically by around 2-3x compared to the leader-only write and > read (most likely due to less leader resource contention). However, the read > throughput becomes worst than leader-only write and read (some can be below > 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 > [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput > remains worse than leader-only write (it even improves the write performance > instead of the read performance). I suspect that because write throughput > increase, the read index increases at a faster rate which causes follower > linearizable read to wait longer. > The target is to improve read throughput by 1.5x - 2x of the leader-only > write and reads. Currently pure reads (no writes) performance improves read > throughput up to 1.7x. > Currently my ideas are > * Sacrificing writes for reads: Can we limit the write QPS so that read QPS > can increase > ** From the benchmark, the read throughput only improves when write > throughput is lower > ** We can try to use backpressure mechanism so that writes do not advance so > quickly that read
[jira] [Updated] (RATIS-2403) Improve linearizable follower read throughput instead of writes
[ https://issues.apache.org/jira/browse/RATIS-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated RATIS-2403: --- Description: While benchmarking linearizable follower read, the observation is that the more requests go to the followers instead of the leader, the write throughput improves dramatically by around 2-3x compared to the leader-only write and read (most likely due to less leader resource contention). However, the read throughput becomes worst than leader-only write and read (some can be below 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput remains worse than leader-only write (it even improves the write performance instead of the read performance). I suspect that because write throughput increase, the read index increases at a faster rate which causes follower linearizable read to wait longer. The target is to improve read throughput by 1.5x - 2x of the leader-only write and reads. Currently my ideas are * Sacrificing writes for reads: Can we limit the write QPS so that read QPS can increase ** From the benchmark, the read throughput only improves when write throughput is lower ** We can try to use backpressure mechanism so that writes do not advance so quickly that read throughput suffer *** Follower gap mechanisms (RATIS-1411), but this might cause leader to stall if follower down for a while (e.g. restarted), which will violated the majority availability guarantee. It's also hard to know which value is optimal for different workloads. Raising this ticket for ideas. [~szetszwo] [~tanxinyu] was: While benchmarking linearizable follower read, the observation is that the more requests go to the followers instead of the leader, the write throughput improves dramatically by around 2-3x compared to the leader-only write and read (most likely due to less leader resource contention). However, the read throughput becomes worst than leader-only write and read (some can be below 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput remains worse than leader-only write (it even improves the write performance instead of the read performance). I suspect that because write throughput increase, the read index increases at a faster rate which causes follower linearizable read to wait longer. The target is to improve read throughput by 1.5x - 2x of the leader-only write and reads. Currently my ideas are * Sacrificing writes for reads: Can we limit the write QPS so that read QPS can increase ** From the benchmark, the read throughput only improves when write throughput is lower ** We can try to use backpressure mechanism so that writes do not advance so quickly that read throughput suffer *** Follower gap mechanisms (RATIS-1411), but this might cause leader to stall if follower down for a while (e.g. restarted), which will violated the majority availability guarantee. It's also hard to know which value is the best. Raising this ticket for ideas. [~szetszwo] [~tanxinyu] > Improve linearizable follower read throughput instead of writes > --- > > Key: RATIS-2403 > URL: https://issues.apache.org/jira/browse/RATIS-2403 > Project: Ratis > Issue Type: Improvement >Reporter: Ivan Andika >Priority: Major > > While benchmarking linearizable follower read, the observation is that the > more requests go to the followers instead of the leader, the write throughput > improves dramatically by around 2-3x compared to the leader-only write and > read (most likely due to less leader resource contention). However, the read > throughput becomes worst than leader-only write and read (some can be below > 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 > [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput > remains worse than leader-only write (it even improves the write performance > instead of the read performance). I suspect that because write throughput > increase, the read index increases at a faster rate which causes follower > linearizable read to wait longer. > The target is to improve read throughput by 1.5x - 2x of the leader-only > write and reads. > Currently my ideas are > * Sacrificing writes for reads: Can we limit the write QPS so that read QPS > can increase > ** From the benchmark, the read throughput only improves when write > throughput is lower > ** We can try to use backpressure mechanism so that writes do not advance so > quickly that read throughput suffer > *** Follower gap mechanisms (RATIS-1411), but this might cause leader to > stall if follower down for a while (e.g. restarted), which will violated the > ma
[jira] [Updated] (RATIS-2403) Improve linearizable follower read throughput instead of writes
[ https://issues.apache.org/jira/browse/RATIS-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated RATIS-2403: --- Description: While benchmarking linearizable follower read, the observation is that the more requests go to the followers instead of the leader, the write throughput improves dramatically by around 2-3x compared to the leader-only write and read (most likely due to less leader resource contention). However, the read throughput becomes worst than leader-only write and read (some can be below 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput remains worse than leader-only write (it even improves the write performance instead of the read performance). I suspect that because write throughput increase, the read index increases at a faster rate which causes follower linearizable read to wait longer. The target is to improve read throughput by 1.5x - 2x of the leader-only write and reads. Currently my ideas are * Sacrificing writes for reads: Can we limit the write QPS so that read QPS can increase ** From the benchmark, the read throughput only improves when write throughput is lower ** We can try to use backpressure mechanism so that writes do not advance so quickly that read throughput suffer *** Follower gap mechanisms (RATIS-1411), but this might cause leader to stall if follower down for a while (e.g. restarted), which violates the majority availability guarantee. It's also hard to know which value is optimal for different workloads. Raising this ticket for ideas. [~szetszwo] [~tanxinyu] was: While benchmarking linearizable follower read, the observation is that the more requests go to the followers instead of the leader, the write throughput improves dramatically by around 2-3x compared to the leader-only write and read (most likely due to less leader resource contention). However, the read throughput becomes worst than leader-only write and read (some can be below 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput remains worse than leader-only write (it even improves the write performance instead of the read performance). I suspect that because write throughput increase, the read index increases at a faster rate which causes follower linearizable read to wait longer. The target is to improve read throughput by 1.5x - 2x of the leader-only write and reads. Currently my ideas are * Sacrificing writes for reads: Can we limit the write QPS so that read QPS can increase ** From the benchmark, the read throughput only improves when write throughput is lower ** We can try to use backpressure mechanism so that writes do not advance so quickly that read throughput suffer *** Follower gap mechanisms (RATIS-1411), but this might cause leader to stall if follower down for a while (e.g. restarted), which will violated the majority availability guarantee. It's also hard to know which value is optimal for different workloads. Raising this ticket for ideas. [~szetszwo] [~tanxinyu] > Improve linearizable follower read throughput instead of writes > --- > > Key: RATIS-2403 > URL: https://issues.apache.org/jira/browse/RATIS-2403 > Project: Ratis > Issue Type: Improvement >Reporter: Ivan Andika >Priority: Major > > While benchmarking linearizable follower read, the observation is that the > more requests go to the followers instead of the leader, the write throughput > improves dramatically by around 2-3x compared to the leader-only write and > read (most likely due to less leader resource contention). However, the read > throughput becomes worst than leader-only write and read (some can be below > 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 > [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput > remains worse than leader-only write (it even improves the write performance > instead of the read performance). I suspect that because write throughput > increase, the read index increases at a faster rate which causes follower > linearizable read to wait longer. > The target is to improve read throughput by 1.5x - 2x of the leader-only > write and reads. > Currently my ideas are > * Sacrificing writes for reads: Can we limit the write QPS so that read QPS > can increase > ** From the benchmark, the read throughput only improves when write > throughput is lower > ** We can try to use backpressure mechanism so that writes do not advance so > quickly that read throughput suffer > *** Follower gap mechanisms (RATIS-1411), but this might cause leader to > stall if follower down for a while (e.g. restarted), which viol
[jira] [Updated] (RATIS-2403) Improve linearizable follower read throughput instead of writes
[ https://issues.apache.org/jira/browse/RATIS-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated RATIS-2403: --- Description: While benchmarking linearizable follower read, the observation is that the more requests go to the followers instead of the leader, the write throughput improves dramatically by around 2-3x compared to the leader-only write and read (most likely due to less leader resource contention). However, the read throughput becomes worst than leader-only write and read (some can be below 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput remains worse than leader-only write (it even improves the write performance instead of the read performance). I suspect that because write throughput increase, the read index increases at a faster rate which causes follower linearizable read to wait longer. The target is to improve read throughput by 1.5x - 2x of the leader-only write and reads. Currently my ideas are * Sacrificing writes for reads: Can we limit the write QPS so that read QPS can increase ** From the benchmark, the read throughput only improves when write throughput is lower ** We can try to use backpressure mechanism so that writes do not advance so quickly that read throughput suffer *** Follower gap mechanisms (RATIS-1411), but this might cause leader to stall if follower down for a while (e.g. restarted), which will violated the majority availability guarantee. It's also hard to know which value is the best. Raising this ticket for ideas. [~szetszwo] [~tanxinyu] was: While benchmarking linearizable follower read, the observation is that the more requests go to the followers instead of the leader, the write throughput improves dramatically by around 2-3x compared to the leader-only write and read (most likely due to less leader resource contention). However, the read throughput becomes worst than leader-only write and read (some can be below 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput remains worse than leader-only write (it even improves the write performance instead of the read performance). I suspect that because write throughput increase, the read index increases at a faster rate which causes follower linearizable read to wait longer. The target is to improve read throughput by 1.5x - 2x of the leader-only write and reads. Currently my ideas are * Sacrificing writes for reads: Can we limit the write QPS so that read QPS can increase ** From the benchmark, the read throughput only improves when write throughput is lower ** We can try to use backpressure mechanism so that writes do not advance so quickly that read throughput suffer *** Follower gap mechanisms (RATIS-1411), but this might cause leader to stall if follower down for a while (e.g. restarted), which will violated the majority availability guarantee. It's also hard to know which value is the best. Raising this ticket for ideas. > Improve linearizable follower read throughput instead of writes > --- > > Key: RATIS-2403 > URL: https://issues.apache.org/jira/browse/RATIS-2403 > Project: Ratis > Issue Type: Improvement >Reporter: Ivan Andika >Priority: Major > > While benchmarking linearizable follower read, the observation is that the > more requests go to the followers instead of the leader, the write throughput > improves dramatically by around 2-3x compared to the leader-only write and > read (most likely due to less leader resource contention). However, the read > throughput becomes worst than leader-only write and read (some can be below > 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 > [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput > remains worse than leader-only write (it even improves the write performance > instead of the read performance). I suspect that because write throughput > increase, the read index increases at a faster rate which causes follower > linearizable read to wait longer. > The target is to improve read throughput by 1.5x - 2x of the leader-only > write and reads. > Currently my ideas are > * Sacrificing writes for reads: Can we limit the write QPS so that read QPS > can increase > ** From the benchmark, the read throughput only improves when write > throughput is lower > ** We can try to use backpressure mechanism so that writes do not advance so > quickly that read throughput suffer > *** Follower gap mechanisms (RATIS-1411), but this might cause leader to > stall if follower down for a while (e.g. restarted), which will violated the > majority availability guarantee. It's also hard to
[jira] [Updated] (RATIS-2403) Improve linearizable follower read throughput instead of writes
[ https://issues.apache.org/jira/browse/RATIS-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Andika updated RATIS-2403: --- Description: While benchmarking linearizable follower read, the observation is that the more requests go to the followers instead of the leader, the write throughput improves dramatically by around 2-3x compared to the leader-only write and read (most likely due to less leader resource contention). However, the read throughput becomes worst than leader-only write and read (some can be below 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput remains worse than leader-only write (it even improves the write performance instead of the read performance). I suspect that because write throughput increase, the read index increases at a faster rate which causes follower linearizable read to wait longer. The target is to improve read throughput by 1.5x - 2x of the leader-only write and reads. Currently my ideas are * Sacrificing writes for reads: Can we limit the write QPS so that read QPS can increase ** From the benchmark, the read throughput only improves when write throughput is lower ** We can try to use backpressure mechanism so that writes do not advance so quickly that read throughput suffer *** Follower gap mechanisms (RATIS-1411), but this might cause leader to stall if follower down for a while (e.g. restarted), which will violated the majority availability guarantee. It's also hard to know which value is the best. Raising this ticket for ideas. was: While benchmarking linearizable follower read, the observation is that the more requests go to the followers instead of the leader, the write throughput improves dramatically by around 2-3x compared to the leader-only write and read (most likely due to less leader resource contention). However, the read throughput becomes worst than leader-only write and read (some can be below 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382[|https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput remains worse than leader-only write. I suspect that because write throughput increase, the read index changes a lot faster which causes follower to wait a lot longer. The target is to improve read throughput by 1.5x - 2x of the leader-only write and reads. Raising this ticket for ideas. Currently my ideas are * Sacrificing writes for reads ** From the benchmark, the read throughput only improves when write throughput is lower ** We can try to use backpressure mechanism so that writes do not advance so quickly that read throughput suffer *** Follower gap mechanisms (RATIS-1411), but this might cause leader to stall if follower down for a while (e.g. restarted), which will violated the majority availability guarantee It's also hard to know which value is the best ** > Improve linearizable follower read throughput instead of writes > --- > > Key: RATIS-2403 > URL: https://issues.apache.org/jira/browse/RATIS-2403 > Project: Ratis > Issue Type: Improvement >Reporter: Ivan Andika >Priority: Major > > While benchmarking linearizable follower read, the observation is that the > more requests go to the followers instead of the leader, the write throughput > improves dramatically by around 2-3x compared to the leader-only write and > read (most likely due to less leader resource contention). However, the read > throughput becomes worst than leader-only write and read (some can be below > 0.2x). Even with optimizations such as RATIS-2392 RATIS-2382 > [https://github.com/apache/ratis/pull/1334] RATIS-2379, the read throughput > remains worse than leader-only write (it even improves the write performance > instead of the read performance). I suspect that because write throughput > increase, the read index increases at a faster rate which causes follower > linearizable read to wait longer. > The target is to improve read throughput by 1.5x - 2x of the leader-only > write and reads. > Currently my ideas are > * Sacrificing writes for reads: Can we limit the write QPS so that read QPS > can increase > ** From the benchmark, the read throughput only improves when write > throughput is lower > ** We can try to use backpressure mechanism so that writes do not advance so > quickly that read throughput suffer > *** Follower gap mechanisms (RATIS-1411), but this might cause leader to > stall if follower down for a while (e.g. restarted), which will violated the > majority availability guarantee. It's also hard to know which value is the > best. > Raising this ticket for ideas. -- This message was sent by Atlassian Jira (v8.20.10#820010)
