[
https://issues.apache.org/jira/browse/IGNITE-18475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Gusakov updated IGNITE-18475:
------------------------------------
Description:
During the YCSB benchmark runs for ignite-3 beta1 we found out, that we have
significant issues with performance for select/insert queries.
One of the root cause of these issues: write every log entry to rocksdb with
enabled sync option (which leads to frequent fsync calls).
These issues can be reproduced by localised jmh benchmarks
[SelectBenchmark|https://github.com/gridgain/apache-ignite-3/blob/4b9de922caa4aef97a5e8e159d5db76a3fc7a3ad/modules/runner/src/test/java/org/apache/ignite/internal/benchmark/SelectBenchmark.java#L39]
and
[InsertBenchmark|https://github.com/gridgain/apache-ignite-3/blob/4b9de922caa4aef97a5e8e159d5db76a3fc7a3ad/modules/runner/src/test/java/org/apache/ignite/internal/benchmark/InsertBenchmark.java#L29]
with RaftOptions.sync=true/false:
* jdbc select queries: 115ms vs 4ms
* jdbc insert queries: 70ms vs 2.5ms
(These results received on MacBook Pro (16-inch, 2019) and it looks like macOS
has slow fsync command in general, but runs on Ubuntu shows the huge different
also (~26 times for insert test). So, your environment can show another, but
still huge difference.)
Why select queries suffers from syncs even more, than inserts, described in
https://issues.apache.org/jira/browse/IGNITE-18474.
Possible solutions for the issue:
* Doesn't sync every raft record in rocksdb by default, but it can break the
raft invariants
* Investigate the inner parts of RocksDB (according syscall tracing, not every
write with sync produce fsync syscall), maybe another strategies wll be
suitable for our cases
was:
During the YCSB benchmark runs for ignite-3 beta1 we found out, that we have
significant issues with performance for select/insert queries.
One of the root cause of these issues: write every log entry to rocksdb with
enabled sync option (which leads to frequent fsync calls).
These issues can be reproduced by localised jmh benchmarks
[SelectBenchmark|https://github.com/gridgain/apache-ignite-3/blob/4b9de922caa4aef97a5e8e159d5db76a3fc7a3ad/modules/runner/src/test/java/org/apache/ignite/internal/benchmark/SelectBenchmark.java#L39]
and
[InsertBenchmark|https://github.com/gridgain/apache-ignite-3/blob/4b9de922caa4aef97a5e8e159d5db76a3fc7a3ad/modules/runner/src/test/java/org/apache/ignite/internal/benchmark/InsertBenchmark.java#L29]
with RaftOptions.sync=true/false:
* jdbc select queries: 115ms vs 4ms
* jdbc insert queries: 70ms vs 2.5ms
(These results received on MacBook Pro (16-inch, 2019) and it looks like macOS
has slow fsync command in general, but runs on Ubuntu shows the huge different
also (~26 times for insert test). So, your environment can show another, but
still huge difference.)
Why select quries suffers from syncs even more, than inserts, described in
https://issues.apache.org/jira/browse/IGNITE-18474.
Possible solutions for the issue:
* Doesn't sync every raft record in rocksdb by default, but it can break the
raft invariants
* Investigate the inner parts of RocksDB (according syscall tracing, not every
write with sync produce fsync syscall), maybe another strategies wll be
suitable for our cases
> Huge performance drop with enabled sync write per log entry for RAFT logs
> -------------------------------------------------------------------------
>
> Key: IGNITE-18475
> URL: https://issues.apache.org/jira/browse/IGNITE-18475
> Project: Ignite
> Issue Type: Task
> Reporter: Kirill Gusakov
> Priority: Major
>
> During the YCSB benchmark runs for ignite-3 beta1 we found out, that we have
> significant issues with performance for select/insert queries.
> One of the root cause of these issues: write every log entry to rocksdb with
> enabled sync option (which leads to frequent fsync calls).
> These issues can be reproduced by localised jmh benchmarks
> [SelectBenchmark|https://github.com/gridgain/apache-ignite-3/blob/4b9de922caa4aef97a5e8e159d5db76a3fc7a3ad/modules/runner/src/test/java/org/apache/ignite/internal/benchmark/SelectBenchmark.java#L39]
> and
> [InsertBenchmark|https://github.com/gridgain/apache-ignite-3/blob/4b9de922caa4aef97a5e8e159d5db76a3fc7a3ad/modules/runner/src/test/java/org/apache/ignite/internal/benchmark/InsertBenchmark.java#L29]
> with RaftOptions.sync=true/false:
> * jdbc select queries: 115ms vs 4ms
> * jdbc insert queries: 70ms vs 2.5ms
> (These results received on MacBook Pro (16-inch, 2019) and it looks like
> macOS has slow fsync command in general, but runs on Ubuntu shows the huge
> different also (~26 times for insert test). So, your environment can show
> another, but still huge difference.)
> Why select queries suffers from syncs even more, than inserts, described in
> https://issues.apache.org/jira/browse/IGNITE-18474.
> Possible solutions for the issue:
> * Doesn't sync every raft record in rocksdb by default, but it can break the
> raft invariants
> * Investigate the inner parts of RocksDB (according syscall tracing, not
> every write with sync produce fsync syscall), maybe another strategies wll be
> suitable for our cases
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
