Ivan Bessonov created IGNITE-23240:
--------------------------------------
Summary: Ignite 3 new log storage
Key: IGNITE-23240
URL: https://issues.apache.org/jira/browse/IGNITE-23240
Project: Ignite
Issue Type: Epic
Reporter: Ivan Bessonov
Attachments: image-2024-09-20-10-39-22-043.png
h1. Preface
Current implementation, based on {{{}RocksDB{}}}, is known to be way slower
then it should be. There are multiple obvious reasons for that:
* Writing into WAL +and+ memtable
* Creating unique keys for every record
* Inability to efficiently serialize data, we must have an intermediate state
before we pass data into {{{}RocksDB{}}}'s API.
h1. Benchmarks
h3. Local benchmarks
Local benchmarks ({{{}LogStorageBenchmarks{}}}) have been performed on my local
environment with fsync disabled. I got the following results:
* {{{}Logit{}}}:
{noformat}
Test write:
Log number : 1024000
Log Size : 16384
Batch Size : 100
Cost time(s) : 23.541
Total size : 16777216000
Throughput(bps) : 712680684
Throughput(rps) : 43498
Test read:
Log number : 1024000
Log Size : 16384
Batch Size : 100
Cost time(s) : 3.808
Total size : 16777216000
Throughput(bps) : 4405781512
Throughput(rps) : 268907
Test done!{noformat}
* {{{}RocksDB{}}}:
{noformat}
Test write:
Log number : 1024000
Log Size : 16384
Batch Size : 100
Cost time(s) : 178.785
Total size : 16777216000
Throughput(bps) : 93840176
Throughput(rps) : 5727
Test read:
Log number : 1024000
Log Size : 16384
Batch Size : 100
Cost time(s) : 13.572
Total size : 16777216000
Throughput(bps) : 1236163866
Throughput(rps) : 75449
Test done!{noformat}
While testing on local environment is not optimal, is still shows a huge
improvement in writing speed (7.5x) and reading speed (3.5x). Enabling
{{fsync}} sort-of equalizes writing speed, but we still expect that simpler log
implementation would be faster dues to smaller overall overhead.
h3. Integration testing
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
