[
https://issues.apache.org/jira/browse/IGNITE-22664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Shergalis updated IGNITE-22664:
---------------------------------------
Description:
Inserting 1 million rows, resulting database after compaction takes around
330MB on the disk. Still, delta files created in the process sum up to 3.3
gigabytes (checked by commenting out compaction code, 256 MB region size), so
checkpointing and compacting can't keep up with write-intensive scenarios. We
should try to optimise amount of writing to disk
Also, RocksDB with 1 million rows used only 95MB of disk space
was:
Every page in persistent page memory delta file uses ~15-20 kilobytes (default
settings).
Examples. Inserting via storageUpdateHandler. No indexes:
- Inserting 1 row at a time, doing checkpoint after every insertion.
Checkpoint saves 7 pages, pageSize is 1024, but delta file created uses up 144
KB.
- Inserting million rows, every checkpoint with ~11000 pages creates delta
files of ~180 megabytes
Also, after compaction one row uses ~300 bytes, while the same row in rocksDB
uses ~100 bytes
Checkpoints / compaction are hardcoded to use 4 threads
> Insanely memory consuming checkpoints in aipersist
> --------------------------------------------------
>
> Key: IGNITE-22664
> URL: https://issues.apache.org/jira/browse/IGNITE-22664
> Project: Ignite
> Issue Type: Improvement
> Components: persistence
> Reporter: Philipp Shergalis
> Priority: Major
> Labels: ignite-3
>
> Inserting 1 million rows, resulting database after compaction takes around
> 330MB on the disk. Still, delta files created in the process sum up to 3.3
> gigabytes (checked by commenting out compaction code, 256 MB region size), so
> checkpointing and compacting can't keep up with write-intensive scenarios. We
> should try to optimise amount of writing to disk
>
> Also, RocksDB with 1 million rows used only 95MB of disk space
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
