[
https://issues.apache.org/jira/browse/IGNITE-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Rakov updated IGNITE-6334:
-------------------------------
Description:
We've received several negative pieces of feedback about LFS performance with
enabled persistence. Ignite node stops responding to user operations under
intensive load. Typical operations/second graph is attached.
!attachment-name.jpg|thumbnail!
Zero dropdowns happen during ongoing checkpoint when checkpoint buffer (memory
segment that accumulates old versions of dirty pages that are not yet written
in current checkpoint) is overflowed.
In general, performance decrease is inevitable - writing in memory is always
faster than writing to disk. Though, we can avoid zero dropdowns if we'll
throttle threads that generate dirty pages.
We can manage amount of throttle time with tocken bucket algorithm:
1) Before checkpoint start, we calculate ratio K = (number of checkpoint pages)
/ (size of checkpoint buffer) and initialize non-negative atomic marker counter
2) Every checkpointing thread increments marker counter once per K written pages
3) Any thread that makes page dirty should decrement marker counter. Thread
should wait if marker counter is zero.
Such algorithm makes buffer overflow impossible. If activity is intensive and
constant, user threads will write at the speed of the disk. On the other hand,
user threads will write at maximum speed in case of burst activity.
was:
We've received several negative pieces of feedback about LFS performance with
enabled persistence. Ignite node stops responding to user operations under
intensive load. Typical operations/second graph is attached. Zero dropdowns
happen during ongoing checkpoint when checkpoint buffer (memory segment that
accumulates old versions of dirty pages that are not yet written in current
checkpoint) is overflowed.
In general, performance decrease is inevitable - writing in memory is always
faster than writing to disk. Though, we can avoid zero dropdowns if we'll
throttle threads that generate dirty pages.
We can manage amount of throttle time with tocken bucket algorithm:
1) Before checkpoint start, we calculate ratio K = (number of checkpoint pages)
/ (size of checkpoint buffer) and initialize non-negative atomic marker counter
2) Every checkpointing thread increments marker counter once per K written pages
3) Any thread that makes page dirty should decrement marker counter. Thread
should wait if marker counter is zero.
Such algorithm makes buffer overflow impossible. If activity is intensive and
constant, user threads will write at the speed of the disk. On the other hand,
user threads will write at maximum speed in case of burst activity.
> Throttle writing threads during ongoing checkpoint with token bucket algorithm
> ------------------------------------------------------------------------------
>
> Key: IGNITE-6334
> URL: https://issues.apache.org/jira/browse/IGNITE-6334
> Project: Ignite
> Issue Type: Improvement
> Components: persistence
> Affects Versions: 2.1
> Reporter: Ivan Rakov
> Fix For: 2.3
>
> Attachments: opsec3.jpg, opsec3.PNG
>
>
> We've received several negative pieces of feedback about LFS performance with
> enabled persistence. Ignite node stops responding to user operations under
> intensive load. Typical operations/second graph is attached.
> !attachment-name.jpg|thumbnail!
> Zero dropdowns happen during ongoing checkpoint when checkpoint buffer
> (memory segment that accumulates old versions of dirty pages that are not yet
> written in current checkpoint) is overflowed.
> In general, performance decrease is inevitable - writing in memory is always
> faster than writing to disk. Though, we can avoid zero dropdowns if we'll
> throttle threads that generate dirty pages.
> We can manage amount of throttle time with tocken bucket algorithm:
> 1) Before checkpoint start, we calculate ratio K = (number of checkpoint
> pages) / (size of checkpoint buffer) and initialize non-negative atomic
> marker counter
> 2) Every checkpointing thread increments marker counter once per K written
> pages
> 3) Any thread that makes page dirty should decrement marker counter. Thread
> should wait if marker counter is zero.
> Such algorithm makes buffer overflow impossible. If activity is intensive and
> constant, user threads will write at the speed of the disk. On the other
> hand, user threads will write at maximum speed in case of burst activity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
