[
https://issues.apache.org/jira/browse/IGNITE-8017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ilya Lantukh updated IGNITE-8017:
---------------------------------
Description:
While handling SupplyMessage, node handles each supplied data entry separately,
which causes a WAL record for each entry to be written. It significantly limits
preloading speed.
We can improve rebalancing speed and reduce pressure on disk by disabling WAL
until all data is loaded. The disadvantage of this approach is that data might
get corrupted if node crashes - but node that crashed during preloading has to
clear all it's data anyway. However, it is important to distinguish situations
when new node joined cluster or added to baseline topology (and doesn't hold
any data) and when additional partitions got assigned to node after baseline
topology changed (in this case node has to keep all data in consistent state).
was:
While handling SupplyMessage, node handles each supplied data entry separately,
which causes a WAL record for each entry to be written. It significantly limits
preloading speed, especially with WALMode == FSYNC - it will perform fsync for
every entry!
We can improve rebalancing speed and reduce pressure on disk by disabling WAL
until all data is loaded. The disadvantage of this approach is that data might
get corrupted if node crashes - but node that crashed during preloading has to
clear all it's data anyway. However, it is important to distinguish situations
when new node joined cluster or added to baseline topology (and doesn't hold
any data) and when additional partitions got assigned to node after baseline
topology changed (in this case node has to keep all data in consistent state).
> Disable WAL during initial preloading
> -------------------------------------
>
> Key: IGNITE-8017
> URL: https://issues.apache.org/jira/browse/IGNITE-8017
> Project: Ignite
> Issue Type: Improvement
> Reporter: Ilya Lantukh
> Assignee: Ilya Lantukh
> Priority: Major
>
> While handling SupplyMessage, node handles each supplied data entry
> separately, which causes a WAL record for each entry to be written. It
> significantly limits preloading speed.
> We can improve rebalancing speed and reduce pressure on disk by disabling WAL
> until all data is loaded. The disadvantage of this approach is that data
> might get corrupted if node crashes - but node that crashed during preloading
> has to clear all it's data anyway. However, it is important to distinguish
> situations when new node joined cluster or added to baseline topology (and
> doesn't hold any data) and when additional partitions got assigned to node
> after baseline topology changed (in this case node has to keep all data in
> consistent state).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
