Hi, Ivan! I have only described ideas. But here are a few more details.
We can take care not to go beyond DataStorageConfiguration#maxWalArchiveSize. Before increasing the size of WAL archive (transferring to archive /rollOver, compression, decompression), we can make sure that there will be enough space in the archive and if there is no such, then we will try to clean it. We cannot delete those segments that are required for recovery (between the last two checkpoints) and reserved for example for historical rebalancing. We can receive a notification about the change of checkpoints and the reservation / release of segments, thus we can know how many segments we can delete right now. 06.11.2020, 09:53, "Ivan Daschinsky" <ivanda...@gmail.com>: >>> For example, when trying to move a segment to the archive. > > We cannot do this, we will lost data. We can truncate archived segment if > and only if it is not required for recovery. If last checkpoint marker > points to segment > with lower index, we cannot delete any segment with higher index. So the > only moment where we can remove truncate segments is a finish of checkpoint. > > пт, 6 нояб. 2020 г. в 09:46, ткаленко кирилл <tkalkir...@yandex.ru>: > >> Hello, everybody! >> >> As far as I know, WAL archive is used for PITP(GridGain feature) and >> historical rebalancing. >> >> Facundo seems to have a problem with running out of directory >> (/opt/work/walarchive) space. >> Currently, WAL archive is cleared at the end of checkpoint. Potentially >> long transaction may prevent checkpoint starting, thereby not cleaning WAL >> archive, which will lead to such an error. >> At the moment, I see such a WA to increase size of directory >> (/opt/work/walarchive) in k8s and avoid long transactions or something like >> that that modifies data and runs for a long time. >> >> And it is best to fix the logic of working with WAL archive. I think we >> should remove WAL archive cleanup from the end of the checkpoint and do it >> on demand. For example, when trying to move a segment to the archive. >> >> 06.11.2020, 01:58, "Denis Magda" <dma...@apache.org>: >> > Folks, >> > >> > In my understanding, you need the archives only for features such as >> PITR. >> > Considering, that the PITR functionality is not provided in Ignite why do >> > we have the archives enabled by default? >> > >> > How about having this feature disabled by default to prevent the >> following >> > issues experienced by our users: >> > >> >> http://apache-ignite-users.70518.x6.nabble.com/WAL-and-WAL-Archive-volume-size-recommendation-td34458.html >> > >> > - >> > Denis > > -- > Sincerely yours, Ivan Daschinskiy
