Re: Using native persistence to "extend" memory
Thank you both for clarifying this. I usually also have large objects 3KB so I thought about increasing the page size to 32KB to reduce the number of pages and thus reduce the speed at which we get to the 2/3 of dirty pages. Good idea? On top of that during the process at some I generate a significant amount of new objects so I also kept increased checkpointPageBufferSize to 4GB. I also writeThrottlingEnabled disabled. Thanks for advising. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Using native persistence to "extend" memory
Evgeniy, Thanks for clarifying, I completely forgot about this behavior! - Denis On Tue, Jun 16, 2020 at 5:10 PM Evgenii Zhuravlev wrote: > Steve, > > Actually, disabling WAL is a good option for your use case. Checkpoint > mechanism is the same with disabled WAL, the only difference is that node > is not writing WAL to the disk on each operation. Usually, it might make > sense to disable WAL for initial loading - when you can lose the data in > case of failure and start data loading again. Four your use case, if you > don't care about restore, you can just disable it: > https://www.gridgain.com/docs/latest/developers-guide/persistence/native-persistence#disabling-wal > > Best Regards, > Evgenii > > > > вт, 16 июн. 2020 г. в 17:02, Denis Magda : > >> Steve, >> >> Please check these generic recommendations if you haven't done so >> already: >> https://apacheignite.readme.io/docs/durable-memory-tuning#native-persistence-related-tuning >> >> Otherwise, send us a note if you come across any bottlenecks or issues so >> that we can give you more specific recommendations. >> >> - >> Denis >> >> >> On Tue, Jun 16, 2020 at 3:25 PM steve.hostettler < >> steve.hostett...@gmail.com> wrote: >> >>> Thanks a lot for the recommendation. So keeping the WAL, disabling >>> archiving. >>> I understand all records are kept on disk. >>> >>> Thanks again. Anything else? >>> >>> >>> >>> -- >>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >>> >>
Re: Using native persistence to "extend" memory
Steve, Actually, disabling WAL is a good option for your use case. Checkpoint mechanism is the same with disabled WAL, the only difference is that node is not writing WAL to the disk on each operation. Usually, it might make sense to disable WAL for initial loading - when you can lose the data in case of failure and start data loading again. Four your use case, if you don't care about restore, you can just disable it: https://www.gridgain.com/docs/latest/developers-guide/persistence/native-persistence#disabling-wal Best Regards, Evgenii вт, 16 июн. 2020 г. в 17:02, Denis Magda : > Steve, > > Please check these generic recommendations if you haven't done so already: > https://apacheignite.readme.io/docs/durable-memory-tuning#native-persistence-related-tuning > > Otherwise, send us a note if you come across any bottlenecks or issues so > that we can give you more specific recommendations. > > - > Denis > > > On Tue, Jun 16, 2020 at 3:25 PM steve.hostettler < > steve.hostett...@gmail.com> wrote: > >> Thanks a lot for the recommendation. So keeping the WAL, disabling >> archiving. >> I understand all records are kept on disk. >> >> Thanks again. Anything else? >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >
Re: Using native persistence to "extend" memory
Steve, Please check these generic recommendations if you haven't done so already: https://apacheignite.readme.io/docs/durable-memory-tuning#native-persistence-related-tuning Otherwise, send us a note if you come across any bottlenecks or issues so that we can give you more specific recommendations. - Denis On Tue, Jun 16, 2020 at 3:25 PM steve.hostettler wrote: > Thanks a lot for the recommendation. So keeping the WAL, disabling > archiving. > I understand all records are kept on disk. > > Thanks again. Anything else? > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Using native persistence to "extend" memory
Thanks a lot for the recommendation. So keeping the WAL, disabling archiving. I understand all records are kept on disk. Thanks again. Anything else? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Using native persistence to "extend" memory
Hi Steve, I think that you can get a performance hit for your write operations by disabling the WAL. It serves two purposes in Ignite. The first is the recovery while the second is fast disk writes. The WAL is an append-only structure and Ignite can persist changes to disk really quick. If the WAL is disabled then it will take more time to update persistence files on disk. Just in case, if you missed that point, once the persistence is enabled Ignite will keep 100% of records on disk. It's not like a OS swap process that kicks in only when you're running out of memory space. Usually, we recommend disabling the WAL during a loading phase: https://apacheignite.readme.io/docs/write-ahead-log#section-wal-activation-and-deactivation In general, I would certainly keep the WAL enabled but disabled the WAL archiving if the recovery is not a big deal for you: https://apacheignite.readme.io/docs/write-ahead-log#disabling-wal-archiving - Denis On Tue, Jun 16, 2020 at 10:50 AM steve.hostettler < steve.hostett...@gmail.com> wrote: > Hello, > > I am trying to use ignite persistence to "extend" memory. That is to > replace > the swap that is not working very well. > > Therefore since I do not care about recovery I disabled the WAL. Are there > other things you would recommend to configure to use the ignite persistence > as a sort of swap. For instance only persisting the less used pages most of > the time. > > Thanks > > Steve > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Using native persistence to "extend" memory
Hello, I am trying to use ignite persistence to "extend" memory. That is to replace the swap that is not working very well. Therefore since I do not care about recovery I disabled the WAL. Are there other things you would recommend to configure to use the ignite persistence as a sort of swap. For instance only persisting the less used pages most of the time. Thanks Steve -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
