Hi Ilya, This data loading is not a one time activity for our scenario. So, I guess turning off the WAL would not be a solution since turning off and on every time is not feasible every time the data loading happens. Please let me know if my understanding is correct.
I'll try to increase time between checkpoints and also increase checkpoint page buffer and let you know. In the meantime, just wanted to check if the Disk I/O and memory also has an effect? Currently, I am using 32 GB machine and increasing the memory will make any difference? Thanks & Regards, Shubham Agrawal On Mon, Apr 27, 2020 at 7:31 PM Ilya Kasnacheev <[email protected]> wrote: > Hello! > > My recommendation is to increase time between checkpoints > (checkpointFrequency) and, maybe, also increase checkpoint page buffer. > > If you do data loading in bursts, it may make sense to turn off WAL why > you do data loading. > > You may still expect some data degradation as time goes on (what's your > total allocated after the last 2.5 million?) > > Regards, > -- > Ilya Kasnacheev > > > пн, 27 апр. 2020 г. в 00:32, Shubham Agrawal <[email protected] > >: > >> Hi Team, >> >> I am using ignite 2.7.5. >> >> I am trying to insert 25 million records in the ignite persistent table. >> >> Observations: >> I am observing that initial 2.5 million records took 5 mins to insert, >> next 2.5 million records took 8 mins to insert, following 2.5 records took >> 12 mins to insert and it kept on increasing . The last 2.5 million records >> took over 40 minutes to insert. The CPU of the client and server is around >> 30%, so nothing on that front. Heap usage looks normal. I have tried >> inserting to the table with and without indexes but nothing seems to make a >> difference. >> >> Configurations: >> I am running a 3 node cluster, 16 Cores 32 GB machines. Ignite Version: >> 2.7.5 >> >> Things I have tried: >> JDBI Implementation - >> 1. @SQLBatch does not work for ignite giving the below error >> org.skife.jdbi.v2.exceptions.TransactionException: Failed to start >> transaction at >> org.skife.jdbi.v2.tweak.transactions.LocalTransactionHandler.begin(LocalTransactionHandler.java:57) >> at org.skife.jdbi.v2.BasicHandle.begin(BasicHandle.java:159) >> >> 2. Tried ?streaming=true, but with no effect >> >> 3. Increasing threads to insert, but not much effect >> >> JDBC Implementation >> 1. Tried ?streaming=true, but with no effect >> 2. Tried batching, but the performance actually degraded >> 3. Tried ?streaming=true with and without batching but with no effect. >> >> Want to achieve a common pattern in insertions, like 5 mins or 8 mins >> constantly for 25 million insertions >> >> Please let me know your thoughts on the same. Your inputs could help me a >> lot. Thanks a lot. >> >> Regards, >> Shubham Agrawal >> >
