Thanks for your analysis on this Ilya. Have some queries/observations on this as below:
1. Disabling WAL won’t be possible as it’s the current scenario that we are having in Production system. Also, currently difficult to upgrade (may be an least wanted option). 2. I have seen that there is still a disk space available on the drive (not actually saturated). 3. Also, we have tried with altering WAL size to 2GB (instead of default of 64 MB) and default chunks of it i.e. 10 but still there was no visible change. 4. Lastly, if we replace HDD with SSDs will this problem occur or it will really help ? Thanks and Regards, Kamlesh Joshi From: Ilya Kasnacheev <[email protected]> Sent: Monday, May 27, 2019 7:49 PM To: [email protected] Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin. Hello! The relevant stack trace as follows: "sys-stripe-43-#44%EDIFCustomer%" #70 prio=5 os_prio=0 tid=0x00007fc68e274800 nid=0x7ebb runnable [0x00007fc4898d6000] java.lang.Thread.State: RUNNABLE at java.nio.MappedByteBuffer.force0(Native Method) at java.nio.MappedByteBuffer.force(MappedByteBuffer.java:203) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.close(FileWriteAheadLogManager.java:2798) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.access$600(FileWriteAheadLogManager.java:2438) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.rollOver(FileWriteAheadLogManager.java:1064) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.log(FileWriteAheadLogManager.java:753) at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.beforeReleaseWrite(PageMemoryImpl.java:1611) at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlockPage(PageMemoryImpl.java:1478) at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:449) at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:443) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writeUnlock(PageHandler.java:377) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.writeUnlock(DataStructure.java:210) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.writeUnlock(DataStructure.java:181) at org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.put(PagesList.java:684) It seems that syncing WAL file to disk takes prolonged period of time, which is indicative of not being able to write to disks fast enough. I think that you have saturated your storage capacity. Maybe it will make sense to upgrade to 2.7 and disable WAL for the period of data ingestion, if you can separate it. Otherwise, look int beefing up your storage. What might also help is fewer WAL segments of larger size. Regards, -- Ilya Kasnacheev пт, 24 мая 2019 г. в 09:27, Kamlesh Joshi <[email protected]<mailto:[email protected]>>: Hi Ilya, PFA thread dumps and JFRs from the given cluster. Any observation and recommendation would be of great help! Thanks and Regards, Kamlesh Joshi From: Ilya Kasnacheev <[email protected]<mailto:[email protected]>> Sent: Thursday, May 23, 2019 7:35 PM To: [email protected]<mailto:[email protected]> Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin. Hello! I recommend gathering thread dumps from cluster as it has performance issues, sharing these dumps with us. Regards, -- Ilya Kasnacheev чт, 23 мая 2019 г. в 10:52, Kamlesh Joshi <[email protected]<mailto:[email protected]>>: Hi ilya, We tried with LOG_ONLY and BACKGROUND both but still behavior remains same. Any other parameter tweak which would help? Thanks and Regards, Kamlesh Joshi From: Ilya Kasnacheev <[email protected]<mailto:[email protected]>> Sent: Wednesday, May 22, 2019 9:14 PM To: [email protected]<mailto:[email protected]> Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin. Hello! If it is 'timeout' then it is likely not the reason of your issues. What is your walMode? Is it LOG_ONLY? If it's not, try to use LOG_ONLY here. Regards, -- Ilya Kasnacheev ср, 22 мая 2019 г. в 16:30, Kamlesh Joshi <[email protected]<mailto:[email protected]>>: Hi Ilya, Looking at the logs, every time checkpoint starts due to ‘timeout’ reason not because the buffer is full. So still do we need to increase checkpointing buffer or only changing checkpointing frequency will help ? Thanks and Regards, Kamlesh Joshi From: Ilya Kasnacheev <[email protected]<mailto:[email protected]>> Sent: Wednesday, May 22, 2019 4:19 PM To: [email protected]<mailto:[email protected]> Subject: Re: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin. Hello! We recommend using SSD and not HDD with Ignite. Otherwise, try to increase size of Checkpoint Page Buffer - if you run out of this buffer, all activity will stop until checkpoint is finished. Maybe you also need to decrease time between checkpoints. Regards, -- Ilya Kasnacheev вт, 7 мая 2019 г. в 13:01, Kamlesh Joshi <[email protected]<mailto:[email protected]>>: Hi, Unfortunately, I cant share the reproducer. We are using Ignite Binary Objects for performing operations on the cluster (Get/Put). We have exposed these operations to other application (i.e. TIBCO BW services) which performs operations on the cluster. Thanks and Regards, Kamlesh Joshi -----Original Message----- From: Maxim.Pudov <[email protected]<mailto:[email protected]>> Sent: Wednesday, April 17, 2019 8:18 PM To: [email protected]<mailto:[email protected]> Subject: [External]Re: Read/query TPS is decreasing after enabling mix load i.e. write services The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin. Hi, could you share a reproducer of your problem? We are missing a lot of information here. The configuration of your nodes, cache configurations, what API you use to query and update the data. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment." "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment." "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment." "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment." "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
