Hello!
The relevant stack trace as follows:
"sys-stripe-43-#44%EDIFCustomer%" #70 prio=5 os_prio=0
tid=0x00007fc68e274800 nid=0x7ebb runnable [0x00007fc4898d6000]
java.lang.Thread.State: RUNNABLE
at java.nio.MappedByteBuffer.force0(Native Method)
at java.nio.MappedByteBuffer.force(MappedByteBuffer.java:203)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.close(FileWriteAheadLogManager.java:2798)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.access$600(FileWriteAheadLogManager.java:2438)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.rollOver(FileWriteAheadLogManager.java:1064)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.log(FileWriteAheadLogManager.java:753)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.beforeReleaseWrite(PageMemoryImpl.java:1611)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlockPage(PageMemoryImpl.java:1478)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:449)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:443)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writeUnlock(PageHandler.java:377)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.writeUnlock(DataStructure.java:210)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.writeUnlock(DataStructure.java:181)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.put(PagesList.java:684)
It seems that syncing WAL file to disk takes prolonged period of time,
which is indicative of not being able to write to disks fast enough. I
think that you have saturated your storage capacity.
Maybe it will make sense to upgrade to 2.7 and disable WAL for the period
of data ingestion, if you can separate it. Otherwise, look int beefing up
your storage. What might also help is fewer WAL segments of larger size.
Regards,
--
Ilya Kasnacheev
пт, 24 мая 2019 г. в 09:27, Kamlesh Joshi <kamlesh.jo...@ril.com>:
> Hi Ilya,
>
>
>
> PFA thread dumps and JFRs from the given cluster. Any observation and
> recommendation would be of great help!
>
>
>
> *Thanks and Regards,*
>
> *Kamlesh Joshi*
>
>
>
> *From:* Ilya Kasnacheev <ilya.kasnach...@gmail.com>
> *Sent:* Thursday, May 23, 2019 7:35 PM
> *To:* user@ignite.apache.org
> *Subject:* Re: [External]Re: Read/query TPS is decreasing after enabling
> mix load i.e. write services
>
>
>
> The e-mail below is from an external source. Please do not open
> attachments or click links from an unknown or suspicious origin.
>
> Hello!
>
>
>
> I recommend gathering thread dumps from cluster as it has performance
> issues, sharing these dumps with us.
>
>
>
> Regards,
>
> --
>
> Ilya Kasnacheev
>
>
>
>
>
> чт, 23 мая 2019 г. в 10:52, Kamlesh Joshi <kamlesh.jo...@ril.com>:
>
> Hi ilya,
>
>
>
> We tried with LOG_ONLY and BACKGROUND both but still behavior remains
> same. Any other parameter tweak which would help?
>
>
>
> *Thanks and Regards,*
>
> *Kamlesh Joshi*
>
>
>
> *From:* Ilya Kasnacheev <ilya.kasnach...@gmail.com>
> *Sent:* Wednesday, May 22, 2019 9:14 PM
> *To:* user@ignite.apache.org
> *Subject:* Re: [External]Re: Read/query TPS is decreasing after enabling
> mix load i.e. write services
>
>
>
> The e-mail below is from an external source. Please do not open
> attachments or click links from an unknown or suspicious origin.
>
> Hello!
>
>
>
> If it is 'timeout' then it is likely not the reason of your issues.
>
>
>
> What is your walMode? Is it LOG_ONLY? If it's not, try to use LOG_ONLY
> here.
>
>
>
> Regards,
>
> --
>
> Ilya Kasnacheev
>
>
>
>
>
> ср, 22 мая 2019 г. в 16:30, Kamlesh Joshi <kamlesh.jo...@ril.com>:
>
> Hi Ilya,
>
>
>
> Looking at the logs, every time checkpoint starts due to
> ‘timeout’ reason not because the buffer is full. So still do we need to
> increase checkpointing buffer or only changing checkpointing frequency will
> help ?
>
>
>
> *Thanks and Regards,*
>
> *Kamlesh Joshi*
>
>
>
> *From:* Ilya Kasnacheev <ilya.kasnach...@gmail.com>
> *Sent:* Wednesday, May 22, 2019 4:19 PM
> *To:* user@ignite.apache.org
> *Subject:* Re: [External]Re: Read/query TPS is decreasing after enabling
> mix load i.e. write services
>
>
>
> The e-mail below is from an external source. Please do not open
> attachments or click links from an unknown or suspicious origin.
>
> Hello!
>
>
>
> We recommend using SSD and not HDD with Ignite. Otherwise, try to increase
> size of Checkpoint Page Buffer - if you run out of this buffer, all
> activity will stop until checkpoint is finished. Maybe you also need to
> decrease time between checkpoints.
>
>
>
> Regards,
>
> --
>
> Ilya Kasnacheev
>
>
>
>
>
> вт, 7 мая 2019 г. в 13:01, Kamlesh Joshi <kamlesh.jo...@ril.com>:
>
> Hi,
>
> Unfortunately, I cant share the reproducer.
> We are using Ignite Binary Objects for performing operations on the
> cluster (Get/Put). We have exposed these operations to other application
> (i.e. TIBCO BW services) which performs operations on the cluster.
>
> Thanks and Regards,
> Kamlesh Joshi
>
> -----Original Message-----
> From: Maxim.Pudov <pudov....@gmail.com>
> Sent: Wednesday, April 17, 2019 8:18 PM
> To: user@ignite.apache.org
> Subject: [External]Re: Read/query TPS is decreasing after enabling mix
> load i.e. write services
>
> The e-mail below is from an external source. Please do not open
> attachments or click links from an unknown or suspicious origin.
>
> Hi, could you share a reproducer of your problem? We are missing a lot of
> information here. The configuration of your nodes, cache configurations,
> what API you use to query and update the data.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
> "Confidentiality Warning: This message and any attachments are intended
> only for the use of the intended recipient(s).
> are confidential and may be privileged. If you are not the intended
> recipient. you are hereby notified that any
> review. re-transmission. conversion to hard copy. copying. circulation or
> other use of this message and any attachments is
> strictly prohibited. If you are not the intended recipient. please notify
> the sender immediately by return email.
> and delete this message and any attachments from your system.
>
> Virus Warning: Although the company has taken reasonable precautions to
> ensure no viruses are present in this email.
> The company cannot accept responsibility for any loss or damage arising
> from the use of this email or attachment."
>
>
> "*Confidentiality Warning*: This message and any attachments are intended
> only for the use of the intended recipient(s), are confidential and may be
> privileged. If you are not the intended recipient, you are hereby notified
> that any review, re-transmission, conversion to hard copy, copying,
> circulation or other use of this message and any attachments is strictly
> prohibited. If you are not the intended recipient, please notify the sender
> immediately by return email and delete this message and any attachments
> from your system.
>
> *Virus Warning:* Although the company has taken reasonable precautions to
> ensure no viruses are present in this email. The company cannot accept
> responsibility for any loss or damage arising from the use of this email or
> attachment."
>
>
> "*Confidentiality Warning*: This message and any attachments are intended
> only for the use of the intended recipient(s), are confidential and may be
> privileged. If you are not the intended recipient, you are hereby notified
> that any review, re-transmission, conversion to hard copy, copying,
> circulation or other use of this message and any attachments is strictly
> prohibited. If you are not the intended recipient, please notify the sender
> immediately by return email and delete this message and any attachments
> from your system.
>
> *Virus Warning:* Although the company has taken reasonable precautions to
> ensure no viruses are present in this email. The company cannot accept
> responsibility for any loss or damage arising from the use of this email or
> attachment."
>
>
> "*Confidentiality Warning*: This message and any attachments are intended
> only for the use of the intended recipient(s), are confidential and may be
> privileged. If you are not the intended recipient, you are hereby notified
> that any review, re-transmission, conversion to hard copy, copying,
> circulation or other use of this message and any attachments is strictly
> prohibited. If you are not the intended recipient, please notify the sender
> immediately by return email and delete this message and any attachments
> from your system.
>
> *Virus Warning:* Although the company has taken reasonable precautions to
> ensure no viruses are present in this email. The company cannot accept
> responsibility for any loss or damage arising from the use of this email or
> attachment."
>
