Hello, Ilya.
Great! Thanks!
Can I extend your idea a bit?
I think it be very useful monitor all-user provided listeners and
callbacks also.
To handle following scenarios:
1а. User starts a job with ExecutorService, IgniteCompute or similar.
1b. User creates ContinuousQuery with remoteFilter and localListener.
2. It take huge amount of time to execute user callback on some node. Or
thread blocks on some monitor inside callback.
In that case Ignite can detect it and print some waring message.
We can cancel user callback to free resources in some cases.
Specific timeouts and cancel policy should be configured by user.
May be it already covered by FailOverSpi [1] but I can't find
description of such feature.
We can take WebSphere hangs detection mechanism [2], [3] as an example.
[1] https://apacheignite.readme.io/docs/fault-tolerance
[2]
https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/hung_thread_detection_in_websphere_application_server?lang=en
[3]
https://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.iseries.doc/ae/ttrb_confighangdet.html
08.09.2017 14:27, Ilya Lantukh пишет:
Igniters,
According to our current design and implementation, unclosed transaction or
unreleased lock can hang ignite cluster forever. This is logical, and with
correct usage of those mechanics such issue should never happen, in real
world developers can make mistakes and leave transaction open. We have a
feature "transaction timeout", but turns out it doesn't work in all cases
(see https://issues.apache.org/jira/browse/IGNITE-6181). Even if all known
issues are fixed, there is still a lot of room for mistake and incorrect
usage.
To make it possible for Ignite users to discover such problem and trace it
to a particular part of code, I've created a very simple utility that
collects and prints information about long running transactions for the
whole cluster. It is available here:
https://github.com/ilantukh/IgniteTxViewer.
One might expect such monitoring utilities to be included in Ignite
codebase. Personally, I think that such information should be available
from public API, without using of additional applications or diving into
Ignite internals.
What do you think?
