The "Illegal invocation from MessageListener callback" issue looks to be a distinct bug in the client. A MessageListener isnt normally allowed to call close() on its session or connection, however the way this restriction is implemented in the client means it is currently also being applied incorrectly during the ExceptionListener callback caused by these unusual failures during async dispatch. Essentially, if your application didnt deadlock upon itself across different sessions/connections in somewhat of a circle as youve seen, then it could have instead hit this bug.
I have raised https://issues.apache.org/jira/browse/QPIDJMS-521 and pushed an initial change with a test exposing the issue and an initial fix resolving it. You can try it out by building yourself from https://github.com/apache/qpid-jms or using a snapshot from http://repository.apache.org/content/groups/snapshots/. The order of your connection/consumer/session closures doesnt make sense. If you close the connection as you do first, then the sessions and their consumers/producers are implicity shut down and marked closed. You can just close the connection in that case. No, you needn't call setMessageListener(null) first if you do close a consumer. You can just close the consumer, or its session, or connection. The overall effect isnt much different - all routes would wait on the delivery lock if needed before proceeding. Setting the fields to null wont really affect cleanup beyond basic GC. If there is a failure to cleanup and there are threads still running they may keep doing so regardless. That mostly shouldnt be a problem(besides obviously the associated resources) since the connections would be largely independent outwith your application. One thing to perhaps look at may be the way you create/use the ExceptionListeners. It might be good to try and ensure each listener can only reference specific objects it was created for, and cant for example reference different objects created later over time, which are perhaps being served by another listener also created later, such that they cant cross-interact that way. The original issue occurred due to unexpected failure during the work surrounding onMessage callback, leading to the ExceptionListener being fired. In the older client, this was done while holding the session delivery lock, leading to interaction with your applications own lock and connection/session usage as I've previously explained, or failing that issue arising then instead it could then hit QPIDJMS-521. One simple way the unexpected failure might occur was for message arriving with a Data body section and having content-type indicating it contained text, but the payload not decoding as such. This seemed to be what you were saw originally. Another simple way presented recently by PROTON-2097 was for an entirely valid UTF-8 encoded string type containing multi-byte character encodings, to be received across multiple transfer sections, such that a single multi-byte character was split across different internal buffer sections. Simplest way for the client to hit this, is likely to receive a > 2MB TextMessage (with amqp-value body section containing string type) with body composed entirely of multi-byte unicode characters, ensuring it breaches the clients default maxFrameSize and will be in multiple chunks. There is still an element of variability where it could work though, depending on the other properties etc in the message shifting the relative position of the body content such that a character isn't split across different chunks. Robbie On Thu, 26 Nov 2020 at 17:13, akabhishek1 <mailbox.abhishek.ku...@gmail.com> wrote: > > Hi Robbie, > > We are trying to replicate this issue from our end, so we can confidently > apply fix and roll out to all. We need some your input/help to replicate > this issue. It could be be some issue from my end, So I am trying to find > trouble code as well. Sorry for many questions. > > In May, we had this issue due to bad Message which was published with .Net, > but now we got issue due to normal messages as well. > > We are having this issue while restarting of ServiceBus connector restart > (due to ServiceBus connectivity loss). I follow below steps for restart - > ------------------------------------------- > 1. Closing and Nullifying JMS Objects > > a. Connection.close(), > Connection = null, > > b. MessageConsumer.setMessageListener(null); --> I think i am doing wrong > here. I don't need to set null here, as i am closing on next step. What's > your opinion? > MessageConsumer.close(); > MessageConsumer=null; > > c. Session.close(); > Session = null > ------------------------------------------ > 2. Creating JMS Objects > a. Create Connection > b. Create Session > c. Create Consumer > ------------------------------------------ > Q1. ) Could you please suggest if I am doing anything wrong while restarting > of JMS Objects ? > > Q2. ) If we got any exception while closing Connection, Session, Consumer > and Producer, then how can we make sure that Connection, Session, Consumer > and Producer closed and clear successfully from system? > Is it good practise set Null to JMS Objects after closing to make sure that > JMS Objects are cleaned up properly? > I want to make sure that JMS object and threads are clear from system > properly before creating new connection, so it will not cause any issue > while starting of JMS connection, Session, etc. > Overall I feel that, this could be an issue with clearance/disposing of > existing JMS Objects. > > Could you please suggest your opinion to handle gracefully for exception > scenario while closing JMS Objects? > > > Q3. ) We got "Illegal invocation from MessageListener callback" error while > closing Connection & Session, and then we started having "Suspected > deadlocked threads" error. > > Could you please suggest any idea to generate "Illegal invocation from > MessageListener callback" error ? It could be one of reason for deadlocked. > If I am able to generate this error then I could generate deadlock error. > > It will be great help if you provide your opinion on above queries. > > Please find below error logs. > > Error Pattern 1 - > "Failed to close connection - 'Illegal invocation from MessageListener > callback'" > "Stopping: 'null'. Object is: ServicebusMessageReceiver" > "Failed to reset message listener on endpoint > 'servicebus://topic:***/subscriptions/***' - 'The MessageConsumer was closed > due to an unrecoverable error.' - 'class javax.jms.IllegalStateException'" > "Suspected deadlocked threads: JmsSession > [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:10] delivery dispatcher > Id=41620240 is in BLOCKED on > lock=org.mule.transport.servicebus.SbusConnector@58e4b68c is owned by > JmsSession [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:7] delivery dispatcher > Id=51620937 | JmsSession [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:7] > delivery dispatcher Id=51620937 is in WAITING on > lock=java.util.concurrent.locks.ReentrantLock$NonfairSync@26cc0a98 is > owned by JmsSession [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:5] delivery > dispatcher Id=41620235 | JmsSession > [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:5] delivery dispatcher > Id=41620235 is in BLOCKED on > lock=org.mule.transport.servicebus.SbusConnector@58e4b68c is owned by > JmsSession [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:7] delivery dispatcher > Id=51620937 | " > "Suspected deadlocked threads: JmsSession > [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:10] delivery dispatcher > Id=41620240 is in BLOCKED on > lock=org.mule.transport.servicebus.SbusConnector@58e4b68c is owned by > JmsSession [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:7] delivery dispatcher > Id=51620937 | JmsSession [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:7] > delivery dispatcher Id=51620937 is in WAITING on > lock=java.util.concurrent.locks.ReentrantLock$NonfairSync@26cc0a98 is > owned by JmsSession [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:5] delivery > dispatcher Id=41620235 | JmsSession > [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:5] delivery dispatcher > Id=41620235 is in BLOCKED on > lock=org.mule.transport.servicebus.SbusConnector@58e4b68c is owned by > JmsSession [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:7] delivery dispatcher > Id=51620937 | " > "Suspected deadlocked threads: JmsSession > [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:10] delivery dispatcher > Id=41620240 is in BLOCKED on > lock=org.mule.transport.servicebus.SbusConnector@58e4b68c is owned by > JmsSession [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:7] delivery dispatcher > Id=51620937 | JmsSession [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:7] > delivery dispatcher Id=51620937 is in WAITING on > lock=java.util.concurrent.locks.ReentrantLock$NonfairSync@26cc0a98 is > owned by JmsSession [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:5] delivery > dispatcher Id=41620235 | JmsSession > [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:5] delivery dispatcher > Id=41620235 is in BLOCKED on > lock=org.mule.transport.servicebus.SbusConnector@58e4b68c is owned by > JmsSession [ID:ge300959-71f8-8fff-9d61-0a3e6dc02b98:9:7] delivery dispatcher > Id=51620937 | " > > Error Sample 2 > "Failed to close connection - 'Illegal invocation from MessageListener > callback'" > "Failed to close session - 'Illegal invocation from MessageListener > callback'" > > Regards, > Abhishek Kumar > > > > -- > Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
