> ...the listener never logs the 'onMessage() received' message that should be called for us to try to put the msg on a remote queue.
The consumer-count, message-count, and delivering-count would be useful metrics to gather from the relevant queue. Perhaps the message was dispatched to a consumer that was already stuck for some reason. It's impossible to say what is happening at this point. > Are there any tools that allow me to 'browse the queues' and see the content and headers for messages on the queues. You can use a JMS QueueBrowser [1] to look at messages on the queue which have not already been dispatched to consumers (i.e. messages which are "in delivery"). Depending on your operating environment you may not even have any messages that fit those requirements. There are also several management operations you could invoke to list the messages on the queue including messages which are in delivery, but I'm not sure those are exposed by WildFly. To be clear, WildFly has its own "subsystem" to configure and manage the broker and it's not the same as it would be if the broker was running standalone. > Are there components to ActiveMQ such that we could turn on some DEBUG logging around say the broker or whatever handles message listeners but not for the entire product? If you're using ActiveMQ Artemis embedded in WildFly then it will control what gets logged. There are many distinct logging categories in the broker for the various components (e.g. org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl). However, I'm not sure about the distinction you're making between the broker and the "entire product." Can you clarify that? Justin [1] https://docs.oracle.com/javaee/7/api/javax/jms/QueueBrowser.html On Tue, Sep 19, 2023 at 12:51 PM Miller, Michael < michael.mill...@flooranddecor.com> wrote: > Thanks again Justin. > > I am working on getting access to the Server and the WildFly console to > see some of the ActiveMQ stats. We have a large amount of logging in our > code to help track if the message in case of errors and also step-by-step > 'got here' type of logging to give us a pretty good idea of where we are in > the process. I may have mentioned this previously, but we have messages > right after we put the message on the local queue, but the listener never > logs the 'onMessage() received' message that should be called for us to try > to put the msg on a remote queue. > > Once I get access to Central, hopefully I can use the WildFly console to > check on the queue stats. Are there any tools that allow me to 'browse the > queues' and see the content and headers for messages on the queues. I > seem to remember years ago using QueueBrowser? but haven't spent much time > looking at that avenue yet. Someone with access to Central did 'show me' > one of the queues in the WildFly console but I believe the only data was a > couple properties, not enough to be very helpful. > > Our code has its own DLQ and process that just spins looking for new > messages on our DLQ and also tries to put the msg on the remote queue. > > Are there components to ActiveMQ such that we could turn on some DEBUG > logging around say the broker or whatever handles message listeners but not > for the entire product? I say this because we have almost 300 stores and > each store generally receives around 5-10 files per night. > > ________________________________ > From: Justin Bertram <jbert...@apache.org> > Sent: Tuesday, September 19, 2023 11:42 AM > To: users@activemq.apache.org <users@activemq.apache.org> > Subject: Re: [EXTERNAL] Re: Sporadic messageListener > > Based on the code you provided the application is sending a persistent > message which means it will wait for a response from the broker confirming > that it has received the message. This rules out the possibility that a > failure to send the message > ZjQcmQRYFpfptBannerStart > This Message is From an Unknown or Untrusted Sender > You have not recently corresponded with this external sender. Do not click > links, download attachments or share sensitive information unless you > recognize the sender and know the content is safe. If this email looks > suspicious, please report via “Report Suspicious” button. > < > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/D60pOmjYakNXXA!oq8qymrG5Kcjs1oATfO6ddEiKlkIQEHpwUtyk2u-CTAWLTHO75lU91SAPhAURoHcHZ5X92CVj08cp6pNQlcpCouq3EcGQpZ0LJjyZ_I0OqfDy2GYDlbpRa2KYGuHwpJi52pvCw$ > > > Report Suspicious > > ZjQcmQRYFpfptBannerEnd > > Based on the code you provided the application is sending a persistent > message which means it will wait for a response from the broker confirming > that it has received the message. This rules out the possibility that a > failure to send the message is being ignored. However, once the message > arrives at the broker a number of things can still go wrong. The consumer > may fail to consume it for some reason resulting either in a message that's > stuck "in delivery" or the message may be sent to the dead-letter address. > In the latter case there must actually be a dead-letter address configured > and there must be a queue on that address to receive the message otherwise > the message will just be dropped with a corresponding WARN message in the > log. A message could also expire after sitting in the queue for too long at > which point it may be sent to an expiry address or simply dropped (e.g. if > there is no expiry address configured). In order to investigate these > possibilities you need to get access to the broker's metrics (and > potentially application logs) to track exactly what's going on at each step > in your process. > > There is, of course, always the chance you're hitting a bug. As mentioned > previously, there have been thousands of fixes in the years since the > version you're using was released. If you exhaust all other possible > explanations perhaps you can work with your software vendor to upgrade and > see if that resolves the issue. > > In any event, the ActiveMQ Artemis User Manual [1] has a wealth of > information about how the broker behaves e.g. with regard to undeliverable > messages, expired messages, etc. If you need any clarification don't > hesitate to ask. > > > Justin > > [1] > https://urldefense.com/v3/__https://activemq.apache.org/components/artemis/documentation/2.6.0/__;!!D60pOmjYakNXXA!LCO--1drFmPs-boQpi9H64aSEdj-A318i3ybc5dEc84t73h87Co3ksemCWEEEchHr62xPslTFlAsk68BJBGjMBOni6tI$ > > On Mon, Sep 18, 2023 at 4:57 PM Miller, Michael < > michael.mill...@flooranddecor.com> wrote: > > > Correct about the code but we do have a copy of the code. > > > > I haven't received any stats but I will need to check with someone else > > and I don't have login access to our Central Server and don't have access > > to the Wildfly console to get any numbers. Problem is - this doesn't > > happen all the time - just sporadically. I also noticed that it does > > look like we have an ExceptionListener and haven't seen any of that > classes > > log entries. > > > > > > > > Here's an excerpt: so we iterate across a list of files and call our > > SendToQueue class to send each file to our local queue. > > > > > > sender = new SendToQueue(data.source.getConnectionFactory(), > > data.source.getTopicJNDIName()); > > > > for (String manifestName : data.filesToExportList.keySet()) > > { > > for (File xmlFile : data.filesToExportList.get(manifestName)) > > { > > //Set up a zip output stream for this file. > > BytesMessage message = sender.preBuildMessageFromFile(xmlFile); > > > > //tell message listeners which remote location to send messages. > > message.setStringProperty(JMSConstants.STORE_NUMBER, > > data.source.getLocationNumber()); > > > > sender.sendMessageToQueue(message); > > > > } > > > > //if all the files get sent, record this manifest as a success > > data.successManifests.add(manifestName); > > } > > > > SendToQueue.java - which does the JMS sending > > ------------------------------- > > //imports and package removed... > > > > public class SendToQueue > > { > > private Context context; > > private QueueConnectionFactory queueFactory; > > private QueueConnection connection; > > private QueueSession session; > > private QueueSender sender; > > private String jndiName; > > private static final String queueDest="Queue_Destination"; > > > > public SendToQueue(String connectionFactory, String queueJNDIName) throws > > Exception > > { > > Properties properties = EJBProperties.getInstance().getEJBProperties(); > > context = new InitialContext(properties); > > queueFactory = (QueueConnectionFactory) > context.lookup(connectionFactory); > > try { > > connection = > > queueFactory.createQueueConnection(JMSUtil.JMS_TOPIC_PRINCIPAL, > > JMSUtil.JMS_TOPIC_CREDENTIALS); > > connection.setExceptionListener(new PPOSJMSExceptionListener()); > > session = connection.createQueueSession(false, > > Session.CLIENT_ACKNOWLEDGE); > > jndiName=queueJNDIName; > > Queue queue = (Queue) context.lookup(queueJNDIName); > > sender = session.createSender(queue); > > } catch(JMSException jmse) { > > cleanup(); > > throw jmse; > > } > > } > > > > /** > > * Build a bytes message from a file and return it to the caller. > > * > > * It is done in this way so that properties may be set on the message > > * besides the contents before the message is sent. > > * @param file - the file to compress and stream to a message. > > * @return the BytesMessage containing the file. > > * @throws Exception > > */ > > public BytesMessage preBuildMessageFromFile(File file) throws > JMSException > > { > > try > > { > > BytesMessage message = session.createBytesMessage(); > > message = JMSFileTransformer.fileToBytesMessage(file, message); > > return message; > > } > > catch (Exception e) > > { > > cleanup(); > > JMSException ex = new JMSException(e.toString()); > > ex.initCause(e); > > Log.getInstance().error("Error on File: " + ((file == null) ? "null" : > > file.getName()), ex); > > throw ex; > > } > > > > } > > > > public BytesMessage createMessage() throws JMSException > > { > > return session.createBytesMessage(); > > } > > > > public BytesMessage testBuildMessageFromFile(File file) throws > JMSException > > { > > try > > { > > BytesMessage message = session.createBytesMessage(); > > // message = JMSFileTransformer.fileToBytesMessage(file, message); > > return message; > > } > > catch (Exception e) > > { > > cleanup(); > > JMSException ex = new JMSException(e.toString()); > > ex.initCause(e); > > Log.getInstance().error("Error on File: " + ((file == null) ? "null" : > > file.getName()), ex); > > throw ex; > > } > > > > } > > /** > > * Build an object message from a serializable object and return it to > the > > caller. > > * > > * It is done in this way so that properties may be set on the message > > * besides the contents before the message is sent. > > * @param object - The object to include in the message > > * @return the ObjectMessage containing the object. > > * @throws Exception > > */ > > public ObjectMessage preBuildMessageFromObject(Serializable object) > throws > > JMSException > > { > > try > > { > > ObjectMessage message = session.createObjectMessage(); > > message.setObject(object); > > return message; > > } > > catch (JMSException e) > > { > > cleanup(); > > Log.getInstance().error(e); > > throw e; > > } > > > > } > > > > /** > > * Build an object message from a serializable object and return it to > the > > caller. > > * > > * It is done in this way so that properties may be set on the message > > * besides the contents before the message is sent. > > * @param string - The string to include in the message > > * @return the TextMessage containing the object. > > * @throws Exception > > */ > > public TextMessage preBuildMessageFromString(String string) throws > > JMSException > > { > > try > > { > > TextMessage message = session.createTextMessage(string); > > return message; > > } > > catch (JMSException e) > > { > > cleanup(); > > Log.getInstance().error(e); > > throw e; > > } > > > > } > > > > > > /** > > * This method is called once a message has been created by the > > * public BytesMessage preBuildMessageFromFile(File file) method > > * and properties have been set on the message as desired. > > * > > * @param message - the BytesMessage being sent. > > * @throws Exception any and all messages up to the calling code. > > */ > > public void sendMessageToQueue(Message message) throws Exception > > { > > try > > { > > send(message); > > } > > catch (Exception e) > > { > > Log.getInstance().error(e); > > throw new JMSException(e.toString()); > > } > > > > } > > > > /** > > * This method is used to put Portfolio Beans on a JMS Queue for > > asynchronous transfer between servers > > * Along with some properties. > > * > > * @param object - the serializable object being sent. > > * @param list - string properties to be attached to message. > > * @return the message that was sent. > > * @throws Exception any and all messages up to the calling code. > > */ > > public Message sendObjectToQueue(Serializable object, HashMap > > hashMapStringProps) throws JMSException > > { > > try > > { > > ObjectMessage message = session.createObjectMessage(object); > > > > if (hashMapStringProps != null) > > { > > for (Iterator iter = > > hashMapStringProps.entrySet().iterator();iter.hasNext();) > > { > > Map.Entry entry = (Map.Entry) iter.next(); > > message.setStringProperty((String)entry.getKey(), > > (String)entry.getValue()); > > } > > } > > send(message); > > return message; > > } > > catch (JMSException e) > > { > > cleanup(); > > Log.getInstance().error(e); > > throw e; > > } > > > > } > > > > /* > > Common code between the above send methods. > > */ > > private void send(Message message) throws JMSException > > { > > message.setStringProperty(queueDest, jndiName); > > message.setLongProperty(JMSConstants.PROPERTY_MSG_ORIGINAL_CREATE_MILLIS, > > System.currentTimeMillis()); > > > > // SAS01032022 - Add code to log message properties ( this is only for > > debug purposes) > > //FDMessageUtil.logMessageProperties(message, "SENDING TO QUEUE", ", "); > > -- commented out for POS-13673 > > > > try { > > sender.send(message); > > //POS-13673 log AFTER the send call to ensure message processed ( this is > > only for debug purposes) > > FDMessageUtil.logMessageProperties(message, "SENT TO QUEUE", ", "); > > } catch(JMSException e) { > > cleanup(); > > throw e; > > } > > } > > > > /** > > * Cleanup - closes the connection and session, if exists > > */ > > public void cleanup() > > { > > if(sender != null){ > > try { > > sender.close(); > > }catch(JMSException e) { > > Log.getInstance().error("Exception caught while closing Queue sender: > "+e); > > } > > finally > > { > > sender = null; > > } > > } > > if (connection != null) > > { > > try { > > connection.close(); > > } catch(JMSException e) { > > Log.getInstance().error("Exception caught while closing JMS connection: > > "+e); > > } finally { > > connection = null; > > } > > } > > > > if (queueFactory != null) > > { > > queueFactory = null; > > } > > > > if (session != null) > > { > > try { > > session.close(); > > } catch(JMSException e) { > > Log.getInstance().error("Exception caught while closing JMS session: > "+e); > > } finally { > > session = null; > > } > > } > > > > if (context != null) > > { > > try > > { > > context.close(); > > } > > catch (NamingException e) > > { > > Log.getInstance().error("Exception caught while closing context: " + e); > > } > > finally > > { > > context = null; > > } > > } > > } > > } > > > > > > ________________________________ > > From: Justin Bertram <jbert...@apache.org> > > Sent: Monday, September 18, 2023 3:26 PM > > To: users@activemq.apache.org <users@activemq.apache.org> > > Subject: Re: [EXTERNAL] Re: Sporadic messageListener > > > > So the software sending and consuming the messages is not something you > > (or your company) has written but instead is something from a vendor? If > > so, do you have the source-code for this product? Unfortunately a list of > > libraries starting with > > ZjQcmQRYFpfptBannerStart > > This Message is From an Unknown or Untrusted Sender > > You have not recently corresponded with this external sender. Do not > click > > links, download attachments or share sensitive information unless you > > recognize the sender and know the content is safe. If this email looks > > suspicious, please report via “Report Suspicious” button. > > < > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/D60pOmjYakNXXA!oq8qymrG5KcjsxzI5vQQnwE7dlzSJjrHenGFQf-8KnLrzqwDldYdXJL_qtyUthWJNOG9QLNXuNZQku88u9DgpklBMotSLODEjE7l_H0UiZiL_HMP4rUoy0SlE1GF2E3jjQkIxw$ > > > > > Report Suspicious > > > > ZjQcmQRYFpfptBannerEnd > > > > So the software sending and consuming the messages is not something you > (or > > your company) has written but instead is something from a vendor? If so, > do > > you have the source-code for this product? > > > > Unfortunately a list of libraries starting with "activemq" doesn't > clarify > > what the software is actually using. > > > > Regarding upgrades...This seems like a false dichotomy. You can't really > > get new features without upgrading. There's been over 30 releases of > > ActiveMQ Artemis since 2.6.3 which have included 4,700+ commits to the > > code-base. Those commits include bug fixes and features. > > > > Couple more questions: > > > > - How are you sending the messages? Can you paste the code from the > > producer? Also, what library are you using to send the messages? > > - Have you gathered any statistics from the queue where you're sending > > messages (e.g. Consumer Count, Message Count, Delivering Count, etc.)? If > > so, what did they show? If not, could you gather these before and after > you > > see the problem? > > > > > > Justin > > > > On Mon, Sep 18, 2023 at 2:34 PM Miller, Michael < > > michael.mill...@flooranddecor.com> wrote: > > > > > Thanks for the quick response. I started working on this project > just a > > > couple months ago, so not completely sure I will be able to answer your > > > questions correctly, but here goes: > > > We have 1 Central Server and X number of Store servers - all running > > > WildFly as the app server. I believe Artemis is the version embedded > in > > > WildFly 16.0.0.Final. The message listener is not an MDB, but a 'task' > > > that implements MessageListener interface. Not sure how the > > activemq-core > > > fits in - I just threw that in just in case it mattered - again these > are > > > libraries I found in our Libraries folder. I just searched our > \Libraries > > > folder where we store all 3rd party software and supplied any that > > started > > > with activemq. 🙂 > > > > > > We (including me) are the retailer and the software comes from our > > > vendor, and I have heard that retailers are never in a hurry to upgrade > > > their software, but prefer new features instead. Not much chance > right > > > now that we would be able to update any packages. > > > > > > > > > > > > ________________________________ > > > From: Justin Bertram <jbert...@apache.org> > > > Sent: Monday, September 18, 2023 1:55 PM > > > To: users@activemq.apache.org <users@activemq.apache.org> > > > Subject: [EXTERNAL] Re: Sporadic messageListener > > > > > > Can you elaborate on your application components? What are you using > as a > > > broker? Is it ActiveMQ Artemis 2. 6. 3 embedded in WildFly 16. 0. 0. > > Final? > > > Are your MessageListener implementations running in WildFly (e. g. as > > MDBs) > > > or elsewhere? How does > > > ZjQcmQRYFpfptBannerStart > > > This Message is From an Unknown or Untrusted Sender > > > You have not recently corresponded with this external sender. Do not > > click > > > links, download attachments or share sensitive information unless you > > > recognize the sender and know the content is safe. If this email looks > > > suspicious, please report via “Report Suspicious” button. > > > < > > > > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/D60pOmjYakNXXA!oq8qymrG5Kcjsx0gp5MxPXHKrr6gJs4f2iGQDt7dMh9X7nXrS00QtnLjXq2Yy58U-Uf0zyDwjUHiC4or-Y1ua0EAyhgXgDQghJWlnvqNuoXnUjX9mCSizI4o16qryEQGGh40zg$ > > > > > > > Report Suspicious > > > > > > ZjQcmQRYFpfptBannerEnd > > > > > > Can you elaborate on your application components? What are you using > as a > > > broker? Is it ActiveMQ Artemis 2.6.3 embedded in WildFly 16.0.0.Final? > > Are > > > your MessageListener implementations running in WildFly (e.g. as MDBs) > or > > > elsewhere? How does "ActiveMQ core 5.7.0" fit in and what exactly do > you > > > mean by "core" in this context? > > > > > > Also, these components are pretty old. Both WildFly 16.0.0.Final and > > > ActiveMQ Artemis 2.6.3 were released in early 2019 and ActiveMQ 5.7.0 > was > > > released in late 2012. Has there been any thought given to upgrading > > these > > > components? > > > > > > > > > Justin > > > > > > On Mon, Sep 18, 2023 at 1:40 PM Miller, Michael < > > > michael.mill...@flooranddecor.com> wrote: > > > > > > > We are encountering a problem where a small number of messages are > not > > > > triggering the messageListener. We have a shared folder where our > host > > > > drops files to be delivered to all our retail stores. We have a > process > > > > that has to collect a full set of files as a set and when the set is > > > found, > > > > we put each file on a local queue on a centralized server. The > > > > messageListener gets called and we attempt to put the new message on > a > > > > remote queue for retail store. > > > > > > > > Generally this works fairly well, but we have started encountering > > > > problems where, according to our logging, we can see a log message > that > > > the > > > > (file)message was put in the local queue, but in some small > percentage > > of > > > > the cases, we never get the onMessage() for the message listener to > put > > > the > > > > message on the remote store, which breaks our process. > > > > > > > > Looking for help/suggestions on how to debug this process to make > sure > > > > that our message listener gets called. Is there a way to request > > activemq > > > > debug logging without debugging logging that entire application? > > > Happening > > > > in production but not in QA or local dev... > > > > > > > > We are running WildFly 16.0.0.Final, with ActiveMQ core 5.7.0 and > > > ActiveMQ > > > > Artemis 2.6.3. > > > > > > > > We have added extra logging around the our executor service and the > > > > message listener but some files/messages seem to either 1) NOT make > it > > > into > > > > the local queue or 2) the message listener doesn't fire. Based on the > > > > logging, it seems like the message gets to the local queue, as we > found > > > no > > > > exceptions around that timeframe, so seems to be the listener. > > > > > > > > > > > > > > > > Mike Miller > > > > > > > > Senior Developer II > > > > > > > > Floor & Decor > > > > > > > > 214-477-9953 > > > > > > > > mailto: michael.mill...@flooranddecor.com > > > > > > > > This email and any files transmitted with it are the property of > Floor > > & > > > > Decor, are confidential, and are intended solely for the use of the > > > > individual or entity to which this email is addressed. If you are not > > one > > > > of the named recipient(s) or otherwise have reason to believe that > you > > > have > > > > received this message in error, please notify the sender at > > 214.477.9953 > > > > and delete this message immediately from your computer. Any other > use, > > > > retention, dissemination, forwarding, printing, or copying of this > > email > > > is > > > > strictly prohibited. > > > > > > > > Please consider the environment before printing this email. > > > > > > > > > > > > > > > >
