Based on the code you provided the application is sending a persistent message which means it will wait for a response from the broker confirming that it has received the message. This rules out the possibility that a failure to send the message is being ignored. However, once the message arrives at the broker a number of things can still go wrong. The consumer may fail to consume it for some reason resulting either in a message that's stuck "in delivery" or the message may be sent to the dead-letter address. In the latter case there must actually be a dead-letter address configured and there must be a queue on that address to receive the message otherwise the message will just be dropped with a corresponding WARN message in the log. A message could also expire after sitting in the queue for too long at which point it may be sent to an expiry address or simply dropped (e.g. if there is no expiry address configured). In order to investigate these possibilities you need to get access to the broker's metrics (and potentially application logs) to track exactly what's going on at each step in your process.
There is, of course, always the chance you're hitting a bug. As mentioned previously, there have been thousands of fixes in the years since the version you're using was released. If you exhaust all other possible explanations perhaps you can work with your software vendor to upgrade and see if that resolves the issue. In any event, the ActiveMQ Artemis User Manual [1] has a wealth of information about how the broker behaves e.g. with regard to undeliverable messages, expired messages, etc. If you need any clarification don't hesitate to ask. Justin [1] https://activemq.apache.org/components/artemis/documentation/2.6.0/ On Mon, Sep 18, 2023 at 4:57 PM Miller, Michael < michael.mill...@flooranddecor.com> wrote: > Correct about the code but we do have a copy of the code. > > I haven't received any stats but I will need to check with someone else > and I don't have login access to our Central Server and don't have access > to the Wildfly console to get any numbers. Problem is - this doesn't > happen all the time - just sporadically. I also noticed that it does > look like we have an ExceptionListener and haven't seen any of that classes > log entries. > > > > Here's an excerpt: so we iterate across a list of files and call our > SendToQueue class to send each file to our local queue. > > > sender = new SendToQueue(data.source.getConnectionFactory(), > data.source.getTopicJNDIName()); > > for (String manifestName : data.filesToExportList.keySet()) > { > for (File xmlFile : data.filesToExportList.get(manifestName)) > { > //Set up a zip output stream for this file. > BytesMessage message = sender.preBuildMessageFromFile(xmlFile); > > //tell message listeners which remote location to send messages. > message.setStringProperty(JMSConstants.STORE_NUMBER, > data.source.getLocationNumber()); > > sender.sendMessageToQueue(message); > > } > > //if all the files get sent, record this manifest as a success > data.successManifests.add(manifestName); > } > > SendToQueue.java - which does the JMS sending > ------------------------------- > //imports and package removed... > > public class SendToQueue > { > private Context context; > private QueueConnectionFactory queueFactory; > private QueueConnection connection; > private QueueSession session; > private QueueSender sender; > private String jndiName; > private static final String queueDest="Queue_Destination"; > > public SendToQueue(String connectionFactory, String queueJNDIName) throws > Exception > { > Properties properties = EJBProperties.getInstance().getEJBProperties(); > context = new InitialContext(properties); > queueFactory = (QueueConnectionFactory) context.lookup(connectionFactory); > try { > connection = > queueFactory.createQueueConnection(JMSUtil.JMS_TOPIC_PRINCIPAL, > JMSUtil.JMS_TOPIC_CREDENTIALS); > connection.setExceptionListener(new PPOSJMSExceptionListener()); > session = connection.createQueueSession(false, > Session.CLIENT_ACKNOWLEDGE); > jndiName=queueJNDIName; > Queue queue = (Queue) context.lookup(queueJNDIName); > sender = session.createSender(queue); > } catch(JMSException jmse) { > cleanup(); > throw jmse; > } > } > > /** > * Build a bytes message from a file and return it to the caller. > * > * It is done in this way so that properties may be set on the message > * besides the contents before the message is sent. > * @param file - the file to compress and stream to a message. > * @return the BytesMessage containing the file. > * @throws Exception > */ > public BytesMessage preBuildMessageFromFile(File file) throws JMSException > { > try > { > BytesMessage message = session.createBytesMessage(); > message = JMSFileTransformer.fileToBytesMessage(file, message); > return message; > } > catch (Exception e) > { > cleanup(); > JMSException ex = new JMSException(e.toString()); > ex.initCause(e); > Log.getInstance().error("Error on File: " + ((file == null) ? "null" : > file.getName()), ex); > throw ex; > } > > } > > public BytesMessage createMessage() throws JMSException > { > return session.createBytesMessage(); > } > > public BytesMessage testBuildMessageFromFile(File file) throws JMSException > { > try > { > BytesMessage message = session.createBytesMessage(); > // message = JMSFileTransformer.fileToBytesMessage(file, message); > return message; > } > catch (Exception e) > { > cleanup(); > JMSException ex = new JMSException(e.toString()); > ex.initCause(e); > Log.getInstance().error("Error on File: " + ((file == null) ? "null" : > file.getName()), ex); > throw ex; > } > > } > /** > * Build an object message from a serializable object and return it to the > caller. > * > * It is done in this way so that properties may be set on the message > * besides the contents before the message is sent. > * @param object - The object to include in the message > * @return the ObjectMessage containing the object. > * @throws Exception > */ > public ObjectMessage preBuildMessageFromObject(Serializable object) throws > JMSException > { > try > { > ObjectMessage message = session.createObjectMessage(); > message.setObject(object); > return message; > } > catch (JMSException e) > { > cleanup(); > Log.getInstance().error(e); > throw e; > } > > } > > /** > * Build an object message from a serializable object and return it to the > caller. > * > * It is done in this way so that properties may be set on the message > * besides the contents before the message is sent. > * @param string - The string to include in the message > * @return the TextMessage containing the object. > * @throws Exception > */ > public TextMessage preBuildMessageFromString(String string) throws > JMSException > { > try > { > TextMessage message = session.createTextMessage(string); > return message; > } > catch (JMSException e) > { > cleanup(); > Log.getInstance().error(e); > throw e; > } > > } > > > /** > * This method is called once a message has been created by the > * public BytesMessage preBuildMessageFromFile(File file) method > * and properties have been set on the message as desired. > * > * @param message - the BytesMessage being sent. > * @throws Exception any and all messages up to the calling code. > */ > public void sendMessageToQueue(Message message) throws Exception > { > try > { > send(message); > } > catch (Exception e) > { > Log.getInstance().error(e); > throw new JMSException(e.toString()); > } > > } > > /** > * This method is used to put Portfolio Beans on a JMS Queue for > asynchronous transfer between servers > * Along with some properties. > * > * @param object - the serializable object being sent. > * @param list - string properties to be attached to message. > * @return the message that was sent. > * @throws Exception any and all messages up to the calling code. > */ > public Message sendObjectToQueue(Serializable object, HashMap > hashMapStringProps) throws JMSException > { > try > { > ObjectMessage message = session.createObjectMessage(object); > > if (hashMapStringProps != null) > { > for (Iterator iter = > hashMapStringProps.entrySet().iterator();iter.hasNext();) > { > Map.Entry entry = (Map.Entry) iter.next(); > message.setStringProperty((String)entry.getKey(), > (String)entry.getValue()); > } > } > send(message); > return message; > } > catch (JMSException e) > { > cleanup(); > Log.getInstance().error(e); > throw e; > } > > } > > /* > Common code between the above send methods. > */ > private void send(Message message) throws JMSException > { > message.setStringProperty(queueDest, jndiName); > message.setLongProperty(JMSConstants.PROPERTY_MSG_ORIGINAL_CREATE_MILLIS, > System.currentTimeMillis()); > > // SAS01032022 - Add code to log message properties ( this is only for > debug purposes) > //FDMessageUtil.logMessageProperties(message, "SENDING TO QUEUE", ", "); > -- commented out for POS-13673 > > try { > sender.send(message); > //POS-13673 log AFTER the send call to ensure message processed ( this is > only for debug purposes) > FDMessageUtil.logMessageProperties(message, "SENT TO QUEUE", ", "); > } catch(JMSException e) { > cleanup(); > throw e; > } > } > > /** > * Cleanup - closes the connection and session, if exists > */ > public void cleanup() > { > if(sender != null){ > try { > sender.close(); > }catch(JMSException e) { > Log.getInstance().error("Exception caught while closing Queue sender: "+e); > } > finally > { > sender = null; > } > } > if (connection != null) > { > try { > connection.close(); > } catch(JMSException e) { > Log.getInstance().error("Exception caught while closing JMS connection: > "+e); > } finally { > connection = null; > } > } > > if (queueFactory != null) > { > queueFactory = null; > } > > if (session != null) > { > try { > session.close(); > } catch(JMSException e) { > Log.getInstance().error("Exception caught while closing JMS session: "+e); > } finally { > session = null; > } > } > > if (context != null) > { > try > { > context.close(); > } > catch (NamingException e) > { > Log.getInstance().error("Exception caught while closing context: " + e); > } > finally > { > context = null; > } > } > } > } > > > ________________________________ > From: Justin Bertram <jbert...@apache.org> > Sent: Monday, September 18, 2023 3:26 PM > To: users@activemq.apache.org <users@activemq.apache.org> > Subject: Re: [EXTERNAL] Re: Sporadic messageListener > > So the software sending and consuming the messages is not something you > (or your company) has written but instead is something from a vendor? If > so, do you have the source-code for this product? Unfortunately a list of > libraries starting with > ZjQcmQRYFpfptBannerStart > This Message is From an Unknown or Untrusted Sender > You have not recently corresponded with this external sender. Do not click > links, download attachments or share sensitive information unless you > recognize the sender and know the content is safe. If this email looks > suspicious, please report via “Report Suspicious” button. > < > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/D60pOmjYakNXXA!oq8qymrG5KcjsxzI5vQQnwE7dlzSJjrHenGFQf-8KnLrzqwDldYdXJL_qtyUthWJNOG9QLNXuNZQku88u9DgpklBMotSLODEjE7l_H0UiZiL_HMP4rUoy0SlE1GF2E3jjQkIxw$ > > > Report Suspicious > > ZjQcmQRYFpfptBannerEnd > > So the software sending and consuming the messages is not something you (or > your company) has written but instead is something from a vendor? If so, do > you have the source-code for this product? > > Unfortunately a list of libraries starting with "activemq" doesn't clarify > what the software is actually using. > > Regarding upgrades...This seems like a false dichotomy. You can't really > get new features without upgrading. There's been over 30 releases of > ActiveMQ Artemis since 2.6.3 which have included 4,700+ commits to the > code-base. Those commits include bug fixes and features. > > Couple more questions: > > - How are you sending the messages? Can you paste the code from the > producer? Also, what library are you using to send the messages? > - Have you gathered any statistics from the queue where you're sending > messages (e.g. Consumer Count, Message Count, Delivering Count, etc.)? If > so, what did they show? If not, could you gather these before and after you > see the problem? > > > Justin > > On Mon, Sep 18, 2023 at 2:34 PM Miller, Michael < > michael.mill...@flooranddecor.com> wrote: > > > Thanks for the quick response. I started working on this project just a > > couple months ago, so not completely sure I will be able to answer your > > questions correctly, but here goes: > > We have 1 Central Server and X number of Store servers - all running > > WildFly as the app server. I believe Artemis is the version embedded in > > WildFly 16.0.0.Final. The message listener is not an MDB, but a 'task' > > that implements MessageListener interface. Not sure how the > activemq-core > > fits in - I just threw that in just in case it mattered - again these are > > libraries I found in our Libraries folder. I just searched our \Libraries > > folder where we store all 3rd party software and supplied any that > started > > with activemq. 🙂 > > > > We (including me) are the retailer and the software comes from our > > vendor, and I have heard that retailers are never in a hurry to upgrade > > their software, but prefer new features instead. Not much chance right > > now that we would be able to update any packages. > > > > > > > > ________________________________ > > From: Justin Bertram <jbert...@apache.org> > > Sent: Monday, September 18, 2023 1:55 PM > > To: users@activemq.apache.org <users@activemq.apache.org> > > Subject: [EXTERNAL] Re: Sporadic messageListener > > > > Can you elaborate on your application components? What are you using as a > > broker? Is it ActiveMQ Artemis 2. 6. 3 embedded in WildFly 16. 0. 0. > Final? > > Are your MessageListener implementations running in WildFly (e. g. as > MDBs) > > or elsewhere? How does > > ZjQcmQRYFpfptBannerStart > > This Message is From an Unknown or Untrusted Sender > > You have not recently corresponded with this external sender. Do not > click > > links, download attachments or share sensitive information unless you > > recognize the sender and know the content is safe. If this email looks > > suspicious, please report via “Report Suspicious” button. > > < > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/D60pOmjYakNXXA!oq8qymrG5Kcjsx0gp5MxPXHKrr6gJs4f2iGQDt7dMh9X7nXrS00QtnLjXq2Yy58U-Uf0zyDwjUHiC4or-Y1ua0EAyhgXgDQghJWlnvqNuoXnUjX9mCSizI4o16qryEQGGh40zg$ > > > > > Report Suspicious > > > > ZjQcmQRYFpfptBannerEnd > > > > Can you elaborate on your application components? What are you using as a > > broker? Is it ActiveMQ Artemis 2.6.3 embedded in WildFly 16.0.0.Final? > Are > > your MessageListener implementations running in WildFly (e.g. as MDBs) or > > elsewhere? How does "ActiveMQ core 5.7.0" fit in and what exactly do you > > mean by "core" in this context? > > > > Also, these components are pretty old. Both WildFly 16.0.0.Final and > > ActiveMQ Artemis 2.6.3 were released in early 2019 and ActiveMQ 5.7.0 was > > released in late 2012. Has there been any thought given to upgrading > these > > components? > > > > > > Justin > > > > On Mon, Sep 18, 2023 at 1:40 PM Miller, Michael < > > michael.mill...@flooranddecor.com> wrote: > > > > > We are encountering a problem where a small number of messages are not > > > triggering the messageListener. We have a shared folder where our host > > > drops files to be delivered to all our retail stores. We have a process > > > that has to collect a full set of files as a set and when the set is > > found, > > > we put each file on a local queue on a centralized server. The > > > messageListener gets called and we attempt to put the new message on a > > > remote queue for retail store. > > > > > > Generally this works fairly well, but we have started encountering > > > problems where, according to our logging, we can see a log message that > > the > > > (file)message was put in the local queue, but in some small percentage > of > > > the cases, we never get the onMessage() for the message listener to put > > the > > > message on the remote store, which breaks our process. > > > > > > Looking for help/suggestions on how to debug this process to make sure > > > that our message listener gets called. Is there a way to request > activemq > > > debug logging without debugging logging that entire application? > > Happening > > > in production but not in QA or local dev... > > > > > > We are running WildFly 16.0.0.Final, with ActiveMQ core 5.7.0 and > > ActiveMQ > > > Artemis 2.6.3. > > > > > > We have added extra logging around the our executor service and the > > > message listener but some files/messages seem to either 1) NOT make it > > into > > > the local queue or 2) the message listener doesn't fire. Based on the > > > logging, it seems like the message gets to the local queue, as we found > > no > > > exceptions around that timeframe, so seems to be the listener. > > > > > > > > > > > > Mike Miller > > > > > > Senior Developer II > > > > > > Floor & Decor > > > > > > 214-477-9953 > > > > > > mailto: michael.mill...@flooranddecor.com > > > > > > This email and any files transmitted with it are the property of Floor > & > > > Decor, are confidential, and are intended solely for the use of the > > > individual or entity to which this email is addressed. If you are not > one > > > of the named recipient(s) or otherwise have reason to believe that you > > have > > > received this message in error, please notify the sender at > 214.477.9953 > > > and delete this message immediately from your computer. Any other use, > > > retention, dissemination, forwarding, printing, or copying of this > email > > is > > > strictly prohibited. > > > > > > Please consider the environment before printing this email. > > > > > > > > >
