I have experienced a similar issue with AMQP on 0.9.3 and it was related to some netty issue which were solved on 0.9.4. We upgraded to that version and it fixed the issue. I would highly recommend using 0.9.4 (or 0.9.6 which is my latest version that I am using with AMQP). It has been stable ever since. ________________________________ From: Cody Lee <[email protected]> Sent: Tuesday, August 16, 2016 9:07:16 AM To: [email protected] Subject: Re: AMQP spout gets stuck
We're on 1.x here, but I might be able to help. Taking off dev@ Topology/Storm questions: 1. The executors have been healthy and haven't been erroring out? 2. An appropriate tuple timeout has been set for the topology? (This will fail a "stuck" tuple and keep things moving, but view it more as a bandaid and not solution. This triggering indicates that tuples are not getting acked or failed in time, or the setting is just too small). Code-Specific 1. Are you ensuring that you fully ack or fail every tuple in the tuple tree coming from this spout? It's easy to miss, especially under edge-cases (I've been guilty of this myself before). Also the collector is not thread safe, be sure to only ack or fail inside the bolt thread if you're doing anything async. 2. Are there dependency failures that are holding up processing? 3. Incoming messages are as expected, ie no unexpected message drift? 4. Are you using any windowing functions? If so, is it possible that some executor windows are able to fill up to a trigger point to execute? Cody From: Rahul Dahiya <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, August 16, 2016 at 6:52 AM To: "[email protected]" <[email protected]>, "[email protected]" <[email protected]> Subject: AMQP spout gets stuck Hi Team, We are using apache-storm v 0.9,3 to read messages from RMQ using AMQP spout and processing the messages to save in the database. Recently we have started facing an issue in the topology where the messages gets stuck in unacked stated in the RMQ and we see failures in the AMQP spout itself. To solve this we have to redeploy that specific topology . It works fine for sometime and process messages at a normal rate. After some time (not specific interval ) the messages gets stuck in unacked state in RMQ and we have to repeat the process all over again. Also, sometimes it runs flawlessly throughout the day and then out of the blue it starts experiencing the issue. We haven't done any code change in months and we tried running the topology in a separate cluster (RMQ and Storm) as well with no success . Also there are multiple topologies deployed on the storm cluster and most of them run perfectly fine (Although we have seen issues with some but they got resolved). We are using Amazon cloud for the cluster. Any help regarding this is highly appreciated. Thanks a ton in advance. Regards, Rahul ________________________________ NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
