this is the AMQP_Spout I was using https://github.com/davidpr/storm-rabbitmq/tree/master/storm-amqp-spout-regression
and this is the Storm topology https://github.com/davidpr/mongo-storm-fast (watch out the line QueueDeclaration qd = new SharedQueueWithBinding("stormqueue", "stormexchange", "stormkey", 30000, 60000, 100); in the MongoTopology.java; I was trying to use Xorlev AMQP_Spout which has the last 3 additional parameters, they don't match with the spout I have posted here) 2014-02-23 19:34 GMT+01:00 Sean Allen <[email protected]>: > Prefetch is how many messages each spout takes from rabbitmq at a time. > > Having it higher than max spout pending can cause undesired behavior. > > Is your code available anywhere online? > > > On Sun, Feb 23, 2014 at 12:42 PM, hyakunin <[email protected]> wrote: > >> I had this constant in the AMQP_Spout, I guess it's something related >> with the outstanding non-ack tuples >> >> private static final long DEFAULT_PREFETCH_COUNT = 100; >> >> when increased this limit I now have the limit in the maxSpoutPending as >> Sean said >> >> For my needs I'll use rabbitmq and storm without any ack or auto-ack... >> >> >> >> >> >> 2014-02-22 20:59 GMT+01:00 Sean Allen <[email protected]>: >> >> are you using maxSpoutPending? >>> >>> >>> >>> On Thu, Feb 20, 2014 at 3:11 PM, hyakunin <[email protected]> wrote: >>> >>>> Hi, I have a Storm Topology composed of a AMPQ_Spout and a bolt capable >>>> of writing to a MongoDB Collection >>>> >>>> The problem is that once I run the Java Rabbitmq program that sends >>>> tuples to Rabbitmq queue; the Storm topology pulls the Rabbitmq taking 100 >>>> tuples and waiting for some minutes before it pulls another bunch of 100 >>>> tuples. >>>> >>>> I have revised all the code and simplified the system as much I could >>>> but I still have the same problem >>>> >>>> does anybody have any clue of what can be happening or at least how to >>>> detect where's this weird bottleneck? >>>> >>>> thanks >>>> David >>>> >>> >>> >>> >>> -- >>> >>> Ce n'est pas une signature >>> >> >> > > > -- > > Ce n'est pas une signature >
