It might be a good fail-safe, but hopefully you'll find something as you step through with the debugger that will highlight what the problem actually is (so you don't need to use the no-selector workaround). Workarounds are great, but fixes are better. :) But it might be a way to at least get production working again so you can solve this at a more reasonable pace rather than spending 15-hour days on it.
Tim On Fri, Apr 24, 2015 at 5:04 PM, Kevin Burton <bur...@spinn3r.com> wrote: > Sounds like a good idea. I just pushed with maxPageSize=5000 > > Interesting that my backlogged queues seems to be locked up at about 5000 > messages. > > … I’m wonder if those are messages that aren’t matching selectors? > > I could have a plan B / fall through that accepts messages without a > selector.. it would be a hack and would give me soft priority but this > would get me closer to my goal. > > throttle-queue:discovery-task:pending:192.0.78.13 > 5,004 4 > throttle-queue:discovery-task:pending:192.0.78.12 > 5,002 4 > throttle-queue:discovery-task:pending:74.125.141.132 > 5,000 0 > throttle-queue:feed-task:pending:74.125.141.132 > 4,997 0 > throttle-queue:source-task:pending:69.171.237.20 > 4,062 0 > > On Fri, Apr 24, 2015 at 3:40 PM, Tim Bain <tb...@alumni.duke.edu> wrote: > > > OK, so based on what you wrote, it sounds like the broker gets into a > state > > where message selectors don't work properly, since the same selector > > matches all messages (except ones with a null priority; have you > confirmed > > that both artemis_priority <= 9 and artemis_priority = null match their > > respective messages when you're in a good state, and that neither one > gets > > any messages when you're in a bad state?). > > > > I'd start the broker with the debug port enabled, pull up the > > selector-evaluation code and set a breakpoint in it (I'd start with > > SelectorWorker.run()), wait till you get into the bad state, and then > > attach a debugger and step through the code to see why messages aren't > > getting handed off to your consumer with the selector. > > > > Tim > > > > On Fri, Apr 24, 2015 at 4:27 PM, Kevin Burton <bur...@spinn3r.com> > wrote: > > > > > > > > > > > > http://imgur.com/a/2myja > > > > > > > > > > > > What are the two screenshots; with and without the selector? If > that's > > > > right, then clearly zero messages are matching the selector (which > > means > > > > this isn't a delivery problem, it's a selector problem), so hopefully > > > > pulling that thread will lead to a solution. > > > > > > > > > > > Actually, maybe it’s just a but related to message selectors? > > > > > > If I remove the selector, I have it print the messages it receives. I > > then > > > confirm that they are ALL have the right headers. So from a > mathematical > > > perspective, they should match. > > > > > > Here’s an example. > > > > > > QueueDumperTask: Got message: using config: Config{sync=true, > > > selector='artemis_priority <= 9'} > > > > > > QueueDumperTask: > > > ID:util0048.wdc.sl.spinn3r.com-38049-1429893012757-6:1:1:1:11179 > > > > > > QueueDumperTask: artemis.link= > http://entertainment-2-all.blogspot.com > > > > > > QueueDumperTask: artemis.throttleKey=74.125.141.132 > > > > > > QueueDumperTask: artemis.pending.timestamp=2015-04-24T16:46:30Z > > > > > > QueueDumperTask: artemis.sequence=1429893990000012414 > > > > > > QueueDumperTask: > > > > artemis.pendingQueue=throttle-queue:discovery-task:pending:74.125.141.132 > > > > > > QueueDumperTask: artemis_priority=0 > > > > > > QueueDumperTask: artemis.throttleQueue=discovery-task > > > > > > QueueDumperTask: artemis.delay=0 > > > > > > QueueDumperTask: JMSXDeliveryCount=3 > > > > > > .. so it should match > > > > > > artemis_priority <= 9 > > > > > > … so I think there are two issues here. I’m aware of the improper > > > scheduling issue do to maxPageSize… so I won’t get REAL priorities. > I’ll > > > get soft priorities. > > > > > > But that’s not the issue that I’m seeing.. What I’m seeing is that the > > > ENTIRE thing locks up. Because at least ONE of my selectors should > work. > > > > > > > > > > > If I’m already an in-memory store, is there any problem just > setting > > > > > maxPageSize to a HUGE number? > > > > > > > > > > > > That's not code I've looked at, but I'd double-check the code to make > > > sure > > > > it's not going to do anything crazy like pre-allocate a huge > > > > array/ArrayList or anything like that... > > > > > > > > > > Yes. That’s what I was worried about.. I might trace it or just try it > in > > > production and see what memory usage looks like. > > > > > > -- > > > > > > Founder/CEO Spinn3r.com > > > Location: *San Francisco, CA* > > > blog: http://burtonator.wordpress.com > > > … or check out my Google+ profile > > > <https://plus.google.com/102718274791889610666/posts> > > > <http://spinn3r.com> > > > > > > > > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > … or check out my Google+ profile > <https://plus.google.com/102718274791889610666/posts> > <http://spinn3r.com> >
