Re: [Catalyst] CatalystX::JobServer (anyone use it?)

[Bill Moseley]

On Tue, Jun 5, 2012 at 11:29 PM, Drew Taylor  wrote:

> On Tue, Jun 5, 2012 at 8:25 AM, Seth Daniel 
> wrote:
> > I found CatalystX::JobServer mentioned in a thread from about a year
> > ago.
>

This is a bit off topic, and maybe not a very useful question (as I asked
on RabbitMQ list w/o a response), but this thread was timely.   It's a
general question about messaging in web apps.   It's not really about any
specific messaging system.

How do you, or perhaps,  do you handle the possibility of message loss when
using messaging systems?

We try and "design for failure" assuming any part of our system can fail.
 So, that means an entire queue could be lost.  But, in some cases we don't
want[1] to lose any messages (which in this case are tasks to be
completed).   It's very possible we could have thousands of messages in a
queue any time.

As a simple example, in our web app we have a way for a user to generate a
report and have it emailed to them.  This happens in the background on a
separate server pool.  So, the web app queues a message to trigger this
task.

If the job is critical what I have done is track the state of the job with
a database table that has start time and a retry count.   Then I have cron
look for stale jobs and re-queue after some timeout and before retry count
exceeds some value.

The problem with this approach is 1) I don't really know if the message was
lost or if the queue is just backed up, and 2) it's quite a bit of overhead
and additional complexity having to also maintain state of the job.

Other than (or in addition to) using an HA setup like
RabbitMQ's is
there a design pattern you use to handle the situation of a message or
entire queue being lost?




[1] perhaps more accurately, we *think* we can't lose any messages.


-- 
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] CatalystX::JobServer (anyone use it?)

[Drew Taylor]

On Tue, Jun 5, 2012 at 8:25 AM, Seth Daniel  wrote:
> I found CatalystX::JobServer mentioned in a thread from about a year
> ago.
>
> It seems to still only be on github.  Does anyone use this?  Can anyone
> recommend it?
>
> https://github.com/bobtfish/CatalystX-JobServer

I too was looking at CatalystX::JobServer, but ended up going with the
perl port of Resque [1]. It was pretty straightforward to setup and
wrap my head around. On the monitoring side I'm using the ruby-based
Resque admin console [2]. My job needs are pretty simple right now,
but I know the uses will only grow over time.

I'm using Ubic [3] to manage multiple worker processes with
Ubic::Multiservice::Simple. Yesterday I launched a Dancer-based REST
API which is run by Ubic::Service::Starman [4]. The learning curve
wasn't bad, and it Just Works.

Drew

[1] https://metacpan.org/release/Resque
[2] https://github.com/defunkt/resque
[3] https://metacpan.org/release/Ubic
[4] https://metacpan.org/release/Ubic-Service-Starman
--

___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] CatalystX::JobServer (anyone use it?)

[Gianni Ceccarelli]

On 2012-06-05 Tomas Doran  wrote:
> Most of the good parts got ripped off by dakkar++, and combined with
> the (ideas from the) Catalyst::Engine::STOMP stuff, to make [3 dists]
> which is ActiveMQ specific, but a nice jobs solution.

To be honest, those distributions are not dependent on ActiveMQ. More
precisely:

* https://metacpan.org/release/CatalystX-ComponentsFromConfig
  is just a pair of plugins to instantiate simple views / models from
  the configuration files, without having to write empty subclasses in
  a file

* https://metacpan.org/release/Catalyst-ActionRole-JMS
  is an Action Role to dispatch to a specific action based on the
  contents of a C request header, or a C PSGI env
  element; the way I use it, these values are filled in by
  https://metacpan.org/release/Plack-Handler-Stomp but it would be
  easy to write handlers for different messaging protocols

* https://metacpan.org/release/CatalystX-ConsumesJMS
  is just a lot of sugar on top of the action role

-- 
Dakkar - 
GPG public key fingerprint = A071 E618 DD2C 5901 9574
 6FE2 40EA 9883 7519 3F88
key id = 0x75193F88

Mal: "Keep on talking... You're not getting a location trace off this
wave..." --"Serenity"


signature.asc
Description: PGP signature
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] CatalystX::JobServer (anyone use it?)

[Tomas Doran]

On 4 Jun 2012, at 23:25, Seth Daniel wrote:

> I found CatalystX::JobServer mentioned in a thread from about a year
> ago.
> 
> It seems to still only be on github.  Does anyone use this?  Can anyone
> recommend it?
> 
> https://github.com/bobtfish/CatalystX-JobServer
> 
> 
> I am looking at using RabbitMQ and I already have a catalyst server
> doing some other things.  It seemed like an interesting idea to combine
> them somehow.

Hi Seth :)

I consider CatalystX::JobServer a failed experiment.

I was trying to do too many things (and too many new things!) at once. There is 
some interesting code there to steal, but whilst it probably 'works', I 
wouldn't recommend trying using that code in production.

Most of the good parts got ripped off by dakkar++, and combined with the (ideas 
from the) Catalyst::Engine::STOMP stuff, to make:

https://metacpan.org/release/CatalystX-ComponentsFromConfig
https://metacpan.org/release/Catalyst-ActionRole-JMS
https://metacpan.org/release/CatalystX-ConsumesJMS

which is ActiveMQ specific, but a nice jobs solution.

I pulled my experience from that and other things out into a stab at something 
more generic:

https://metacpan.org/module/Message::Passing

You can use ZeroMQ, STOMP or AMQP!

However, I haven't yet CPAN'd a Message::Passing specific process manager to 
manage a worker pool, or blessed a specific way of running workers, or got 
workers which can consume messages from multiple possible work pools etc..

What you're looking for is easily possible to do with the framework, I just 
haven't yet got anything reusable enough to share generally yet…

I'm hoping to expand the docs a lot in the next couple of weeks, and I'm 
talking about it at YAPC::NA, but feel free to drop into #message-passing to 
pick my brains and/or tell me how my documentation sucks!!

Cheers
t0m




___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/