I agree that Rails shouldn't force a specific serialization scheme, and I definitely think that it should be message-based. So many of the problems people already encounter with queue have to do with poor design in this regard so it'd be nice if Rails pushed the design standard forward.
On Mon, Apr 30, 2012 at 3:53 PM, Chris Eppstein <[email protected]> wrote: > Rails should enforce the best practices required to keep a user from > making a mistake that would be hard to recover from as their application > scales. An In-Memory queue is good, but it's trivial to build an > implementation that can't be offloaded to another process. So I would like > to see this API *enforce* that all queued objects be > serializable/marshal-able at a minimum. This is a best practice and users > who don't like it can trivially build their own simple queueing solution. > > Regarding serializing to non-ruby protocols, I agree this is best left as > an implementation detail. > > Chris > > On Mon, Apr 30, 2012 at 11:48 AM, Aaron Patterson < > [email protected]> wrote: > >> On Sat, Apr 28, 2012 at 07:31:27PM -0700, Mike Perham wrote: >> > Hi guys, I wanted to discuss the new Queueing API for those of us who >> > are implementing an out-of-process version. In my case, I write >> > Sidekiq [1] and would like to support the new API once Rails 4 is >> > released. My issue is that because the API is object-oriented rather >> > than message-oriented, implementation of out-of-process workers is >> > difficult. >> > >> > The API is Queue#push(job) where job has a run method. Ruby doesn't >> > have a great solution for serializing a Ruby object across the wire. >> > Marshal limits the API to Ruby solutions (which rules out RabbitMQ, et >> > al), JSON can't fully serialize Ruby objects (e.g. symbols) and YAML >> > has a number of issues in practice that make it painful to use (e.g. >> > see the monkeypatches DelayedJob has to use [2]). >> >> I don't understand what you mean by this. Marshal returns a string. If >> your producers and consumers are both Ruby, why would this rule out >> anything? Are you saying you want to mix languages between producers >> and consumers? >> >> > So I love the simplicity of the API but think it will lead to painful >> > implementation issues. What do you think about defining a simpler >> > message format that can be fully serialized and deserialized via JSON >> > / YAML / etc instead of using a Ruby object? >> >> I don't think the serialization format is something that Rails should >> define. It's an implementation detail of the queuing system, and is >> something a user must understand when using a queue. >> >> For example: >> >> * an in-memory queue has the advantage doesn't require serialization >> but is volatile. But maybe that's all the programmer needs. >> >> * a DRb based queue can be distributed and uses marshal, so the user >> doesn't need to understand serialization as much. >> >> * An object that has references to an IO object must take precautions >> when being serialized. >> >> The other problem is that maybe the JSON format that sidekiq requires >> could possibly be different than the JSON format that some other queuing >> system requires. >> >> tl;dr every queue has unique aspects regarding transactions, wire >> protocol, and storage facility. Users should take this in to account >> when selecting a queue for their application's requirements. >> >> -- >> Aaron Patterson >> http://tenderlovemaking.com/ >> > > -- > You received this message because you are subscribed to the Google Groups > "Ruby on Rails: Core" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/rubyonrails-core?hl=en. > -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
