Rails should enforce the best practices required to keep a user from making a mistake that would be hard to recover from as their application scales. An In-Memory queue is good, but it's trivial to build an implementation that can't be offloaded to another process. So I would like to see this API *enforce* that all queued objects be serializable/marshal-able at a minimum. This is a best practice and users who don't like it can trivially build their own simple queueing solution.
Regarding serializing to non-ruby protocols, I agree this is best left as an implementation detail. Chris On Mon, Apr 30, 2012 at 11:48 AM, Aaron Patterson <[email protected]>wrote: > On Sat, Apr 28, 2012 at 07:31:27PM -0700, Mike Perham wrote: > > Hi guys, I wanted to discuss the new Queueing API for those of us who > > are implementing an out-of-process version. In my case, I write > > Sidekiq [1] and would like to support the new API once Rails 4 is > > released. My issue is that because the API is object-oriented rather > > than message-oriented, implementation of out-of-process workers is > > difficult. > > > > The API is Queue#push(job) where job has a run method. Ruby doesn't > > have a great solution for serializing a Ruby object across the wire. > > Marshal limits the API to Ruby solutions (which rules out RabbitMQ, et > > al), JSON can't fully serialize Ruby objects (e.g. symbols) and YAML > > has a number of issues in practice that make it painful to use (e.g. > > see the monkeypatches DelayedJob has to use [2]). > > I don't understand what you mean by this. Marshal returns a string. If > your producers and consumers are both Ruby, why would this rule out > anything? Are you saying you want to mix languages between producers > and consumers? > > > So I love the simplicity of the API but think it will lead to painful > > implementation issues. What do you think about defining a simpler > > message format that can be fully serialized and deserialized via JSON > > / YAML / etc instead of using a Ruby object? > > I don't think the serialization format is something that Rails should > define. It's an implementation detail of the queuing system, and is > something a user must understand when using a queue. > > For example: > > * an in-memory queue has the advantage doesn't require serialization > but is volatile. But maybe that's all the programmer needs. > > * a DRb based queue can be distributed and uses marshal, so the user > doesn't need to understand serialization as much. > > * An object that has references to an IO object must take precautions > when being serialized. > > The other problem is that maybe the JSON format that sidekiq requires > could possibly be different than the JSON format that some other queuing > system requires. > > tl;dr every queue has unique aspects regarding transactions, wire > protocol, and storage facility. Users should take this in to account > when selecting a queue for their application's requirements. > > -- > Aaron Patterson > http://tenderlovemaking.com/ > -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
