Martin, thanks for the detailed explanation. Now, it's clear to me.
I see the light ;-) you are queuing the PAYLOAD, not the POST request, the
request object and transfer that it was born with the post.
Now I understand the situation (I hope ;-) I see three main use cases:
a- you need to queue the payload, process it, without sending a response to
the client.
b- you need to queue the payload, process it, and the client needs a
response in the associated (to original request) response.
c- you need to queue the payload, process it, and the client needs a
result, in the future.
Case a: can be solved easily, only en-queue the data/message, and process
it in a consumer process.
Case c: it's like case a, and, as you say, a client could poll the result.
I used this case in a game demo past year: the user started a new game,
send the request to main web site (not node yet), the web site send the
payload to an Azure queue, Azure workers process the command, making the
provisioning of the new game (board, initial position, maybe notify to
other players), and the client (browser) polls a blob storage that has the
game board/room status (ready, in progress.. etc).
Case b: hmmm.. I could make a callback function:
var newfn = function(result) {
///....
res.write(result); // res is a free "outer" variable, in this closure
res.end();
}
save it in an object as
var nc = ncallbacks++;
callbacks[nc] = newfn;
and put the nc value in the message to put into the queue. Many consumers
can process that messages, and then, sent a result to a second queue, with
the nc value in the result. The original web site consumes that queue, and
then, something like:
nc = message.nc;
result = message.result;
cb = callbacks[nc];
delete callback[nc];
cb(result);
If you have many web apps, there is a result queue by each one. The message
sent to the first queue, should have not only nc, but the name of the queue
associated to the inicial web app.
Maybe, instead of a result queue, a pub/sub-like solution could be used.
But I guess case b is the difficult one.
Another alternative: instead of queue, use an RPC solution, like dnode. And
the web app connects and balance the loads to many RPC servers.
I used the callback "old trick" in
https://github.com/ajlopez/SimpleRemote/blob/master/lib/simpleremote.js#L37
On Sat, Jul 7, 2012 at 5:55 PM, Martin Wawrusch <[email protected]> wrote:
> There are a couple ways to do this, for example:
>
> POST request with payload -> Machine A
> within request, machine A takes the payload and adds a message with it to
> the queue,
> then returns with some status message, perhaps containing a reference to
> the task entry.
>
> Machine B, upon pulling the message from the queue, processes the message,
> updates the database(s), updates caching layers (this is the tricky part),
> marks the message as processed, and perhaps sends a push notification to
> the originator of the POST request, probably through a common gateway, or
> the client just polls.
>
> For long file uploads, if you have sticky load balancing you just store
> the file chunks locally, once assembled upload it to s3 or something, then
> create a queue message that references the s3 file.
> If you don't have sticky load balancing, then you need to save the data
> chunks directly to s3, and keep some status info in redis or something.
>
> Note that this is not just for large file uploads, in a lot of cases you
> do this for normal database requests as well. In addition to that it can
> get very complicated very fast if you have intermediate caching layers.
>
>
>
> On Sat, Jul 7, 2012 at 1:39 PM, Angel Java Lopez <[email protected]>wrote:
>
>> Thanks! Hmm... interesting... but I missed some part, Martin. If I
>> understand your idea, the consumer(s) of the queue is in another machine.
>> That's is Ok, it compiles in my neuron ;-) My problem is about "POST
>> request".
>>
>> The initial POST request reach Machine A, from Client 1. At the end, it's
>> a socket connection, that http module gives to us as req/res. How to attend
>> that incoming request in Machine B? The client (maybe a browser) knows
>> Machine A, the open socket channel knows Machine A, not B, and it already
>> started to send POST data to it, maybe a file upload, I guess. The only
>> solution I can see: open a socket from A to B, and pipe to it the incoming
>> request (I apologize... my English, I'm struggling trying to express the
>> idea ;-).
>>
>> But I could misunderstand the initial question..
>>
>>
>> On Sat, Jul 7, 2012 at 5:23 PM, Martin Wawrusch <[email protected]>wrote:
>>
>>> Here are some docs, just a piece of advice - this can easily result in
>>> overkill, only use when it is really necessary:
>>>
>>> http://www.soapatterns.org/synchronous_response_deferred_processing.php
>>> http://martinfowler.com/bliki/CQRS.html
>>>
>>> Also, it might be feasible to check out iron.io - iron worker product,
>>> which works with node.js too. It basically takes care of message queues,
>>> worker processes etc and is super fast (< 1s response time)
>>>
>>> @Angel
>>> You do this to prevent clogging of the queue on a single machine, and to
>>> easier distribute load, and to recover in case of error scenarios.
>>>
>>>
>>> On Sat, Jul 7, 2012 at 1:13 PM, Angel Java Lopez
>>> <[email protected]>wrote:
>>>
>>>> Just curious...
>>>>
>>>> Why you need to queue the POST request?
>>>>
>>>> An in-memory queue is enough? Or you need to process the POST request
>>>> in another process? (I can't imagine why/how)
>>>>
>>>>
>>>>
>>>> On Sat, Jul 7, 2012 at 5:04 PM, Alan Hoffmeister <
>>>> [email protected]> wrote:
>>>>
>>>>> The major problem is the logic, I had mastered amqp already.
>>>>>
>>>>> Em Jul 7, 2012, às 4:55 PM, Martin Wawrusch <[email protected]>
>>>>> escreveu:
>>>>>
>>>>> No docs that I know of, but take a look at the amqp npm module, it
>>>>> works fine with RabbitMQ (at least it used to a year back when I worked
>>>>> with it).
>>>>>
>>>>> Other then that it is simply a matter of defining a message format,
>>>>> pushing that onto the queue, and have some separate process(es) working
>>>>> the
>>>>> queue.
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Jul 7, 2012 at 12:49 PM, Alan Hoffmeister <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hello fellows,
>>>>>>
>>>>>> I want to queue my POST requests, someone known a link where I can
>>>>>> read more about requests, Node.js and queue manager (my preference is
>>>>>> RabbitMQ).
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> --
>>>>>> Job Board: http://jobs.nodejs.org/
>>>>>> Posting guidelines:
>>>>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "nodejs" group.
>>>>>> To post to this group, send email to [email protected]
>>>>>> To unsubscribe from this group, send email to
>>>>>> [email protected]
>>>>>> For more options, visit this group at
>>>>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Job Board: http://jobs.nodejs.org/
>>>>> Posting guidelines:
>>>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "nodejs" group.
>>>>> To post to this group, send email to [email protected]
>>>>> To unsubscribe from this group, send email to
>>>>> [email protected]
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>>>>
>>>>> --
>>>>> Job Board: http://jobs.nodejs.org/
>>>>> Posting guidelines:
>>>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "nodejs" group.
>>>>> To post to this group, send email to [email protected]
>>>>> To unsubscribe from this group, send email to
>>>>> [email protected]
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>>>>
>>>>
>>>> --
>>>> Job Board: http://jobs.nodejs.org/
>>>> Posting guidelines:
>>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>>>> You received this message because you are subscribed to the Google
>>>> Groups "nodejs" group.
>>>> To post to this group, send email to [email protected]
>>>> To unsubscribe from this group, send email to
>>>> [email protected]
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>>>
>>>
>>>
>>>
>>> --
>>> Job Board: http://jobs.nodejs.org/
>>> Posting guidelines:
>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>>> You received this message because you are subscribed to the Google
>>> Groups "nodejs" group.
>>> To post to this group, send email to [email protected]
>>> To unsubscribe from this group, send email to
>>> [email protected]
>>> For more options, visit this group at
>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>>
>>
>> --
>> Job Board: http://jobs.nodejs.org/
>> Posting guidelines:
>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> You received this message because you are subscribed to the Google
>> Groups "nodejs" group.
>> To post to this group, send email to [email protected]
>> To unsubscribe from this group, send email to
>> [email protected]
>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>
>
>
>
> --
>
>
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to [email protected]
> To unsubscribe from this group, send email to
> [email protected]
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
