This is actually something different than what Giuseppe is asking about. While related to data storage, this is more about live streaming of workload directly to the client, without any storage in between. For things like images, it makes sense to store somewhere else like mogile fs, since they will be written to someplace eventually (assuming you keep the originals).
In regards to Giuseppe's original question, I've thought about creating a simple wrapper API that would allow client<->worker direct communication rather than bouncing everything through the job server. If you are familiar with simple socket programming, you can do this now with a little bit of information in the client/worker. For example, before a client submits a job, it can bind() a port and send it's IP/port in the workload. When the worker gets this, it connects to the IP/port and streams the data directly to it. This can obviously break down if you start bouncing through NAT and so on, but it should work just fine for applications that really need this. -Eric On Sat, Oct 24, 2009 at 05:44:27PM +0800, Brian Aker wrote: > Hi! > > I've beeb discussing this with Eric and others. When I can find the time > to write up the protocol specification for this I can add some simple > methods for it so that we can add some native storage methods. The > storage backend will be some sort abstraction like what we do with > queue. > > This is a common complaint it needs to be solved sooner then later. > > Cheers, > --Brian > > On Oct 24, 2009, at 2:41 PM, Giuseppe Maxia <[email protected]> wrote: > >> I gave a well received talk on Gearman at the Italian Perl Workshop a >> few days ago. >> One of the questions from the audience was this: When a client sends a >> task request and provides a large chunk of data to be processed, does >> the data travel from client to server and to server to worker, or does >> the server somehow establish a communication channel between client >> and worker to minimize traffic? >> >> Example: >> >> gearman -w -f wordcount wc >> gearman -f wordcount < hugefile >> >> Does the content of hugefile travel from the client to the server and >> then from the server to the worker? >> >> Thanks >> >> Giuseppe >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~drizzle-discuss >> Post to : [email protected] >> Unsubscribe : https://launchpad.net/~drizzle-discuss >> More help : https://help.launchpad.net/ListHelp > > _______________________________________________ > Mailing list: https://launchpad.net/~drizzle-discuss > Post to : [email protected] > Unsubscribe : https://launchpad.net/~drizzle-discuss > More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~drizzle-discuss Post to : [email protected] Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
