On Tue, Mar 27, 2012 at 3:24 PM, Morgan Goose <[email protected]>wrote:
> Use xargs:
> def parallel_local(cmd, list_of_args, procs=4):
> local("echo '%s' | xargs -P %d -n1 %s" % (" ".join(list_of_args),
> num_of_procs, cmd))
>
>
> I find that most everything you wanna do has a gnu command or flag for
> it already. thankfully.
>
> -goose
>
>
While I agree this is a solution... ;) it still seems like a bit of a
kludge. Also, i'd typically be calling fab again because I want to write
the command with fabric APIs. So make it "...-n1 fab %s ", sure.
But still, my point is that while Fabric is good at executing commands in
parallel for some unique set of hosts, it seems like execute() should have
some mode of operation to do the complement, to execute commands in
parallel for some set of hosts as well as some set of inputs.
Rather than shell out to xargs to get this done, I'd just as soon call
multiprocessing.imap() to do what I need to do directly - my observation
here is that this is very close to what execute() does already, perhaps
there is room for it within execute().
T
>
> On Tue, Mar 27, 2012 at 2:57 PM, Tyler Pirtle <[email protected]> wrote:
> >
> >
> > On Tue, Mar 27, 2012 at 1:13 PM, Morgan Goose <[email protected]>
> > wrote:
> >>
> >> What is it you even mean by staging? You might be over looking GNU
> >> commands that will do exactly what you need. If you give us some more
> >> information, we might be able to assist.
> >>
> >> -goose
> >>
> >
> > Hi goose - let me try and clarify.
> >
> > My use case is something like invoking a function over one host many
> times
> > with
> > different argument values (Or many hosts with many values for extra
> credit).
> >
> > So I've got some function i want to execute on localhost,
> >
> > def foo(i): ...
> >
> > What I'd like to do is imap(foo, xrange(1, 10)) in parallel.
> >
> > The execute() function doesn't give me a way to express this (the way
> I've
> > done this below is to generate fake host lists). The host list seems to
> be
> > the unit controlling the parallelism, I'd like to be able to control it
> via
> > input arguments
> > as well.
> >
> > To illustrate:
> >
> > (with env.parallel = True)
> > hosts = ["localhost", "localhost", "localhost"]
> > execute(foo, hosts, ... )
> >
> > Would execute "foo" 3 times in parallel (except it wouldnt because host
> > lists are uniq'd)
> >
> > What I'm looking for:
> >
> > hosts = ["localhost"]
> > input = xrange(10)
> > execute(foo, hosts, input)
> >
> > Should be equivalent to imap(foo, input) over hosts.
> >
> >
> >
> >
> >>
> >> On Fri, Mar 23, 2012 at 7:42 PM, Tyler Pirtle <[email protected]> wrote:
> >> > I realize this use case in particular may sound a little strange, but
> >> > bear
> >> > with me.
> >> >
> >> > I've got 10 files lets say, locally, and would like to launch 10 tasks
> >> > to
> >> > stage them to some other target. I'd like to do this in parallel,
> since
> >> > the
> >> > files are named appropriately (they're sharded), and what I'd like to
> do
> >> > is
> >> > to construct a call in fabric that executed some function over the
> same
> >> > host
> >> > with varying input.
> >> >
> >> > At a first pass, I thought I would just try to get one task to
> execute N
> >> > times in parallel at localhost:
> >> >
> >> > @parallel
> >> > def Stuff():
> >> > print "Sleep!"
> >> > time.sleep(2)
> >> >
> >> >
> >> > $ fab -H localhost,localhost,localhost Stuff
> >> > [localhost] Executing task 'Stuff'
> >> > Sleep!
> >> >
> >> > Done.
> >> >
> >> > This doesn't work because the host list gets merged (merge in
> >> > task_utils.py).
> >> >
> >> > But, merge() isn't that smart.
> >> >
> >> > $ fab -P -H foo@localhost,bar@localhost,baz@localhost Stuff
> >> > [foo@localhost] Executing task 'Stuff'
> >> > [bar@localhost] Executing task 'Stuff'
> >> > [baz@localhost] Executing task 'Stuff'
> >> > Sleep!
> >> > Sleep!
> >> > Sleep!
> >> >
> >> > So this is a workaround and a pretty silly hack - does anyone have a
> >> > better
> >> > way to do this? As a further hack, I wrap it in an easy-to-use
> command:
> >> >
> >> >
> >> > @parallel
> >> > def Stuff():
> >> > print "Sleep - %s" % env.host
> >> >
> >> > @hosts("localhost")
> >> > @parallel
> >> > def PLaunch(target="", times=1):
> >> > h = []
> >> > for i in xrange(int(times)):
> >> > h.append("%d@localhost" % i)
> >> > execute(target, hosts=h)
> >> >
> >> > $ time fab -f fab2.py -P PLaunch:Stuff,10
> >> > [localhost] Executing task 'PLaunch'
> >> > [0@localhost] Executing task 'Stuff'
> >> > [1@localhost] Executing task 'Stuff'
> >> > [2@localhost] Executing task 'Stuff'
> >> > [3@localhost] Executing task 'Stuff'
> >> > [4@localhost] Executing task 'Stuff'
> >> > [5@localhost] Executing task 'Stuff'
> >> > [6@localhost] Executing task 'Stuff'
> >> > [7@localhost] Executing task 'Stuff'
> >> > [8@localhost] Executing task 'Stuff'
> >> > [9@localhost] Executing task 'Stuff'
> >> > Sleep - 9@localhost
> >> > Sleep - 8@localhost
> >> > Sleep - 7@localhost
> >> > Sleep - 4@localhost
> >> > Sleep - 6@localhost
> >> > Sleep - 3@localhost
> >> > Sleep - 2@localhost
> >> > Sleep - 5@localhost
> >> > Sleep - 1@localhost
> >> > Sleep - 0@localhost
> >> >
> >> > Done.
> >> >
> >> > real 0m5.522s
> >> > user 0m0.352s
> >> > sys 0m0.251s
> >> >
> >> >
> >> > So i can strip off everything up to the @ and use that as the arg. Any
> >> > better ideas? ;)
> >> >
> >> > Thanks,
> >> >
> >> >
> >> > Tyler
> >> >
> >> >
> >> > _______________________________________________
> >> > Fab-user mailing list
> >> > [email protected]
> >> > https://lists.nongnu.org/mailman/listinfo/fab-user
> >> >
> >
> >
>
_______________________________________________
Fab-user mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/fab-user
