On 03/29/2012 12:00 PM, Nick Shelley wrote:
In my limited experience with parallel computing, it seems like the
master-worker paradigm is somewhat common. However, it seems like
Racket's places (or at least the way events are done with
place-channels) makes this inconvenient. Since there is nothing that I
can find in the result of syncing on a place-channel that would allow
me to then send some new work to the same channel, I have to implement
this manually by having each channel keep track of it's place in the
list of workers and send that information explicitly. This approach
seems hacky and more prone to bugs.
You need to use wrap-evt or handle-evt with a closure that includes the
channel.
I think it would make more sense for the result of syncing on a
place-channel to return the channel itself as a result. This would
make an explicit call to place-channel-get necessary, which may be a
downside, but the upside is I could then put something on the same
channel or do whatever else I may want with it.
This is one way to do it, however we choose to make place-channels work
like plain channels in Racket.
Its a trade off. Either you have to explicitly call place-channel-get
or you have to use wrap-evt or handle-evt to maintain a reference to the
channel.
If changing how place-channel events are treated isn't feasible, I
think it would at least be useful to provide an abstraction that makes
master-worker more convenient. One idea might be a sync-channel
function that returns the channel.
wrap-evt and handle-evt are those abstractions.
Place-channel is meant to be a simple primitive, enabling more
complicated parallel constructs to be built on top of place-channels.
We don't yet have a master-worker library. It would be nice if we did.
Some example you can look at include:
The places paper at
http://www.cs.utah.edu/plt/publications/dls11-tsffd.pdf, demonstrates a
master-worker parallel build using a jobqueue and handle-evt.
The file collects/setup/parallel-do.rkt is a very complicated example of
a master-worker implementation using work-queues and wrap-evt
As a sort of side note, it would be nice to be able to treat this
problem similar to a user thread problem where you can conceptually
imagine having infinite workers and just giving one chunk of work to
each of them. I tried to do this at first, but because of the way
places are implemented, I ran out of file descriptors relatively
quickly (and the overhead of starting a new VM for each chunk of work
might have been too much anyway, I don't know). I don't know if an
abstraction like that is possible or useful in general, but it may be
something to consider.
Places are too resource expensive to spawn one for each work item. You
have to use a work-queue and a small number of worker places.
Kevin
_
Racket Developers list:
http://lists.racket-lang.org/dev
On 03/29/2012 12:00 PM, Nick Shelley wrote:
In my limited experience with parallel computing, it seems like the
master-worker paradigm is somewhat common. However, it seems like
Racket's places (or at least the way events are done with
place-channels) makes this inconvenient. Since there is nothing that I
can find in the result of syncing on a place-channel that would allow
me to then send some new work to the same channel, I have to implement
this manually by having each channel keep track of it's place in the
list of workers and send that information explicitly. This approach
seems hacky and more prone to bugs.
I think it would make more sense for the result of syncing on a
place-channel to return the channel itself as a result. This would
make an explicit call to place-channel-get necessary, which may be a
downside, but the upside is I could then put something on the same
channel or do whatever else I may want with it.
If changing how place-channel events are treated isn't feasible, I
think it would at least be useful to provide an abstraction that makes
master-worker more convenient. One idea might be a sync-channel
function that returns the channel.
As a sort of side note, it would be nice to be able to treat this
problem similar to a user thread problem where you can conceptually
imagine having infinite workers and just giving one chunk of work to
each of them. I tried to do this at first, but because of the way
places are implemented, I ran out of file descriptors relatively
quickly (and the overhead of starting a new VM for each chunk of work
might have been too much anyway, I don't know). I don't know if an
abstraction like that is possible or useful in general, but it may be
something to consider.
-Nick
_
Racket Developers list:
http://lists.racket-lang.org/dev
_
Racket Developers list:
http://lists.racket-lang.org/dev
