Hi! Hmmm, that code example looks simple enough. If that works for arbitrary serializable-lambda with only serializable parts, I could continue my process pool project. The only issue would then be, that any user of it would have to know in advance, that they cannot define their lambdas as usual, but have to use serializable-lambda, potentially through their entire code (as there might be references to things which contain references to things, which ...). The abstraction is in this way leaky, but it would make a working library then.
`n` does not need to be serializable, because it is defined in the module and will be defined in the module "on the other side" as well? If I understand this correctly. When saying, that in other languages it is possible to simply define a lambda and send it to another actor, I meant of course with immutable data, or at least with data, which is not actually mutated. I assumed that already, but yes, of course you are right about that. I was thinking of Erlang, Elixir and maybe Pony. When hearing actors, then Threads could be thought of a means of implementing them, but I think might not be useful to do so when thinking about performance. Architecturally yes, maybe. That is, why I would think of places as a means of implementing an actor model kind of thing. When you say, that loci extends the idea of places to multiple machines, what do you mean? I thought places can already run on multiple machines. I might try to use your example code to finally finish the process pool implementation I started. If everything works like that, then I probably have to retract any statements about parallelism being too hard in Racket : ) It would be nice however, to not have to use a different construct to define serializable lambdas and to be able to go to any program and simply use existing lambdas to send them to a process pool to make use of multiple cores, instead of having to refactor many things into serializable things. Regards, Zelphir On 10/9/19 11:58 PM, Philip McGrath wrote: > On Wed, Oct 9, 2019 at 2:09 PM Zelphir Kaltstahl > <zelphirkaltst...@gmail.com <mailto:zelphirkaltst...@gmail.com>> wrote: > > I was wrongly under the impression, that serializable-lambda are > supposed to work out of the box, when sending them over channels, > without needing to do any further work ("are serialized > automatically" instead of "can be serialized"). … If you know how > to serialize those serializable-lambdas, it is possible, that you > could solve the problems they faced. > > … Is there a generic way to serialize such lambdas, no matter what > they look like? > > You can serialize the procedures produced by `serial-lambda` using the > `racket/serialize > <https://docs.racket-lang.org/reference/serialization.html>` library, > the same way you serialize other kinds of Racket values. Here is an > extended example: > #lang racket > > (require web-server/lang/serial-lambda > racket/serialize > rackunit) > > (define (make-serializable-adder n) > (serial-lambda (x) > (+ n x))) > > (define serializable-add5 > (make-serializable-adder 5)) > > ;; The value of a serial-lambda expression is a procedure. > (check-eqv? (serializable-add5 2) > 7) > > ;; The procedure can't be sent over a place-channel directly ... > (check-false > (place-message-allowed? serializable-add5)) > ;; ... but it is serializable: > (check-true > (serializable? serializable-add5)) > > (define serialized > (serialize serializable-add5)) > ;; The serialized form can be sent over a place-channel. > (check-true > (place-message-allowed? serialized)) > > (define-values [in out] > (place-channel)) > > (place-channel-put in serialized) > > ;; When we deserialize the received value, potentially in a new place ... > (define received > (place-channel-get out)) > (define deserialized > (deserialize received)) > > ;; ... it works like the original, including closing over the lexical > environment. > (check-eqv? (deserialized 11) > 16) > > One thing to note is that the procedures returned by `serial-lambda` > do close over their lexical environments, which is desirable for the > reasons you mention in your previous message. That means that values > which are captured as part of the closure must also be serializable, > or `serialize` will raise an exception. (The same is true for lists: > they are serializable as long as their contents are also serializable.) > > Concretely, in the example above, `n` is part of the closure and > therefore must be serialized—which works out great, because `n` must > be a number, and numbers are serializable. On the other hand, the > procedure `+` isn't serializable, but that's ok, because `+` isn't > part of the closure: it's a reference to a module-level identifier. > > … it seems to be the case in other programming languages, where > one can simply send lambdas to other actors, which can, but don't > have to, run on other cores or other machines. … It also seems to > be necessary for eventually creating a library, which provides a > process pool, as such library possibly should be easy to use and > not force the user to think about difficult thing, when the > intention the user has seems so simple "just do that on another core". > > > This depends on what you mean by "actors." You can simply send > closures to other Racket-level threads (in the sense of `thread > <https://docs.racket-lang.org/reference/threads.html>`), which share > state and run concurrently (but not in parallel) within the same > OS-level thread. While not framed specifically in the vocabulary of > the actor model, Racket threads seem to fit the model pretty well: > they are what I think of first when I hear "actors" in a Racket context. > > When you want machine-level parallelism (i.e. multiple OS threads), > that shared mutable state becomes a problem, because hardware provides > only very low-level mechanisms for coordinating parallel threads so > that they have a consistent view of the state (i.e. no variables are > half-assigned). This is true regardless of the language you work in: > languages differ in what tools, if any, they give you to manage the > problem. > > Racket provides two mechanisms for parallelism, places and futures > <https://docs.racket-lang.org/reference/futures.html>. Futures share > state like Racket-level threads, but block when attempting an > operation that can't safely be done in parallel: in other words, they > provide "best-effort" parallelism. On traditional Racket, there are > lots of operations that can't safely be done in parallel, so futures > have mostly been useful so far for numeric computation. There's hope > that will change with Racket-on-Chez, but I think we're still at the > stage of needing more people to try it and see how it goes in practice. > > Places, on the other hand, provide "shared nothing" parallelism: each > exists in its own world, without shared mutable state, and they > communicate by explicitly sending messages over place-channels (which > are different than normal channels, e.g. by being asynchronous, but > have much the same API and integrate with Racket's general system for > threads and events). (Actually, places do support a very restricted > form of shared mutable state through things like `make-shared-bytes`.) > There are also distributed places and Paulo Matos's loci > <https://github.com/LinkiTools/racket-loci>, which extend the idea of > places to multiple processes or machines. > > The point is that "just do that on another core" doesn't work in > general, because "that" might rely on mutable state or other aspects > of the context where it is run. Of course there are lots of cases that > are trivially parallelizable, but, even in those cases, starting > parallel threads and communicating between them involves overhead. I > don't know of any language or library that completely relieves the > programmer of thinking about what can be parallelized well and how > much parallelism to use, and then writing those decisions down somehow > in the programming language. In my experience, Racket makes it easier > to reason about parallel programming than languages that expose you to > the wild west of the hardware, and I don't find writing down parallel > programs in Racket especially onerous. > > -Philip -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to racket-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/bd8271fe-35e3-ea0c-ad38-067fce378207%40gmail.com.
