> Sorry for the short reply. I know you are busy so any reply at all is appreciated; thank you. **I 'll try to phrase the following so you can give short answers that I can use...**
> `spawn` and `parallel` should move to external packages It seems to me that it would be easy enough to move `spawn` to an external package, **but doesn 't `parallel` require some "computer magic" that means it needs to stay in core**, and if so doesn't it make sense that `spawn` would stay with it? From [the Experimental Features section of the manual on Parallel](https://nim-lang.org/docs/manual_experimental.html#parallel-amp-spawn-parallel-statement) **" The parallel statement is the preferred mechanism to introduce parallelism in a Nim program. A subset of the Nim language is valid within a parallel section. This subset is checked during semantic analysis to be free of data races. A sophisticated disjoint checker ensures that no data races are possible even though shared memory is extensively supported!"; that seems to imply some "computer magic"?** It seems to be your implementation of a `forall` available in many multi-processing intended languages such as Fortran, Chapel, etc., and of course is very useful for a suitable use case such as in array processing but doesn't fit all general use cases. I suppose that `FlowVar[T]` doesn't require "compiler magic" as @mratsim was able to implement it as well as `sync()` in his own "Weave" multi-processing package. His **package** is very useful for some advanced applications to multi-processing including work stealing, but may be overkill for someone who is just looking for an equivalent to GoLang's go routines. Considering `FlowVar[T]` as compared to `Channel[T]`, there are some interesting thought experiments to be made: Haskell's equivalent to `FlowVar[T]` is `Control.Concurrent.MVar` and these are considered the base of multi-threading control and actually used to create the Haskell version of `Control.Concurrent.Chan`. Also, in my distant past when I first tried GoLang, I used single buffer element channels as `FlowVar[T]`'s rather than using `sync.Mutex` to protect a critical section accessing shared memory when the type was wrong to be able to use atomic access, as it was simpler since I could block/wait and get a return value in the same operation, so I was inverting the Haskell relationship by using a `Channel[T]` to create a `FlowVar[T]`. It would seem to me that the same could be done in Nim, although I don't know if there would be a performance cost one way or the other. > Btw without `Isolated[T]` the design is incomplete (and it needs compiler > magic) so we got that one into Nim's core. I think I saw `Isolated[T]` go by while I was just monitoring Nim development, but I forgot to follow it up. Interesting. **If the `threadpool` library were to switch to using `Isolated[T]`, would that mean that they no longer need to be in core as the "computer magic" parts required are already encapsulated there?** I can't see that would include `parallel` as per the above documentation observation. Also, `parallel` seems to be the only part within the `threadpool` library that is marked experimental. **Does that mean it should be separated out into its own library that needs to be part of core?** **Is `isolation` considered to be a main stream library that can be used to update standard libraries or is it just experimental even though it isn 't marked as such?** I note that your commit calls this a "builtin", so that would suggest that it is considered to be a standard. As an aside, what is the significance of some libraries being in the "/std/" sub folder where the `isolated` library is found? As a documentation issue, **shouldn 't the `isolated` library be added to the Standard Library main documentation page as it as already been added to the Searchable Index?** If so, I can do that for you. As a second documentation issue: the statment in the experimental section of the manual regarding `spawn` that "ref parameters are deeply copied which is a subtle semantic change and can cause performance problems but ensures memory safety. This deep copy is performed via system.deepCopy and so can be overridden." is no longer exactly correct, as with `sink` parameters that are last use, `ref` parameters are moved rather than copied. **Again, this should be corrected?** **Do you consider it a goal for version 2.0 to get `threadpool` out of "Unstable API" status and into the mainstream?** I think it should be, or at least everything except maybe `parallel` should be, as I would like to see basic multi-threading in Nim be as easy as using "go routines" in GoLang, which isn't quite there even using `Isolated[T]`: # Simple "go routine" in Nim language... import threadpool; import std/isolation proc goFunc(istr: sink Isolated[string]): string {.thread.} = var isv = istr; echo isv.extract result = "Hello there, Go lover!" echo ^spawn goFunc("Hello there!".isolate) Run First, the above code doesn't seem to be able to return an `Isolated[T]`, **although a re-written `FlowVar[T]` could take care of the isolation itself, but again, maybe that 's "compiler magic" and would mean that `FlowVar[T]` would need to be core, right? Or is it possible to return a `ResultVar[sink Isolated[T]]` without "compiler magic" with some library tweaking?** Second, the `{.thread.}` pragma as per the documentation says that it only flags the reader that this is a threaded function/proc and allows the compiler to check whether the function/proc uses cross thread memory accesses (more "compiler magic"); however **with the `--gc:arc/orc` runtime, is there a distinction between shared and non-shared memory so is there anything for the compiler to check?** I note that this pragma is not used in the manual's examples for `spawn` and `parallel`. Finally, the above code isn't really equivalent to GoLang code as `go routines` can't return anything (no automatic generation of a return `FlowVar`, which GoLang doesn't have) and inter-thread communication needs to be done by Channel's, which we have covered whether we elect to use `Isolated[T]` or not, or by the usual mutex/`Lock` protected shared memory accesses or atomic access, all of which we have covered. So the encouraged GoLang form of code using `Channel[T]`'s would be more as follows: # Simple "go routine" with return channel in Nim language... # No call to `allocateShared0` is made as the memory pointed is on the stack which # persists as long as the function `main` which waits until threading is finished! import threadpool; import std/isolation proc main() = # [ var ch: Channel[sink Isolated[string]]; ch.open proc goProc(istr: sink Isolated[string], rchp: ptr Channel[sink Isolated[string]]) {.thread.} = var isv = istr; echo isv.extract rchp[].send("Hello there, Go lover!".isolate) spawn goProc("Hello there!".isolate, ch.addr) var irstr = ch.recv echo irstr.extract # ]# #[ var ch: Channel[sink string]; ch.open proc goProc(istr: sink Isolated[string], rchp: ptr Channel[sink string]) {.thread.} = var isv = istr; echo isv.extract rchp[].send("Hello there, Go lover!") spawn goProc("Hello there!".isolate, ch.addr) echo ch.recv # ]# ch.close when isMainModule: main() Run A minor "whoops" there as the top version using `Channel[sink Isolated[string]]` doesn't compile due to "channels_builtin.nim(369, 73) Error: =destroy needs to have the 'nimcall' calling convention" as of Nim devel version 1.5.1 built yesterday. **Guess I had better file an issue on that?** Even when it is fixed, it's getting a little verbose. **Is the process in converting to and fro with `Isolated[T]` expensive?** It seems your original idea as per the RFC was to make `Channel[T]` automatically expect a `Isolated` for `T` but some objected in order to get the flexibility either way and you relented. **If it isn 't expensive, perhaps your original idea would be better? It would certainly reduce the verbosity. Even if the conversion is expensive, could we "spoof" the compiler with some type "punning"? Or have an overload? If using `Isolated[T]` is optional, I don't see how it makes the design complete?** In summary, as I mentioned about @mratsim's Weave package, that is likely the package to use for major applications by power users. However, it can't have `parallel` because that would seem to require "compiler magic" as currently specified and as implemented as `forall`/`coforall` in other languages. I don't think Nim core/basic libraries need to have what Weave offers, but it should have an easy-to-use equivalent to GoLang's go routines that tap into a thread pool without requiring that one implement a thread pool oneself (which I have done when all these things were uglier than now). As this seems to be mostly just library/documentation changes (other than `parallel`), I could likely help with this. **As to `parallel`, do you see any way of stabilizing it and getting it off of the experimental list, at least for use with `--gc:arc/orc`, for version 2.0?**
