There are limits on how many processes can exist at the same time and having more 'later processes than cores is wasteful and Linux only allows for 1024 file descriptors per process. ? (for N 10000 (later (cons) (* N N))) !? (pipe (pr (prog (* N N)))) Pipe error: Too many open files ? N -> 339
Is there some "worker" implementation which keeps a queue of task to do and creates N worker processes which take tasks from the queue when they are finished? Ideas: Task would be copy of environment + function + values to apply the function too. There would be some optimization potential in case the function and environment is static, or reseted each time to a constant value but it should be possible to fetch a copy of the current environment to. Such a pool of workers is best represented with a symbol on which certain functions are applied which are may stored inside the symbol. This could be archived with some overhead with the picolisp object system when compared to a manual implementation since the worker queue should be accessible fast and putting it in val of the symbol would be the easiest way to achieve that. Further thoughts: It might be wise to give the possibility to register a callback on the return value. This would allow to make things more efficient. For example: (prog1 # Parallel background calculation of square numbers (mapcan '((N) (later (cons) )) (range 1 100)) (wait NIL (full @)) ) could be: (pool "squares.db") (with-workerpool W '((workers . 2) (callback . '((R) (put *DB (car R) (cadr R)))) )) (mapcar '((N) (add_task 'W '((N) (cons N (* N N))) N ) (range 1 100)) )] #Normally the with-workerpool should not finish until the queue is empty How ever it should also be possible to do something like that: (with-workerpool W '((callback . NIL) (return . nil) (output . "+file") (wait . NIL)) (for X 10000000 (add_task 'W '((N) (ifn (= N (apply '* (prime-factors N))) (print N))) X ] # workers run in background main thread can continue This usage may seem overly complex at first and the proposed interface is horrible. However i think that such an library could be an valuable addition to picolisp. Lets discuss the idea itself, your proposals for the interface and the implementation.