branch: externals/el-job
commit 2ede4a3183a49598961d1d586c4724fda11fdf60
Author: Martin Edström <[email protected]>
Commit: Martin Edström <[email protected]>
Docs comments
---
README.org | 58 +++++++++++++++++++++++++++++-
el-job-ng.el | 30 +++++++++++-----
el-job.el | 16 +++++----
el-job.texi | 116 +++++++++++++++++++++++++++++++++++++++++++++++++++++++----
4 files changed, 199 insertions(+), 21 deletions(-)
diff --git a/README.org b/README.org
index 8c17c01915..cb7c42e3eb 100644
--- a/README.org
+++ b/README.org
@@ -19,7 +19,63 @@ That's it in a nutshell. You can look at real-world usage
by searching for "el-
-
[[https://raw.githubusercontent.com/meedstrom/org-mem/refs/heads/main/org-mem.el][org-mem.el]]
-
[[https://raw.githubusercontent.com/meedstrom/org-roam-async/refs/heads/main/org-roam-async.el][org-roam-async.el]]
-* Since 2.5.0
+* 2.7.0
+- New entry point: =el-job-parallel-mapcar=! Read more below.
+- Argument =:inputs= are now split differently, for the sake of a predictable
order of outputs. It is likely less optimal, but hopefully not too much.
+- Argument =:require= now accepts strings in addition to symbols. Strings are
passed to =load= instead of =require=.
+- Value of =temporary-file-directory= no longer added to =:inject-vars= for
you.
+ - Now the only values added are =load-path= and =native-comp-eln-load-path=.
+- Function =el-job-ng-job= renamed to =el-job-ng-get-job=.
+ - New class =el-job-ng-job=.
+- Delete many aliases to el-job-old.el.
+
+** New entry point: =el-job-parallel-mapcar=
+Until now, we had to make do with an unwieldy =el-job-ng-run= with ~5 keyword
arguments. And we had to understand asynchronous programming to use it.
+
+I've long dreamed to be able to brainlessly rewrite a form =(mapcar #'FN
INPUTS)= to something like =(multicore-mapcar #'FN INPUTS)= and have it Just
Work and Just Be Faster.
+
+Finally, it is done!
+
+The calling convention is
+
+: (el-job-parallel-mapcar FN INPUTS &optional INJECT-VARS)
+
+Please see the docstring for what you need to know.
+
+*** Weaknesses
+Do not use =el-job-parallel-mapcar= with overly trivial functions. It adds
some overhead per item, so it actually slows you down if the function itself
only takes microseconds or nanoseconds per invocation.
+
+Example of a bad use-case:
+
+#+BEGIN_SRC elisp
+(let ((spam (make-list 1000000 "spam")))
+ (list (benchmark-elapse (mapcar #'upcase spam))
+ (benchmark-elapse (el-job-parallel-mapcar #'upcase spam))))
+#+END_SRC
+
+Return value:
+
+: (0.482017027 4.793430311)
+
+Another bad use-case is when plain =mapcar= would've been fast enough. There
is some constant overhead related to spinning up subprocesses.
+
+Changing 1000000 from the earlier expression to just 100 reveals the constant
to be ~210ms on my machine:
+
+#+BEGIN_SRC elisp
+(let ((spam (make-list 100 "spam")))
+ (list (benchmark-elapse (mapcar #'upcase spam))
+ (benchmark-elapse (el-job-parallel-mapcar #'upcase spam))))
+#+END_SRC
+
+Return value:
+
+: (0.000041974 0.219570936)
+
+* 2.6.0
+
+- For =el-job-old-launch=, new argument =:eval=
+
+* 2.5.0
Released [2025-10-06 Mon], v2.5.0 comes with a variant library "el-job-ng".
diff --git a/el-job-ng.el b/el-job-ng.el
index 8fb2211a4d..c13076e8f1 100644
--- a/el-job-ng.el
+++ b/el-job-ng.el
@@ -39,7 +39,7 @@ if making too many processes, so capping it can help."
:group 'processes)
-;;; Subroutines
+;;;; Subroutines
(defvar el-job-ng--debug-level 0
"Increase this to 1 or 2 to see more debug messages.")
@@ -138,7 +138,7 @@ Unlike `locate-library', this can actually find the .eln."
(error "el-job-ng: Library not found: %S" name))))
-;;; Entry point
+;;;; Entry point
(defvar el-job-ng--jobs (make-hash-table :test 'eq))
(defclass el-job-ng-job ()
@@ -172,22 +172,26 @@ At a glance:
that to CALLBACK, a function called precisely once.
In other words, CALLBACK should be expected to receive one list that
is equal in length to INPUTS.
+ Also, the order of values is preserved.
Details:
- INJECT-VARS is an alist of symbols and values to pass to `set'.
It has some default members, including `load-path'.
-- REQUIRE is a list of symbols for `require' or strings for `load'.
+- REQUIRE is a list of symbols for `require', or strings for `load'.
- EVAL is a list of quoted forms.
- FUNCALL-PER-INPUT must be a symbol with a function definition,
not an anonymous lambda.
+ That definition must be discoverable via `load-path' or REQUIRE.
It is passed two arguments: the current item, and the remaining items.
\(You probably will not need the second argument.\)
Finally, ID is an optional symbol. Passing an ID has two effects:
- Automatically cancel a running job with the same ID, before starting.
- Use benchmarks from previous runs to better balance the INPUTS split.
+ See `el-job-ng--split-optimally'.
ID can also be passed to these helpers:
+- `el-job-ng-get-job'
- `el-job-ng-await'
- `el-job-ng-await-or-die'
- `el-job-ng-ready-p'
@@ -279,10 +283,12 @@ ID can also be passed to these helpers:
(setf process-outputs (nreverse process-outputs)))))))
-;;; Code used by child processes
+;;;; Code used by child processes
(defvar el-job-ng--child-args 2)
(defun el-job-ng--child-work ()
+ "Read a few lines from stdin, then work according to that info.
+Finally print to stdout and die."
(let* ((coding-system-for-write 'utf-8-emacs-unix)
(coding-system-for-read 'utf-8-emacs-unix)
(vars (read-from-minibuffer "" nil nil t))
@@ -310,7 +316,7 @@ ID can also be passed to these helpers:
(print (nreverse benchmarked-outputs)))))
-;;; Sentinel; receiving what the child printed
+;;;; Sentinel; receiving what the child printed
(defun el-job-ng--sentinel (proc event)
"Handle changed state of a child process.
@@ -350,6 +356,10 @@ and run `el-job-ng--handle-finished-child'."
(el-job-ng-kill-keep-bufs id)))))
(defun el-job-ng--handle-finished-child (proc buf job)
+ "Handle output returned by PROC, presuming that is in buffer BUF.
+Then kill BUF.
+Once this has handled all outputs for JOB, run the CALLBACK function
+specified in `el-job-ng-run'."
(with-slots (id process-outputs callback benchmarks do-bench) job
(with-current-buffer buf
(unless (and (eobp) (> (point) 2) (eq (char-before) ?\n))
@@ -371,7 +381,7 @@ and run `el-job-ng--handle-finished-child'."
(el-job-ng--dbg 0 "Quit while executing :callback for %s" id))))))
-;;; API
+;;;; API
(defmacro el-job-ng-sit-until (test max-secs &optional message)
"Block until form TEST evaluates to non-nil, or MAX-SECS elapse.
@@ -399,12 +409,14 @@ A typical TEST would check if something in the
environment has changed."
,last)))
(defun el-job-ng-await (id max-secs &optional message)
- "Like `el-job-ng-sit-until' but take ID and return t if job finishes."
+ "Like `el-job-ng-sit-until' but take ID and return t if job finishes.
+MAX-SECS and MESSAGE as in `el-job-ng-sit-until'."
(el-job-ng-sit-until (el-job-ng-ready-p id) max-secs message))
(defun el-job-ng-await-or-die (id max-secs &optional message)
"Like `el-job-ng-await', but kill the job on timeout or any signal.
-Otherwise, a keyboard quit would let it continue in the background."
+Otherwise, a keyboard quit would let it continue in the background.
+ID, MAX-SECS and MESSAGE as in `el-job-ng-await'."
(condition-case sig
(if (el-job-ng-await id max-secs message)
t
@@ -440,6 +452,7 @@ Otherwise, a keyboard quit would let it continue in the
background."
(delete-process proc)))
(defun el-job-ng-stderr (id)
+ "Get the stderr buffer for ID."
(let ((job (el-job-ng-get-job id)))
(and job (oref job stderr))))
@@ -450,6 +463,7 @@ Otherwise, a keyboard quit would let it continue in the
background."
(mapcar #'car (oref job process-outputs))))))
(defun el-job-ng-get-job (id-or-process)
+ "Get the job object associated with ID-OR-PROCESS."
(if (processp id-or-process)
(cl-loop for job being each hash-value of el-job-ng--jobs
when (assq id-or-process (oref job process-outputs))
diff --git a/el-job.el b/el-job.el
index fa86620874..97c69ecc83 100644
--- a/el-job.el
+++ b/el-job.el
@@ -53,10 +53,11 @@
"Apply FN to LIST like `mapcar' in one or more parallel processes.
Function FN must be known in `load-history' to be defined in some file.
-The parallel processes inherit `load-path' and then load that file.
+At spin-up, the parallel processes inherit `load-path', then load that
+file \(even if it is not on `load-path'\), and then get to work.
-Function FN must not depend on side effects from previous invocations of
-itself, because each process gets a different subset of LIST.
+Function FN should not depend on side effects from previous invocations
+of itself, because each process gets a different subset of LIST.
Unlike the more general `el-job-ng-run', this is meant as a close
drop-in for `mapcar'. It behaves like a synchronous function by
@@ -76,9 +77,12 @@ since FN runs in external processes.
That means FN will not see let-bindings, runtime variables and the like,
that you might have meant to have in effect where
`el-job-parallel-mapcar' is invoked.
-Nor can it mutate such variables for you -- the only way it can affect
-the current Emacs session is if the caller of
-`el-job-parallel-mapcar' does something with the return value."
+That is why you may need INJECT-VARS.
+
+N/B: The aforementioned loss of scope also means that FN cannot set or
+mutate any variables for you -- the only way it can affect the current
+Emacs session is if the caller of `el-job-parallel-mapcar' does
+something with the return value."
(let* (result
(vars (el-job-ng-vars (cons '(el-job-ng--child-args . 1)
inject-vars)))
(id (intern (format "parallel-mapcar.%S.%d" fn (sxhash vars)))))
diff --git a/el-job.texi b/el-job.texi
index 4a6bf1c963..83937728e3 100644
--- a/el-job.texi
+++ b/el-job.texi
@@ -40,13 +40,23 @@ That's it in a nutshell. You can look at real-world usage
by searching for "el-
@end ifnottex
@menu
-* Since 2.5.0: Since 250.
+* 2.7.0: 270.
+* 2.6.0: 260.
+* 2.5.0: 250.
* README for 2.4.8: README for 248.
@detailmenu
--- The Detailed Node Listing ---
-Since 2.5.0
+2.7.0
+
+* New entry point @samp{el-job-parallel-mapcar}::
+
+New entry point: @samp{el-job-parallel-mapcar}
+
+* Weaknesses::
+
+2.5.0
* Future work::
@@ -69,10 +79,104 @@ Design rationale
@end detailmenu
@end menu
-@node Since 250
-@chapter Since 2.5.0
+@node 270
+@chapter 2.7.0
+
+@itemize
+@item
+New entry point: @samp{el-job-parallel-mapcar}! Read more below.
+@item
+Argument @samp{:inputs} are now split differently@comma{} for the sake of a
predictable order of outputs. It is likely less optimal@comma{} but hopefully
not too much.
+@item
+Argument @samp{:require} now accepts strings in addition to symbols. Strings
are passed to @samp{load} instead of @samp{require}.
+@item
+Value of @samp{temporary-file-directory} no longer added to
@samp{:inject-vars} for you.
+@itemize
+@item
+Now the only values added are @samp{load-path} and
@samp{native-comp-eln-load-path}.
+@end itemize
+@item
+Function @samp{el-job-ng-job} renamed to @samp{el-job-ng-get-job}.
+@itemize
+@item
+New class @samp{el-job-ng-job}.
+@end itemize
+@item
+Delete many aliases to el-job-old.el.
+@end itemize
+
+@menu
+* New entry point @samp{el-job-parallel-mapcar}::
+@end menu
+
+@node New entry point @samp{el-job-parallel-mapcar}
+@section New entry point: @samp{el-job-parallel-mapcar}
+
+Until now@comma{} we had to make do with an unwieldy @samp{el-job-ng-run} with
~5 keyword arguments. And we had to understand asynchronous programming to use
it.
+
+I've long dreamed to be able to brainlessly rewrite a form @samp{(mapcar #'FN
INPUTS)} to something like @samp{(multicore-mapcar #'FN INPUTS)} and have it
Just Work and Just Be Faster.
+
+Finally@comma{} it is done!
+
+The calling convention is
+
+@example
+(el-job-parallel-mapcar FN INPUTS &optional INJECT-VARS)
+@end example
+
+Please see the docstring for what you need to know.
+
+@menu
+* Weaknesses::
+@end menu
+
+@node Weaknesses
+@subsection Weaknesses
+
+Do not use @samp{el-job-parallel-mapcar} with overly trivial functions. It
adds some overhead per item@comma{} so it actually slows you down if the
function itself only takes microseconds or nanoseconds per invocation.
+
+Example of a bad use-case:
+
+@lisp
+(let ((spam (make-list 1000000 "spam")))
+ (list (benchmark-elapse (mapcar #'upcase spam))
+ (benchmark-elapse (el-job-parallel-mapcar #'upcase spam))))
+@end lisp
+
+Return value:
+
+@example
+(0.482017027 4.793430311)
+@end example
+
+Another bad use-case is when plain @samp{mapcar} would've been fast enough.
There is some constant overhead related to spinning up subprocesses.
+
+Changing 1000000 from the earlier expression to just 100 reveals the constant
to be ~210ms on my machine:
+
+@lisp
+(let ((spam (make-list 100 "spam")))
+ (list (benchmark-elapse (mapcar #'upcase spam))
+ (benchmark-elapse (el-job-parallel-mapcar #'upcase spam))))
+@end lisp
+
+Return value:
+
+@example
+(0.000041974 0.219570936)
+@end example
+
+@node 260
+@chapter 2.6.0
+
+@itemize
+@item
+For @samp{el-job-old-launch}@comma{} new argument @samp{:eval}
+@end itemize
+
+@node 250
+@chapter 2.5.0
-Released @emph{<2025-Oct-06>}@comma{} v2.5.0 comes with a variant library
"el-job-ng".
+Released @emph{[2025-10-06 Mon]}@comma{} v2.5.0 comes with a variant library
"el-job-ng".
I find it simpler and easier to reason about. 400 lines of code instead of
800.
@@ -108,7 +212,7 @@ I may write yet another variant.
Something that came with experience is that it's best to make a new variant
library for a narrow use-case@comma{} rather than complicate one library with
different code flows. When it comes to this type of library@comma{} you really
want to keep it easy to reason about!
-Ideas as of @emph{<2025-Oct-06>}:
+Ideas as of @emph{[2025-10-06 Mon]}:
@table @asis
@item File IPC
