Re: Define a Simple Echo-Op
> > > ASDF as it exists now is not capable of running anything in parallel. > Faré has an experimental project to do this, called POIU, but I > > can't speak for its status. Until very recently, SBCL was incapable of > compiling files in parallel, and still today, lots of code does > > not-thread-safe stuff at load-time, so I'm not convinced it's > practically possible to make ASDF parallel via shared-memory threaded > > concurrency. I believe POIU compiles each component in a separate > process, then loads them sequentially. > > I see, that's good to know. At least it's an implementation limitation > and not a design limitation I guess. > POIU compiles files in a fork, on supported implementations that it knows how to fork on (SBCL, CCL with a patch, CLISP, ACL). It used to work well, but it wasn't updated to take into account the new planning infrastructure of ASDF 3.3, and requires some love. It looks like it's still working, but it's probably not doing all the right things across build phases. Good luck untangling it. -#f
Re: Define a Simple Echo-Op
Hi phoebe, > [[PGP Signed Part:Undecided]] > I'll still get the warnings (even with one new warning: "No dependency > propagating scheme specified..."): > > Perhaps it was imprecise of me to say that defining a method on > COMPONENT-DEPENDS-ON will prevent the deprecation warning. > Causing the default method to be invoked on an OPERATION that is not a > direction-OPERATION is deprecated; defining a more > specific method which does CALL-NEXT-METHOD still invokes the deprecated > behavior. > > What I meant was subclassing ASDF:OPERATION only > > You can subclass OPERATION without a direction-OPERATION, but you have to > completely override COMPONENT-DEPENDS-ON > with your own behavior, never invoking the built-in method. I strongly > recommend against doing this. > > So, all operations are by default downward and sideway unless they are > subclass of (OR DOWNWARD-OPERATION UPWARD-OPERATION SIDEWAY-OPERATION > SELFWARD-OPERATION NON-PROPAGATING-OPERATION) > > Er... I'm not comfortable with saying that. In ASDF 2, all operations behaved > in a way that is now called being downward and > sideway. ASDF 3 treats operations which don't specify their dependency > relationship in a way compatible with ASDF 2 to give > maintainers a chance to update old extensions. The beautiful dream is that, > one day, the compatibility behavior will be replaced > with a hard error. Please do not treat the compatibility behavior as a part > of ASDF 3's interface. > >From our exchange, I think its safe to summarise into this table: "Table of ASDF operation classes to subclass and the corresponding methods to define on your custom class." | | OPERATION | -OPERATION| *-op | |--++--+--| | perform | X | X| X| | component-depends-on | X (note 1) | | | | input-files | X | (Depending on your need) | | | output-files | X | (Depending on your need) | | Note 1: You have to completely override COMPONENT-DEPENDS-ON for your custom class without invoking (CALL-NEXT-METHOD). As a new ASDF extension writer, I think having a table like this helps a lot to easily getting started. > Is it a linear chain or a "network chain" of > operations? > > The latter. When you define a subclass of SELFWARD-OPERATION, you specify > what other operation(s) it is selfward to by > overriding the :INITFORM of the slot SELFWARD-OPERATION. The value you > provide may be either a single operation designator, > or a list of operation designators. For example, (LOAD-OP COMPONENT) depends > on both (COMPILE-OP COMPONENT) and > (PREPARE-OP COMPONENT), so its SELFWARD-OPERATION slot is defined: > > (selfward-operation :initform '(prepare-op compile-op) :allocation :class) > > can be done in parallel, > > But how can I > express the operation dependencies such that ASDF knows PRINT-OP depends > on ECHO-OP? > > (defclass print-op (asdf:selfward-operation) > ((asdf:selfward-operation :initform 'echo-op :allocation :class))) > Thanks! I have tried it out and it works. Still need some time to understand its behaviour though. > ASDF as it exists now is not capable of running anything in parallel. Faré > has an experimental project to do this, called POIU, but I > can't speak for its status. Until very recently, SBCL was incapable of > compiling files in parallel, and still today, lots of code does > not-thread-safe stuff at load-time, so I'm not convinced it's practically > possible to make ASDF parallel via shared-memory threaded > concurrency. I believe POIU compiles each component in a separate process, > then loads them sequentially. I see, that's good to know. At least it's an implementation limitation and not a design limitation I guess. -- Regards, zacque
Re: Define a Simple Echo-Op
(Ignore my previous email with the wrong "From:" sender field)
> For some frustrating reason I can't email the asdf-devel list, so cc'ing
> rpgoldman & he can forward.
Noted, I'll reply to rpgoldman and cc you then (for better thread viewing?).
> The various UPWARD- DOWNWARD- etc -OPERATION classes are subclasses of
> OPERATION. You can explicitly add OPERATION to your superclasses list
> alongside SIDEWAY-OPERATION if you want to, but it's redundant.
What I meant was subclassing ASDF:OPERATION only, as shown in the
following code.
> In ASDF 2, the direction-OPERATION classes didn't exist; you just
> subclassed OPERATION directly. All operations were implicitly DOWNWARD
> and SIDEWAY unless you wrote your own COMPONENT-DEPENDS-ON
> method. ASDF 3 added some built-in operations that were not
> SIDEWAY/DOWNWARD (PREPARE-OP is UPWARD, for example), so Faré codified
> all the sane dependency relationships as the direction-OPERATION
> classes, and deprecated the default COMPONENT-DEPENDS-ON method for
> OPERATION.
I see, thank you for your explanation.
> In ASDF 3, if you subclass OPERATION directly and don't
> define a method on COMPONENT-DEPENDS-ON, you get a deprecation
> warning.
This is not true. For my ASDF version "3.3.5.7". If I do this:
--- file: echo-op.lisp ---
(in-package #:echo-op)
(defclass echo-op (asdf:operation)
())
(defmethod asdf:component-depends-on ((op echo-op) c)
(call-next-method))
(defmethod asdf:perform ((op echo-op) c)
(format t "~ ~a on component ~a depends on ~{~% ~a~}~%"
op c (asdf:component-depends-on op c)))
-
I'll still get the warnings (even with one new warning: "No dependency
propagating scheme specified..."):
- REPL --
CL-USER> (asdf:operate 'echo-op:echo-op :echo-op-test :force t)
WARNING:
No dependency propagating scheme specified for operation class ECHO-OP:ECHO-OP.
The class needs to be updated for ASDF 3.1 and specify appropriate propagation
mixins.
WARNING:
DEPRECATED-FUNCTION-WARNING: Using deprecated function
(ASDF/ACTION::BACKWARD-COMPATIBLE-DEPENDS-ON
:FOR-OPERATION
#) --
please update your code to use a newer API.
WARNING:
DEPRECATED-FUNCTION-WARNING: Using deprecated function
(ASDF/ACTION::BACKWARD-COMPATIBLE-DEPENDS-ON
:FOR-OPERATION
#) --
please update your code to use a newer API.
WARNING:
DEPRECATED-FUNCTION-WARNING: Using deprecated function
(ASDF/ACTION::BACKWARD-COMPATIBLE-DEPENDS-ON
:FOR-OPERATION
#) --
please update your code to use a newer API.
WARNING:
DEPRECATED-FUNCTION-WARNING: Using deprecated function
(ASDF/ACTION::BACKWARD-COMPATIBLE-DEPENDS-ON
:FOR-OPERATION
#) --
please update your code to use a newer API.
Operation # on component #
depends on
(#)
Operation # on component #
depends on
(#)
Operation # on component #
depends on
(#)
Operation # on component # depends on
(DEFINE-OP echo-op-test)
(#)
(# #
#
#)
#
#
-
Note:
1. "No dependency propagating scheme specified..." warning only occurs
once with a fresh Lisp image. It won't occur again with OPERATE :force t
or :force :all.
>> I'm following the manual: "If the action of performing the operation on a
>> component has
>> dependencies, you must define a method on component-depends-on."
>
>
> This is wrong; I don't know why the manual says that. You still can
> define a method on COMPONENT-DEPENDS-ON, but there's no reason to
> unless you have a really weird operation. Writing your own
> COMPONENT-DEPENDS-ON methods is error-prone and a pain in the ass, so
> it's much easier to subclass one of the direction-DEPENDS-ON methods
> and get a built-in method that does the right thing.
>
>> But still I don't get why echo-op depends on itself. I didn't specify it
>> anyway. Or it that the default behaviour for all *-operation classes?
>
>
> Honestly, I have no clue why there's an (ECHO-OP . NIL) pair at the
> head of the COMPONENT-DEPENDS-ON list for an ECHO-OP. But
> MAP-DIRECT-DEPENDENCIES ignores it. Shrug.
>
>> "A method for [INPUT-FILES] is often not needed, since ASDF has a pretty
>> clever default input-files mechanism."
>
>
> Again, I don't know why the manual says that. You don't need to define
> INPUT-FILES methods for built-in operations, but you will need to
> define them for custom operations that take files as input. Most
> user-defined operations subclass either COMPILE-OP or LOAD-OP, which
> already have their
Re: Define a Simple Echo-Op
On 27 Apr 2022, at 11:15, phoebe Goldman wrote: >> According to the docs, most operations inherit from self ward-operation, >> which would take the component-pathname as an input file (and might take >> others). > > I think input file sets are orthogonal to dependency direction. > SELFWARD-OPERATION means that (OP COMPONENT) depends on (DIFFERENT-OP > COMPONENT), like how (LOAD-OP LISP-SOURCE-FILE) depends on (COMPILE-OP > LISP-SOURCE-FILE). That doesn't mean that (LOAD-OP LISP-SOURCE-FILE) takes > COMPONENT-PATHNAME as an INPUT-FILE. This is what I see in the source, but I haven't had enough time to read it over carefully (or really, at all): ``` ;; An action with a selfward-operation by default gets its input-files from the output-files of ;; the actions using selfward-operations it depends on (and the same component), ;; or if there are none, on the component-pathname of the component if it's a file ;; -- and then on the results of the next-method. (defmethod input-files ((o selfward-operation) (c component)) `(,@(or (loop :for dep-o :in (ensure-list (selfward-operation o)) :append (or (output-files dep-o c) (input-files dep-o c))) (if-let ((pathname (component-pathname c))) (and (file-pathname-p pathname) (list pathname ,@(call-next-method ``` signature.asc Description: OpenPGP digital signature
Re: Define a Simple Echo-Op
A quick note to point everyone at the doc fixes: https://gitlab.common-lisp.net/asdf/asdf/-/merge_requests/207 signature.asc Description: OpenPGP digital signature
Re: Define a Simple Echo-Op
On 27 Apr 2022, at 10:26, Robert Goldman wrote: > Phoebe Goldman writes (but can't post): > >> I'm following the manual: "If the action of performing the operation on a >> component has >> dependencies, you must define a method on component-depends-on." > > > This is wrong; I don't know why the manual says that. You still can define a > method on COMPONENT-DEPENDS-ON, but there's no reason to unless you have a > really weird operation. Writing your own COMPONENT-DEPENDS-ON methods is > error-prone and a pain in the ass, so it's much easier to subclass one of the > direction-DEPENDS-ON methods and get a built-in method that does the right > thing. I will fix the manual RN. > >> But still I don't get why echo-op depends on itself. I didn't specify it >> anyway. Or it that the default behaviour for all *-operation classes? > > > Honestly, I have no clue why there's an (ECHO-OP . NIL) pair at the head of > the COMPONENT-DEPENDS-ON list for an ECHO-OP. But MAP-DIRECT-DEPENDENCIES > ignores it. Shrug. > >> "A method for [INPUT-FILES] is often not needed, since ASDF has a pretty >> clever default input-files mechanism." > > > Again, I don't know why the manual says that. You don't need to define > INPUT-FILES methods for built-in operations, but you will need to define them > for custom operations that take files as input. Most user-defined operations > subclass either COMPILE-OP or LOAD-OP, which already have their INPUT-FILES > (and OUTPUT-FILES) methods, but I've had to define INPUT-FILES methods in the > past. I will update that, too. TBH, I'm not sure that this is up-to-date with Fare's refactoring that introduced the `additional-input-files` generic function, but I don't have enough time today to dig into this burger. >> Is there a way to obtain the pathname to (:FILE "foo") from the >> CL-SOURCE-FILE object like what COMPILE-OP does? > > > COMPONENT-PATHNAME. > >> Why do I get nothing for my input-files? I'm expecting it to print out >> the pathname to (:FILE "foo") object, very much like the input files >> into COMPILE-OP. > > Taking the COMPONENT-PATHNAME as the sole INPUT-FILE can't be the default, > because lots of operations do something different. If you think of > PREPARE-OP, COMPILE-OP and LOAD-OP as being the "normal" operations on a > LISP-SOURCE-FILE, only COMPILE-OP takes the source file as an INPUT-FILE. > LOAD-OP takes as an INPUT-FILE the compiled fasl, which is the OUTPUT-FILES > of COMPILE-OP. I don't think PREPARE-OP takes any INPUT-FILES at all, but I > could be wrong. According to the docs, most operations inherit from `self ward-operation`, which would take the component-pathname as an input file (and might take others). If this is wrong, please let me know and I will adjust the documentation. Getting more of this discussion into the manual would be great, since there's lots of good guidance here. signature.asc Description: OpenPGP digital signature
Re: Define a Simple Echo-Op
Phoebe Goldman writes (but can't post): For some frustrating reason I can't email the asdf-devel list, so cc'ing rpgoldman & he can forward. > Why can't I subclass ASDF:OPERATION? This is unexpected since > ASDF:OPERATION is the base class for all operations. The various UPWARD- DOWNWARD- etc -OPERATION classes are subclasses of OPERATION. You can explicitly add OPERATION to your superclasses list alongside SIDEWAY-OPERATION if you want to, but it's redundant. In ASDF 2, the direction-OPERATION classes didn't exist; you just subclassed OPERATION directly. All operations were implicitly DOWNWARD and SIDEWAY unless you wrote your own COMPONENT-DEPENDS-ON method. ASDF 3 added some built-in operations that were not SIDEWAY/DOWNWARD (PREPARE-OP is UPWARD, for example), so Faré codified all the sane dependency relationships as the direction-OPERATION classes, and deprecated the default COMPONENT-DEPENDS-ON method for OPERATION. In ASDF 3, if you subclass OPERATION directly and don't define a method on COMPONENT-DEPENDS-ON, you get a deprecation warning. > I'm following the manual: "If the action of performing the operation on a > component has > dependencies, you must define a method on component-depends-on." This is wrong; I don't know why the manual says that. You still can define a method on COMPONENT-DEPENDS-ON, but there's no reason to unless you have a really weird operation. Writing your own COMPONENT-DEPENDS-ON methods is error-prone and a pain in the ass, so it's much easier to subclass one of the direction-DEPENDS-ON methods and get a built-in method that does the right thing. > But still I don't get why echo-op depends on itself. I didn't specify it > anyway. Or it that the default behaviour for all *-operation classes? Honestly, I have no clue why there's an (ECHO-OP . NIL) pair at the head of the COMPONENT-DEPENDS-ON list for an ECHO-OP. But MAP-DIRECT-DEPENDENCIES ignores it. Shrug. > "A method for [INPUT-FILES] is often not needed, since ASDF has a pretty > clever default input-files mechanism." Again, I don't know why the manual says that. You don't need to define INPUT-FILES methods for built-in operations, but you will need to define them for custom operations that take files as input. Most user-defined operations subclass either COMPILE-OP or LOAD-OP, which already have their INPUT-FILES (and OUTPUT-FILES) methods, but I've had to define INPUT-FILES methods in the past. > Is there a way to obtain the pathname to (:FILE "foo") from the > CL-SOURCE-FILE object like what COMPILE-OP does? COMPONENT-PATHNAME. > Why do I get nothing for my input-files? I'm expecting it to print out > the pathname to (:FILE "foo") object, very much like the input files > into COMPILE-OP. Taking the COMPONENT-PATHNAME as the sole INPUT-FILE can't be the default, because lots of operations do something different. If you think of PREPARE-OP, COMPILE-OP and LOAD-OP as being the "normal" operations on a LISP-SOURCE-FILE, only COMPILE-OP takes the source file as an INPUT-FILE. LOAD-OP takes as an INPUT-FILE the compiled fasl, which is the OUTPUT-FILES of COMPILE-OP. I don't think PREPARE-OP takes any INPUT-FILES at all, but I could be wrong. > One question: How do you know whether to define PERFORM primary method > or an :AROUND method for the custom operation class? It's not obvious to > me which one to choose for different use cases. Write an :AROUND method if you're subclassing an existing operation like LOAD-OP to modify its behavior, and you intend to use CALL-NEXT-METHOD to invoke the default behavior for your superclass at some point in your method. This is useful if you want to invoke the next method in a dynamic context e.g. with handlers or restarts bound, or with local bindings of special variables; or if you want to conditionally invoke the next method sometimes but not always. Actually, you can do all of these things in a primary method via CALL-NEXT-METHOD, so now that I think of it, just define a primary method. Write an :AROUND method only if you're defining a mix-in class that other people will subclass which wants to do those things. If you're subclassing an existing operation and you want to add additional behavior, but you still want the default behavior to run unconditionally in a default dynamic context, define a :BEFORE or :AFTER method as appropriate. Write a primary method if you're defining a new method, or if you want to completely override the behavior of an existing operation. Or if you want to do the things I said earlier with CALL-NEXT-METHOD. When in doubt, write a primary method. cheers, phoebe > On Apr 27, 2022, at 3:21 AM, zacque wrote: > > > Hi phoebe, > > Thanks for your thoughtful reply! > >> A few things. >> >> 1. ECHO-OP should not be selfward. SELFWARD-OPERATION is for (OPERATION >> COMPONENT) pairs which depend on >> (DIFFERENT-OPERATION COMPONENT) for the same COMPONENT. For example, LOAD-OP >> is selfward with
Re: Define a Simple Echo-Op
Hi phoebe,
Thanks for your thoughtful reply!
> A few things.
>
> 1. ECHO-OP should not be selfward. SELFWARD-OPERATION is for (OPERATION
> COMPONENT) pairs which depend on
> (DIFFERENT-OPERATION COMPONENT) for the same COMPONENT. For example, LOAD-OP
> is selfward with respect to
> COMPILE-OP, because in order to perform (load-op FILE), you must first
> perform (compile-op FILE). Your ECHO-OP has no such
> dependency. In this case, I believe you want ECHO-OP to be downward and
> sideways, meaning that (ECHO-OP MODULE)
> depends on (ECHO-OP MODULE-COMPONENT) for each of the children
> MODULE-COMPONENTs of the MODULE, and that
> (ECHO-OP SOURCE-FILE) depends on (ECHO-OP EARLIER-SOURCE-FILE) for each of
> the EARLIER-SOURCE-FILEs in
> SOURCE-FILE's :DEPENDS-ON list. This way, when you call (OPERATE 'ECHO-OP
> (FIND-SYSTEM "whatever")), ASDF will do a
> depth-first dependency-order traversal of your system.
After playing a little with downward and sideway, I can observe that
downward-operation provides a depth-search/"postorder" traversal for the
current system; sideway-operation provides a dependency-first traversal
for dependent systems.
* Note:
Use :force all with OPERATE to force reloading all dependent
systems. E.g. (asdf:operate 'echo-op:echo-op :echo-op-test :force :all)
* Question:
Why can't I subclass ASDF:OPERATION? This is unexpected since
ASDF:OPERATION is the base class for all operations.
My code:
--- file: echo-op.lisp ---
(in-package #:echo-op)
(defclass echo-op (asdf:operation) ())
(defmethod asdf:perform ((op echo-op) c)
(format t "~ ~a on component ~a has input files:~{~% ~a~}~%"
op c (asdf:input-files op c)))
-
Output:
- REPL --
CL-USER> (asdf:operate 'echo-op:echo-op :echo-op-test :force t)
WARNING:
DEPRECATED-FUNCTION-WARNING: Using deprecated function
(ASDF/ACTION::BACKWARD-COMPATIBLE-DEPENDS-ON
:FOR-OPERATION
#) --
please update your code to use a newer API.
WARNING:
DEPRECATED-FUNCTION-WARNING: Using deprecated function
(ASDF/ACTION::BACKWARD-COMPATIBLE-DEPENDS-ON
:FOR-OPERATION
#) --
please update your code to use a newer API.
WARNING:
DEPRECATED-FUNCTION-WARNING: Using deprecated function
(ASDF/ACTION::BACKWARD-COMPATIBLE-DEPENDS-ON
:FOR-OPERATION
#) --
please update your code to use a newer API.
WARNING:
DEPRECATED-FUNCTION-WARNING: Using deprecated function
(ASDF/ACTION::BACKWARD-COMPATIBLE-DEPENDS-ON
:FOR-OPERATION
#) --
please update your code to use a newer API.
Operation # on component #
has input files:
Operation # on component #
has input files:
Operation # on component # has
input files:
Operation # on component # has input files:
#
#
-
> 2. Your COMPONENT-DEPENDS-ON method is wrong. No pair of (OPERATION
> COMPONENT) should ever depend on the same
> (OPERATION COMPONENT). What you're saying is, "in order to perform (ECHO-OP
> FILE), you must first perform (ECHO-OP
> FILE)."
>
> 3. For operations that subclass one or more of DOWNWARD- UPWARD- SIDEWAY-
> SELFWARD- or
> NON-PROPOGATING-OPERATION, you don't need to define a COMPONENT-DEPENDS-ON
> method.
Why is my COMPONENT-DEPENDS-ON method wrong? It's not obvious that one
doesn't need to define a COMPONENT-DEPENDS-ON method for >=1
*-OPERATION. Is it stated anywhere?
I'm following the manual: "If the action of performing the operation on a
component has
dependencies, you must define a method on component-depends-on." and following
the examples in cffi-grovel:
https://github.com/cffi/cffi/blob/3c76afe7ba03ce015e0df99ac9ddcd61320a44a4/grovel/asdf.lisp#L66
That said, my code works fine by removing my COMPONENT-DEPENDS-ON
method. And I can see what you meant by (ECHO-OP FILE) depending on
itself by printing it out:
My code:
--- file: echo-op.lisp ---
(in-package #:echo-op)
(defclass echo-op (asdf:sideway-operation asdf:downward-operation) ())
(defmethod asdf:perform ((op echo-op) c)
(format t "~ ~a on component ~a depends on ~{~% ~a~}~%"
op c (asdf:component-depends-on op c)))
-
Output:
- REPL --
CL-USER> (asdf:operate 'echo-op:echo-op :echo-op-test :force t)
Operation # on component #
depends on
(#)
Operation # on component #
depends on
(#)
Operation # on component #
depends on
(#)
Operation # on component # depends on
(#)
(# #
#
