Re: "deprecated recursive use" warning

[Faré]

On Wed, Mar 21, 2018 at 7:25 AM, Attila Lendvai  wrote:
> yeah, it feels like a lot of pain. it would be nice if there was a
> fork-like API in ASDF for implementing such exec'd compilation, but
> then i guess ASDF itself has no clue which /foo/bin/ directory has the
> underlying lisp binary, and/or which lisp to launch at all, let alone
> how...
>
Well, if you want to fork+exec a subprocess, then asdf:run-program does it,
and lisp-invocation can find the lisp binary,
at least on a lot of implementations, and/or with user help.

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
Where did the water from the Deluge come from, and where did it go?

Re: "deprecated recursive use" warning

[Robert P. Goldman]

You're welcome, but really the intuition is due to Drew McDermott, I think in 
his ILC 2005 paper, which I recommend. (He describes a system like ASDF, but 
finer-grained, so that even individual data structures can be updated).

This is key to understanding some of the ways ASDF is NOT like "make."

Best,
R

Sent from my iPad

On Mar 21, 2018, at 06:25, Attila Lendvai  wrote:

>> I think the Right Thing is to realize that what ASDF does is not so much
>> transform files, as to maintain the consistency of the running lisp image.
>> Now, in no case does the running lisp image need either the JSON generation
>> system or the JSON to lisp translation system. All the running lisp image
>> needs to function correctly is an up-to-date lisp file produced by this
>> pipeline. So
> 
> 
> this is a very useful insight, thanks for formulating it explicitly!
> it's a good candidate for an introductory bird's eye view section of
> the manual.
> 
> i already made the JSON generation a separate operation, and now i'm
> considering to do the same with the JSON -> lisp generation.
> 
> in a way it makes sense: the lib author tested the code that he
> saw. if the lisp generation gets changed/fixed (by a new patch to
> CFFI), it may even break the rest of the lib that may be running with
> some old assumptions... so, maybe it's better to just make both of
> those as a separate process that needs to be initiated by the lib
> author explicitly, and then checked into the git history.
> 
> 
>> Once you think about what ASDF does and doesn't do, I think this makes
>> perfect sense. But, of course, it might be a big pain to do so.
> 
> 
> yeah, it feels like a lot of pain. it would be nice if there was a
> fork-like API in ASDF for implementing such exec'd compilation, but
> then i guess ASDF itself has no clue which /foo/bin/ directory has the
> underlying lisp binary, and/or which lisp to launch at all, let alone
> how...
> 
> -- 
> • attila lendvai
> • PGP: 963F 5D5F 45C7 DFCD 0A39
> --
> “Far from helping students to develop into mature, self-reliant,
> self-motivated individuals, schools seem to do everything they can to
> keep youngsters in a state of chronic, almost infantile, dependency.
> The pervasive atmosphere of distrust, together with rules covering the
> most minute aspects of existence, teach students every day that they
> are not people of worth, and certainly not individuals capable of
> regulating their own behavior.”
> — Alfie Kohn, 'Punished by Rewards: The Trouble with Gold Stars,
> Incentive Plans, A's, Praise, and Other Bribes'
> 

Re: "deprecated recursive use" warning

[Attila Lendvai]

> I think the Right Thing is to realize that what ASDF does is not so much
> transform files, as to maintain the consistency of the running lisp image.
> Now, in no case does the running lisp image need either the JSON generation
> system or the JSON to lisp translation system. All the running lisp image
> needs to function correctly is an up-to-date lisp file produced by this
> pipeline. So


this is a very useful insight, thanks for formulating it explicitly!
it's a good candidate for an introductory bird's eye view section of
the manual.

i already made the JSON generation a separate operation, and now i'm
considering to do the same with the JSON -> lisp generation.

in a way it makes sense: the lib author tested the code that he
saw. if the lisp generation gets changed/fixed (by a new patch to
CFFI), it may even break the rest of the lib that may be running with
some old assumptions... so, maybe it's better to just make both of
those as a separate process that needs to be initiated by the lib
author explicitly, and then checked into the git history.


> Once you think about what ASDF does and doesn't do, I think this makes
> perfect sense. But, of course, it might be a big pain to do so.


yeah, it feels like a lot of pain. it would be nice if there was a
fork-like API in ASDF for implementing such exec'd compilation, but
then i guess ASDF itself has no clue which /foo/bin/ directory has the
underlying lisp binary, and/or which lisp to launch at all, let alone
how...

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Far from helping students to develop into mature, self-reliant,
self-motivated individuals, schools seem to do everything they can to
keep youngsters in a state of chronic, almost infantile, dependency.
The pervasive atmosphere of distrust, together with rules covering the
most minute aspects of existence, teach students every day that they
are not people of worth, and certainly not individuals capable of
regulating their own behavior.”
— Alfie Kohn, 'Punished by Rewards: The Trouble with Gold Stars,
Incentive Plans, A's, Praise, and Other Bribes'

Re: "deprecated recursive use" warning

[Robert Goldman]

On 30 Jan 2018, at 15:53, Attila Lendvai wrote:


I haven't used CFFI in a while.


TL;DR: is this a sane fix?

https://github.com/cffi/cffi/commit/4b9b06f15912e823581b1aeb8a0d5c2ef11f702d

--
and here follows the elaborate email that led me to find the above
solution:

a bit of background: it's a subsystem of CFFI that generates the CFFI
bindings from a json file, that is in turn autogenerated from the C
source.

so,

 1) C -> json
 2) json -> lisp (CFFI definitions)
 3) asdf compiles/loads the generated lisp file

1) requires a heavyweight infrastructure (a binary run by
run-program), so there's support to do that lazily, and just
distribute the json files generated once by the lib author.

2) is relatively lightweight, but it still requires loading a broader
scope of lisp dependencies, so there's support for the lib author to
run the generation and distribute the generated lisp file.

now, whether the generator code (extra dependencies) is needed is
decided by whether or not the generated lisp file is up-to-date.

unfortunately i cannot test it properly because of another bug/change
that i'll report in a separate thread.


Thinking out loud:

I'll have a bit more of a look.  IIUC what you are saying is that there 
should be an OP that covers the C -> JSON translation, and that requires 
some infrastructure that you don't want to load unless it's necessary.


Then there's JSON -> lisp that has additional dependencies that are 
optional.


The problem I see is that, IIRC, ASDF creates a build plan that is 
*unconditional*, and then *executes* it conditionally, by calling 
`OPERATION-DONE-P`, and skipping unnecessary operations.  It does this 
because earlier steps in the process can change what `OPERATION-DONE-P` 
would return for later stages in plan execution.


The problem with this is that you want what comes *later* in the plan 
(whether or not the lisp derived from JSON) to affect what comes 
*earlier* in the plan (whether or not the JSON to lisp translation 
library gets loaded).


I think I know what is the Right Answer to this, but it might be so much 
more work that you would rather just keep your current hack working


I think the Right Thing is to realize that what ASDF does is not so much 
transform files, as to maintain the consistency of the running lisp 
image.  Now, in no case does the running lisp image need *either* the 
JSON generation system *or* the JSON to lisp translation system.  All 
the *running lisp image* needs to function correctly is an up-to-date 
lisp file produced by this pipeline.  So


The Right Thing is to kick that pipeline out of the current lisp 
process.  Instead of making ASDF use its `LOAD-OP`, etc. to do this 
translation for you, you should just create an external application 
(which might be a lisp program) to do the JSON generation (if needed) 
and the JSON to lisp translation (if needed).  That external program 
could well use ASDF to manage itself.  But all that would be in your 
*main* ASDF system definition would be a `JSON-GEN-OP` and a 
`JSON-TO-LISP-OP`, each of which would be implemented by invoking an 
external program.  As in, for example

```
(DEFMETHOD PERFORM ((OP JSON-GEN-OP) (c json-component))
   (uiop:run-program ))
```

Once you think about what ASDF does and doesn't do, I think this makes 
perfect sense.  But, of course, it might be a big pain to do so.


Best,
R

Re: "deprecated recursive use" warning

[Faré]

On Tue, Jan 30, 2018 at 4:53 PM, Attila Lendvai  wrote:
>> I haven't used CFFI in a while.
>
> TL;DR: is this a sane fix?
>
> https://github.com/cffi/cffi/commit/4b9b06f15912e823581b1aeb8a0d5c2ef11f702d
>
(the (not null) ...) is redundant around find-system without the nil
optional argument.

Also, having a component depend on a system violates a weak tacit
invariant in ASDF, though it should only matter for bundle operations,
and only for things required at link-time, which shouldn't be the case
here. Still, something to keep in mind.

> now, whether the generator code (extra dependencies) is needed is
> decided by whether or not the generated lisp file is up-to-date.
>
... except that ASDF only considers it "up-to-date" if its date is
later than that of all dependencies, including the compiled objects
for c2ffi, now that staging is done properly.

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
Eric Schmidt rule of management: always assume that
the other person is as intelligent as you are.

Re: "deprecated recursive use" warning

[Attila Lendvai]

> I haven't used CFFI in a while.

TL;DR: is this a sane fix?

https://github.com/cffi/cffi/commit/4b9b06f15912e823581b1aeb8a0d5c2ef11f702d

--
and here follows the elaborate email that led me to find the above
solution:

a bit of background: it's a subsystem of CFFI that generates the CFFI
bindings from a json file, that is in turn autogenerated from the C
source.

so,

 1) C -> json
 2) json -> lisp (CFFI definitions)
 3) asdf compiles/loads the generated lisp file

1) requires a heavyweight infrastructure (a binary run by
run-program), so there's support to do that lazily, and just
distribute the json files generated once by the lib author.

2) is relatively lightweight, but it still requires loading a broader
scope of lisp dependencies, so there's support for the lib author to
run the generation and distribute the generated lisp file.

now, whether the generator code (extra dependencies) is needed is
decided by whether or not the generated lisp file is up-to-date.

unfortunately i cannot test it properly because of another bug/change
that i'll report in a separate thread.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Enlightenment is a destructive process. It has nothing to do with
becoming better or being happier. Enlightenment is the crumbling away
of untruth. It's seeing through the facade of pretence. It's the
complete eradication of everything we imagined to be true.”
— Adyashanti (1962–)

Re: "deprecated recursive use" warning

[Robert Goldman]

I haven't used CFFI in a while.  Can I ask why this needs to be here 
instead of there being a


```
:depends-on ((generate-lisp-op (load-op "cffi/c2ffi-generator")))
```
?

Thanks,
r


On 30 Jan 2018, at 12:20, Attila Lendvai wrote:


dear list,

shall i be concerned about this?

WARNING:
   Deprecated recursive use of (ASDF/OPERATE:OPERATE
'ASDF/LISP-ACTION:LOAD-OP '("cffi/c2ffi-generator")) while
   visiting (CFFI/C2FFI::GENERATE-LISP-OP "hu.dwim.zlib" "c2ffi-spec"
"zlib.h") - please use proper dependencies instead

it happens due to this in cffi/src/c2ffi/asdf.lisp:

(defmethod perform ((op generate-lisp-op) (c c2ffi-file))
...
  (unless (component-loaded-p :cffi/c2ffi-generator)
(load-system :cffi/c2ffi-generator))
...
)

that system only needs to get loaded when the action is executed.

or is it safe to ignore that warning long term?

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“If we have the truth, it cannot be harmed by investigation. If we
have not the truth, it ought to be harmed.”
— J. Reuben Clark (1871–1961)

"deprecated recursive use" warning

[Attila Lendvai]

dear list,

shall i be concerned about this?

WARNING:
   Deprecated recursive use of (ASDF/OPERATE:OPERATE
'ASDF/LISP-ACTION:LOAD-OP '("cffi/c2ffi-generator")) while
   visiting (CFFI/C2FFI::GENERATE-LISP-OP "hu.dwim.zlib" "c2ffi-spec"
"zlib.h") - please use proper dependencies instead

it happens due to this in cffi/src/c2ffi/asdf.lisp:

(defmethod perform ((op generate-lisp-op) (c c2ffi-file))
...
  (unless (component-loaded-p :cffi/c2ffi-generator)
(load-system :cffi/c2ffi-generator))
...
)

that system only needs to get loaded when the action is executed.

or is it safe to ignore that warning long term?

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“If we have the truth, it cannot be harmed by investigation. If we
have not the truth, it ought to be harmed.”
— J. Reuben Clark (1871–1961)