Re: Multiple processing compiling the same file

[Robert Goldman]

Sorry for the late response.

What you have seems like it will work, but couldn't you just as easily 
use the ASDF output translations configuration facility that is 
described here: 
https://common-lisp.net/project/asdf/asdf/Controlling-where-ASDF-saves-compiled-files.html#Controlling-where-ASDF-saves-compiled-files


There's an example there that you could probably adapt by inserting 
`uid` and `pid`, as appropriate.


Best,
r


On 23 Jan 2018, at 6:51, Jim Newton wrote:

Apparently, this approach seems to work.   I’m not sure if it is the 
best approach.
Here is what my code looks like.  It creates a directory in /tmp/ and 
asdf:load-system

seems to compile the .fasl files into there.


(require :asdf)
(require :sb-posix)
(let ((home (directory-namestring (user-homedir-pathname)))
  (uid (sb-posix:getuid))
  (pid  (sb-posix:getpid)))
  (setf asdf::*user-cache* (ensure-directories-exist (format nil 
"/tmp~A~D/~D/" home uid pid


#-quicklisp
(let ((quicklisp-init
"/lrde/home/jnewton/quicklisp/setup.lisp"))
  (if (probe-file quicklisp-init)
  (load quicklisp-init)
  (error "file not found ~S" quicklisp-init)))
(asdf:load-system :lisp-types-test)


On 23 Jan 2018, at 12:47, Pascal Bourguignon  
wrote:




On 23 Jan 2018, at 12:00, Jim Newton > wrote:


If I run several sbcl processes on different nodes in my compute 
cluster, it might happen that two different runs notice the same 
file needs to be recompiled (via asdf),
and they might try to compile it at the same time.  What is the best 
way to prevent this?


I see in the asdf documentation that there is an asdf:*user-cache* 
variable whose
value is the path name of the directory where asdf compiles into.
Would it be advisable for my to arrange so that asdf:*user-cache* is 
a function of the pid and hostname and perhaps thread-id (if such a 
thing exists) to avoid such collisions?


Or is there some better way to handle this which is build into asdf?


I had requested that ASDF includes the hostname (or 
machine-instance), in the built path for the cache.
Unfortunately, for some reason, the maintainers of ASDF thought it 
was a good read to remove it.

There you are!




--
__Pascal J. Bourguignon__

Re: Multiple processing compiling the same file

[Robert Goldman]

On 23 Jan 2018, at 5:47, Pascal Bourguignon wrote:


> On 23 Jan 2018, at 12:00, Jim Newton  wrote:


If I run several sbcl processes on different nodes in my compute 
cluster, it might happen that two different runs notice the same file 
needs to be recompiled (via asdf),
and they might try to compile it at the same time.  What is the best 
way to prevent this?


I see in the asdf documentation that there is an asdf:*user-cache* 
variable whose
value is the path name of the directory where asdf compiles into.
Would it be advisable for my to arrange so that asdf:*user-cache* is 
a function of the pid and hostname and perhaps thread-id (if such a 
thing exists) to avoid such collisions?


Or is there some better way to handle this which is build into asdf?


I had requested that ASDF includes the hostname (or machine-instance), 
in the built path for the cache.
Unfortunately, for some reason, the maintainers of ASDF thought it was 
a good read to remove it.

There you are!


This would be a poor solution for the many users who connect via DHCP on 
different networks and have hostnames that change.


There are relatively few people who have an issue with contention over 
their home directories, and those users can simply reconfigure their 
ASDF output translations.  But there are relatively many who might find 
their hostname changing.

Re: Multiple processing compiling the same file

[Faré]

(Sorry for delayed response)

>>>: Jim Newton
>>> If I run several sbcl processes on different nodes in my compute cluster, 
>>> it might happen that
>>> two different runs notice the same file needs to be recompiled (via asdf),
>>> and they might try to compile it at the same time.  What is the best way to 
>>> prevent this?
>>>
You mean that this machines share the same host directory? Interesting.

"Normal" rules of ASDF compile to a temporary file and rename the
output at the end,
thus providing some kind of race resistance. But for
backward-compatibility reasons,
this requires every extension to manually follow a protocol for ASDF
to remain robust.

>>> I see in the asdf documentation that there is an asdf:*user-cache* variable 
>>> whose
>>> value is the path name of the directory where asdf compiles into.
>>> Would it be advisable for my to arrange so that asdf:*user-cache*
>>> is a function of the pid and hostname and perhaps thread-id (if such a 
>>> thing exists) to avoid such collisions?
>>>
That's an option. It is expensive, though: it means no sharing of fasl
files between hosts. If you have cluster of 200 machines, that means
200x the disk space.

What about instead building your application as an executable and
delivering that to the cluster?

My rule of thumb is that there is one home directory per human, and
the human is only interactively building one thing at a time (and/or
can set several accounts and/or $HOME variants for as many
"personalities"). Thus you only need one fasl cache for interactive
compilation. If you want non-interactive deployment, use tools like
bazel, nix, etc., to build your software deterministically.

>>> Or is there some better way to handle this which is build into asdf?
>
You can have different ASDF_OUTPUT_TRANSLATIONS or
asdf:*output-translations-parameter*
on each machine, or you can indeed have the user cache depend on
uiop:hostname and more.

The Right Thing™ is still to build and test then deploy, rather than
deploy then build.
Using Bazel, you might even be able to build in parallel on your cluster.

>>: pjb
>> I had requested that ASDF includes the hostname (or machine-instance), in
>> the built path for the cache.
>> Unfortunately, for some reason, the maintainers of ASDF thought it was a
>> good read to remove it.
>> There you are!
I still think it's a bad idea. If your $HOME is shared by many
machines, you probably want what's in $HOME to be shared, too. Go
build in /var/tmp or use Bazel or whatever. Or use uiop:hostname in
your ASDF configuration.

On Tue, Jan 23, 2018 at 7:51 AM, Jim Newton  wrote:
> Apparently, this approach seems to work.   I’m not sure if it is the best
> approach.
> Here is what my code looks like.  It creates a directory in /tmp/ and
> asdf:load-system
> seems to compile the .fasl files into there.
>
>
> (require :asdf)
> (require :sb-posix)
> (let ((home (directory-namestring (user-homedir-pathname)))
>   (uid (sb-posix:getuid))
>   (pid  (sb-posix:getpid)))
>   (setf asdf::*user-cache* (ensure-directories-exist (format nil
> "/tmp~A~D/~D/" home uid pid
>
I still don't understand why your use case uses deploy-then-build
rather than build-then-deploy.

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
A child of five would understand this. Send someone to fetch a child of five.
— Groucho Marx

Re: spurious reloads with systems not following the foo/bar naming convention?

[Faré]

On Tue, Jan 30, 2018 at 1:18 PM, Attila Lendvai  wrote:
>> I believe this is the bug that was fixed in 3.3.1.3.
>
> FYI, there's no such tag pushed into the official repo.
> https://gitlab.common-lisp.net/asdf/asdf/tags
>
Robert and I failed to push the 3.3.1.2 and 3.3.1.3 tags. This is now fixed.
But the code was already there in master, and should contain a bug
to the issue you're experiencing (with my apologies again).

> using HEAD, the test case i sent doesn't fail anymore, but our own
> issue still prevails. it's a large environment with many systems, but
> the symptom looks the same, although for us due to cxml.
>
If the issue is the c2ffi issue, see the other thread.

> i couldn't easily test with an older version of ASDF because my SBCL
> already ships with 3.3.0.1 (preventing an easy downgrade), but it used
> to work until recently.
>
One weird trick: use the install-asdf script to overwrite your SBCL's
asdf with an older (or newer) version.

> after i edit cxml.asd to follow the foo/bar convention, i see a few of
> these, that may or may not be the cause of the spurious reload that i
> still see:
>
> WARNING:
>   Computing just-done stamp in plan NIL for action
>   (ASDF/LISP-ACTION:COMPILE-OP "iolib/syscalls" "ffi-types"), but
>   dependency (ASDF/LISP-ACTION:PREPARE-OP "iolib/syscalls"
>   "ffi-types") wasn't done yet!
>
> WARNING:
>Computing just-done stamp in plan NIL for action
>(ASDF/LISP-ACTION:COMPILE-OP "iolib/os" "ffi-types"), but
>dependency (ASDF/LISP-ACTION:PREPARE-OP "iolib/os" "ffi-types")
>wasn't done yet!
>
> WARNING:
>Computing just-done stamp in plan NIL for action
>(ASDF/LISP-ACTION:COMPILE-OP "iolib/sockets" "grovel"), but
>dependency (ASDF/LISP-ACTION:PREPARE-OP "iolib/sockets" "grovel")
>wasn't done yet!
>
> please note that this is not a burning issue for us, it's just a
> friendly heads up with the hopes that it's useful.
>
Thanks for raising the issue while ASDF is still in my L3 cache.

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
A word for the epoch of free software and universal publishing:
voluntocracy n 1. governance by those who do the work. 2. the volunteers
who do the work. — Aubrey Jaffer, http://swissnet.ai.mit.edu/~jaffer/

Re: "deprecated recursive use" warning

[Robert Goldman]

On 30 Jan 2018, at 15:53, Attila Lendvai wrote:


I haven't used CFFI in a while.


TL;DR: is this a sane fix?

https://github.com/cffi/cffi/commit/4b9b06f15912e823581b1aeb8a0d5c2ef11f702d

--
and here follows the elaborate email that led me to find the above
solution:

a bit of background: it's a subsystem of CFFI that generates the CFFI
bindings from a json file, that is in turn autogenerated from the C
source.

so,

 1) C -> json
 2) json -> lisp (CFFI definitions)
 3) asdf compiles/loads the generated lisp file

1) requires a heavyweight infrastructure (a binary run by
run-program), so there's support to do that lazily, and just
distribute the json files generated once by the lib author.

2) is relatively lightweight, but it still requires loading a broader
scope of lisp dependencies, so there's support for the lib author to
run the generation and distribute the generated lisp file.

now, whether the generator code (extra dependencies) is needed is
decided by whether or not the generated lisp file is up-to-date.

unfortunately i cannot test it properly because of another bug/change
that i'll report in a separate thread.


Thinking out loud:

I'll have a bit more of a look.  IIUC what you are saying is that there 
should be an OP that covers the C -> JSON translation, and that requires 
some infrastructure that you don't want to load unless it's necessary.


Then there's JSON -> lisp that has additional dependencies that are 
optional.


The problem I see is that, IIRC, ASDF creates a build plan that is 
*unconditional*, and then *executes* it conditionally, by calling 
`OPERATION-DONE-P`, and skipping unnecessary operations.  It does this 
because earlier steps in the process can change what `OPERATION-DONE-P` 
would return for later stages in plan execution.


The problem with this is that you want what comes *later* in the plan 
(whether or not the lisp derived from JSON) to affect what comes 
*earlier* in the plan (whether or not the JSON to lisp translation 
library gets loaded).


I think I know what is the Right Answer to this, but it might be so much 
more work that you would rather just keep your current hack working


I think the Right Thing is to realize that what ASDF does is not so much 
transform files, as to maintain the consistency of the running lisp 
image.  Now, in no case does the running lisp image need *either* the 
JSON generation system *or* the JSON to lisp translation system.  All 
the *running lisp image* needs to function correctly is an up-to-date 
lisp file produced by this pipeline.  So


The Right Thing is to kick that pipeline out of the current lisp 
process.  Instead of making ASDF use its `LOAD-OP`, etc. to do this 
translation for you, you should just create an external application 
(which might be a lisp program) to do the JSON generation (if needed) 
and the JSON to lisp translation (if needed).  That external program 
could well use ASDF to manage itself.  But all that would be in your 
*main* ASDF system definition would be a `JSON-GEN-OP` and a 
`JSON-TO-LISP-OP`, each of which would be implemented by invoking an 
external program.  As in, for example

```
(DEFMETHOD PERFORM ((OP JSON-GEN-OP) (c json-component))
   (uiop:run-program ))
```

Once you think about what ASDF does and doesn't do, I think this makes 
perfect sense.  But, of course, it might be a big pain to do so.


Best,
R

Re: "deprecated recursive use" warning

[Faré]

On Tue, Jan 30, 2018 at 4:53 PM, Attila Lendvai  wrote:
>> I haven't used CFFI in a while.
>
> TL;DR: is this a sane fix?
>
> https://github.com/cffi/cffi/commit/4b9b06f15912e823581b1aeb8a0d5c2ef11f702d
>
(the (not null) ...) is redundant around find-system without the nil
optional argument.

Also, having a component depend on a system violates a weak tacit
invariant in ASDF, though it should only matter for bundle operations,
and only for things required at link-time, which shouldn't be the case
here. Still, something to keep in mind.

> now, whether the generator code (extra dependencies) is needed is
> decided by whether or not the generated lisp file is up-to-date.
>
... except that ASDF only considers it "up-to-date" if its date is
later than that of all dependencies, including the compiled objects
for c2ffi, now that staging is done properly.

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
Eric Schmidt rule of management: always assume that
the other person is as intelligent as you are.

Re: hu.dwim.zlib broke; broken operation-done-p

[Faré]

On Tue, Jan 30, 2018 at 5:01 PM, Attila Lendvai  wrote:
> if you issue the following (available in quicklisp):
>
> (asdf:load-system :hu.dwim.zlib)
>
> then for the first time it should generate a lisp file, which then gets
> compiled and loaded.
>
> issuing it for the second time shouldn't do anything, but since some
> revisions it regenerates the lisp file every time.
>
> the relevant code is somewhere around here:
>
> https://github.com/cffi/cffi/blob/master/src/c2ffi/asdf.lisp#L135
>
> i tried to trace the INPUT-FILES, OUTPUT-FILES, PERFORM,
> OPERATION-DONE-P methods, but i don't see anything wrong.
>
> one unusual thing is that the output of GENERATE-LISP-OP goes into the
> src/ directory, not to the usual fasl output dir. same applies to
> GENERATE-SPEC-OP.
>
> any ideas how to debug this? or what to look at?
>
I can reproduce the issue using asdf 3.3.1.3 from master (the tag
hadn't been pushed, so I pushed it, but the code was already there).

(plan-actions (nth-value 1 (operate 'load-op :hu.dwim.zlib)))

starts with:
((# . #)
 (# . #)
 (# . #) (# . #)

So somehow the first action above is considered not done. I suppose
it's a bug in c2ffi.

Looking at the c2ffi source, I don't see anything obviously wrong,
though you shouldn't the (the (not null) (find-system ...)) since
find-system will error out for you if the system is not found (to
avoid error, add optional argument nil, as in find-class).

My theory is that that you're failing to re-generate the .spec when
it's present, even though your dependencies say it will be
re-generated. Therefore, ASDF decides that it's out-of-date and must
be re-generated again the next time over, etc. In other words, you lie
to ASDF, and ASDF punishes you right back.

To make ASDF happy, you might instead want to have two disjoint
operation modes, one to generate the spec files, and one to use it.
Normal use mode wouldn't try to generate the file, but instead bork
with a useful error message telling you how to generate it and submit
it upstream.

Another strategy to make ASDF happy would be to detect whether the
file exists as part of input-files and output-files. If it exists,
it's in input-files and not output-files; if not, it's in output-files
and not input-files. There's something I don't like about this
strategy, but it might work (somehow I can't swear it won't bite you
back somehow at some point).

Good luck!

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
There is only one thing more harmful to society than an elected official
forgetting the promises he made in order to get elected; that's when he
doesn't forget them.  — John McCarthy

hu.dwim.zlib broke; broken operation-done-p

[Attila Lendvai]

if you issue the following (available in quicklisp):

(asdf:load-system :hu.dwim.zlib)

then for the first time it should generate a lisp file, which then gets
compiled and loaded.

issuing it for the second time shouldn't do anything, but since some
revisions it regenerates the lisp file every time.

the relevant code is somewhere around here:

https://github.com/cffi/cffi/blob/master/src/c2ffi/asdf.lisp#L135

i tried to trace the INPUT-FILES, OUTPUT-FILES, PERFORM,
OPERATION-DONE-P methods, but i don't see anything wrong.

one unusual thing is that the output of GENERATE-LISP-OP goes into the
src/ directory, not to the usual fasl output dir. same applies to
GENERATE-SPEC-OP.

any ideas how to debug this? or what to look at?

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“For an instant I think I saw. I saw the loneliness of man as a
gigantic wave which had been frozen in front of me, held back by the
invisible wall of a metaphor.”
— Carlos Castaneda (1925–1998), 'Journey to Ixtlan' (1972)

Re: "deprecated recursive use" warning

[Attila Lendvai]

> I haven't used CFFI in a while.

TL;DR: is this a sane fix?

https://github.com/cffi/cffi/commit/4b9b06f15912e823581b1aeb8a0d5c2ef11f702d

--
and here follows the elaborate email that led me to find the above
solution:

a bit of background: it's a subsystem of CFFI that generates the CFFI
bindings from a json file, that is in turn autogenerated from the C
source.

so,

 1) C -> json
 2) json -> lisp (CFFI definitions)
 3) asdf compiles/loads the generated lisp file

1) requires a heavyweight infrastructure (a binary run by
run-program), so there's support to do that lazily, and just
distribute the json files generated once by the lib author.

2) is relatively lightweight, but it still requires loading a broader
scope of lisp dependencies, so there's support for the lib author to
run the generation and distribute the generated lisp file.

now, whether the generator code (extra dependencies) is needed is
decided by whether or not the generated lisp file is up-to-date.

unfortunately i cannot test it properly because of another bug/change
that i'll report in a separate thread.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Enlightenment is a destructive process. It has nothing to do with
becoming better or being happier. Enlightenment is the crumbling away
of untruth. It's seeing through the facade of pretence. It's the
complete eradication of everything we imagined to be true.”
— Adyashanti (1962–)

Re: "deprecated recursive use" warning

[Robert Goldman]

I haven't used CFFI in a while.  Can I ask why this needs to be here 
instead of there being a


```
:depends-on ((generate-lisp-op (load-op "cffi/c2ffi-generator")))
```
?

Thanks,
r


On 30 Jan 2018, at 12:20, Attila Lendvai wrote:


dear list,

shall i be concerned about this?

WARNING:
   Deprecated recursive use of (ASDF/OPERATE:OPERATE
'ASDF/LISP-ACTION:LOAD-OP '("cffi/c2ffi-generator")) while
   visiting (CFFI/C2FFI::GENERATE-LISP-OP "hu.dwim.zlib" "c2ffi-spec"
"zlib.h") - please use proper dependencies instead

it happens due to this in cffi/src/c2ffi/asdf.lisp:

(defmethod perform ((op generate-lisp-op) (c c2ffi-file))
...
  (unless (component-loaded-p :cffi/c2ffi-generator)
(load-system :cffi/c2ffi-generator))
...
)

that system only needs to get loaded when the action is executed.

or is it safe to ignore that warning long term?

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“If we have the truth, it cannot be harmed by investigation. If we
have not the truth, it ought to be harmed.”
— J. Reuben Clark (1871–1961)

"deprecated recursive use" warning

[Attila Lendvai]

dear list,

shall i be concerned about this?

WARNING:
   Deprecated recursive use of (ASDF/OPERATE:OPERATE
'ASDF/LISP-ACTION:LOAD-OP '("cffi/c2ffi-generator")) while
   visiting (CFFI/C2FFI::GENERATE-LISP-OP "hu.dwim.zlib" "c2ffi-spec"
"zlib.h") - please use proper dependencies instead

it happens due to this in cffi/src/c2ffi/asdf.lisp:

(defmethod perform ((op generate-lisp-op) (c c2ffi-file))
...
  (unless (component-loaded-p :cffi/c2ffi-generator)
(load-system :cffi/c2ffi-generator))
...
)

that system only needs to get loaded when the action is executed.

or is it safe to ignore that warning long term?

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“If we have the truth, it cannot be harmed by investigation. If we
have not the truth, it ought to be harmed.”
— J. Reuben Clark (1871–1961)

Re: spurious reloads with systems not following the foo/bar naming convention?

[Attila Lendvai]

> I believe this is the bug that was fixed in 3.3.1.3.

FYI, there's no such tag pushed into the official repo.

https://gitlab.common-lisp.net/asdf/asdf/tags

using HEAD, the test case i sent doesn't fail anymore, but our own
issue still prevails. it's a large environment with many systems, but
the symptom looks the same, although for us due to cxml.

i couldn't easily test with an older version of ASDF because my SBCL
already ships with 3.3.0.1 (preventing an easy downgrade), but it used
to work until recently.

after i edit cxml.asd to follow the foo/bar convention, i see a few of
these, that may or may not be the cause of the spurious reload that i
still see:

WARNING:
  Computing just-done stamp in plan NIL for action
  (ASDF/LISP-ACTION:COMPILE-OP "iolib/syscalls" "ffi-types"), but
  dependency (ASDF/LISP-ACTION:PREPARE-OP "iolib/syscalls"
  "ffi-types") wasn't done yet!

WARNING:
   Computing just-done stamp in plan NIL for action
   (ASDF/LISP-ACTION:COMPILE-OP "iolib/os" "ffi-types"), but
   dependency (ASDF/LISP-ACTION:PREPARE-OP "iolib/os" "ffi-types")
   wasn't done yet!

WARNING:
   Computing just-done stamp in plan NIL for action
   (ASDF/LISP-ACTION:COMPILE-OP "iolib/sockets" "grovel"), but
   dependency (ASDF/LISP-ACTION:PREPARE-OP "iolib/sockets" "grovel")
   wasn't done yet!

please note that this is not a burning issue for us, it's just a
friendly heads up with the hopes that it's useful.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“A true friendship involves two people who are both intensely focused
on their own healing processes, on working on their own issues and
maturing and evolving. Basically two people who are growing toward
truth, and they both individually, in their own specific individual
lives, have that as their purpose. And when they come together it
becomes synergistic.”
— Daniel Mackler