Re: A modest proposition: DEFSYSTEM-DEPENDS-ON should die [was Re: What's the right way to extend ASDF with new symbols?]

[73budden .]

Hi!

>In the old days, we would just put
>(asdf:load-system "extension")
That's good. This is declarative enough to be processed by tools like
quicklisp.

What goes to side effects, they are just inevitable if we want

i) asdf to be extensible
ii) some dependency language is used to declarate which
system-extension relationships

The only way to avoid side effects at asd file processing time is to
load all extensions together with asdf, but this is what we want to
avoid when we talk about build system, isnt' it?

Re: A modest proposition: DEFSYSTEM-DEPENDS-ON should die [was Re: What's the right way to extend ASDF with new symbols?]

[Robert Goldman]

On 2/15/16 Feb 15 -3:06 PM, Stelian Ionescu wrote:
> On Mon, 2016-02-15 at 14:48 -0600, Robert Goldman wrote:
>> Trying again:
>>
>> Can someone please state what it is that Quicklisp needs?
>>
>> IIUC Quicklisp does *something* with .asd files that does not involve
>> the defsystem-depends-on being resolved.
>>
>> Is this reading?  Or loading?
>>
>> If it's loading, then the DEFSYSTEM-DEPENDS-ON entries are resolved by
>> REGISTER-SYSTEM-DEFINITION, so the status quo doesn't solve the problem
>> of large scale analysis without side-effects.
>>
>> If it's reading, then my solution is benign.
>>
>> At least as far as I can tell.
>>
>> Can someone give me a clue here?
> 
> Quicklisp needs to compute the complete dependency graph between
> systems, given a set of .asd files. I'd like to extend the current Slime
> integration to search for systems in all registered .asd files and
> autocomplete. All sorts of things become possible if one could treat a
> set of .asd files as a database, with the guarantee that querying would
> never trigger a compile-file operation, ever.
> 

If that's the case, then I believe that the existing implementation
*already* fails your requirement.

When you register an ASDF system, the DEFSYSTEM-DEPENDS-ON dependencies
must be loaded (and therefore, possibly compiled).  ASDF needs to do
this to parse the defsystem form correctly.

So I think that we may have been arguing over nothing.

Thanks,
r

Re: A modest proposition: DEFSYSTEM-DEPENDS-ON should die [was Re: What's the right way to extend ASDF with new symbols?]

[Faré]

For some historical perspective, defsystem-depends-on was added at a
time when we wanted a more declarative solution than using load-system
in a .asd file, and wasn't initially thought through. It took over a
year for it to become an actually useful feature, after realizing the
package issue and allowing and recommending the use of symbols in the
asdf and/or keyword packages as a workaround.

At the time, the dependency graph of ASDF wasn't understood by anyone,
and indeed was broken, and there was no hope of ever turning a
load-system into a dependency. Now, the dependency graph is understood
and fixed, and there's a way to handle such dependencies right,
whether through load-system or through defsystem-depends-on — but it
requires a new maintainer to do some non-trivial re-coding of load-asd
and/or parse-defsystem. That would mean a major release of ASDF 3.2
and/or ASDF 4 as a result. Basically, you'll need to (a) add new
dependency nodes for loading an asd file, (b) properly handle
recursive calls to operate from find-system and/or make-plan, and (c)
use special variables to track context and maintain dependency graph
in presence of such calls.

Will anyone do it? Neither Robert nor I has the will to do it, though
I will gladly consult with whoever does it (if he cares from my
opinions).

Once this capability is added to ASDF, the rationale for
defsystem-depends-on vanishes: just use load-system or load-systems
from a .asd file. You may still want to restrict .asd file to a short
whitelist of allowed forms to keep it "declarative".

As for what Quicklisp needs, I suppose it is a way to reliably find
*all* the transitive dependencies of a system, if possible without
loading it, though perhaps after loading its defsystem-depends-on
dependencies, which might be required in the general case. Right now,
the best you can do is load into an empty image, and inspect all
that's loaded afterwards.

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
Constitution, article 1: any person who claims that a document called
"constitution" possesses any authority whatsoever on other people is an
impostor guilty of high treason and liable to immediate capital punishment.

Re: A modest proposition: DEFSYSTEM-DEPENDS-ON should die [was Re: What's the right way to extend ASDF with new symbols?]

[Robert Goldman]

Trying again:

Can someone please state what it is that Quicklisp needs?

IIUC Quicklisp does *something* with .asd files that does not involve
the defsystem-depends-on being resolved.

Is this reading?  Or loading?

If it's loading, then the DEFSYSTEM-DEPENDS-ON entries are resolved by
REGISTER-SYSTEM-DEFINITION, so the status quo doesn't solve the problem
of large scale analysis without side-effects.

If it's reading, then my solution is benign.

At least as far as I can tell.

Can someone give me a clue here?

thanks!
r

Re: A modest proposition: DEFSYSTEM-DEPENDS-ON should die [was Re: What's the right way to extend ASDF with new symbols?]

[Stelian Ionescu]

On Mon, 2016-02-15 at 11:13 -0600, Robert Goldman wrote:
> On 2/15/16 Feb 15 -10:26 AM, Stelian Ionescu wrote:
> >> On 2/12/16 Feb 12 -3:15 PM, Stelian Ionescu wrote:
> >>> On Fri, 2016-02-12 at 16:07 -0500, Faré wrote:
>  I'm OK with declaring DEFSYSTEM-DEPENDS-ON a failure, and load-system
>  (or load-systems) the official way to go. But
> 
>  1- This of course requires heads up, updating all users before
>  retiring the feature, etc. From my experience, if you start seriously
>  deprecating today and sending patches to all authors who use it in
>  quicklisp, you can expect to be removing that part of the code in two
>  years or so.
> 
>  2- To make these dependencies work properly still requires modifying
>  ASDF to add explicit plan nodes for loading ASD files, that will
>  contain the systems loaded by load-system. The same trick will also
>  automatically make the :defsystem-depends-on work, since it itself
>  calls load-system.
> 
>  3- Yes, defining things in the ASDF package is ugly, but extensions
>  are few enough, and using a prefix is a good enough namespace
>  management strategy. Not the most horrible thing that working with CL
>  does to you.
> >>>
> >>> Please don't. It's a net improvement compared to the previous situation.
> >>> It's easy to simply name your class with a keyword :package/foo-file.
> >>>
> >>
> >> With all due respect, no, it is not.  It is a net *negative* with
> >> respect to the previous situation.  Here are the reasons why, briefly
> >> reiterated:
> >>
> >> 1. It effectively forces you to stick new symbols into the ASDF namespace.
> > 
> > Technically yes, but that's not the essential requirement. Because the CL 
> > reader interns eagerly, ASDF extension classes need to be interned into a 
> > package that is owned by ASDF. Currently that's the ASDF package itself, 
> > but it would be a good idea to add a special package for extensions, and 
> > start encouraging people to use that by showing appropriate deprecation 
> > messages.
> 
> This doesn't do anything to resolve the underlying problem, AFAICT.
> I.e., instead of getting name collisions in ASDF (I believe that's
> ASDF/INTERFACE), we get name collisions in ASDF/EXTENSIONS.

I don't see this as a problem. A few times there were collisions in
Quicklisp, so somebody had to rename the project/package, etc... If two
different authors want to use asdf/extensions::foo/file then one will
have to give up. This is a technical problem(the design of the CL
reader) that is best solved socially. Also, it's not like people write
new ASDF extensions everyday.

> 
> Am I missing something about this suggestion?
> > 
> > 
> >> I don't understand your proposed rebuttal, involving slash-named
> >> packages. I don't see any evidence of this being legal ASDF syntax,
> >> looking at FIND-CLASS*, and trying an experiment.  If it works, it needs
> >> documentation.  If it does not work, it will not be added -- ASDF must
> >> get simpler, not more complex.  Please amplify, thanks!
> > 
> > It's not "syntax", it's just manual namespacing. The symbol 
> > 'asdf::pkg/class is symply a legal symbol. And I don't agree that ASDF must 
> > get simpler at all costs, rather that it should have as simple an 
> > implementation as possible, while still allowing the use cases that its 
> > users require.
> 
> With all due respect, I believe ASDF to be unmaintainable now, except
> for a Hail Mary Pass to Faré at regular intervals.  Again, you are
> welcome to have a whack at it.  I won't be upset if you prove me wrong!
> > 
> > 
> >> 2. If you are arguing that we can just solve this with a naming
> >> convention, I don't buy it.  Consider different libraries, each of which
> >> hook ASDF to normal "make" by creating MAKE-FILE and MAKE-OP classes.
> >> This is not a far-fetched example; I have seen it.  They both try to jam
> >> these symbols into ASDF.
> > 
> > See my previous reply.
> > 
> > 
> >> This behavior should be *strongly* discouraged, even to the extent of
> >> (as I said earlier) package-locking ASDF when possible.  Currently, it
> >> is actively *ENCOURAGED* -- close to mandated, in fact.
> > 
> > By dedicating a package for naming extension classes, we can even lock ASDF 
> > while still making QA easier.
> 
> Per earlier response, this seems to me to just kick the problem from one
> namespace/package to another.
> > 
> >> 3. If we make DEFSYSTEM-DEPENDS-ON into a declaration, a lot of
> >> simplification ensues, including eliminating the complex double-parsing
> >> of DEFSYSTEM.  ASDF is currently over-complicated and over-long.
> > 
> > In what sense is it not currently a declaration ?
> 
> What I meant is that it's not *only* a declaration.  It has extra
> operational import.   I would like to change D-D-O to be only
> declarative (except that we will also check it).
> 
> > 
> > 
> >> 4.  The double-parsing doesn't even work, because the packages don't

Re: Warnings: towards 3.1.7

[Robert Goldman]

On 2/15/16 Feb 15 -2:00 PM, Faré wrote:
> On Mon, Feb 15, 2016 at 2:18 PM, Robert Goldman  wrote:
>> I was having some trouble with warnings causing build failures in ASDF,
>> despite (correctly?) setting variables.
>>
>> Please see the test-warnings branch on cl.net.  This contains a small
>> number of additional tests.
>>
>> I'll continue to look these over to see why they are failing.  I'll move
>> to release after these are fixed/dismissed as wrong.
>>
> I didn't try to think about those tests you added, but they pass for
> me on sbcl and ccl.

I got failures on a number of the implementations:

 Unexpected test failures on these implementations:
build/results/ccl-test.text
build/results/clisp-test.text
build/results/lispworks-test.text
build/results/allegro_64-test.text
build/results/allegromodern_64-test.text
build/results/allegro8_64-test.text
build/results/allegromodern8_64-test.text
build/results/allegro_64_s-test.text
build/results/allegromodern_64_s-test.text
build/results/allegro8_64_s-test.text
build/results/allegromodern8_64_s-test.text
build/results/abcl-test.text


This is on mac.

For me test-undeferred-warnings fails on CCL.  But I'm still running
1.10, in case that matters.

More details soon.

Note that I'm not entirely sure how the deferred/undeferred warnings
should be working.

I was just alerted to this when I tried to quash a redefined variable
warning that was expected, but was breaking a build on Allegro, when we
set warnings to cause failures.  I expected that ignoring warnings on
ASDF would also suppress the build failures.

R

Re: Warnings: towards 3.1.7

[Faré]

On Mon, Feb 15, 2016 at 2:18 PM, Robert Goldman  wrote:
> I was having some trouble with warnings causing build failures in ASDF,
> despite (correctly?) setting variables.
>
> Please see the test-warnings branch on cl.net.  This contains a small
> number of additional tests.
>
> I'll continue to look these over to see why they are failing.  I'll move
> to release after these are fixed/dismissed as wrong.
>
I didn't try to think about those tests you added, but they pass for
me on sbcl and ccl.

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
The main difference between a computer salesman and a used car salesman is
that the used car salesman can probably drive and knows when he's lying.
- Peter da Silva

Warnings: towards 3.1.7

[Robert Goldman]

I was having some trouble with warnings causing build failures in ASDF,
despite (correctly?) setting variables.

Please see the test-warnings branch on cl.net.  This contains a small
number of additional tests.

I'll continue to look these over to see why they are failing.  I'll move
to release after these are fixed/dismissed as wrong.

cheers,
r