Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[Robert Goldman]

On 12 Oct 2017, at 12:09, 73budden . wrote:


This tracing tool should help a lot.

I believe this tool should be supplied by asdf team. Even I begin to
be more positive towards efforts of ASDF team to clean up all the mess
that was in ASDF initially, but obviously society is not quite happy
with breaking changes, so some small tool with a good manual would
make life easier.


This will probably not happen for a while -- my work on other aspects, 
and just staying on top of bugs and testing takes most of the available 
time.


But I think I can provide pieces of a recipe for this kind of debugging 
and if someone out there could give feedback, I will see to it that the 
recipe gets into the manual.


I think if you want to see the plan that ASDF produces to perform a 
requested operation, you should use something like:

```
(setf *plan* (asdf/plan:make-plan nil (make-operation 'load-op) 
(find-system "sysname")))

```

Then, to figure out what's happening, I would suggest
```
(trace asdf:operation-done-p)
```
...to see if ASDF is wrong about whether or not it needs to do some 
operations.


Then I would try something like tracing `PERFORM`.

I'd have to think a little about what to do if `MAKE-PLAN` gives you a 
plan you don't expect.


cheers,
r



Printing readtable before loading, I think, requires just a line or
two. Dumping log of operations might be one (trace) call, so that's
trivial for the person who knows how ASDF is organized. Writing a
small two-paragraph addition to manual would relief a lot of pain and
stress for all.

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[Faré]

On Thu, Oct 12, 2017 at 6:53 PM, Stelian Ionescu  wrote:
> On Thu, 2017-10-12 at 13:03 -0500, Robert Goldman wrote:
>> That may be, but it was unfair to get angry at the ASDF maintainers
>> about this.  This is just a pre-existing error that was *manifested*
>> because of a change in ASDF.  It's not our fault that this error
>> appeared, it's not our fault that the puri library is no longer
>> maintained, and we can't be expected to avoid releasing improved
>> versions of ASDF because there exists buggy, unmaintained code in
>> Quicklisp.
>
> No it's not your fault, but I think it would be a very sensitive idea
> to avoid annoying ASDF users, simply for practical reasons, because
> you'll find yourself with people who fork or refuse to update ASDF.
>
Well, in this case, it was a combination of things that are not our
fault, and things that are. So my apologies for the things that were
my fault.

> Something more useful would be to:
>  * introduce the notion of ASDF compilation options, as a set of key-
> value pairs which control different compilation modes or effects
ASDF already kind of has that, as keyword arguments to operate. They
was a bit of a cleanup in ASDF 3.3.0 already: the arguments are not
passed blindly into non-sensical arguments to the operation class
anymore, and are now well-defined. Thus they can now be used the way
you propose.

>  * make the new strictness modes, like preserving the readtable, depend
> on those toggles, but upon introduction the default should be perfect
> backwards-compatibility, even if that is something you consider broken
Agreed. Any new strictness should be added cautiously and slowly,
disabled by default or adding only a (style-)warning. In this case,
the strictness was added by accident (because the forms that prevented
the strictness were victim of a refactoring I botched while adding a
different feature, and the strictness was only added in a corner case
for which I had no test, but just added one now).

>  * blog about the fancy new way toggle and explain why it's better to
> use it than not
Will do, if I keep hacking on it.

>  * let the libraries' users nag the developers to change the code to be
> compatible with the new strictness checks
Maybe. Historically, the main problem has been unmaintained libraries.

>  * wait a couple of years (at least) until you see that most of
> Quicklisp libraries have been ported to the new way of doing things and
> if that happens maybe consider turning it on by default. In that case,
> announce it publicly
Yes, we've typically waited for a year or two before we added any new
strictness (e.g. everything is now UTF-8 by default).

>  * if adoption didn't happen, keep it disabled happily knowing that you
> can always turn it on in your company's internal projects.
>
This has happened in the past, e.g. for properly catching deferred
warnings. Most users never bothered to do it, maintainers never
bothered to fix their libraries, etc. Maybe with more outreach they
would, but I'm kind of burned out with CL right now.

> This is more or less how we do things at work. It has a certain amount
> of overhead but it gains you good will from the community; on the other
> hand enabling things by default, and on a short notice, only makes you
> seem like you're imposing your preference on everybody else just for
> the sake of it. I think it's better to let things sink in and allow the
> users of ASDF to come to a consensus, although that's a slow process.
>
I've tried very hard not to do that this time. It was a bug. (Unlike
four to five years ago, when I learned the hard way not to push too
hard the deferred warning support or the syntax-control branch; the
latter just came back and bit me.)

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
Amateur bureaucrats are often even worse than professional bureaucrats.
— John McCarthy

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[Stelian Ionescu]

On Thu, 2017-10-12 at 13:03 -0500, Robert Goldman wrote:
> That may be, but it was unfair to get angry at the ASDF maintainers 
> about this.  This is just a pre-existing error that was *manifested* 
> because of a change in ASDF.  It's not our fault that this error 
> appeared, it's not our fault that the puri library is no longer 
> maintained, and we can't be expected to avoid releasing improved 
> versions of ASDF because there exists buggy, unmaintained code in 
> Quicklisp.

No it's not your fault, but I think it would be a very sensitive idea
to avoid annoying ASDF users, simply for practical reasons, because
you'll find yourself with people who fork or refuse to update ASDF.

Something more useful would be to:
 * introduce the notion of ASDF compilation options, as a set of key-
value pairs which control different compilation modes or effects
 * make the new strictness modes, like preserving the readtable, depend
on those toggles, but upon introduction the default should be perfect
backwards-compatibility, even if that is something you consider broken
 * blog about the fancy new way toggle and explain why it's better to
use it than not
 * let the libraries' users nag the developers to change the code to be
compatible with the new strictness checks
 * wait a couple of years (at least) until you see that most of
Quicklisp libraries have been ported to the new way of doing things and
if that happens maybe consider turning it on by default. In that case,
announce it publicly
 * if adoption didn't happen, keep it disabled happily knowing that you
can always turn it on in your company's internal projects.

This is more or less how we do things at work. It has a certain amount
of overhead but it gains you good will from the community; on the other
hand enabling things by default, and on a short notice, only makes you
seem like you're imposing your preference on everybody else just for
the sake of it. I think it's better to let things sink in and allow the
users of ASDF to come to a consensus, although that's a slow process.

-- 
Stelian Ionescu a.k.a. fe[nl]ix
Quidquid latine dictum sit, altum videtur.


signature.asc
Description: This is a digitally signed message part

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[Faré]

On Thu, Oct 12, 2017 at 6:11 PM, Robert Goldman  wrote:
>> I will merge this into master, but do not have time to make a new release,
>> or even test much this week or next.
>>
>> So this will simply remain in master for now.
>
> This patch is now available in ASDF 3.3.0.1
>
I also pushed a regression test directly to master.

I will also try to salvage the most important stuff to salvage from
the syntax-control branch, namely some *shared-readtable* and
*shared-print-pprint-dispatch* tables in asdf (or uiop?) to restore
around the build (e.g. in asdf:operate) so it is well-defined what
readtable is used when building with asdf (i.e. the readtable active
when asdf was initially loaded, usually your Lisp's initial
readtable).

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
[T]he “evolution stops at the neck” reflex. They're willing to accept that
everything else about humans has evolved through evolution but the thing
that is most important in explaining your personhood, namely your mind,
somehow evolution doesn’t apply to it. — Gad Saad

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[Robert Goldman]

On 12 Oct 2017, at 13:08, Robert Goldman wrote:


On 12 Oct 2017, at 13:04, Faré wrote:


For those in a hurry for a fix, here is the merge request:
https://gitlab.common-lisp.net/asdf/asdf/merge_requests/85


I will merge this into master, but do not have time to make a new 
release, or even test much this week or next.


So this will simply remain in master for now.


This patch is now available in ASDF 3.3.0.1

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[73budden .]

Wow! Good news.

2017-10-12 21:08 GMT+03:00, Robert Goldman :
> On 12 Oct 2017, at 13:04, Faré wrote:
>
>> For those in a hurry for a fix, here is the merge request:
>> https://gitlab.common-lisp.net/asdf/asdf/merge_requests/85
>
> I will merge this into master, but do not have time to make a new
> release, or even test much this week or next.
>
> So this will simply remain in master for now.
>
>

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[Robert Goldman]

On 12 Oct 2017, at 13:04, Faré wrote:


For those in a hurry for a fix, here is the merge request:
https://gitlab.common-lisp.net/asdf/asdf/merge_requests/85


I will merge this into master, but do not have time to make a new 
release, or even test much this week or next.


So this will simply remain in master for now.

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[Faré]

On Thu, Oct 12, 2017 at 5:09 PM, 73budden .  wrote:
> Warning about modifying standard readtable is issued by SBCL (at least
> in my SBCL 1.3.18), please grep for the message in SBCL sources, and
> it seem to be introduced in 1.0.24. I took a look at puri's definition
>  (maybe old, in my local copy of quicklisp) and it looks like it
> actually tries to modify standard readtable.
>
More precisely, puri modifies the _current_ *readtable*, which is very rude.
If anyone maintains puri (fat luck), he should remove this bug
and instead offer an interface using named-readtables.

The error happens when the current readtable happens to be the
standard readtable, which will happen if someone uses
with-standard-io-syntax.

Looking at ASDF sources, there is one place that uses it, and indeed,
a bug was introduced during the refactoring of ASDF 3.3.0: whereas
earlier variants of ASDF
saved the *readtable* from outside the with-standard-io-syntax to
restore it inside,
the new ASDF fails to do it. Reduced test case: try to
(asdf:load-system "ddop") with the following ddop.asd.

(defsystem "ddop" :defsystem-depends-on ("puri"))

I believe this justifies a 3.3.1 bugfix release indeed.

It also justifies spending more time on the syntax-control branch
supposed to cleanup the readtable issues, and/or getting all software
fixed so it never modifies the current readtable unless it explicitly
makes and sets a new readtable.

> It'd be very nice to have instructions on how to trace what happens
> while system is being built:that is, which files are compiled, which
> are loaded and what is a readtable in the beginning of each load
> operation.
>
You can pass a :verbose t flag to load-system and its friends.

As for tracing what readtable are used, there does not exist
infrastructure for even identifying readtables. You're welcome to
write one and seek assistance in writing
a hook in ASDF (e.g. as a :before method on perform compile-op or load-op,
which would not have helped you much in this case, or as a local
modification to uiop:compile-file*, if you have a local ASDF source
checkout).

> Running builds in old and new asdf versions and comparing logs it
> would be relatively easy to figure out what is wrong.
>
Only if given enough information to reproduce, which the complainer
failed to provide. Luckily, I investigated nevertheless.

For those in a hurry for a fix, here is the merge request:
https://gitlab.common-lisp.net/asdf/asdf/merge_requests/85

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
The two most common things in the Universe are hydrogen and stupidity.
— Harlan Ellison

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[Robert Goldman]

On 12 Oct 2017, at 12:53, Konstanski, Carlos wrote:

All signs point at puri being the culprit. I am absolutely not 
disagreeing

with that. I provided the offending code in the bug report myself. The
question is: what to do about it? The project is dead. The home page 
is
gone. Here is what I will do: I'll see if I can get drakma to steer 
clear
of it. They will undoubtedly be interested in not having their fine 
code

dragged down by a dead library.

The only other option is to fix puri. I don't even know how to begin 
doing

that with a dead project. Where did Kevin Rosenberg disappear off to?

2017-10-12 11:45 GMT-06:00 Robert Goldman <rpgold...@sift.info>:




Forwarded message:


From: 73budden . <budde...@gmail.com>
To: Robert Goldman <rpgold...@sift.info>
Cc: Faré <fah...@gmail.com>, ASDF-devel 
<asdf-devel@common-lisp.net>

Subject: Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER
Date: Thu, 12 Oct 2017 20:09:44 +0300

Warning about modifying standard readtable is issued by SBCL (at 
least

in my SBCL 1.3.18), please grep for the message in SBCL sources, and
it seem to be introduced in 1.0.24. I took a look at puri's 
definition

 (maybe old, in my local copy of quicklisp) and it looks like it
actually tries to modify standard readtable. But here it does not
happen when it is being loaded via my patched asdf 3.1.6 and my own
IDE with forked SLIME with many systems pre-loaded. I don't know why
and have no time to investigate further. Obviously, my insights 
would

be of limited use because my environment is very different from that
of OP.


So we might be able to help you debug this

It'd be very nice to have instructions on how to trace what happens
while system is being built:that is, which files are compiled, which
are loaded and what is a readtable in the beginning of each load
operation.

Running builds in old and new asdf versions and comparing logs it
would be relatively easy to figure out what is wrong.

I guess that in OP's setup something had changed readtable before, 
but

this does not happen anymore. And obviously many people would face
similar problems.

This tracing tool should help a lot.

I believe this tool should be supplied by asdf team. Even I begin to
be more positive towards efforts of ASDF team to clean up all the 
mess

that was in ASDF initially, but obviously society is not quite happy
with breaking changes, so some small tool with a good manual would
make life easier.

Printing readtable before loading, I think, requires just a line or
two. Dumping log of operations might be one (trace) call, so that's
trivial for the person who knows how ASDF is organized. Writing a
small two-paragraph addition to manual would relief a lot of pain 
and

stress for all.

WBR, Budden

2017-10-12 18:46 GMT+03:00, Robert Goldman <rpgold...@sift.info>:
I'm with Faré on this one.  I don't see evidence that this change 
is
because ASDF is doing something bad.  I believe it's consistent 
with the
hypothesis that there was some imperfectly-controlled aspect of 
building
that is done differently now (e.g., files loaded in a different 
order

where both orders are compatible with the constraints in the system
definitions).

This doesn't even look like an ASDF error to me -- it looks like an
error that occurred while loading a system that ASDF has 
re-packaged.


So we might be able to help you debug this, but without more 
evidence,

there's no reason to believe that ASDF has done anything to you: it
looks like the change in ASDF just reveals a pre-existing bug.









--
Carlos Konstanski
MTS IV Cslt
Verizon Cloud Platform
carlos.konstan...@verizonwireless.com
Cell: 15126218301
Slack: vzw-vsi.slack.com username: @ckonstanski




That may be, but it was unfair to get angry at the ASDF maintainers 
about this.  This is just a pre-existing error that was *manifested* 
because of a change in ASDF.  It's not our fault that this error 
appeared, it's not our fault that the puri library is no longer 
maintained, and we can't be expected to avoid releasing improved 
versions of ASDF because there exists buggy, unmaintained code in 
Quicklisp.


An apology from you would be appropriate at this point.

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[73budden .]

Warning about modifying standard readtable is issued by SBCL (at least
in my SBCL 1.3.18), please grep for the message in SBCL sources, and
it seem to be introduced in 1.0.24. I took a look at puri's definition
 (maybe old, in my local copy of quicklisp) and it looks like it
actually tries to modify standard readtable. But here it does not
happen when it is being loaded via my patched asdf 3.1.6 and my own
IDE with forked SLIME with many systems pre-loaded. I don't know why
and have no time to investigate further. Obviously, my insights would
be of limited use because my environment is very different from that
of OP.

> So we might be able to help you debug this
It'd be very nice to have instructions on how to trace what happens
while system is being built:that is, which files are compiled, which
are loaded and what is a readtable in the beginning of each load
operation.

Running builds in old and new asdf versions and comparing logs it
would be relatively easy to figure out what is wrong.

I guess that in OP's setup something had changed readtable before, but
this does not happen anymore. And obviously many people would face
similar problems.

This tracing tool should help a lot.

I believe this tool should be supplied by asdf team. Even I begin to
be more positive towards efforts of ASDF team to clean up all the mess
that was in ASDF initially, but obviously society is not quite happy
with breaking changes, so some small tool with a good manual would
make life easier.

Printing readtable before loading, I think, requires just a line or
two. Dumping log of operations might be one (trace) call, so that's
trivial for the person who knows how ASDF is organized. Writing a
small two-paragraph addition to manual would relief a lot of pain and
stress for all.

WBR, Budden

2017-10-12 18:46 GMT+03:00, Robert Goldman :
> I'm with Faré on this one.  I don't see evidence that this change is
> because ASDF is doing something bad.  I believe it's consistent with the
> hypothesis that there was some imperfectly-controlled aspect of building
> that is done differently now (e.g., files loaded in a different order
> where both orders are compatible with the constraints in the system
> definitions).
>
> This doesn't even look like an ASDF error to me -- it looks like an
> error that occurred while loading a system that ASDF has re-packaged.
>
> So we might be able to help you debug this, but without more evidence,
> there's no reason to believe that ASDF has done anything to you: it
> looks like the change in ASDF just reveals a pre-existing bug.
>
>
>

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[73budden .]

Warning about modifying standard readtable is issued by SBCL (at least
in my SBCL 1.3.18), please grep for the message in SBCL sources, and
it seem to be introduced in 1.0.24. I took a look at puri's definition
 (maybe old, in my local copy of quicklisp) and it looks like it
actually tries to modify standard readtable. But here it does not
happen when it is being loaded via my patched asdf 3.1.6 and my own
IDE with forked SLIME with many systems pre-loaded. I don't know why
and have no time to investigate further. Obviously, my insights would
be of limited use because my environment is very different from that
of OP.

> So we might be able to help you debug this
It'd be very nice to have instructions on how to trace what happens
while system is being built:that is, which files are compiled, which
are loaded and what is a readtable in the beginning of each load
operation.

Running builds in old and new asdf versions and comparing logs it
would be relatively easy to figure out what is wrong.

I guess that in OP's setup something had changed readtable before, but
this does not happen anymore. And obviously many people would face
similar problems.

This tracing tool should help a lot.

I believe this tool should be supplied by asdf team. Even I begin to
be more positive towards efforts of ASDF team to clean up all the mess
that was in ASDF initially, but obviously society is not quite happy
with breaking changes, so some small tool with a good manual would
make life easier.

Printing readtable before loading, I think, requires just a line or
two. Dumping log of operations might be one (trace) call, so that's
trivial for the person who knows how ASDF is organized. Writing a
small two-paragraph addition to manual would relief a lot of pain and
stress for all.

WBR, Budden

2017-10-12 18:46 GMT+03:00, Robert Goldman :
> I'm with Faré on this one.  I don't see evidence that this change is
> because ASDF is doing something bad.  I believe it's consistent with the
> hypothesis that there was some imperfectly-controlled aspect of building
> that is done differently now (e.g., files loaded in a different order
> where both orders are compatible with the constraints in the system
> definitions).
>
> This doesn't even look like an ASDF error to me -- it looks like an
> error that occurred while loading a system that ASDF has re-packaged.
>
> So we might be able to help you debug this, but without more evidence,
> there's no reason to believe that ASDF has done anything to you: it
> looks like the change in ASDF just reveals a pre-existing bug.
>
>
>

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[Robert Goldman]

I'm with Faré on this one.  I don't see evidence that this change is 
because ASDF is doing something bad.  I believe it's consistent with the 
hypothesis that there was some imperfectly-controlled aspect of building 
that is done differently now (e.g., files loaded in a different order 
where both orders are compatible with the constraints in the system 
definitions).


This doesn't even look like an ASDF error to me -- it looks like an 
error that occurred while loading a system that ASDF has re-packaged.


So we might be able to help you debug this, but without more evidence, 
there's no reason to believe that ASDF has done anything to you: it 
looks like the change in ASDF just reveals a pre-existing bug.