Shims for old systems

[73budden]

Hi!

Every time I read "asdf", I feel a pain. I've read that there is an
attempt to gain resources to improve asdf. I have a sort of plan.

1. Shims. Recent tightening of rules for system definitions is ok, but
there are old systems with no maintainers. If such system does not
obey the rules, one can introduce "shim" concept. I've met them in JS
culture where they serve as third-party adapters to connect two
mismatching things.

In the simplest way shim is just an alternative directory hierarchy
with shim asd files, isomorphic to local lisp directory structure.

When looking for system, asdf must search in shims directory first,
and only then in the directory of the file itself. Also things like
quicklisp might take care of installing shims where they exist.
Maintanence of shims for all popular systems can be done within a
separate git repository.

2. Get rid of upgrade. Upgrade feature requires to maintain a 3d array
of possible cases, where dimensions are "old asdf version", "new asdf
version" and "lisp implementation". It is hard to maintain and it will
get harder and harder to maintain as the time goes on. Also upgrade is
a good test of CLOS, but running tests at the very beginning of image
bootstrap is not a good idea because there is no e.g. SLIME to work
with convenience.

3. Last, but most important actually. Prioritize manual, FAQ, Wiki and
all like this.

Instead, when loading asdf, allow the user to pass the parameter that
fills asdf database with the initial loaded system information. It
would be also good to have an utility to extract this database from
some old asdf versions. This way we have a slight chance to make
things easier, upgrade process explicit and under user's control.

Re: Syntax control code

[73budden .]

Hi! Where the document is found?

2018-01-06 3:53 GMT+03:00, Robert Goldman :
> I just pushed an edit of syntax-control.md in which I try to capture the
> terminology.
>
> Status: several Allegro failures break for me on test-syntax-control.
> Results from Linux:
>
> build/results/allegro8_64-test.text
> build/results/allegro8_64_s-test.text
> build/results/allegromodern8_64-test.text
> build/results/allegromodern8_64_s-test.text
> build/results/allegromodern_64-test.text
> build/results/allegromodern_64_s-test.text
>
> These failures seem to be due to NAMED-READTABLES not working properly
> on these platforms, rather than on anything ASDF itself does.
>
> Concern:  As I was reading over syntax-control.md, it was brought home
> to me that the ASDF shared syntax is initialized to the *initial syntax*
> on the host implementation, rather than the standard syntax of ANSI CL.
> My understanding is that this is done for backwards-compatibility with
> some QL systems that assume they have access to extended syntax from
> some implementation(s).  I'm concerned that this will create a
> maintenance headache going forward just so someone *else* can avoid
> making some minor clean-up.  Should we just suck it up and make the
> shared syntax start out with the initial syntax?  Why not break it now,
> and save ourselves trouble later?  Also, it seems like "initial syntax"
> is not well-defined, even on a single implementation, since ASDF might
> be loaded at arbitrary times, possibly after modifications to the
> "initial initial" readtable.  Finally, going forward, people will be
> yelling at *us* if implementations change their initial syntax.
>
> Unless there's a really important reason to keep this, I think we should
> kill it.
>
>

Re: Relativising FASL Cache Paths

[73budden .]

Hi!

Nicolas, can you resolve source locations at SWANK level? E.g.
functions like SWANK/SBCL::SOURCE-FILE-SOURCE-LOCATION might be
appropriate for patching: you take the location that SBCL gives you,
but then redirect SWANK to other place. I guess this is not the only
place you have to patch, it would require patching private functions
and it might not work at all. But if that works for you please let me
know. I'm also maintaining a thing a bit similar to portacle and it
looks like I would suffer from the same issue. Also let me remind you
that you can set up the path to SBCL sources by

sb-ext:set-sbcl-source-location

AFAIR locating sources of SBCL functions didn't work in portacle when
I saw it, and it was not very long ago.

Re: syntax-control

[73budden .]

Hi! My 2 cents (a draft for a proposal).

1. As I said, there should be a way to set up tracing and maybe
assertable preconditions that should hold before loading the system.
This way we can _diagnose_ what happens. This technique should be
supplied not for newer versions only, but for some old versions too,
so that one can compare the state before and after the change, and all
should be documented. I believe trace and CLOS should suffice to avoid
patching asdf sources for old versions, but if not, patches can also
go.

2. By default, bind readtable to (copy-readtable *shared-readtable*)
or (copy-readtable nil) before loading each system (I hope that
systems are loaded continuously, that is, components of one system are
not intermitted by components of another, otherwise we would do that
at a component granularity). Obviously, it would break some setups.

3. If system wants to do something special, that special must be
declared. Obvious special behavior is not to copy current readtable
before loading system - in this case all changes would affect global
readtable. One can think that another option is to use some named
readtable, but we don't need that: one can specify readtable on a
per-file basis (in fact, currently there should be no more than one
readtable per package).

4. For any additional declarations and settings, there should be a way
to declare them either in system definition itself, or in some another
place which is read before system is loaded (some per-site asd
configuration). This is for inserting declarations into unmaintained
3-d party systems.

Does that work? I guess no due to some reason, but I'm not an ASDF
expert, so I can be optimistic :)

5 (offtopic) there is a perfect solution for macro-character conflict,
which is a plain (non breaking) extension of CL. It is named
symbol-readmacros. I use them for several (maybe 7) years already, and
they just work. No pain anymore :) The idea is that macro reader is
associated to symbol, not to character. This way issue of conflicting
characters is reduced to that of conflicting symbols. We all know that
issue of conflicting symbols is resolved by package prefixes, import,
use-package and so on, so many implementations of custom this-lib:{
and that-lib:{ can coexist peacefully in the same image. Of course,
any client code should be modified to take the advantage of this, so I
usually add couple of lines for every library I use that wants to
extend syntax.

I'm not sure it can be implemented in a portable way or even w/o
changes to the implementation, but I have the implementation for SBCL
and at some time in the past I had one for LW.

I believe this symbol-readmacros is the only right thing here. There
are thousands of libraries and only dozens of characters which can be
used. So when using macro-characters, conflicts are absolutely
inevitable.



2017-10-13 23:31 GMT+03:00, Faré :
> Due to the readtable bug in ASDF 3.3.0 I updated the "syntax-control"
> branch that for the last three years was supposed to solve all
> readtable bugs when building with ASDF.
>
> I merged 3.3.0.1 into the branch, but the branch itself is not ready
> for merging into master:
> it does either too much or too little, and must be either completed or
> simplified.
> I think it stands a better chance of bieng merged soon if it is simplified,
> so I re-read and re-wrote the documentation for what this branch
> should be doing,
> and detailed a "Proposal 1" for minimal changes to ASDF that would go
> a long way towards bringing sanity in builds of software that modify
> the syntax.
>
> I sollicit your feedback:
> https://gitlab.common-lisp.net/asdf/asdf/blob/syntax-control/doc/syntax-control.md
>
> —♯ƒ • François-René ÐVB Rideau •Reflection•
> http://fare.tunes.org
> Problem of the day: finding suction cups that don't suck.
>
>

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[73budden .]

Wow! Good news.

2017-10-12 21:08 GMT+03:00, Robert Goldman :
> On 12 Oct 2017, at 13:04, Faré wrote:
>
>> For those in a hurry for a fix, here is the merge request:
>> https://gitlab.common-lisp.net/asdf/asdf/merge_requests/85
>
> I will merge this into master, but do not have time to make a new
> release, or even test much this week or next.
>
> So this will simply remain in master for now.
>
>

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[73budden .]

Warning about modifying standard readtable is issued by SBCL (at least
in my SBCL 1.3.18), please grep for the message in SBCL sources, and
it seem to be introduced in 1.0.24. I took a look at puri's definition
 (maybe old, in my local copy of quicklisp) and it looks like it
actually tries to modify standard readtable. But here it does not
happen when it is being loaded via my patched asdf 3.1.6 and my own
IDE with forked SLIME with many systems pre-loaded. I don't know why
and have no time to investigate further. Obviously, my insights would
be of limited use because my environment is very different from that
of OP.

> So we might be able to help you debug this
It'd be very nice to have instructions on how to trace what happens
while system is being built:that is, which files are compiled, which
are loaded and what is a readtable in the beginning of each load
operation.

Running builds in old and new asdf versions and comparing logs it
would be relatively easy to figure out what is wrong.

I guess that in OP's setup something had changed readtable before, but
this does not happen anymore. And obviously many people would face
similar problems.

This tracing tool should help a lot.

I believe this tool should be supplied by asdf team. Even I begin to
be more positive towards efforts of ASDF team to clean up all the mess
that was in ASDF initially, but obviously society is not quite happy
with breaking changes, so some small tool with a good manual would
make life easier.

Printing readtable before loading, I think, requires just a line or
two. Dumping log of operations might be one (trace) call, so that's
trivial for the person who knows how ASDF is organized. Writing a
small two-paragraph addition to manual would relief a lot of pain and
stress for all.

WBR, Budden

2017-10-12 18:46 GMT+03:00, Robert Goldman :
> I'm with Faré on this one.  I don't see evidence that this change is
> because ASDF is doing something bad.  I believe it's consistent with the
> hypothesis that there was some imperfectly-controlled aspect of building
> that is done differently now (e.g., files loaded in a different order
> where both orders are compatible with the constraints in the system
> definitions).
>
> This doesn't even look like an ASDF error to me -- it looks like an
> error that occurred while loading a system that ASDF has re-packaged.
>
> So we might be able to help you debug this, but without more evidence,
> there's no reason to believe that ASDF has done anything to you: it
> looks like the change in ASDF just reveals a pre-existing bug.
>
>
>

Re: [E] new compiler error re: SET-DISPATCH-MACRO-CHARACTER

[73budden .]

Warning about modifying standard readtable is issued by SBCL (at least
in my SBCL 1.3.18), please grep for the message in SBCL sources, and
it seem to be introduced in 1.0.24. I took a look at puri's definition
 (maybe old, in my local copy of quicklisp) and it looks like it
actually tries to modify standard readtable. But here it does not
happen when it is being loaded via my patched asdf 3.1.6 and my own
IDE with forked SLIME with many systems pre-loaded. I don't know why
and have no time to investigate further. Obviously, my insights would
be of limited use because my environment is very different from that
of OP.

> So we might be able to help you debug this
It'd be very nice to have instructions on how to trace what happens
while system is being built:that is, which files are compiled, which
are loaded and what is a readtable in the beginning of each load
operation.

Running builds in old and new asdf versions and comparing logs it
would be relatively easy to figure out what is wrong.

I guess that in OP's setup something had changed readtable before, but
this does not happen anymore. And obviously many people would face
similar problems.

This tracing tool should help a lot.

I believe this tool should be supplied by asdf team. Even I begin to
be more positive towards efforts of ASDF team to clean up all the mess
that was in ASDF initially, but obviously society is not quite happy
with breaking changes, so some small tool with a good manual would
make life easier.

Printing readtable before loading, I think, requires just a line or
two. Dumping log of operations might be one (trace) call, so that's
trivial for the person who knows how ASDF is organized. Writing a
small two-paragraph addition to manual would relief a lot of pain and
stress for all.

WBR, Budden

2017-10-12 18:46 GMT+03:00, Robert Goldman :
> I'm with Faré on this one.  I don't see evidence that this change is
> because ASDF is doing something bad.  I believe it's consistent with the
> hypothesis that there was some imperfectly-controlled aspect of building
> that is done differently now (e.g., files loaded in a different order
> where both orders are compatible with the constraints in the system
> definitions).
>
> This doesn't even look like an ASDF error to me -- it looks like an
> error that occurred while loading a system that ASDF has re-packaged.
>
> So we might be able to help you debug this, but without more evidence,
> there's no reason to believe that ASDF has done anything to you: it
> looks like the change in ASDF just reveals a pre-existing bug.
>
>
>

Re: Depending on and generating files in system definitions

[73budden .]

What about #. ?

Re: How to fix arnesi:clean-op.

[73budden .]

I just noticed that touching sources helps a bit to work around this
bug, nothing more. Maybe :force helps too, but I didn't check that.

>Note that I'm not interested in working on this bug at this point, but
>will happily help clean up and merge a patch that you'd submit.

>>> 3. Am I right that bug with incorrect system definition which loads
>>> "successfully" is not fixed yet? Touching the source would help to
>>> work around it.

Re: How to fix arnesi:clean-op.

[73budden .]

>> 3. Am I right that bug with incorrect system definition which loads
>> "successfully" is not fixed yet? Touching the source would help to
>> work around it.
>>
>  But yes, if you have a bug in your .asd file, it's a bug.

I mean just the following known bug in asdf:
; file wrong-system.asd
(defsystem :wrong-system :components
 (:some-trash)
)
; EOF

This is SBCL 1.3.4, an implementation of ANSI Common Lisp.
* (require 'asdf)

("ASDF" "asdf" "UIOP" "uiop")
* (asdf:asdf-version)

"3.1.6"
* (setf asdf:*central-registry* '("c:/clcon/lp/"))

* (asdf:load-system :wrong-system)
While evaluating the form starting at line 1, column 0
  of #P"c:/clcon/lp/wrong-system.asd":

debugger invoked on a LOAD-SYSTEM-DEFINITION-ERROR ... error while parsing
arguments to DESTRUCTURING-BIND ...

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY] Retry EVAL of current toplevel form.
  1: [CONTINUE ] Ignore error and continue loading "wrong-system.asd".
  2: [ABORT] Abort loading file "wrong-system.asd"
  ...

0] 2

T
* (asdf:load-system :wrong-system)

T
*

Re: How to fix arnesi:clean-op.

[73budden .]

Hi! Some new thoughts:

1. Sometimes we (developers) fail to describe system dependencies
correctly. Sometimes we just don't know the exact dependency graph.
But we feel that something is wrong with A system. Just deleting fasls
or touching the source or specific system helps to diagnose this.

2. Sometimes we want to clean system A and then reload system B which
depends on A.

3. Am I right that bug with incorrect system definition which loads
"successfully" is not fixed yet? Touching the source would help to
work around it.

These are just some reasons to extend possible use-case list of
clean-op or what whould stand for it.

>I am inclined to agree with Andreas.  I don't see that CLEAN-OP is
>impossible for ASDF.  I can see that there are some issues that require
>thought, and for some systems additional coding by the developer, but I
>don't see that as an insuperable barrier.

Re: How to fix arnesi:clean-op.

[73budden .]

Hi!

Maybe :force works well now, but I had problems with it in the past
(don't remember which problems exactly). TANIGUCHI Masaya, you can
consult asdf source as the "canonical" example of operations.

Re: How to fix arnesi:clean-op.

[73budden .]

Hi Masaya, list!

  I use batch file which deletes all fasl files to do complete
cleanup. If I wanted to force recompilation of a particular system, I
would just "touch" all files.

  See also 
https://en.wikibooks.org/wiki/Common_Lisp/External_libraries/ASDF/Budden's_infrequently_asked_questions
- it describes a way to find fasls of the system.

Re: ASDF3: How to globally disable output translations on Windows

[73budden .]

Hi!
  This works for me with asdf 3.1.6

  
https://en.wikibooks.org/wiki/Common_Lisp/External_libraries/ASDF/Budden%27s_infrequently_asked_questions#How_do_I_make_portable_lisp_program_with_it.27s_own_fasl_cache_in_a_given_directory.3F

Re: A modest proposition: DEFSYSTEM-DEPENDS-ON should die [was Re: What's the right way to extend ASDF with new symbols?]

[73budden .]

Hi!

>In the old days, we would just put
>(asdf:load-system "extension")
That's good. This is declarative enough to be processed by tools like
quicklisp.

What goes to side effects, they are just inevitable if we want

i) asdf to be extensible
ii) some dependency language is used to declarate which
system-extension relationships

The only way to avoid side effects at asd file processing time is to
load all extensions together with asdf, but this is what we want to
avoid when we talk about build system, isnt' it?

Re: bug: after erring on undefined component type, system is reported to load successfully

[73budden .]

Hi!

I added new bugs

https://bugs.launchpad.net/asdf/+bug/1532584
https://bugs.launchpad.net/asdf/+bug/1532586

and reported that bug affects me:
https://bugs.launchpad.net/asdf/+bug/1500578

Re: bug: after erring on undefined component type, system is reported to load successfully

[73budden .]

Hi! I will open the bug, but first of all there seem to be another issue:

My system :s3 :defsystem-depends-on :gro.

I touch some file belonging to :gro.
Then I (asdf:load-system :s3).
Gro is not recompiled.

Is this the same issue or another one?

Should I combine both into one bug report or open two separate bugs?
Or is this a feature? I  didn't find its description in the manual.

bug: after erring on undefined component type, system is reported to load successfully

[73budden .]

Version 3.1.5,

Example system definition:

(defsystem :s3
  :components
  ((cl-user::cons "file-name-does-not-matter")
   ))
;; eof

I try to load it, it says:

"Error while trying to load definition for system s3 from
pathname c:/clcon/lp/test/s3.asd:
   don't recognize component type CONS"

Then I try to load it another time, it says:

T

I believe that load-system is described incorrectly in the manual

[73budden .]

Quote from the manual:
"But for code that you are actively developing, debugging, or
otherwise modifying, you should use load-system, so ASDF will pick on
your modifications and transitively re-build the modified files and
everything that depends on them. "

What does "everything that depends on them" mean?

I have
sys1.asd:
(asdf:defsystem :sys1 :serial t :components ((:file "sys1-file1")))
; EOF

and
sys2.asd
(asdf:defsystem :sys2 :serial t :depends-on (:sys1) :components
((:file "sys2-file2")))
; EOF

I do (asdf:load-system :sys2), then I touch sys1-file1.lisp, then I do
(asdf:load-system :sys1)

:sys2 does not get recompiled.

This is what I always knew and expected to see, but, according to
manual, it should have been recompiled, as :sys2 depends on :sys1,
and, hence, from the modified sys1-file1.lisp.

Now I don't know is this a "misbug" in asdf or a defeciency in the
manual. If this is a misbug, another question arises: is there a
"canonical" way to reload all systems that are currently loaded into
the image and sources of which have changed since last load?

Re: [Q] figuring out a system / source file

[73budden .]

I defined variables which show current component and current system.
They can be used by IDEs to navigate to sources, or by your code to do
what you need.

I suggested this as a modification of asdf, but asdf maintainers said
that special variables are dangerous. Maybe.

My code is here:
https://bitbucket.org/budden/budden-tools/src/default/asdf-3.1.4-tools.lisp?at=default=file-view-default

Relocating asdf fasl cache

[73budden .]

Hi! I make a "portable file release" of my software. I want that user could just
unpack archive to some "my" directory and than have the software completely
independent from any other possible CL installations on this computer.
I found (too) many words about output translations in asdf manual, but
I was unable to find out how I will do what I need.

My solution and relevant portions in my .sbclrc (for SBCL) follow:

(require :uiop)
(setf uiop::*user-cache* #P"c:/clcon/fasl-cache/asdf/")
(require 'asdf)
;;; The following lines added by ql:add-to-init-file:
#-quicklisp
(let ((quicklisp-init "c:/clcon/quicklisp/setup.lisp"
   ))
  (when (probe-file quicklisp-init)
(load quicklisp-init)))
; swank is handled separatedly (maybe irrelevant to asdf, but relevant
; to task of creating "portable file releases"
(load (merge-pathnames "swank-loader.lisp" (ql:where-is-system :swank)))
(setq swank-loader::*fasl-directory* "c:/clcon/fasl-cache/swank/")
(swank-loader:init)

Problem with current docs is that *user-cache* is mentioned in docs, but
it is not stated _when_ the default is calculated. This variable have no default
value, but it has a function which calculates it. It is not evident
when this function
is called. So I needed an experiment. I even don't know when value I supplied
will be reset back to hard-coded value.

If you find it useful, you can consider including it into asdf
doc/wiki/FAQ, or address
me to the appropriate place.

Thank you!

Re: restarts vs debugger command

[73budden .]

Hi!

Thanks for the hint. I took a look at sources. Both solutions have
their advantages.

Slime-asdf works in any CL which supports SLIME. My solution works in
SBCL and Lispworks only but does not require SLIME.

Either variant can be prefferred in different situations.
What goes to name coersion, I'll try to take a look at it in a couple
weeks after my vacations.

Re: Almost there

[73budden .]

Hi!
 SBCL on Windows isn't quite as good (and is not able to call out to CMD.EXE)
I encountered the problem recently, reported a bug and I seem to have
a workaround. Feel free to experiment with it.

https://bugs.launchpad.net/sbcl/+bug/1470500

Re: restarts vs debugger command

[73budden .]

Hi!
 I believe *current-component* is rather innocent
NOT AT ALL.
Design choices is a great topic for holywar :)  If variables are not
innocent for you, then you just not accept my patch. I'll continue to
use it for myself. Unfortunately, I have no time and desire to reshape
it, at least in the near future.

The other topic is finding definitions.
 What do you think about
 (keywordize (string-upcase (coerce-name name)))
 ?
 Sounds good (at least assuming no one's using mlisp), but then issue
 an error if it doesn't convert back to the same name when you
 coerce-name it back.
This is not I who coerce it back. It does lisp reader. In standard
readtable no problem should occur IMO. In non-standard readtable all
source location handling must be reviewed. So I see no additional work
to be done. If I'm wrong, please give a counter-example. If this
suggestion can be accepted, tell me how I should reshape it.

And third topic is of-system. There are cases where relation between
file and system is essential. Sometimes there are several systems in
one directory (e.g. cl-ppcre). Sometimes code is located in several
subdirectories.

(of-system :mysystem) declares that the file is in the system. I press
M-. on :mysystem and jump to its definition. This is convenient.

Runtime checking guarantees that declaration is always correct. I
can't forget changing it when I refactor my systems.

But there is other possibility: in latest projects I use EMACS
modeline variable instead of (of-system). I write -*- system
:my-system; -*- and I have lispworks command find current system
definition which takes system name from modeline and jumps to system
definition. Similar command can be written for SLIME/EMACS. This is
even more convenient than of-system, but requires more effort to port.
Asdf can even check correctness of modeline if it would open file,
parse its modeline and compare declared system with real prior to
compilation. Thus you can get rid of special variables. But this is
not extremely important.

Re: restarts vs debugger command

[73budden .]

I see you mean situation where we have systems

(defsystem Bad2)
(defsystem bAd2)
(defsystem :|baD2|)

In this case we can navigate to :|baD2| by name :bad2, and unable to
navigate to first two systems. Where should I err? According to
coerce-name rules, :|baD2|, :bad2 and bad2 are identical system name
designators, so my code works correctly. I can also find bad2 by
:bad2

What goes to Bad2 and bAd2, slime-edit-definition IMO accepts
symbol names, not strings. So user will never be able to navigate to
them via M-. . The only reasonable thing to do IMO is to avoid
recording definition for systems named by strings. Right?

Re: restarts vs debugger command

[73budden .]

 Why would a file be loaded through the wrong system?
Sorry, I have no time to dig into techical details of that kind. I
believe *current-component* is rather innocent and sometimes useful,
but this is not so important. What I want, is that editing feature
would be available to all asdf users. However, I'm not sure I'm ready
to change something in my implementaion. Code I have just works in the
way which satisfies me. I see you catched the idea, so now you can
implement it yourself in a form which you think is better. In fact, it
requres about a dozen lines of code only together with customizible
editor hook.

If this is a limitation in your debugger, by all means, convert to
keyword, but then:
1- do something magic (case inversion if it's reversible, or else
error?) to convert from string to keyword.
2- use coerce-name (or the reverse of your magic transformation) to o
back to system name.
What you must not do is silently do the wrong thing when the system
name isn't equal to the string-downcase of your keywordification.
What do you think about
(keywordize (string-upcase (coerce-name name)))
?

Re: Editing component being compiled when compilation failed.

[73budden .]

I see my previous reply was unclear

It looks like what you are building is an
extension to ASDF

not an extension but a patch

that hooks it into an editor

yes

(perhaps specifically the
lispworks IDE?).

No. I use lispworks IDE for Lispworks and EMACS (via SWANK callback)
for SBCL. EMACS/SWANK should work with all lisps supported by SLIME.