Re: monolithic-concatenate-source-op misses a few dependencies

[Ben Vulpes]

Fare et al.,

The Postmodern/usocket problem I encountered was in fact due to
cl-postgres not setting a feature flag in an :execute context:
https://github.com/marijnh/Postmodern/blob/57dc8cb4acc4599aee757e8f81999a0fa83c7111/cl-postgres/features.lisp#L8
. To digress from the list topic for half a second, this confuses me,
as that code refused to execute until I patched cl-postgres at that
point to also eval-when in :execute, even when loading the
concatenated sources with sbcl ... --load full-concatenation.lisp

More relevant to the list, other observations include: chipz mutates
*features* in :chipz-system, a package defined in chipz.asd, which
causes problems because the .asd files are not concatenated with the
sources (not suggesting this is something asdf should even concern
itself with); and both drakma and usocket call (asdf:component-version
(asdf:find-system ...)) at read time, which fails to find the package
in question when loading the concatenated source on a machine without
those packages in *central-registry*, because (I believe) as noted
above, monolithic-concatenate-source-op doesn't include system .asd
files. Again, I'm not suggesting that this behavior change.

Yours,
Ben

On Sat, Oct 14, 2017 at 11:52 AM, Faré  wrote:
> On Sat, Oct 14, 2017 at 1:17 PM, Robert Goldman  wrote:
>> Will you please clarify for my benefit, since I don't actually use any of
>> the image operations.
>>
>> Is the problem that somewhere in the process of loading Postmodern, or one
>> of its dependencies, some bit of code invokes REQUIRE? Or is this an issue
>> with ASDF's REQUIRE-SYSTEM.
>>
> monolithic-concatenate-source-op is supposed to create a source file
> that has "all the (transitive) source code required to load a system".
> Problem is, it is naive in that the current implementation believes
> there are only source files to concatenate. And so it doesn't handle
> (:require "foo") dependencies which would require to generate a
> temporary "source file" that contains (require "FOO") in it — or
> better a refactoring such that instead of source files, you use
> arbitrary arguments to vomit-output, and for require dependencies you
> use a function that outputs that instead of a pathname  (and wrap a
> map () 'vomit-output in a with-output-file).
>
>> If it's the former, then I believe this is simply a bad implementation in
>> the relevant system.
>>
>> If it's the latter, we should do something about it. However, I believe that
>> REQUIRE-SYSTEM, despite its name, doesn't actually use REQUIRE, instead it
>> "acts like require."
>>
> Well, there are unhappily two things called "require-system": the
> class that backs the (:require "foo") dependencies, and the function
> that is now deprecated because it didn't have a good composable
> meaning in presence of multiple build phases. The one that matters
> here is the former.
>
> —♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
> All problems in computer science can be solved by another level of indirection
> — David Wheeler
> Almost all programming can be viewed as an exercise in caching
> — Terje Mathisen, well-known programming optimization guru



-- 
Ben Vulpes
503-313-5838
MAVN

Re: syntax-control

[73budden .]

Hi! My 2 cents (a draft for a proposal).

1. As I said, there should be a way to set up tracing and maybe
assertable preconditions that should hold before loading the system.
This way we can _diagnose_ what happens. This technique should be
supplied not for newer versions only, but for some old versions too,
so that one can compare the state before and after the change, and all
should be documented. I believe trace and CLOS should suffice to avoid
patching asdf sources for old versions, but if not, patches can also
go.

2. By default, bind readtable to (copy-readtable *shared-readtable*)
or (copy-readtable nil) before loading each system (I hope that
systems are loaded continuously, that is, components of one system are
not intermitted by components of another, otherwise we would do that
at a component granularity). Obviously, it would break some setups.

3. If system wants to do something special, that special must be
declared. Obvious special behavior is not to copy current readtable
before loading system - in this case all changes would affect global
readtable. One can think that another option is to use some named
readtable, but we don't need that: one can specify readtable on a
per-file basis (in fact, currently there should be no more than one
readtable per package).

4. For any additional declarations and settings, there should be a way
to declare them either in system definition itself, or in some another
place which is read before system is loaded (some per-site asd
configuration). This is for inserting declarations into unmaintained
3-d party systems.

Does that work? I guess no due to some reason, but I'm not an ASDF
expert, so I can be optimistic :)

5 (offtopic) there is a perfect solution for macro-character conflict,
which is a plain (non breaking) extension of CL. It is named
symbol-readmacros. I use them for several (maybe 7) years already, and
they just work. No pain anymore :) The idea is that macro reader is
associated to symbol, not to character. This way issue of conflicting
characters is reduced to that of conflicting symbols. We all know that
issue of conflicting symbols is resolved by package prefixes, import,
use-package and so on, so many implementations of custom this-lib:{
and that-lib:{ can coexist peacefully in the same image. Of course,
any client code should be modified to take the advantage of this, so I
usually add couple of lines for every library I use that wants to
extend syntax.

I'm not sure it can be implemented in a portable way or even w/o
changes to the implementation, but I have the implementation for SBCL
and at some time in the past I had one for LW.

I believe this symbol-readmacros is the only right thing here. There
are thousands of libraries and only dozens of characters which can be
used. So when using macro-characters, conflicts are absolutely
inevitable.



2017-10-13 23:31 GMT+03:00, Faré :
> Due to the readtable bug in ASDF 3.3.0 I updated the "syntax-control"
> branch that for the last three years was supposed to solve all
> readtable bugs when building with ASDF.
>
> I merged 3.3.0.1 into the branch, but the branch itself is not ready
> for merging into master:
> it does either too much or too little, and must be either completed or
> simplified.
> I think it stands a better chance of bieng merged soon if it is simplified,
> so I re-read and re-wrote the documentation for what this branch
> should be doing,
> and detailed a "Proposal 1" for minimal changes to ASDF that would go
> a long way towards bringing sanity in builds of software that modify
> the syntax.
>
> I sollicit your feedback:
> https://gitlab.common-lisp.net/asdf/asdf/blob/syntax-control/doc/syntax-control.md
>
> —♯ƒ • François-René ÐVB Rideau •Reflection•
> http://fare.tunes.org
> Problem of the day: finding suction cups that don't suck.
>
>

Re: monolithic-concatenate-source-op misses a few dependencies

[Faré]

On Sat, Oct 14, 2017 at 1:17 PM, Robert Goldman  wrote:
> Will you please clarify for my benefit, since I don't actually use any of
> the image operations.
>
> Is the problem that somewhere in the process of loading Postmodern, or one
> of its dependencies, some bit of code invokes REQUIRE? Or is this an issue
> with ASDF's REQUIRE-SYSTEM.
>
monolithic-concatenate-source-op is supposed to create a source file
that has "all the (transitive) source code required to load a system".
Problem is, it is naive in that the current implementation believes
there are only source files to concatenate. And so it doesn't handle
(:require "foo") dependencies which would require to generate a
temporary "source file" that contains (require "FOO") in it — or
better a refactoring such that instead of source files, you use
arbitrary arguments to vomit-output, and for require dependencies you
use a function that outputs that instead of a pathname  (and wrap a
map () 'vomit-output in a with-output-file).

> If it's the former, then I believe this is simply a bad implementation in
> the relevant system.
>
> If it's the latter, we should do something about it. However, I believe that
> REQUIRE-SYSTEM, despite its name, doesn't actually use REQUIRE, instead it
> "acts like require."
>
Well, there are unhappily two things called "require-system": the
class that backs the (:require "foo") dependencies, and the function
that is now deprecated because it didn't have a good composable
meaning in presence of multiple build phases. The one that matters
here is the former.

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
All problems in computer science can be solved by another level of indirection
— David Wheeler
Almost all programming can be viewed as an exercise in caching
— Terje Mathisen, well-known programming optimization guru

Re: monolithic-concatenate-source-op misses a few dependencies

[Robert Goldman]

Will you please clarify for my benefit, since I don't actually use any 
of the image operations.


Is the problem that somewhere in the process of loading Postmodern, or 
one of its dependencies, some bit of code invokes `REQUIRE`?  Or is this 
an issue with ASDF's `REQUIRE-SYSTEM`.


If it's the former, then I believe this is simply a bad implementation 
in the relevant system.


If it's the latter, we should do something about it.  However, I believe 
that `REQUIRE-SYSTEM`, despite its name, doesn't actually use `REQUIRE`, 
instead it "acts like `require`."


Thanks,
R

On 14 Oct 2017, at 11:45, Faré wrote:


Dear Ben,

sorry I won't be developing this feature, but I'll happily merge your
patch if you do. "Just" add support for dumping source code for a
(:require ...) dependency.

—♯ƒ • François-René ÐVB Rideau •Reflection• 
http://fare.tunes.org
Guns & bullets don't kill people — blood loss and organ damage kills 
people.



On Thu, Oct 12, 2017 at 4:07 PM, Ben Vulpes  wrote:

On SBCL 1.3.11, when producing a monolithic source concatenation with
the library "Postmodern", asdf produces a file that needs a Lisp 
image
to need manual calls to (require :usocket) and (require :md5) in 
order

to load completely.

Given:

concatenatrix.asd as:

(asdf:defsystem :concatenatrix
  :build-operation monolithic-concatenate-source-op
  :build-pathname "build/full-concatenation"
  :depends-on (:postmodern)
  :components
  ((:file "concatenatrix")))

concatenatrix.lisp as:

(defpackage :concatenatrix
  (:use :cl :postmodern))
(in-package :concatenatrix)

(defun wat (it)
  (format t "~A~%" it))

Concatenated sources produced with:

(asdf:make :concatenatrix)

Loading tested with:

sbcl --noinform --disable-debugger --load 
build/full-concatenation.lisp


Produces:

Unhandled SB-C::INPUT-ERROR-IN-LOAD in thread #"main

thread" RUNNING
 {10027FE873}>:
  READ error during LOAD:

Package SB-ROTATE-BYTE does not exist.

  Line: 221, Column: 29, File-Position: 8706

  Stream: #


Backtrace for: #
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK # #)
1: (SB-DEBUG::RUN-HOOK SB-EXT:*INVOKE-DEBUGGER-HOOK*
#)
2: (INVOKE-DEBUGGER #)
3: (ERROR #)
4: (SB-C:COMPILER-ERROR SB-C::INPUT-ERROR-IN-LOAD :CONDITION
# :STREAM #)
5: (SB-C::%DO-FORMS-FROM-INFO # #
SB-C::INPUT-ERROR-IN-LOAD)
6: (SB-INT:LOAD-AS-SOURCE # :VERBOSE NIL :PRINT NIL :CONTEXT "loading")
7: ((FLET SB-FASL::LOAD-STREAM :IN LOAD) 
#for "file 
/home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp"

{1003CFCA03}> NIL)
8: (LOAD #P"build/full-concatenation.lisp" :VERBOSE NIL :PRINT NIL
:IF-DOES-NOT-EXIST T :EXTERNAL-FORMAT :DEFAULT)
9: (SB-IMPL::PROCESS-EVAL/LOAD-OPTIONS ((:LOAD .
"build/full-concatenation.lisp")))
10: (SB-IMPL::TOPLEVEL-INIT)
11: ((FLET #:WITHOUT-INTERRUPTS-BODY-90 :IN 
SB-EXT:SAVE-LISP-AND-DIE))

12: ((LABELS SB-IMPL::RESTART-LISP :IN SB-EXT:SAVE-LISP-AND-DIE))

There is another complaint about usocket, which is confusing, as
cl-postgres explicitly doesn't require usocket on sbcl by my read (
http://marijnhaverbeke.nl/git/?p=postmodern;a=blob;f=cl-postgres.asd;h=683edf0f131a4ebe172b44425f597d7f67656e70;hb=HEAD#l16
).

Loading is resolved by requiring both libraries in question, as:

sbcl --eval "(require :md5)" --eval "(require :usocket)" --load
build/full-concatenation.lisp

Yours,
Benjamin

Re: monolithic-concatenate-source-op misses a few dependencies

[Faré]

Dear Ben,

sorry I won't be developing this feature, but I'll happily merge your
patch if you do. "Just" add support for dumping source code for a
(:require ...) dependency.

—♯ƒ • François-René ÐVB Rideau •Reflection• http://fare.tunes.org
Guns & bullets don't kill people — blood loss and organ damage kills people.


On Thu, Oct 12, 2017 at 4:07 PM, Ben Vulpes  wrote:
> On SBCL 1.3.11, when producing a monolithic source concatenation with
> the library "Postmodern", asdf produces a file that needs a Lisp image
> to need manual calls to (require :usocket) and (require :md5) in order
> to load completely.
>
> Given:
>
> concatenatrix.asd as:
>
> (asdf:defsystem :concatenatrix
>   :build-operation monolithic-concatenate-source-op
>   :build-pathname "build/full-concatenation"
>   :depends-on (:postmodern)
>   :components
>   ((:file "concatenatrix")))
>
> concatenatrix.lisp as:
>
> (defpackage :concatenatrix
>   (:use :cl :postmodern))
> (in-package :concatenatrix)
>
> (defun wat (it)
>   (format t "~A~%" it))
>
> Concatenated sources produced with:
>
> (asdf:make :concatenatrix)
>
> Loading tested with:
>
> sbcl --noinform --disable-debugger --load build/full-concatenation.lisp
>
> Produces:
>
> Unhandled SB-C::INPUT-ERROR-IN-LOAD in thread # thread" RUNNING
>  {10027FE873}>:
>   READ error during LOAD:
>
> Package SB-ROTATE-BYTE does not exist.
>
>   Line: 221, Column: 29, File-Position: 8706
>
>   Stream: # /home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp"
> {1003CFCA03}>
>
>
> Backtrace for: #
> 0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK # {1004014AC3}> #)
> 1: (SB-DEBUG::RUN-HOOK SB-EXT:*INVOKE-DEBUGGER-HOOK*
> #)
> 2: (INVOKE-DEBUGGER #)
> 3: (ERROR #)
> 4: (SB-C:COMPILER-ERROR SB-C::INPUT-ERROR-IN-LOAD :CONDITION
> # {10040149E3}> :STREAM # /home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp"
> {1003CFCA03}>)
> 5: (SB-C::%DO-FORMS-FROM-INFO # :CURRENT-INDEX ) :IN SB-INT:LOAD-AS-SOURCE)
> {1003D004FB}> #
> SB-C::INPUT-ERROR-IN-LOAD)
> 6: (SB-INT:LOAD-AS-SOURCE # /home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp"
> {1003CFCA03}> :VERBOSE NIL :PRINT NIL :CONTEXT "loading")
> 7: ((FLET SB-FASL::LOAD-STREAM :IN LOAD) # for "file 
> /home/b/quicklisp/local-projects/concatenatrix/build/full-concatenation.lisp"
> {1003CFCA03}> NIL)
> 8: (LOAD #P"build/full-concatenation.lisp" :VERBOSE NIL :PRINT NIL
> :IF-DOES-NOT-EXIST T :EXTERNAL-FORMAT :DEFAULT)
> 9: (SB-IMPL::PROCESS-EVAL/LOAD-OPTIONS ((:LOAD .
> "build/full-concatenation.lisp")))
> 10: (SB-IMPL::TOPLEVEL-INIT)
> 11: ((FLET #:WITHOUT-INTERRUPTS-BODY-90 :IN SB-EXT:SAVE-LISP-AND-DIE))
> 12: ((LABELS SB-IMPL::RESTART-LISP :IN SB-EXT:SAVE-LISP-AND-DIE))
>
> There is another complaint about usocket, which is confusing, as
> cl-postgres explicitly doesn't require usocket on sbcl by my read (
> http://marijnhaverbeke.nl/git/?p=postmodern;a=blob;f=cl-postgres.asd;h=683edf0f131a4ebe172b44425f597d7f67656e70;hb=HEAD#l16
> ).
>
> Loading is resolved by requiring both libraries in question, as:
>
> sbcl --eval "(require :md5)" --eval "(require :usocket)" --load
> build/full-concatenation.lisp
>
> Yours,
> Benjamin
>