Re: [PATCH] add language/wisp to Guile?

[Christine Lemmer-Webber]

"Philip McGrath"  writes:

> I've sort of alluded above to my pipe dream of a grand unified future
> for Racket-and-Guile-on-Chez, Guile-and-Racket-on-the-Guile-VM, and
> endless other possibilities. I wrote about it in more detail on the
> guix-devel list at [10]. (These thoughts were inspired by
> conversations with Christine Lemmer-Webber, though she bears no
> responsibility for my zany imaginings.)

It's a long email so I'm only quoting the part that mentions me.  ;)

In general I think the hash-lang idea in Racket is a neat one.  File
extensions are also possible I suppose, but ultimately, when the program
boots, there has to be some sort of way of finding the "current
configuration" of languages.

Note that #langs have an ambient authority problem, if we had a more
ocap'y system, but... we're far from that, any module can do a mess of
things, so I suppose if we're relying on the module system with ambient
authority already, the same situation may apply.

One thing that's worth noting is that languages do *two things* (or
three): they provide a reader, and they provide an execution model
(including an initial set of bindings).  It would be nice if we could
separate those two things as much as possible, when it is possible.
I think readers are the least interesting part of language design
usually.  But obviously I support Wisp here, so... ;)

 - Christine

Re: [PATCH] add language/wisp to Guile?

[Christine Lemmer-Webber]

Maxime Devos  writes:

> [[PGP Signed Part:Undecided]]
> Op 30-09-2023 om 15:17 schreef Christine Lemmer-Webber:
>> Haven't fully caught up on this thread, but as a side note I have a
>> mostly-finished implementation of a Wisp parser which takes a very
>> different approach than Arne's, and was more understandable to me
>> personally, a bit more functional and recursive-descent style.
>> I could make it available if anyone is curious.
>> Would love to see Wisp in Guile proper!
>
> I think I technically can't count as ‘curious’ here, as I don't think
> I'll actually read this other implementation, but I would be
> interested in it nonetheless (to potentially replace Arne's
> implementation later, once it's complete), because it sounds like you
> might have avoided the REPR (*) bug thing from Arne's implementation.
>
> (*) Extremely unlikely to be a problem in practice, but still a bug.
>
> Best regards,
> Maxime Devos.

Well it looks like I did make it available already, I just forgot, and
didn't advertise it much:

https://gitlab.com/dustyweb/rewisp/-/blob/main/rewisp.scm?ref_type=heads

It really isn't very complete, but here's some of the example data I was
using:

(define wisp-fac-noinfix-repeat
  "\
define : factorial n  ; foo
__  if : zero? n
   . 1
   * n : factorial : - n 1

define : factorial n
__  if : zero? n
   . 1
   * n : factorial : - n 1")


scheme@(rewisp)> ,pp (parse-lines->sexp (call-with-input-string 
wisp-fac-noinfix-repeat read-wisp-lines))
$6 =
((define (factorial n)
   (if (zero? n) 1 (* n (factorial (- n 1)
 (define (factorial n)
   (if (zero? n) 1 (* n (factorial (- n 1))


What's kind of interesting is to look at it before the parse-lines->sexp
step though:

scheme@(rewisp)> ,pp (call-with-input-string wisp-fac-noinfix-repeat 
read-wisp-lines)
$7 =
(#< indent: 0 args: (# (# #))>
 #< indent: 4 args: (# (# #))>
 #< indent: 7 args: (#< source-loc: #(#f 2 7)> #)>
 #< indent: 7 args: (# # (# (# # #)))>
 #f
 #< indent: 0 args: (# (# #))>
 #< indent: 4 args: (# (# #))>
 #< indent: 7 args: (#< source-loc: #(#f 7 7)> #)>
 #< indent: 7 args: (# # (# (# # #)))>)

So what it does is rewisp builds up a set of lines and parses each
line individually, noting its indentation level and the "arguments" that
appear in it.  It then has a separate step to assemble it into a sexp
structure by examining the indentation level and whether or not it sees
the special  record.  Guile's syntax records are used so that this
would be able to work correctly with the rest of Guile's tools.

So, that's it... first, read each line one by one, don't try to figure
out its relationship to the other lines, and we have a flat structure.
Next, turn that flat structure into a nested structure.

I thought the design was pretty good.

 - Christine

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

Op 30-09-2023 om 15:17 schreef Christine Lemmer-Webber:

Haven't fully caught up on this thread, but as a side note I have a
mostly-finished implementation of a Wisp parser which takes a very
different approach than Arne's, and was more understandable to me
personally, a bit more functional and recursive-descent style.

I could make it available if anyone is curious.

Would love to see Wisp in Guile proper!


I think I technically can't count as ‘curious’ here, as I don't think 
I'll actually read this other implementation, but I would be interested 
in it nonetheless (to potentially replace Arne's implementation later, 
once it's complete), because it sounds like you might have avoided the 
REPR (*) bug thing from Arne's implementation.


(*) Extremely unlikely to be a problem in practice, but still a bug.

Best regards,
Maxime Devos.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Christine Lemmer-Webber]

Haven't fully caught up on this thread, but as a side note I have a
mostly-finished implementation of a Wisp parser which takes a very
different approach than Arne's, and was more understandable to me
personally, a bit more functional and recursive-descent style.

I could make it available if anyone is curious.

Would love to see Wisp in Guile proper!


"Dr. Arne Babenhauserheide"  writes:

> [[PGP Signed Part:Undecided]]
> Hi,
>
> Since (language wisp)¹ has been rock stable for years now and is used in
> the Guix Workflow Language and supported in the Chickadee and the
> Tsukundere game engines, I thought it coud be a good time to merge Wisp
> into Guile itself.
>
> So I prepared a patch that adds language/wisp, some texinfo for
> SRFI-119, and some tests.
>
>
> Why add Wisp?
>
> For Wisp: it is then available directly wherever Guile is available.
>   This will make it much easier for people to follow tutorials.
>
> For Guile:
>
> - Wisp has proven to be good at enabling people to get an
>   entrance to Scheme² without pulling them out of the community.
>
> - It has also been shown to enable people who are used to other
>   programming languages to get a quick start at tools written in Guile.
>
> - And it provides access to the full capabilities of Guile with minimal
>   maintenance effort, because it is just the thinnest possible layer
>   around Scheme. The last required change was in 2020 while I used it
>   continuously.
>
>
> The attached patch provides just the wisp reader, but not the
> wisp->scheme transformer, because the latter has known broken edge-cases
> (and who needs the transformer can get it from the wisp repo and execute
> it directly with a Guile that then already supports wisp without any
> path adaptions).
>
>
> So I’d like to ask: can we merge Wisp as supported language into Guile?
>
>
> Best wishes,
> Arne
>
>
> ¹: https://www.draketo.de/software/wisp
>
> ²: »Wisp allows people to see code how Lispers perceive it. Its
> structure becomes apparent.« — Ricardo Wurmus in IRC
>
> From 4d4759f9fc67b01c40bde41b93e3998f7d64eabd Mon Sep 17 00:00:00 2001
> From: Arne Babenhauserheide 
> Date: Fri, 3 Feb 2023 22:20:04 +0100
> Subject: [PATCH] Add language/wisp, wisp tests, and srfi-119 documentation
>
> * doc/ref/srfi-modules.texi (srfi-119): add node
> * module/language/wisp.scm: New file.
> * module/language/wisp/spec.scm: New file.
> * test-suite/tests/srfi-119.test: New file.
> ---
>  doc/ref/srfi-modules.texi  |  30 ++
>  module/language/wisp.scm   | 796 +
>  module/language/wisp/spec.scm  | 107 +
>  test-suite/tests/srfi-119.test |  81 
>  4 files changed, 1014 insertions(+)
>  create mode 100644 module/language/wisp.scm
>  create mode 100644 module/language/wisp/spec.scm
>  create mode 100644 test-suite/tests/srfi-119.test
>
> diff --git a/doc/ref/srfi-modules.texi b/doc/ref/srfi-modules.texi
> index 0ef136215..759e293ec 100644
> --- a/doc/ref/srfi-modules.texi
> +++ b/doc/ref/srfi-modules.texi
> @@ -64,6 +64,7 @@ get the relevant SRFI documents from the SRFI home page
>  * SRFI-98:: Accessing environment variables.
>  * SRFI-105::Curly-infix expressions.
>  * SRFI-111::Boxes.
> +* SRFI-119::Wisp: simpler indentation-sensitive scheme.
>  * SRFI-171::Transducers
>  @end menu
>  
> @@ -5662,6 +5663,34 @@ Return the current contents of @var{box}.
>  Set the contents of @var{box} to @var{value}.
>  @end deffn
>  
> +@node SRFI-119
> +@subsection SRFI-119 Wisp: simpler indentation-sensitive scheme.
> +@cindex SRFI-119
> +@cindex wisp
> +
> +The languages shipped in Guile include SRFI-119 (wisp), an encoding of
> +Scheme that allows replacing parentheses with equivalent indentation and
> +inline colons. See
> +@uref{http://srfi.schemers.org/srfi-119/srfi-119.html, the specification
> +of SRFI-119}.  Some examples:
> +
> +@example
> +display "Hello World!" @result{}  (display "Hello World!")
> +@end example
> +
> +@example
> +define : factorial n   @result{}  (define (factorial n)
> +if : zero? n   @result{}  (if (zero? n)
> +   . 1 @result{}  1
> +   * n : factorial @{n - 1@} @result{}(* n (factorial @{n - 1@}
> +@end example
> +
> +To execute a file with wisp code, select the language and filename
> +extension @code{.w} vie @code{guile --language=wisp -x .w}.
> +
> +In files using Wisp, @xref{SRFI-105} (Curly Infix) is always activated.
> +
> +
>  @node SRFI-171
>  @subsection Transducers
>  @cindex SRFI-171
> @@ -5705,6 +5734,7 @@ left-to-right, due to how transducers are initiated.
>  * SRFI-171 Helpers::  Utilities for writing your own 
> transducers
>  @end menu
>  
> +
>  @node SRFI-171 General Discussion
>  @subsubsection SRFI-171 General Discussion
>  @cindex transducers discussion
> diff --git a/module/language/wisp.scm 

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Hi,

"Dr. Arne Babenhauserheide"  writes:
> Ludovic Courtès  writes:
>>> Why should Wisp be a separate package when other SRFIs are made part
>>> of Guile?  Your point about maintenance and evolving applies equally
>>> to other SRFIs.
>>
>> That’s a good point.  Making it available as (srfi srfi-119) would make
>> sense I guess.  I need to take a closer look…
>
> That’s where the documentation and tests are located:
>
> - test-suite/tests/srfi-119.test
> - doc/ref/srfi-modules.texi::5666:@node SRFI-119
>
> The language implementation is in (language wisp) because that’s in the
> language search path.

Given the complexities in changing the way languages are handled (the
required discussions, as we’ve seen in the not yet resolved discussion),
would you be OK with keeping the question about adding support for
SRFI-119 to Guile separate from the general discussion about language
handling?

Are there improvements needed besides the ones I did thanks to the
review by Maxime or is this good to go?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: [PATCH] add language/wisp to Guile?

[Marc Nieper-Wißkirchen]

Am Di., 28. Feb. 2023 um 05:27 Uhr schrieb Philip McGrath
:
>
> Hi,
>
> On Monday, February 27, 2023 2:26:47 AM EST Marc Nieper-Wißkirchen wrote:

[...]

> > Nevertheless, I am not sure whether it is relevant to the point I
> > tried to make.  The "#!r6rs" does not indicate a particular language
> > (so tools scanning for "#!r6rs" cannot assume that the file is indeed
> > an R6RS program/library).
>
> I think I had missed that some of your remarks are specifically  about the
> "#!r6rs" directive, not directives of the form "#!" more 
> generally.
> I agree that implementations have more responsibilities with respect to
> "#!r6rs", that the presence of "#!r6rs" in a file is not enough to conclude
> that the file is an R6RS program/library, and that a straightforward
> implementation of "#!r6rs" as reading like "#lang r6rs" in the manner of my
> previous examples would not conform to R6RS.

Yes, this summarizes it well.

> Also, on the broader question, my first preference would be for Guile to
> implement `#lang language/wisp`, not least to avoid the confusing subtleties
> here and the potential for humans to confuse `#!language/wisp` with a shebang
> line. I raise the possibility of `#!language/wisp` only as an alternative if
> people are more comfortable using a mechanism that R6RS specifically designed
> for implementation-defined extensions.

When wisp only changes the lexical syntax, `#!wisp` would be fine (and
it cannot be confused with a shebang line IMO because a shebang line
must begin with `#! ` or `#!/`.  However, the authors of the R6RS
clearly had minor changes of the lexical syntax in mind when they
introduced comments like `#!r6rs` or `#!chezscheme` (e.g. Chez Scheme
adds a syntax for gensyms).  As wisp radically changes how the text is
tokenized, something like `#!wisp` probably only follows the latter
but not the spirit of R6RS.

> Nonetheless, I'll try to explain why I think "#!r6rs" can be handled, and is
> handled by Racket, consistently with both "#lang r6rs" and the behavior
> specified in the report.
>
> >
> > Of course, R6RS gives implementations the freedom to modify the reader
> > in whatever way after, say, "#!foo-baz" was read.  Thus, "#!foo-baz"
> > could be defined to work like Racket's "#lang foo-baz," reading the
> > rest of the source as "(module ...)".  But as long as we stay within
> > the confines of R6RS, this will only raise an undefined exception
> > because, in general, "module" is not globally bound.
> >
>
> Before getting to the general point, specifically about "module" not being
> bound: in Racket, a root-level `module` form is handled quite similarly to the
> `library` form in R6RS, which says in 7.1 [1]:
>
>  The names `library`, `export`, `import`, [...] appearing in the library
> syntax are part of the syntax and are not reserved, i.e., the same names can
> be used for other purposes within the library or even exported from or
> imported into a library with different meanings, without affecting their use 
> in
> the `library` form.
>
> None of the libraries defined in R6RS export a binding for `library`: instead,
> the implementation must recognize it somehow, whether by handling it as a
> built-in or binding it in some environment not standardized by R6RS.
>
> (The `racket/base` library/language does in fact export a binding for `module`
> which can be used to create submodules with the same syntax as a root-level
> `module`, but that isn't relevant to the handling of a `root-level` module
> form itself.)

Sure, but not relevant.  I didn't say that module is bound at the
Racket top-level, only that an R6RS implementation wouldn't expect it
(and cannot interpret it because it is not bound).

> > I don't want to contradict you; I just mean that a plain "#!r6rs"
> > without a top-level language where "module" is bound is not equivalent
> > to "#lang" and that trying to switch to, say,  Elisp mode with
> > "#!elisp" would leave the boundaries of the Scheme reports (and when
> > this is done, this specific discussion is moot).
> >
> > [...]
> >
> > (It must be compatible with calling the procedures "read" and "eval"
> > directly, so "#!r6rs" must not wrap everything in some module form,
> > say.)
>
> Now I'll try to sketch Racket's handling of "#!r6rs" from an R6RS perspective.
> For the sake of a concrete example, lets consider this program:

It is obvious that one can do it; it is just outside the realm of R6RS
because the "non-conformant mode" can be any (even a C-interpreting
mode) that can listen to whatever magic numbers there may be in the
input.  That said, the use of "#!r6rs" as such a magic marker is not
in the spirit of the lexical syntax of R6RS (where it was introduced).
This has been my original point.  Doable, of course.

[...]

> > In an implementation that supports, say,
> > R6RS and R7RS, "#!r6rs" can only switch the lexical syntax but cannot
> > introduce forms that make the implementation change the semantics from
> > R7RS to R6RS, e.g., 

Re: [PATCH] add language/wisp to Guile?

[Philip McGrath]

Hi,

On Monday, February 27, 2023 2:26:47 AM EST Marc Nieper-Wißkirchen wrote:
> Am Mo., 27. Feb. 2023 um 00:22 Uhr schrieb Philip McGrath
> 
> :
> > Hi,
> > 
> > On Sunday, February 26, 2023 6:02:04 AM EST Marc Nieper-Wißkirchen wrote:
> > > Am So., 26. Feb. 2023 um 08:46 Uhr schrieb :
> > > > Message: 1
> > > > Date: Sun, 26 Feb 2023 02:45:12 -0500
> > > > From: "Philip McGrath" 
> > > > To: "Maxime Devos" , Ludovic Courtès
> > > > 
> > > >     , "Matt Wette" ,
> > > > guile-devel@gnu.org
> > > > 
> > > > Cc: "Christine Lemmer-Webber" 
> > > > Subject: Re: [PATCH] add language/wisp to Guile?
> > > > Message-ID: <981b0e74-96c0-4430-b693-7fc8026e3...@app.fastmail.com>
> > > > Content-Type: text/plain;charset=utf-8
> > > 
> > > [...]
> > > 
> > > I would like to make two remarks, which I think are essential to get
> > > the semantics right.
> > > 
> > > The R6RS comments of the form "#!r6rs" are defined to modify the
> > > lexical syntax of the reader; possibly, they don't change the language
> > > semantics (after reading).  In particular, "#!r6rs" also applies to
> > > data files but does not affect the interpretation of the data after it
> > > is read. It cannot because the reader otherwise ignores and does not
> > > report comments.
> > > 
> > > Thus a comment of the form "#!r6rs" may be suitable for Wisp, but it
> > > is not a substitute for Racket's "#lang" (or a similar mechanism).
> > > Guile shouldn't confuse these two different levels of meaning.
> > 
> > I agree that it's important to distinguish between lexical syntax (`read`)
> > and the semantics of what is read.
> > 
> > However, Racket's `#lang` in fact operates entirely at the level of
> > `read`.
> > (Racketeers contribute to confusion on this point by using `#lang` as a
> > shorthand for Racket's entire language-creation infrastructure, when in
> > fact `#lang` specifically has a fairly small, though important, role.)
> > When `read` encounters `#lang something`, it looks up a reader extension
> > procedure in the module indicated by `something` and uses that procedure
> > to continue parsing the input stream into data. Importantly, while syntax
> > objects may be used to attach source location information, there is no
> > "lexical context" or binding information at this stage, as one familiar
> > with syntax objects from macro writing might expect: those semantics come
> > after `read` has finished parsing the input stream from bytes to values.
> 
> [...]
> 
> Thank you for the reminder on Racket's #lang mechanism; it is a long
> time ago since I wrote some #lang extensions myself when experimenting
> with Racket.
> 
> Nevertheless, I am not sure whether it is relevant to the point I
> tried to make.  The "#!r6rs" does not indicate a particular language
> (so tools scanning for "#!r6rs" cannot assume that the file is indeed
> an R6RS program/library). 

I think I had missed that some of your remarks are specifically  about the
"#!r6rs" directive, not directives of the form "#!" more generally. 
I agree that implementations have more responsibilities with respect to
"#!r6rs", that the presence of "#!r6rs" in a file is not enough to conclude 
that the file is an R6RS program/library, and that a straightforward 
implementation of "#!r6rs" as reading like "#lang r6rs" in the manner of my 
previous examples would not conform to R6RS.

Also, on the broader question, my first preference would be for Guile to 
implement `#lang language/wisp`, not least to avoid the confusing subtleties 
here and the potential for humans to confuse `#!language/wisp` with a shebang 
line. I raise the possibility of `#!language/wisp` only as an alternative if 
people are more comfortable using a mechanism that R6RS specifically designed 
for implementation-defined extensions.

Nonetheless, I'll try to explain why I think "#!r6rs" can be handled, and is 
handled by Racket, consistently with both "#lang r6rs" and the behavior 
specified in the report.

> 
> Of course, R6RS gives implementations the freedom to modify the reader
> in whatever way after, say, "#!foo-baz" was read.  Thus, "#!foo-baz"
> could be defined to work like Racket's "#lang foo-baz," reading the
> rest of the source as "(module ...)".  But as long as we stay within
> the confines of R6RS

Re: [PATCH] add language/wisp to Guile?

[Marc Nieper-Wißkirchen]

Am Mo., 27. Feb. 2023 um 00:22 Uhr schrieb Philip McGrath
:
>
> Hi,
>
> On Sunday, February 26, 2023 6:02:04 AM EST Marc Nieper-Wißkirchen wrote:
> > Am So., 26. Feb. 2023 um 08:46 Uhr schrieb :
> > > Message: 1
> > > Date: Sun, 26 Feb 2023 02:45:12 -0500
> > > From: "Philip McGrath" 
> > > To: "Maxime Devos" , Ludovic Courtès
> > >
> > > , "Matt Wette" ,
> > > guile-devel@gnu.org
> > >
> > > Cc: "Christine Lemmer-Webber" 
> > > Subject: Re: [PATCH] add language/wisp to Guile?
> > > Message-ID: <981b0e74-96c0-4430-b693-7fc8026e3...@app.fastmail.com>
> > > Content-Type: text/plain;charset=utf-8
> >
> > [...]
> >
> > I would like to make two remarks, which I think are essential to get
> > the semantics right.
> >
> > The R6RS comments of the form "#!r6rs" are defined to modify the
> > lexical syntax of the reader; possibly, they don't change the language
> > semantics (after reading).  In particular, "#!r6rs" also applies to
> > data files but does not affect the interpretation of the data after it
> > is read. It cannot because the reader otherwise ignores and does not
> > report comments.
> >
> > Thus a comment of the form "#!r6rs" may be suitable for Wisp, but it
> > is not a substitute for Racket's "#lang" (or a similar mechanism).
> > Guile shouldn't confuse these two different levels of meaning.
> >
>
> I agree that it's important to distinguish between lexical syntax (`read`) and
> the semantics of what is read.
>
> However, Racket's `#lang` in fact operates entirely at the level of `read`.
> (Racketeers contribute to confusion on this point by using `#lang` as a
> shorthand for Racket's entire language-creation infrastructure, when in fact
> `#lang` specifically has a fairly small, though important, role.) When `read`
> encounters `#lang something`, it looks up a reader extension procedure in the
> module indicated by `something` and uses that procedure to continue parsing
> the input stream into data. Importantly, while syntax objects may be used to
> attach source location information, there is no "lexical context" or binding
> information at this stage, as one familiar with syntax objects from macro
> writing might expect: those semantics come after `read` has finished parsing
> the input stream from bytes to values.

[...]

Thank you for the reminder on Racket's #lang mechanism; it is a long
time ago since I wrote some #lang extensions myself when experimenting
with Racket.

Nevertheless, I am not sure whether it is relevant to the point I
tried to make.  The "#!r6rs" does not indicate a particular language
(so tools scanning for "#!r6rs" cannot assume that the file is indeed
an R6RS program/library).  In an implementation that supports, say,
R6RS and R7RS, "#!r6rs" can only switch the lexical syntax but cannot
introduce forms that make the implementation change the semantics from
R7RS to R6RS, e.g., in the case of unquoted vector literals.

(It must be compatible with calling the procedures "read" and "eval"
directly, so "#!r6rs" must not wrap everything in some module form,
say.)

Racket's "#lang" mechanism has more freedom (regardless of how it is
implemented).

Of course, R6RS gives implementations the freedom to modify the reader
in whatever way after, say, "#!foo-baz" was read.  Thus, "#!foo-baz"
could be defined to work like Racket's "#lang foo-baz," reading the
rest of the source as "(module ...)".  But as long as we stay within
the confines of R6RS, this will only raise an undefined exception
because, in general, "module" is not globally bound.

I don't want to contradict you; I just mean that a plain "#!r6rs"
without a top-level language where "module" is bound is not equivalent
to "#lang" and that trying to switch to, say,  Elisp mode with
"#!elisp" would leave the boundaries of the Scheme reports (and when
this is done, this specific discussion is moot).

[...]

> > The second comment concerns the shebang line in R6RS scripts (as
> > described in the non-normative appendices).  The shebang line is not a
> > comment in the R6RS lexical syntax; it does not even reach the reader
> > - at least, conceptionally.  The Scheme reader only sees the lines
> > following the shebang line.
> >
> > For example, a conforming R6RS implementation must raise an exception
> > when trying to read (using get-datum, for example) a file that begins
> > with a shebang line.
> >
> > Thus, the shebang line

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Matt Wette  writes:

> On 2/26/23 10:03 AM, Dr. Arne Babenhauserheide wrote:
>> Matt Wette  writes:
>>
>>> You can name a file containing Scheme code "foo.js" and "guile foo.js"

>> guile -L . -e  '(foo)' -c ''
>>
>> won’t find it. But it will find it in foo.scm.
>>
>
> Guile does not use file extensions consistently, then?

Those are two different use-cases. With guile foo.js, guile does not
have to search for the file: it just uses the one you passed.

With -e '(foo)' it has to search for a matching module in the known /
defined load paths.

Note also guile -L . -e '(foo)' -c '' has lower startup time than guile
foo.scm. If I understand it correctly that’s because it simply mmaps the
compiled *.go file while guile foo.scm always reads the file.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: [PATCH] add language/wisp to Guile?

[Matt Wette]

On 2/26/23 10:03 AM, Dr. Arne Babenhauserheide wrote:

Matt Wette  writes:


With respect to file extensions, guile does not use file extension:
You can name a file containing Scheme code "foo.js" and "guile foo.js"
will execute it.

The module-system uses file extensions: If you

(define-module (foo) #:export (main))
(define (main args) (display 'foo))

in foo.js, then

guile -L . -e  '(foo)' -c ''

won’t find it. But it will find it in foo.scm.



Guile does not use file extensions consistently, then?

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Matt Wette  writes:

> With respect to file extensions, guile does not use file extension:
> You can name a file containing Scheme code "foo.js" and "guile foo.js"
> will execute it.

The module-system uses file extensions: If you

(define-module (foo) #:export (main))
(define (main args) (display 'foo))

in foo.js, then

guile -L . -e  '(foo)' -c ''

won’t find it. But it will find it in foo.scm.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: [PATCH] add language/wisp to Guile?

[Matt Wette]

With respect to file extensions, guile does not use file extension:
You can name a file containing Scheme code "foo.js" and "guile foo.js"
will execute it.

The code in the wip-load-lang branch provides file-extension support:
1) Currently emacscript, with "js" extension, elisp, with "el" extension,
    and Scheme, with "scm" extension, are supported.
2) Additional languages can be added by calling add-lang-extension,
    from (system base compile): (add-lang-extension "m" 'nx-mlang)

Matt

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

Op 26-02-2023 om 08:45 schreef Philip McGrath:

Hi,

On Sat, Feb 18, 2023, at 10:58 AM, Maxime Devos wrote:

On 18-02-2023 04:50, Philip McGrath wrote:

I haven't read the patch or this thread closely,


I'll assume you have read it non-closely.


but R6RS has an answer to any concerns about compatibility with `#lang`. At the beginning of 
Chapter 4, "Lexical and Datum Syntax" 
() the report specifies:


   An implementation must not extend the lexical or datum syntax in any way, with one 
exception: it need not treat the syntax `#!`, for any  
(see section 4.2.4) that is not `r6rs`, as a syntax violation, and it may use specific 
`#!`-prefixed identifiers as flags indicating that subsequent input contains extensions to 
the standard lexical or datum syntax. The syntax `#!r6rs` may be used to signify that the 
input afterward is written with the lexical syntax and datum syntax described by this 
report. `#!r6rs` is otherwise treated as a comment; see section 4.2.3.


That is for '#!lang', not '#lang'.  R6RS allows the former, but the
patch does the latter.  As such, R6RS does not have an answer about
incompatibility with `#lang', unless you count ‘it's incompatible’ as an
answer.



Let me try to be more concrete.

If you want a portable, RnRS-standardized lexical syntax for `#lang`, use 
`#!`, and systems that understand `#lang` will treat it (in 
appropriate contexts) as an alias for `#lang `.


RnRS only standardises #!r6rs, not #!.  Even if RnRS 
standardised #! for values of  that aren't rnrs, 
the RnRS only holds sway for Scheme, and one of the main points of 
Guile's language system is to support more than only Scheme.




Alternatively, you could embrace that Guile (like every other Scheme system I'm aware of) 
starts by default in a mode with implementation-specific extensions. Indeed, R6RS 
Appendix A specifically recognizes that "the default mode offered by a Scheme 
implementation may be non-conformant, and such a Scheme implementation may require 
special settings or declarations to enter the report-conformant mode" [1]. Then you 
could just write `#lang` and worry about the non-portable block comments some other day. 
This is what I would personally prefer.


Emphasis on 'non-conformant'.  The appendix states that Scheme 
implementations don't need to be R6RS by default; it doesn't state that 
things non-conformant things are conformant with R6RS.


Remember that this part of the discussion started with:

‘The '#lang whatever' stuff makes Scheme (*) files unportable between
implementations, as '#lang scheme' is not a valid comment’.

The R6RS might permit non-R6RS implementations, but this does not make 
non-R6RS constructs like '#lang scheme' portable.




In Racket, in the initial configuration of the reader when reading a file, "`#!` is an alias 
for `#lang` followed by a space when `#!` is followed by alphanumeric ASCII, `+`, `-`, or 
`_`." (See 
.) [...] > 
(Guile does not handle `#!r6rs` properly, presumably because of the

legacy `#!`/`!#` block comments. I think this should be a surmountable
obstacle, though, especially since Guile does support standard `#|`/`|#`
block comments.)

‘#! ... !#’ comments aren't legacy; they exist to allow putting the
shebang in the first line of a script, and to pass additional arguments
to the Guile interpreter (see: (guile)The Top of a Script File) (*).  As
such, you can't just replace them with #| ... |# (unless you patch the
kernel to recognise "#| ..." as a shebang line).

(*) Maybe they exist for other purposes too.


According to "(guile)Block Comments", the `#!...!#` syntax existed before Guile 
2.0 added support for `#|...|#` comments from SRFI 30 and R6RS.


I agree, and I don't follow what your point is here.



Furthermore, according to the kernel, #!r6rs would mean that the script
needs to be interpreted by a program named 'r6rs', but 'guile' is named
'guile', not 'r6rs'.  (I assume this is in POSIX somewhere, though I
couldn't find it.)

(This is an incompatibility between R6RS and any system that has shebangs.)



This is not an incompatibility, because the `#!r6rs` lexeme (or 
`#!`, more generally) is not the shebang line for the script. R6RS 
Appendix D [2] gives this example of a Scheme script:

```
#!/usr/bin/env scheme-script
#!r6rs
(import (rnrs base)
 (rnrs io ports)
 (rnrs programs))
(put-bytes (standard-output-port)
(call-with-port
(open-file-input-port
  (cadr (command-line)))
  get-bytes-all))
```


OK, didn't notice that appendix.  Only covers Scheme, though.


--
The appendix says that, "if the first line of a script begins with `#!/` or `#!`, 
implementations should ignore it on all platforms, even if it does not conform to the recommended 
syntax". Admittedly this is not handled as consistently as I would 

Re: [PATCH] add language/wisp to Guile?

[Marc Nieper-Wißkirchen]

Am So., 26. Feb. 2023 um 08:46 Uhr schrieb :

> Message: 1
> Date: Sun, 26 Feb 2023 02:45:12 -0500
> From: "Philip McGrath" 
> To: "Maxime Devos" , Ludovic Courtès
> , "Matt Wette" ,
> guile-devel@gnu.org
> Cc: "Christine Lemmer-Webber" 
> Subject: Re: [PATCH] add language/wisp to Guile?
> Message-ID: <981b0e74-96c0-4430-b693-7fc8026e3...@app.fastmail.com>
> Content-Type: text/plain;charset=utf-8

[...]

I would like to make two remarks, which I think are essential to get
the semantics right.

The R6RS comments of the form "#!r6rs" are defined to modify the
lexical syntax of the reader; possibly, they don't change the language
semantics (after reading).  In particular, "#!r6rs" also applies to
data files but does not affect the interpretation of the data after it
is read. It cannot because the reader otherwise ignores and does not
report comments.

Thus a comment of the form "#!r6rs" may be suitable for Wisp, but it
is not a substitute for Racket's "#lang" (or a similar mechanism).
Guile shouldn't confuse these two different levels of meaning.

The second comment concerns the shebang line in R6RS scripts (as
described in the non-normative appendices).  The shebang line is not a
comment in the R6RS lexical syntax; it does not even reach the reader
- at least, conceptionally.  The Scheme reader only sees the lines
following the shebang line.

For example, a conforming R6RS implementation must raise an exception
when trying to read (using get-datum, for example) a file that begins
with a shebang line.

Thus, the shebang line doesn't need to be considered when discussing
comment formats in lexical syntax.

This also shows how a mechanism like "#lang" can be made compatible
with R6RS, namely similar to scripts starting with a shebang line: Let
us define a "multi-language script" whose first line may be a shebang
and whose second line may be of the form "#lang ...".  If "..." is,
say, "r6rs", the following lines are then fed to the reader and
interpreted as an R6RS top-level program.

Best,

Marc

Re: [PATCH] add language/wisp to Guile?

[Philip McGrath]

Hi,

On Sat, Feb 18, 2023, at 10:58 AM, Maxime Devos wrote:
> On 18-02-2023 04:50, Philip McGrath wrote:
>> I haven't read the patch or this thread closely,
>
> I'll assume you have read it non-closely.
>
>> but R6RS has an answer to any concerns about compatibility with `#lang`. At 
>> the beginning of Chapter 4, "Lexical and Datum Syntax" 
>> () the 
>> report specifies:
>> 
>>>   An implementation must not extend the lexical or datum syntax in any way, 
>>> with one exception: it need not treat the syntax `#!`, for any 
>>>  (see section 4.2.4) that is not `r6rs`, as a syntax violation, 
>>> and it may use specific `#!`-prefixed identifiers as flags indicating that 
>>> subsequent input contains extensions to the standard lexical or datum 
>>> syntax. The syntax `#!r6rs` may be used to signify that the input afterward 
>>> is written with the lexical syntax and datum syntax described by this 
>>> report. `#!r6rs` is otherwise treated as a comment; see section 4.2.3.
>
> That is for '#!lang', not '#lang'.  R6RS allows the former, but the 
> patch does the latter.  As such, R6RS does not have an answer about 
> incompatibility with `#lang', unless you count ‘it's incompatible’ as an 
> answer.
>

Let me try to be more concrete.

If you want a portable, RnRS-standardized lexical syntax for `#lang`, use 
`#!`, and systems that understand `#lang` will treat it (in 
appropriate contexts) as an alias for `#lang `.

Alternatively, you could embrace that Guile (like every other Scheme system I'm 
aware of) starts by default in a mode with implementation-specific extensions. 
Indeed, R6RS Appendix A specifically recognizes that "the default mode offered 
by a Scheme implementation may be non-conformant, and such a Scheme 
implementation may require special settings or declarations to enter the 
report-conformant mode" [1]. Then you could just write `#lang` and worry about 
the non-portable block comments some other day. This is what I would personally 
prefer.

>> In Racket, in the initial configuration of the reader when reading a file, 
>> "`#!` is an alias for `#lang` followed by a space when `#!` is followed by 
>> alphanumeric ASCII, `+`, `-`, or `_`." (See 
>> .)
>>  [...] > (Guile does not handle `#!r6rs` properly, presumably because of the 
> legacy `#!`/`!#` block comments. I think this should be a surmountable 
> obstacle, though, especially since Guile does support standard `#|`/`|#` 
> block comments.)
>
> ‘#! ... !#’ comments aren't legacy; they exist to allow putting the 
> shebang in the first line of a script, and to pass additional arguments 
> to the Guile interpreter (see: (guile)The Top of a Script File) (*).  As 
> such, you can't just replace them with #| ... |# (unless you patch the 
> kernel to recognise "#| ..." as a shebang line).
>
> (*) Maybe they exist for other purposes too.

According to "(guile)Block Comments", the `#!...!#` syntax existed before Guile 
2.0 added support for `#|...|#` comments from SRFI 30 and R6RS.

>
> Furthermore, according to the kernel, #!r6rs would mean that the script 
> needs to be interpreted by a program named 'r6rs', but 'guile' is named 
> 'guile', not 'r6rs'.  (I assume this is in POSIX somewhere, though I 
> couldn't find it.)
>
> (This is an incompatibility between R6RS and any system that has shebangs.)
>

This is not an incompatibility, because the `#!r6rs` lexeme (or 
`#!`, more generally) is not the shebang line for the script. R6RS 
Appendix D [2] gives this example of a Scheme script:

```
#!/usr/bin/env scheme-script
#!r6rs
(import (rnrs base)
(rnrs io ports)
(rnrs programs))
(put-bytes (standard-output-port)
   (call-with-port
   (open-file-input-port
 (cadr (command-line)))
 get-bytes-all))
```

The appendix says that, "if the first line of a script begins with `#!/` or 
`#!`, implementations should ignore it on all platforms, even if it does 
not conform to the recommended syntax". Admittedly this is not handled as 
consistently as I would prefer: I wish they had just standardized `#!/` and `#! 
` as special comment syntax, as Racket does, and clarified the interaction with 
`#!`. But Matt points out that JavaScript also has very similar 
special treatment for a single initial shebang comment. Lua has a similar 
mechanism: my vague recollection is that many languages do. 

>>>
>>> (^) it doesn't integrate with the module system -- more concretely,
>>> (use-modules (foo)) wouldn't try loading foo.js -- adding '-x' arguments
>>> would solve that, but we agree that that would be unreasonable in many
>>> situations.  (Alternatively one could place ECMAScript code in a file
>>> with extension '.scm' with a '#lang' / '-*- mode: ecmascript -*-', but
>>> ... no.)

Generally I would use `.scm` (or `.rkt`), and certainly I would do so if there 

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

On 25-02-2023 01:15, Matt Wette wrote:

On 2/24/23 3:51 PM, Maxime Devos wrote:

On 25-02-2023 00:48, Maxime Devos wrote:

(**) For compatibility with Racket, it's not like we couldn't
implement both "#lang" and "-*- stuff: language -*-".


TBC, I mean ‘only support #lang' for values of 'lang' that Racket 
supports’, or alternatively ‘support #lang for compatibility, but 
recommend -*- ... -*- in the Guile manual.’.


The point of this is to tell the compiler what language it needs to parse,
if not the default (Scheme).  Using `#lang pascal' or `#!lang pascal' 
allows

the compiler to figure this out in the first few characters read. Using
`-*- lang: pascal -*-' might require the compiler to read a lot of text 
(not knowing
if what text is comment or not) to determine if language has been 
specified.

For the `-*-' based design, is there a maximum number of characters to be
read before deciding?


Guile already searches for '-*- ... -*-' lines, as mentioned in a 
previous message:



(*) The port encoding detection supports "-*- coding: whatever -*-",
presumably that functionality could be reused. 


As such, it shouldn't incur an overly large cost to support '-*- 
something: modula -*-' too.


On the character limit: according to Emacs documentation, there is a 
line limit: the '-*- ... -*-' must be on the first line.  (See: 
‘(emacs)Specifying File Variables’.) This seems the same situation like 
'#lang' to me; not ‘a lot of text’.


A more explicit character limit of 'first few hundred bytes' is 
mentioned in see the documentation of 'file-encoding':


 -- Scheme Procedure: file-encoding port
 -- C Function: scm_file_encoding (port)
 Attempt to scan the **first few hundred bytes** from the PORT for
 hints about its character encoding.  Return a string containing the
 encoding name or ‘#f’ if the encoding cannot be determined.  The
 port is rewound.

 Currently, the only supported method is to look for an Emacs-like
 character coding declaration (*note how Emacs recognizes file
 encoding: (emacs)Recognize Coding.).  The coding declaration is of
 the form ‘coding: X’ and must appear in a Scheme comment.
 Additional heuristics may be added in the future.

Greetings,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Matt Wette]

On 2/24/23 3:51 PM, Maxime Devos wrote:

On 25-02-2023 00:48, Maxime Devos wrote:

(**) For compatibility with Racket, it's not like we couldn't
implement both "#lang" and "-*- stuff: language -*-".


TBC, I mean ‘only support #lang' for values of 'lang' that Racket 
supports’, or alternatively ‘support #lang for compatibility, but 
recommend -*- ... -*- in the Guile manual.’.


The point of this is to tell the compiler what language it needs to parse,
if not the default (Scheme).  Using `#lang pascal' or `#!lang pascal' 
allows

the compiler to figure this out in the first few characters read. Using
`-*- lang: pascal -*-' might require the compiler to read a lot of text 
(not knowing
if what text is comment or not) to determine if language has been 
specified.

For the `-*-' based design, is there a maximum number of characters to be
read before deciding?

Matt

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

On 25-02-2023 00:48, Maxime Devos wrote:

(**) For compatibility with Racket, it's not like we couldn't
implement both "#lang" and "-*- stuff: language -*-".


TBC, I mean ‘only support #lang' for values of 'lang' that Racket 
supports’, or alternatively ‘support #lang for compatibility, but 
recommend -*- ... -*- in the Guile manual.’.


Greetings,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

On 24-02-2023 16:45, Ludovic Courtès wrote:

Adding #lang support in Guile would be nice.  As discussed on IRC, it
can be experimented with in a WIP branch.

Have you seen my messages on how the "#lang" construct is problematic
for some languages, and how alternatives like "[comment delimiter] -*-
stuff: scheme/ecmascript/... -*- [comment delimiter]" appear to be
equally simple (*) and not have any downsides (**).

(*) The port encoding detection supports "-*- coding: whatever -*-",
presumably that functionality could be reused.

(**) For compatibility with Racket, it's not like we couldn't
implement both "#lang" and "-*- stuff: language -*-".

I haven’t seen your messages yet, I just wanted to express support of
the general idea.  For years, we have discussed #lang support; I know
Andy is enthusiastic about it, and while I was initially reluctant, I’ve
come to appreciate the idea.

What you point out is worth considering, but note that Guile already
supports #!r6rs for instance.


While I initially thought that #!r6rs was problematic, it was pointed 
out to me that #!r6rs is actually in the R6RS (unlike #lang, and even if 
#lang was in the R6RS, it isn't in the de-jure or de-factor standards of 
many other languages).


As such, #!r6rs is a different (unproblematic) situation compared to 
#lang -- #lang is not the same as #!r6rs from a compatibility perspective.


Greetings,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Hi,

Ludovic Courtès  writes:

>> Why should Wisp be a separate package when other SRFIs are made part
>> of Guile?  Your point about maintenance and evolving applies equally
>> to other SRFIs.
>
> That’s a good point.  Making it available as (srfi srfi-119) would make
> sense I guess.  I need to take a closer look…

That’s where the documentation and tests are located:

- test-suite/tests/srfi-119.test
- doc/ref/srfi-modules.texi::5666:@node SRFI-119

The language implementation is in (language wisp) because that’s in the
language search path.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: [PATCH] add language/wisp to Guile?

[Ludovic Courtès]

Hello!

Maxime Devos  skribis:

> Why should Wisp be a separate package when other SRFIs are made part
> of Guile?  Your point about maintenance and evolving applies equally
> to other SRFIs.

That’s a good point.  Making it available as (srfi srfi-119) would make
sense I guess.  I need to take a closer look…

>> Adding #lang support in Guile would be nice.  As discussed on IRC, it
>> can be experimented with in a WIP branch.
>
> Have you seen my messages on how the "#lang" construct is problematic
> for some languages, and how alternatives like "[comment delimiter] -*-
> stuff: scheme/ecmascript/... -*- [comment delimiter]" appear to be
> equally simple (*) and not have any downsides (**).
>
> (*) The port encoding detection supports "-*- coding: whatever -*-",
> presumably that functionality could be reused.
>
> (**) For compatibility with Racket, it's not like we couldn't
> implement both "#lang" and "-*- stuff: language -*-".

I haven’t seen your messages yet, I just wanted to express support of
the general idea.  For years, we have discussed #lang support; I know
Andy is enthusiastic about it, and while I was initially reluctant, I’ve
come to appreciate the idea.

What you point out is worth considering, but note that Guile already
supports #!r6rs for instance.

Thanks,
Ludo’.

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Hi,

Ludovic Courtès  writes:

> FWIW, I think it might be best to keep Wisp as a separate package: that
> allows it to evolve independently of Guile (and possibly more quickly
> :-)), and it might simplify maintenance in some way.

While this reasoning is true for most languages, I don’t think it
applies to wisp: wisp will not evolve faster independently of Guile,
because it is pretty much finished, and it should not need much
maintenance (and if some should be needed, I can do that directly in
Guile). It is such a thin layer that (different from other languages)
there is no need to take care of changing syntax to follow or new
semantics.

It is defined such that its implementation can actually be finished.

Every change in Scheme can automatically also be used with Wisp. While
ecmascript, elisp, lokke, and python-on-guile must be kept up to date to
stay useful, Wisp does not need that.

The only larger changes in the past 4 years were in editor-support and
some scripts, and these can stay separate, just like emacs-geiser-guile
is separate from Guile.


I propose adding Wisp now, because while I have been using Wisp for many
projects in the past years, the changes to wisp itself since declaring
1.0 at FOSDEM 2019 were minimal, and the biggest were due to the review
of Maxime here:

wisp 1.0.10 (2023-02-16):
- only extend the reader for ## while reading wisp. Thanks to Maxime Devos for 
help to use fluids!

wisp 1.0.9 (2023-02-16):
- remove limitation of the number of prefix underscores (_). Thanks to Maxime 
Devos for a much cleaner algorithm!
- only set *port* encoding to UTF-8, do not change encoding for the 
application. Thanks to Maxime Devos!

wisp 1.0.8 (2022-12-09):
- wisp2lisp can now process stdin when called with - as filename.
  And it has help output.

wisp 1.0.7 (2021-12-20):
- fix: a lisp-style comment in the bash-cript had broken the wisp REPL

wisp 1.0.6 (2021-11-30):
- allow (and ignore!) a single space indentation for the first line of a chunk 
to support meta-commands
- ensure that (language wisp) is compiled in the wisp REPL

wisp 1.0.5 (2021-05-02):
- explicitly allow using wisp as language under the expat-license for easier 
embedding in Guile-using games like Tsukundere Shitsumon: 
https://gitlab.com/leoprikler/tsukundere-shitsumon/

wisp 1.0.4 (2021-02-08):
- add one more setlocale fallback: If it cannot use unicode, wisp now proceeds 
with degraded operation rather than failing outright.

wisp 1.0.3 (2020-09-15):
- provide wisp script that wraps guile --language=wisp -x .w
- add Guile 3.0 to supported versions
- fix documentation: wisp allows up to 12 underscores
- You can create wisp-projects with conf via `conf new -l wisp PROJNAME`. See 
https://hg.sr.ht/~arnebab/conf
- wisp moved to sourcehut: https://hg.sr.ht/~arnebab/wisp

wisp 1.0.2 (2019-04-09):
- guild compile is missing the load path

wisp 1.0.1 (2019-03-23):
- fix install directory, thanks to James-Adam Renquinha Henri and Ludovic 
Courtès who both
  discovered a mistake in the paths: correct module path is 
/usr/share/guile/site//…,
  but I used /usr/share/guile//site
- simplify install logic and pre-compile installed modules.
- add beautiful make help

wisp 1.0 (2019-02-08):
- add FOSDEM 2019 slides: docs/fosdem2019.org
- As presented at FOSDEM, wisp the language is complete.
  Tooling, documentation, and porting are still work in progress.


I plan to then only install language/wisp from the wisp-repo when the
local Guile does not provide wisp.


The reasoning for adding wisp to Guile is at the start of the thread:
https://lists.gnu.org/archive/html/guile-devel/2023-02/msg00012.html


> Adding #lang support in Guile would be nice.  As discussed on IRC, it
> can be experimented with in a WIP branch.

I now created the branch wip-load-lang with the two patches by Matt.


Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

On 23-02-2023 12:36, Ludovic Courtès wrote:

Hi!

Sorry for the late reply.

FWIW, I think it might be best to keep Wisp as a separate package: that
allows it to evolve independently of Guile (and possibly more quickly
:-)), and it might simplify maintenance in some way.


To my understanding, Wisp is pretty much finished -- it is standardised 
as surfie 119: , which is in final 
status.  As such, there is no room for evolving (beside bugfixes, perhaps).


The maintenance aspect (and also the evolving) is addressed in the cover 
letter:



- And it provides access to the full capabilities of Guile with minimal
  maintenance effort, because it is just the thinnest possible layer
  around Scheme. The last required change was in 2020 while I used it
  continuously.


There also were several other points in the cover letter for keeping 
Wisp as a _non-separate_ package.


Why should Wisp be a separate package when other SRFIs are made part of 
Guile?  Your point about maintenance and evolving applies equally to 
other SRFIs.



Adding #lang support in Guile would be nice.  As discussed on IRC, it
can be experimented with in a WIP branch.


Have you seen my messages on how the "#lang" construct is problematic 
for some languages, and how alternatives like "[comment delimiter] -*- 
stuff: scheme/ecmascript/... -*- [comment delimiter]" appear to be 
equally simple (*) and not have any downsides (**).


(*) The port encoding detection supports "-*- coding: whatever -*-", 
presumably that functionality could be reused.


(**) For compatibility with Racket, it's not like we couldn't implement 
both "#lang" and "-*- stuff: language -*-".


Greetingss,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

On 23-02-2023 19:04, Maxime Devos wrote:
(*) Java actually allows "#!", but only in 'Shebang' files (see: 
https://openjdk.org/jeps/330#Shebang_files).  It remains invalid to put 
a '#!java' line in files with a class definition that is supposed to be 
found by Java's class loaders and compiler (in Guile terms, the source 
code of a module).


(Also, #java is unconditionally invalid, to my understanding.)

(I pressed the 'Send button' a little to early when composing the 
previous version of this e-mail.)


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

On 23-02-2023 19:04, Maxime Devos wrote:
(*) Java actually allows "#!", but only in 'Shebang' files (see: 
https://openjdk.org/jeps/330#Shebang_files).  It remains invalid to put 
a '#!java' line in files with a class definition that is supposed to be 
found by Java's class loaders and compiler (in Guile terms, the source 
code of a module).


(Also, to my understanding, "#lang" is unconditionally invalid.)


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

On 23-02-2023 19:04, Maxime Devos wrote:

Still doesn't really address the problem though, as Scheme scripts (or
scripts in another language) may need to start with a shebang and
"#!lang" or "#lang" is not a valid comment in all languages.  (E.g., I
don't think it's valid Pascal, though I only have read some Pascal
code, I haven't looked at the specification.)

I think itmust be ignored in all languages that work as scripts in
POSIX. So I would expect that support for ignoring #!... in the first
line is very widespread.


The problem is that not all languages were made with POSIX-style scripts 
in mind, e.g. Pascal, BASIC and Java (*).


I forgot about the following:


Also since the language implementation is in Guile, this could simply be
added for Guile. That may prevent using this file from other
implementations of the language, but it should work well enough as a
first step.


I disagree, because there is an alternative solution that should be 
straightforward and avoids the downside of making an extension to the 
language that could confuse other implementations.


Quoting myself:


The '#lang whatever' stuff makes Scheme (*) files unportable between 
implementations, as '#lang scheme' is not a valid comment -- there exist 
Schemes beyond Guile and Racket.  If it were changed to recognising
'-*- mode: scheme -*-' or '-*- language: scheme -*-' or such, it would be 
better IMO, [...]

E.g.:

;; -*- insert-bikeshed-here: scheme -*-
// -*- insert-bikeshed-here: c++ -*-
#  -*- insert-bikeshed-here: ecmascript -*-
/* -*- insert-bikeshed-here: c -*- */
{*** *- insert-bikeshed-here: pascal -*- ***}
REM -*- insert-bikeshed-here: basic -*-

I suppose the decision could be made to add support for "#lang" for 
compatibility with Racket when 'lang' is some language Racket supports, 
despite the fact that "#lang" is invalid in some of those languages, but 
IMO we should avoid adding new extensions that confuse ‘native’ 
implementations when something compatible like "-*- ... -*-" can be done 
instead.


Greetings,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

On 23-02-2023 09:51, Dr. Arne Babenhauserheide wrote:

Thinking a bit more about it, it should be possible to special-case
Guile's interpretation of "#!" such that "#!r6rs" doesn't require a
closing "!#".  (Technically backwards-incompatible, but I don't think
people are writing #!r6rs ...!# in the wild.)

Do you need the closing !# if you restrict yourself to the first line?


I thought so at first, but doing a little experiment, it appears you 
don't need to:


$ guile
scheme@(guile-user)> #!r6rs
(display "hi") (newline)

(output: hi)

Apparently Guile already has required behaviour.


Still doesn't really address the problem though, as Scheme scripts (or
scripts in another language) may need to start with a shebang and
"#!lang" or "#lang" is not a valid comment in all languages.  (E.g., I
don't think it's valid Pascal, though I only have read some Pascal
code, I haven't looked at the specification.)

I think itmust be ignored in all languages that work as scripts in
POSIX. So I would expect that support for ignoring #!... in the first
line is very widespread.


The problem is that not all languages were made with POSIX-style scripts 
in mind, e.g. Pascal, BASIC and Java (*).


Greetings,
Maxime.

(*) Java actually allows "#!", but only in 'Shebang' files (see: 
https://openjdk.org/jeps/330#Shebang_files).  It remains invalid to put 
a '#!java' line in files with a class definition that is supposed to be 
found by Java's class loaders and compiler (in Guile terms, the source 
code of a module).


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Maxime Devos  writes:

>> ‘#! ... !#’ comments aren't legacy; they exist to allow putting the
>> shebang in the first line of a script, and to pass additional
>> arguments to the Guile interpreter (see: (guile)The Top of a Script
>> File) (*).

This is awesome, by the way.

It’s what allowed me to write wisp scripts that just work without having
wisp shipped by starting as bash script, pre-compiling the language
files, and then exec'ing guile with the right arguments that interprets
the file as module and runs the code inside.

>> Furthermore, according to the kernel, #!r6rs would mean that the
>> script needs to be interpreted by a program named 'r6rs', but
>> 'guile' is named 'guile', not 'r6rs'.  (I assume this is in POSIX
>> somewhere, though I couldn't find it.)

We could fix that by installing a binary named r6rs.

> Thinking a bit more about it, it should be possible to special-case
> Guile's interpretation of "#!" such that "#!r6rs" doesn't require a
> closing "!#".  (Technically backwards-incompatible, but I don't think
> people are writing #!r6rs ...!# in the wild.)

Do you need the closing !# if you restrict yourself to the first line?

> Still doesn't really address the problem though, as Scheme scripts (or
> scripts in another language) may need to start with a shebang and
> "#!lang" or "#lang" is not a valid comment in all languages.  (E.g., I
> don't think it's valid Pascal, though I only have read some Pascal
> code, I haven't looked at the specification.)

I think itmust be ignored in all languages that work as scripts in
POSIX. So I would expect that support for ignoring #!... in the first
line is very widespread.

Also since the language implementation is in Guile, this could simply be
added for Guile. That may prevent using this file from other
implementations of the language, but it should work well enough as a
first step.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

In Racket, in the initial configuration of the reader when reading a file, "`#!` is an alias 
for `#lang` followed by a space when `#!` is followed by alphanumeric ASCII, `+`, `-`, or 
`_`." (See 
.) [...] > 
(Guile does not handle `#!r6rs` properly, presumably because of the legacy `#!`/`!#` block 
comments. I think this should be a surmountable obstacle, though, especially since Guile does 
support standard `#|`/`|#` block comments.)


‘#! ... !#’ comments aren't legacy; they exist to allow putting the shebang in the first 
line of a script, and to pass additional arguments to the Guile interpreter (see: 
(guile)The Top of a Script File) (*).  As such, you can't just replace them with #| ... 
|# (unless you patch the kernel to recognise "#| ..." as a shebang line). [...]

Furthermore, according to the kernel, #!r6rs would mean that the script needs 
to be interpreted by a program named 'r6rs', but 'guile' is named 'guile', not 
'r6rs'.  (I assume this is in POSIX somewhere, though I couldn't find it.)

(This is an incompatibility between R6RS and any system that has shebangs.)


Thinking a bit more about it, it should be possible to special-case 
Guile's interpretation of "#!" such that "#!r6rs" doesn't require a 
closing "!#".  (Technically backwards-incompatible, but I don't think 
people are writing #!r6rs ...!# in the wild.)


Still doesn't really address the problem though, as Scheme scripts (or 
scripts in another language) may need to start with a shebang and 
"#!lang" or "#lang" is not a valid comment in all languages.  (E.g., I 
don't think it's valid Pascal, though I only have read some Pascal code, 
I haven't looked at the specification.)


Greetings,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Hi Matt,

Please tell me once you know for which patch exactly you need a
WIP-branch.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: [PATCH] add language/wisp to Guile?

[Matt Wette]

On 2/18/23 7:58 AM, Maxime Devos wrote:

On 18-02-2023 04:50, Philip McGrath wrote:

I haven't read the patch or this thread closely,


I'll assume you have read it non-closely.

but R6RS has an answer to any concerns about compatibility with 
`#lang`. At the beginning of Chapter 4, "Lexical and Datum Syntax" 
() 
the report specifies:


  An implementation must not extend the lexical or datum syntax in 
any way, with one exception: it need not treat the syntax 
`#!`, for any  (see section 4.2.4) that is 
not `r6rs`, as a syntax violation, and it may use specific 
`#!`-prefixed identifiers as flags indicating that subsequent input 
contains extensions to the standard lexical or datum syntax. The 
syntax `#!r6rs` may be used to signify that the input afterward is 
written with the lexical syntax and datum syntax described by this 
report. `#!r6rs` is otherwise treated as a comment; see section 4.2.3.


That is for '#!lang', not '#lang'.  R6RS allows the former, but the 
patch does the latter.  As such, R6RS does not have an answer about 
incompatibility with `#lang', unless you count ‘it's incompatible’ as 
an answer.


I just looked on the web and it appears that #! as first line is a 
comment in JavaScript also.


https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Lexical_grammar#hashbang_comments

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

On 18-02-2023 04:50, Philip McGrath wrote:

I haven't read the patch or this thread closely,


I'll assume you have read it non-closely.


but R6RS has an answer to any concerns about compatibility with `#lang`. At the beginning of 
Chapter 4, "Lexical and Datum Syntax" 
() the report specifies:


  An implementation must not extend the lexical or datum syntax in any way, with one 
exception: it need not treat the syntax `#!`, for any  
(see section 4.2.4) that is not `r6rs`, as a syntax violation, and it may use specific 
`#!`-prefixed identifiers as flags indicating that subsequent input contains extensions to 
the standard lexical or datum syntax. The syntax `#!r6rs` may be used to signify that the 
input afterward is written with the lexical syntax and datum syntax described by this 
report. `#!r6rs` is otherwise treated as a comment; see section 4.2.3.


That is for '#!lang', not '#lang'.  R6RS allows the former, but the 
patch does the latter.  As such, R6RS does not have an answer about 
incompatibility with `#lang', unless you count ‘it's incompatible’ as an 
answer.



Chez Scheme uses such comments to support extensions to lexical syntax, as documented 
in :


The Chez Scheme lexical extensions described above are disabled in an input 
stream after an `#!r6rs` comment directive has been seen, unless a 
`#!chezscheme` comment directive has been seen since. Each library loaded 
implicitly via import and each RNRS top-level program loaded via the 
`--program` command-line option, the `scheme-script` command, or the 
`load-program` procedure is treated as if it begins implicitly with an `#!r6rs` 
comment directive. [...]


Again, that's '#!whatever', not '#lang' -- Chez does the former, not the 
latter.


In Racket, in the initial configuration of the reader when reading a file, "`#!` is an alias for `#lang` followed by a space when `#!` is followed by alphanumeric ASCII, `+`, `-`, or `_`." (See .) [...] > (Guile does not handle `#!r6rs` properly, presumably because of the 
legacy `#!`/`!#` block comments. I think this should be a surmountable 
obstacle, though, especially since Guile does support standard `#|`/`|#` 
block comments.)


‘#! ... !#’ comments aren't legacy; they exist to allow putting the 
shebang in the first line of a script, and to pass additional arguments 
to the Guile interpreter (see: (guile)The Top of a Script File) (*).  As 
such, you can't just replace them with #| ... |# (unless you patch the 
kernel to recognise "#| ..." as a shebang line).


(*) Maybe they exist for other purposes too.

Furthermore, according to the kernel, #!r6rs would mean that the script 
needs to be interpreted by a program named 'r6rs', but 'guile' is named 
'guile', not 'r6rs'.  (I assume this is in POSIX somewhere, though I 
couldn't find it.)


(This is an incompatibility between R6RS and any system that has shebangs.)



(^) it doesn't integrate with the module system -- more concretely,
(use-modules (foo)) wouldn't try loading foo.js -- adding '-x' arguments
would solve that, but we agree that that would be unreasonable in many
situations.  (Alternatively one could place ECMAScript code in a file
with extension '.scm' with a '#lang' / '-*- mode: ecmascript -*-', but
... no.)



Racket has a mechanism to enable additional source file extensions without needing 
explicit command-line arguments by defining `module-suffixes` or `doc-modules-suffixes` 
in a metadata module that is consulted when the collection is "set up": 
https://docs.racket-lang.org/raco/setup-info.html However, this mechanism is not widely 
used.


I guess this is an improvement over the runtime 'guile -x extension'.
However, if I'm understanding 'setup-info.html' correctly, the downside 
is that you now need a separate file containing compilation settings.


I have previously proposed a mechanism that makes the '-x' + 
'--language' a compile-time thing (i.e., embed the source file extension 
in the compiled .go; see previous e-mails in this thread), without 
having to make a separate file containing compilation settings.


How is Racket's method an improvement over my proposal?


Overall, the experience of the Racket community strongly suggests that a file 
should say what language it is written in. Furthermore, that language is a 
property of the code, not of its runtime environment, so environment variables, 
command-line options, and similar extralinguistic mechanism are a particularly 
poor fit for controlling it.


Agreed on the 'no environment variables' thing, disagreed on the 'no 
command-line options'.  In the past e-mails in this thread, there was 
agreement on the ‘embed the source file extension in the compiled .go or 
something like that; and add -x extension stuff _when compiling_ (not 
runtime!) the software 

Re: [PATCH] add language/wisp to Guile?

[Philip McGrath]

Hi,

On Fri, Feb 17, 2023, at 6:06 PM, Maxime Devos wrote:
> On 16-02-2023 22:38, Dr. Arne Babenhauserheide wrote:
>> 
>> Matt Wette  writes:
>> 
>>> You may be interested in the load-lang patch I generated a few years ago
>>> to allow file-extension based loading, in addition to '#lang elisp"
>>> type hooks.
>>>
>>> https://github.com/mwette/guile-contrib/blob/main/patch/3.0.8/load-lang.patch
>> 
>> @Maxime: Is this something you’d be interested in championing?
>
> For the '#lang whatever stuff', no:
>
> The '#lang whatever' stuff makes Scheme (*) files unportable between 
> implementations, as '#lang scheme' is not a valid comment -- there exist 
> Schemes beyond Guile and Racket.  If it were changed to recognising
> '-*- mode: scheme -*-' or '-*- language: scheme -*-' or such, it would 
> be better IMO, but insufficient, because (^).
>

I haven't read the patch or this thread closely, but R6RS has an answer to any 
concerns about compatibility with `#lang`. At the beginning of Chapter 4, 
"Lexical and Datum Syntax" 
() the report 
specifies:

>  An implementation must not extend the lexical or datum syntax in any way, 
> with one exception: it need not treat the syntax `#!`, for any 
>  (see section 4.2.4) that is not `r6rs`, as a syntax violation, 
> and it may use specific `#!`-prefixed identifiers as flags indicating that 
> subsequent input contains extensions to the standard lexical or datum syntax. 
> The syntax `#!r6rs` may be used to signify that the input afterward is 
> written with the lexical syntax and datum syntax described by this report. 
> `#!r6rs` is otherwise treated as a comment; see section 4.2.3.

Chez Scheme uses such comments to support extensions to lexical syntax, as 
documented in 
:

> The Chez Scheme lexical extensions described above are disabled in an input 
> stream after an `#!r6rs` comment directive has been seen, unless a 
> `#!chezscheme` comment directive has been seen since. Each library loaded 
> implicitly via import and each RNRS top-level program loaded via the 
> `--program` command-line option, the `scheme-script` command, or the 
> `load-program` procedure is treated as if it begins implicitly with an 
> `#!r6rs` comment directive.

> The case of symbol and character names is normally significant, as required 
> by the Revised6 Report. Names are folded, as if by string-foldcase, following 
> a `#!fold-case` comment directive in the same input stream unless a 
> `#!no-fold-case` has been seen since. Names are also folded if neither 
> directive has been seen and the parameter `case-sensitive` has been set to 
> `#f`. 

In Racket, in the initial configuration of the reader when reading a file, 
"`#!` is an alias for `#lang` followed by a space when `#!` is followed by 
alphanumeric ASCII, `+`, `-`, or `_`." (See 
.) 
This does not conflict with Racket's support for script shebangs: "A `#!`  
(which is `#!` followed by a space) or `#!/` starts a line comment that can be 
continued to the next line by ending a line with `\`. This form of comment 
normally appears at the beginning of a Unix script file." (See 
.)
 Furthermore, the lexical syntax for the rest of the file is entirely under 
control of the specified language. Most languages parameterize the reader to 
reject further uses of `#lang` or its `#!` alias. Some "meta-languages" 
chain-load another language but parameterize it in some way (e.g. 
). The `#!r6rs` language, of 
course, handles `#!` exactly as specified by R6RS, with no extensions. 

(Guile does not handle `#!r6rs` properly, presumably because of the legacy 
`#!`/`!#` block comments. I think this should be a surmountable obstacle, 
though, especially since Guile does support standard `#|`/`|#` block comments.)

>
> (^) it doesn't integrate with the module system -- more concretely, 
> (use-modules (foo)) wouldn't try loading foo.js -- adding '-x' arguments 
> would solve that, but we agree that that would be unreasonable in many 
> situations.  (Alternatively one could place ECMAScript code in a file 
> with extension '.scm' with a '#lang' / '-*- mode: ecmascript -*-', but 
> ... no.)
>

Racket has a mechanism to enable additional source file extensions without 
needing explicit command-line arguments by defining `module-suffixes` or 
`doc-modules-suffixes` in a metadata module that is consulted when the 
collection is "set up": https://docs.racket-lang.org/raco/setup-info.html 
However, this mechanism is not widely used.

Overall, the experience of the Racket community strongly suggests that a file 
should say what language it is written in. Furthermore, that language is a 
property of the code, not of its 

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

On 16-02-2023 22:38, Dr. Arne Babenhauserheide wrote:


Matt Wette  writes:


You may be interested in the load-lang patch I generated a few years ago
to allow file-extension based loading, in addition to '#lang elisp"
type hooks.

https://github.com/mwette/guile-contrib/blob/main/patch/3.0.8/load-lang.patch


@Maxime: Is this something you’d be interested in championing?


For the '#lang whatever stuff', no:

The '#lang whatever' stuff makes Scheme (*) files unportable between 
implementations, as '#lang scheme' is not a valid comment -- there exist 
Schemes beyond Guile and Racket.  If it were changed to recognising
'-*- mode: scheme -*-' or '-*- language: scheme -*-' or such, it would 
be better IMO, but insufficient, because (^).


(*) Same argument applies for some, but not all, other non-Scheme 
languages too.


I'm assuming you don't meant the '%file-extension-map' stuff, because of 
your previous ‘[...] and also to avoid stumbling over files that just 
take that extension’ response to the proposal for such a thing.  Even if 
you meant that, (^) also applies.


(^) it doesn't integrate with the module system -- more concretely, 
(use-modules (foo)) wouldn't try loading foo.js -- adding '-x' arguments 
would solve that, but we agree that that would be unreasonable in many 
situations.  (Alternatively one could place ECMAScript code in a file 
with extension '.scm' with a '#lang' / '-*- mode: ecmascript -*-', but 
... no.)


Aside from the '#lang ...' -> '-*- language: scheme -*-' stuff, I think 
the idea behind the patch is good (**) -- it solves the problem it aims 
to solve, AFAICT.  However, this problem is not the module system 
problem that Wisp currently has.


As such, I suppose you could say that I would 'champion' the patch on 
its own, but not champion it in relation to '[PATCH] add language/wisp 
to Guile'.


(**) It needs some documentation though.

Greetings,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Matt Wette]

On 2/16/23 1:38 PM, Dr. Arne Babenhauserheide wrote:

Matt Wette  writes:


You may be interested in the load-lang patch I generated a few years ago
to allow file-extension based loading, in addition to '#lang elisp"
type hooks.

https://github.com/mwette/guile-contrib/blob/main/patch/3.0.8/load-lang.patch

@Maxime: Is this something you’d be interested in championing?

@Matt: Who needs to ack your patch for it to go into the repo?



If by repo you mean main branch, I'm guessing this is a Ludo/Andy call.
Maybe someone (you?) with write priv's could make a wip-branch for it?

Matt

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Matt Wette  writes:

> You may be interested in the load-lang patch I generated a few years ago
> to allow file-extension based loading, in addition to '#lang elisp"
> type hooks.
>
> https://github.com/mwette/guile-contrib/blob/main/patch/3.0.8/load-lang.patch

@Maxime: Is this something you’d be interested in championing?

@Matt: Who needs to ack your patch for it to go into the repo?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Maxime Devos  writes:
> On 16-02-2023 09:03, Dr. Arne Babenhauserheide wrote:
>> Do you mean using it similar to this test?
>>(pass-if "R6RS/SRFI-30 block comment syntax overridden"
>>  ;; To be compatible with 1.8 and earlier, we should be able to override
>>  ;; this syntax.
>>  (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures)))
>>(read-hash-extend #\| (lambda args 'not))
>>(fold (lambda (x y result)
>>(and result (eq? x y)))
>>  #t
>>  (read-string "(this is #| a comment)")
>>  `(this is not a comment
>
> That appears to me a valid (and slightly simpler and more robust) way
> of doing things, yes.

I finally got it working with passing tests — thank you!

The reason that it did not work was that in the wisp repo there are
tests that use wisp.scm directly without going through spec.scm.

From 12aa7314ad85f442f8bfe85839127bf1929be2ba Mon Sep 17 00:00:00 2001
From: Arne Babenhauserheide 
Date: Thu, 16 Feb 2023 22:34:00 +0100
Subject: [PATCH] Only extend the reader while reading wisp. Thanks to Maxime
 Devos!

* module/language/wisp/spec.scm (read-one-wisp-sexp): extend hash only in fluid
* module/language/wisp.scm (wisp-scheme-read-chunk): extend hash in fluid
---
 module/language/wisp.scm  | 18 ++
 module/language/wisp/spec.scm |  8 ++--
 2 files changed, 12 insertions(+), 14 deletions(-)

diff --git a/module/language/wisp.scm b/module/language/wisp.scm
index 65ef8e65a..7a12e126a 100644
--- a/module/language/wisp.scm
+++ b/module/language/wisp.scm
@@ -725,14 +725,16 @@ Match is awesome!"
  a))
 
 (define (wisp-scheme-read-chunk port)
- "Read and parse one chunk of wisp-code"
- (let (( lines (wisp-scheme-read-chunk-lines port)))
-  (wisp-make-improper
-(wisp-replace-empty-eof
-  (wisp-unescape-underscore-and-colon
-(wisp-replace-paren-quotation-repr
-  (wisp-propagate-source-properties
-(wisp-scheme-indentation-to-parens lines
+"Read and parse one chunk of wisp-code"
+(with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures)))
+  (read-hash-extend #\# (lambda args #\#))
+  (let ((lines (wisp-scheme-read-chunk-lines port)))
+   (wisp-make-improper
+ (wisp-replace-empty-eof
+   (wisp-unescape-underscore-and-colon
+ (wisp-replace-paren-quotation-repr
+   (wisp-propagate-source-properties
+ (wisp-scheme-indentation-to-parens lines)
 
 (define (wisp-scheme-read-all port)
  "Read all chunks from the given port"
diff --git a/module/language/wisp/spec.scm b/module/language/wisp/spec.scm
index 3ba248aa9..477036c71 100644
--- a/module/language/wisp/spec.scm
+++ b/module/language/wisp/spec.scm
@@ -45,12 +45,8 @@
   ;; Don't use the globally-acting read-hash-extend, because this
   ;; doesn't make much sense in parenthese-y (non-Wisp) Scheme.
   ;; Instead, use fluids to temporarily add the extension.
-  (read-hash-extend #\# (λ (chr port) #\#))
-  (define %read-hash-procedures/parameter
-(fluid->parameter %read-hash-procedures))
-  (parameterize ((%read-hash-procedures/parameter
-   `((#\# ,(λ (chr port) #\#))
- ,@(%read-hash-procedures/parameter
+  (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures)))
+(read-hash-extend #\# (lambda args #\# ))
 ;; Read Wisp files as UTF-8, to support non-ASCII characters.
 ;; TODO: would be nice to support ';; coding: whatever' lines
 ;; like in parenthese-y Scheme.
-- 
2.39.1


Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

On 16-02-2023 09:03, Dr. Arne Babenhauserheide wrote:


Maxime Devos  writes:

+ ;; allow using "# foo" as #(foo).
+ (read-hash-extend #\# (λ (chr port) #\#))


That's a rather Wisp-specific extension, but it appears you are
extending things globally.  Instead, I propose extending it
temporarily, with the undocumented '%read-hash-procedures' fluid.


I tried the spec-example, but it didn’t work for me.


Assuming that with 'spec-example', you meant

  (parameterize ((%read-hash-procedures/parameter
   `((#\# ,(λ (chr port) #\#))
 ,@(%read-hash-procedures/parameter
[...]):

I forgot to place a '.' between #\# and ,(λ ...).


Do you mean using it similar to this test?

   (pass-if "R6RS/SRFI-30 block comment syntax overridden"
 ;; To be compatible with 1.8 and earlier, we should be able to override
 ;; this syntax.
 (with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures)))
   (read-hash-extend #\| (lambda args 'not))
   (fold (lambda (x y result)
   (and result (eq? x y)))
 #t
 (read-string "(this is #| a comment)")
 `(this is not a comment


That appears to me a valid (and slightly simpler and more robust) way of 
doing things, yes.


Greetings,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Maxime Devos  writes:
>> + ;; allow using "# foo" as #(foo).
>> + (read-hash-extend #\# (λ (chr port) #\#))
>
> That's a rather Wisp-specific extension, but it appears you are
> extending things globally.  Instead, I propose extending it
> temporarily, with the undocumented '%read-hash-procedures' fluid.

I tried the spec-example, but it didn’t work for me.

Do you mean using it similar to this test?

  (pass-if "R6RS/SRFI-30 block comment syntax overridden"
;; To be compatible with 1.8 and earlier, we should be able to override
;; this syntax.
(with-fluids ((%read-hash-procedures (fluid-ref %read-hash-procedures)))
  (read-hash-extend #\| (lambda args 'not))
  (fold (lambda (x y result)
  (and result (eq? x y)))
#t
(read-string "(this is #| a comment)")
`(this is not a comment

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Maxime Devos  writes:

>> [...]
>> Which begs an important question: How would you like to be attributed? I
>> plan to also merge this back to the wisp repo and I’d like to attribute
>> you there, too.
>
> You could add a ";; Copyright © 2023 Maxime Devos
> " line next to yours in the file that contains
> the read-one-wisp-sexp I modified, a line like
> 'https://hg.sr.ht/~arnebab/wisp/browse/NEWS?rev=tip#L63' in the NEWS
> and an entry in 'Specific Contributions' in AUTHORS.in.

I implemented that now — thank you!

In the wisp-repo I added the copyright line in both the source file for
wisp.scm and spec.scm, because I see the unlimited underscores as
significant, too.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

[...]
Which begs an important question: How would you like to be attributed? I
plan to also merge this back to the wisp repo and I’d like to attribute
you there, too.


You could add a ";; Copyright © 2023 Maxime Devos 
" line next to yours in the file that contains 
the read-one-wisp-sexp I modified, a line like 
'https://hg.sr.ht/~arnebab/wisp/browse/NEWS?rev=tip#L63' in the NEWS and 
an entry in 'Specific Contributions' in AUTHORS.in.


Greetings,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Maxime Devos  writes:

>> [...]
>> That would be nice, but would require doing changes in a critical core
>> part of Guile. It would change this addition from a risk-free added
>> feature to a risky core change.
>
> I maintain that a new language shouldn't be merged until the
> Scheme-specific load path stuff is fixed/extended to work for
> non-Scheme things (e.g. Wisp) too -- if this requires somewhat risky
> (*) changes to core parts, then that just means we'll have to do some
> risky stuff, then.
>
> I also expect that Guile maintainers will have the opposite opinion
> (i.e., ‘fixing the load path stuff isn't necessary for merging a new
> language implementation’).
>
> (*) FWIW I disagree on the 'risky' assessment -- it seems like a ‘if
> it runs, it will work’ thing to me.  That it modifies a core part of
> Guile, makes it less risky IMO, as it would automatically be more
> tested.
>
> Aside from the (*) and the 'I also expect [...],', I don't have
> anything new to say about this, so I'll stop here.

Thank you for your suggestions and contributions so far. They help me a
lot!

>> [...]
>> That would also enable shipping pre-compiled software without
>> sourcecode,
>
> That can already be done -- besides legalities, nothing stops people
> from putting [^] or [^] .scm files in $GUILE_LOAD_PATH and putting .go
> in $GUILE_LOAD_COMPILED_PATH.
>
> [^]: Redacted to not give people ideas on how to circumvent stuff.
> I can elaborate by non-public e-mail if you like.

Thank you! (for redacting) — I hope I’ll never need that :-)

> On 14-02-2023 22:24, Dr. Arne Babenhauserheide wrote:
>> PS: So what’s still missing here is to avoid setting the locale. Do you
>>  happen to have a hint how to actually do this right?
>
> I think you might have forgotten about this:
>
>>  -- Scheme Procedure: set-port-encoding! port enc
>>  -- C Function: scm_set_port_encoding_x (port, enc)
>>  Sets the character encoding that will be used to interpret I/O to
>>  PORT.  ENC is a string containing the name of an encoding.  Valid
>>  encoding names are those defined by IANA
>>  (http://www.iana.org/assignments/character-sets), for example
>>  ‘"UTF-8"’ or ‘"ISO-8859-1"’.
>> As such, I propose calling set-port-encoding! right in the beginning
>> of read-one-wisp-sexp.

Yikes, yes. I shouldn’t spend so much time thinking about implications
when I haven’t yet applied all the clear and uncontested improvements.

Thank you!

> More concretely, replace
>
> (define (read-one-wisp-sexp port env)
>  ;; allow using "# foo" as #(foo).
>  (read-hash-extend #\# (λ (chr port) #\#))
>  (cond
> ((eof-object? (peek-char port))
>   (read-char port )); return eof: we’re done
> (else
>   (let ((chunk (wisp-scheme-read-chunk port)))
> (cond
>   ((not (null? chunk))
> (car chunk))
>   (else
> #f))
>
> by
>
> (define (read-one-wisp-sexp port env)
>   ;; Allow using "# foo" as #(foo).
>   ;; Don't use the globally-acting read-hash-extend, because this
>   ;; doesn't make much sense in parenthese-y (non-Wisp) Scheme.
>   ;; Instead, use fluids to temporarily add the extension.
>   (define %read-hash-procedures/parameter
> (fluid->parameter %read-hash-procedures))
>   (parameterize ((%read-hash-procedures/parameter
>`((#\# ,(λ (chr port) #\#))
>  ,@(%read-hash-procedures/parameter
> ;; Read Wisp files as UTF-8, to support non-ASCII characters.
> ;; TODO: would be nice to support ';; coding: whatever' lines
> ;; like in parenthese-y Scheme.
> (set-port-encoding! port "UTF-8")
> (if (eof-object? (peek-char port))
> (read-char port) ; return eof: we’re done
> (let ((chunk (wisp-scheme-read-chunk port)))
>   (and (not (null? chunk)) ; < XXX: maybe (pair? chunk)
>(car chunk))
>
> (untested).
>
> (I've also done the read-hash-extend stuff and simplified the 'cond'
> expressions.)

Thank you again for that!

Which begs an important question: How would you like to be attributed? I
plan to also merge this back to the wisp repo and I’d like to attribute
you there, too.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: [PATCH] add language/wisp to Guile?

[Matt Wette]

You may be interested in the load-lang patch I generated a few years ago
to allow file-extension based loading, in addition to '#land elisp" type 
hooks.


https://github.com/mwette/guile-contrib/blob/main/patch/3.0.8/load-lang.patch

Matt

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

> [...]

That would be nice, but would require doing changes in a critical core
part of Guile. It would change this addition from a risk-free added
feature to a risky core change.


I maintain that a new language shouldn't be merged until the 
Scheme-specific load path stuff is fixed/extended to work for non-Scheme 
things (e.g. Wisp) too -- if this requires somewhat risky (*) changes to 
core parts, then that just means we'll have to do some risky stuff, then.


I also expect that Guile maintainers will have the opposite opinion 
(i.e., ‘fixing the load path stuff isn't necessary for merging a new 
language implementation’).


(*) FWIW I disagree on the 'risky' assessment -- it seems like a ‘if it 
runs, it will work’ thing to me.  That it modifies a core part of Guile, 
makes it less risky IMO, as it would automatically be more tested.


Aside from the (*) and the 'I also expect [...],', I don't have anything 
new to say about this, so I'll stop here.


> [...]

That would also enable shipping pre-compiled software without
sourcecode,


That can already be done -- besides legalities, nothing stops people 
from putting [^] or [^] .scm files in $GUILE_LOAD_PATH and putting .go 
in $GUILE_LOAD_COMPILED_PATH.


[^]: Redacted to not give people ideas on how to circumvent stuff.
I can elaborate by non-public e-mail if you like.


so there may be strategic reasons to avoid it. Always
providing the sourcecode also makes compliance with automatic copyleft
licenses automatic.


Mm, yes, I guess.  If only people weren't careless and didn't try to 
circumvent copyleft, then things would be easier ...


On 14-02-2023 22:24, Dr. Arne Babenhauserheide wrote:

PS: So what’s still missing here is to avoid setting the locale. Do you
 happen to have a hint how to actually do this right?


I think you might have forgotten about this:


 -- Scheme Procedure: set-port-encoding! port enc
 -- C Function: scm_set_port_encoding_x (port, enc)
 Sets the character encoding that will be used to interpret I/O to
 PORT.  ENC is a string containing the name of an encoding.  Valid
 encoding names are those defined by IANA
 (http://www.iana.org/assignments/character-sets), for example
 ‘"UTF-8"’ or ‘"ISO-8859-1"’.

As such, I propose calling set-port-encoding! right in the beginning of read-one-wisp-sexp. 


More concretely, replace

(define (read-one-wisp-sexp port env)
 ;; allow using "# foo" as #(foo).
 (read-hash-extend #\# (λ (chr port) #\#))
 (cond
((eof-object? (peek-char port))
  (read-char port )); return eof: we’re done
(else
  (let ((chunk (wisp-scheme-read-chunk port)))
(cond
  ((not (null? chunk))
(car chunk))
  (else
#f))

by

(define (read-one-wisp-sexp port env)
  ;; Allow using "# foo" as #(foo).
  ;; Don't use the globally-acting read-hash-extend, because this
  ;; doesn't make much sense in parenthese-y (non-Wisp) Scheme.
  ;; Instead, use fluids to temporarily add the extension.
  (define %read-hash-procedures/parameter
(fluid->parameter %read-hash-procedures))
  (parameterize ((%read-hash-procedures/parameter
   `((#\# ,(λ (chr port) #\#))
 ,@(%read-hash-procedures/parameter
;; Read Wisp files as UTF-8, to support non-ASCII characters.
;; TODO: would be nice to support ';; coding: whatever' lines
;; like in parenthese-y Scheme.
(set-port-encoding! port "UTF-8")
(if (eof-object? (peek-char port))
(read-char port) ; return eof: we’re done
(let ((chunk (wisp-scheme-read-chunk port)))
  (and (not (null? chunk)) ; < XXX: maybe (pair? chunk)
   (car chunk))

(untested).

(I've also done the read-hash-extend stuff and simplified the 'cond' 
expressions.)


Greetings,
Maxime.


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

PS: So what’s still missing here is to avoid setting the locale. Do you
happen to have a hint how to actually do this right?

-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Maxime Devos  writes:

> On 04-02-2023 22:35, Dr. Arne Babenhauserheide wrote:
>> Maxime Devos  writes:
>> 
 This needs an addition to the extensions via guile -x .w — I wrote
 that
 in the documentation. I didn’t want to do that unconditionally, because
 detecting a wisp file as scheme import would cause errors.
>>>
>>> If done carefully, I don't think this situations would happen.
>>> More precisely:
>>>
>>>* .w would be in the file extensions list.
>>>
>>>* Instead of a list, it would actually be a map from extensions to
>>>  languages:
>>>
>>>.scm -> scheme
>>>.w -> wisp
>>>
>>>  With this change, (use-modules (foo)) will load 'foo.scm' as Scheme
>>>  and 'foo.w' as Wisp.  (Assuming that foo.go is out-of-date or
>>>  doesn't exist.)
>>>
>>>  (For backwards compatibility, I think %load-extensions needs to
>>>  remain a list of strings, but a %extension-language variable could
>>>  be defined.)
>>>
>>>* "guile --language=whatever foo" loads foo as whatever, regardless
>>>  of the extension of 'foo' (if a specific language is requested,
>>>  then the user knows best).
>>>
>>>* "guile foo" without --language will look up the extension of foo in
>>>  the extension map. If an entry exists, it would use the
>>>  corresponding language.  If no entry exists, it would use
>>>  a default language (scheme).
>> This sounds good, though a bit more complex than I think it should
>> be.
>> I think this should stick to only load Scheme if no language is
>> detected
>> to keep Scheme the default language for Guile
>
> To my knowledge, this is the case in my proposal.  Detecting the
> language is done via the file extension, and if no known mapping
> exists, it defaults to Scheme.

I’ve been thinking about this for a while and I expect that this will
take quite a bit of discussion, because it is a change that would affect
every language already shipped by Guile, and it is a change that needs
strategic decisions.

I like the idea but would like to separate out changes to how languages
are treated.

>> — and also to avoid
>> stumbling over files that just take that extension.
>
> While I suppose it is theoretically possible someone will write a .w
> file that contains Scheme code instead of Wisp, I'm not convinced by
> this argument.  It sounds very unlikely, and also a 'don't do that,
> then' situation.

I don’t mean Scheme files, but rather some other tool using the
extension for another file format.

> Maybe the .go could contain some information on what the corresponding
> source code file name is, and Guile could read the .go without
> checking first checking for up-to-dateness. (But only reading; not
> loading yet!)

That would be nice, but would require doing changes in a critical core
part of Guile. It would change this addition from a risk-free added
feature to a risky core change.

> Once read, it should be easy to look up the source code file name from
> the .go and then verify whether the .go is up to date, and proceed
> with actually loading the .go (as in, put stuff in the module system,
> run top-level code, ...).

That’s what I’l like to see, yes.

> IIUC, that would be fully backwards compatible, and not cause any
> non-negligible slowdowns.

It could actually speed up loading, because there are fewer locations
for caching bytecode files that locations for source-files.

>  * when doing (use-module (foo)) and foo.go exists in the
>$GUILE_LOAD_COMPILED_PATH, load it, and don't
>bother checking whether foo.scm, foo.w or foo.whatever exists
>or is up-to-date.  (If not done already in Guile.)
>
> That should solve the 'I don't care what language the library Foo is
> written in, and my library Bar isn't written in Wisp so it seems
> unreasonable to have to add -x w.’, and would also avoid the need for
> a 'extension -> language map' thing.  It should also be a little
> faster than what we had before.

That would also enable shipping pre-compiled software without
sourcecode, so there may be strategic reasons to avoid it. Always
providing the sourcecode also makes compliance with automatic copyleft
licenses automatic.

 readable uses
>>>
>>> This sentence appears to be incomplete; I might have misinterpreted it
>>> below (I don't know what you mean with 'readable' -- its an adjective
>>> and you are using it as a noun?).
>> readable is a noun, yes: the readable lisp project.
>
> Looks like you meant this: .
 Can I write it into a string and then read it back?
>>>
>>> No.  [...]
>> This sounds like I cannot go that way, because there’s a necessary
>> pre-processing step in wisp-read via (match-charlist-to-repr peeked):
>> [...] > This actually needs to be able to write the replacement symbols back
>> into the port.
>>
>>> ..., for which I proposed a replacement, so do you still need to turn
>>> it in a string & back?
>> Sadly yes. Otherwise the normal reader will 

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

On 04-02-2023 22:35, Dr. Arne Babenhauserheide wrote:


Maxime Devos  writes:


This needs an addition to the extensions via guile -x .w — I wrote
that
in the documentation. I didn’t want to do that unconditionally, because
detecting a wisp file as scheme import would cause errors.


If done carefully, I don't think this situations would happen.
More precisely:

   * .w would be in the file extensions list.

   * Instead of a list, it would actually be a map from extensions to
 languages:

   .scm -> scheme
   .w -> wisp

 With this change, (use-modules (foo)) will load 'foo.scm' as Scheme
 and 'foo.w' as Wisp.  (Assuming that foo.go is out-of-date or
 doesn't exist.)

 (For backwards compatibility, I think %load-extensions needs to
 remain a list of strings, but a %extension-language variable could
 be defined.)

   * "guile --language=whatever foo" loads foo as whatever, regardless
 of the extension of 'foo' (if a specific language is requested,
 then the user knows best).

   * "guile foo" without --language will look up the extension of foo in
 the extension map. If an entry exists, it would use the
 corresponding language.  If no entry exists, it would use
 a default language (scheme).


This sounds good, though a bit more complex than I think it should be.

I think this should stick to only load Scheme if no language is detected
to keep Scheme the default language for Guile


To my knowledge, this is the case in my proposal.  Detecting the 
language is done via the file extension, and if no known mapping exists, 
it defaults to Scheme.



— and also to avoid
stumbling over files that just take that extension.


While I suppose it is theoretically possible someone will write a .w 
file that contains Scheme code instead of Wisp, I'm not convinced by 
this argument.  It sounds very unlikely, and also a 'don't do that, 
then' situation.



Checking more files
could slow down startup and I think having multiple languages fully
equal would risk splintering the development community.

Guile is first and foremost Scheme and fast startup time is essential.

More complicated is what should be done if a *.go file is detected
during import. There I could see Guile check if a file with any
supported extension is up to date.


Maybe the .go could contain some information on what the corresponding 
source code file name is, and Guile could read the .go without checking 
first checking for up-to-dateness. (But only reading; not loading yet!)


(There is already debugging information with such information, but to my
understanding that's for individual procedures, not the .go as a whole, 
and by using stuff like 'include' or macros there can be multiple source 
files.)


Once read, it should be easy to look up the source code file name from 
the .go and then verify whether the .go is up to date, and proceed with 
actually loading the .go (as in, put stuff in the module system, run 
top-level code, ...).


IIUC, that would be fully backwards compatible, and not cause any 
non-negligible slowdowns.


I also have an alternative proposal, more complicated and 
backwards-incompatible -- I wouldn't recommend it, but for completeness:


 * when doing (use-module (foo)) and foo.go exists in the
   $GUILE_LOAD_COMPILED_PATH, load it, and don't
   bother checking whether foo.scm, foo.w or foo.whatever exists
   or is up-to-date.  (If not done already in Guile.)

That should solve the 'I don't care what language the library Foo is 
written in, and my library Bar isn't written in Wisp so it seems 
unreasonable to have to add -x w.’, and would also avoid the need for a 
'extension -> language map' thing.  It should also be a little faster 
than what we had before.


That's for "make install", "apt-get install", "guix install" ...-like 
uses of compiled .go -- let's call them 'installed .go'.  It won't work 
for ~/.cache/guile/ccache/3.0-LE-8-4 (‘cached .go’) as for that it's 
actually important to check up-to-dateness because, well, cache.


Additionally, to support compiling software that is already installed, 
there needs to be an option to treat certain modules with the 'cache' 
behaviour even if not in the actual ~/.cache, maybe with some 
'--local-module=(stuff ...)' option (name pending).


This would be backwards-incompatible, but it could be done.


readable uses


This sentence appears to be incomplete; I might have misinterpreted it
below (I don't know what you mean with 'readable' -- its an adjective
and you are using it as a noun?).


readable is a noun, yes: the readable lisp project.


Looks like you meant this: .

Can I write it into a string and then read it back?


No.  [...]


This sounds like I cannot go that way, because there’s a necessary
pre-processing step in wisp-read via (match-charlist-to-repr peeked):
[...] > This actually needs to be able to write the replacement symbols back
into the port.


..., for which I 

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Maxime Devos  writes:

>> This needs an addition to the extensions via guile -x .w — I wrote
>> that
>> in the documentation. I didn’t want to do that unconditionally, because
>> detecting a wisp file as scheme import would cause errors.
>
> If done carefully, I don't think this situations would happen.
> More precisely:
>
>   * .w would be in the file extensions list.
>
>   * Instead of a list, it would actually be a map from extensions to
> languages:
>
>   .scm -> scheme
>   .w -> wisp
>
> With this change, (use-modules (foo)) will load 'foo.scm' as Scheme
> and 'foo.w' as Wisp.  (Assuming that foo.go is out-of-date or
> doesn't exist.)
>
> (For backwards compatibility, I think %load-extensions needs to
> remain a list of strings, but a %extension-language variable could
> be defined.)
>
>   * "guile --language=whatever foo" loads foo as whatever, regardless
> of the extension of 'foo' (if a specific language is requested,
> then the user knows best).
>
>   * "guile foo" without --language will look up the extension of foo in
> the extension map. If an entry exists, it would use the
> corresponding language.  If no entry exists, it would use
> a default language (scheme).

This sounds good, though a bit more complex than I think it should be.

I think this should stick to only load Scheme if no language is detected
to keep Scheme the default language for Guile — and also to avoid
stumbling over files that just take that extension. Checking more files
could slow down startup and I think having multiple languages fully
equal would risk splintering the development community.

Guile is first and foremost Scheme and fast startup time is essential.

More complicated is what should be done if a *.go file is detected
during import. There I could see Guile check if a file with any
supported extension is up to date.

>> Is there a way to only extend the loading mechanism to detect .w when
>> language is changed to wisp?

> I don't care what language the library Foo is written in, and my
> library Bar isn't written in Wisp so it seems unreasonable to have to
> add -x w.

I think you’re right with that. For any already compiled library, the
language should not matter.

>> readable uses
>
> This sentence appears to be incomplete; I might have misinterpreted it
> below (I don't know what you mean with 'readable' -- its an adjective
> and you are using it as a noun?).

readable is a noun, yes: the readable lisp project.

>> (set! %load-extensions (cons ".sscm" %load-extensions))
>> Would that be the correct way of doing this?

> FWIW, it appears to be an answer to the following unasked question:
>
>   How to make Guile accept "foo.go" when "foo.w" exists and is
>   up-to-date.

Yes, I think that is the most important question. If that is solved,
guile provides a multi-language environment in which only the build
tools of the libraries themselves have to know the languages used.

> This sounds like the second proposal ('alternatively ...'), but the
> way it is written, you appear to proposing it as a third proposal.  Is
> this the case?

It only differs in details (keeping Scheme more central and only
checking for non-scheme languages if a *.go file is detected).

> (I mean, after this patch, Wisp is a supported language, so it seems
> equivalent to me.)

Pretty close, yes.

 +; Set locale to something which supports unicode. Required to avoid
 using fluids.
 +(catch #t
>>>
>>>   * Why avoid fluids?
>> I’m not sure anymore. It has been years since I wrote that code …
>> I think it was because I did not understand what that would mean for
>> the
>> program. And I actually still don’t know …
>> Hoow would I do that instead with fluids?
>> 
>>>   * Assuming for sake of argument that fluids are to be avoided,
>>> what is the point of setting the locale to something supporting
>>> Unicode?
>> I had problems with reading unicode symbols. Things like
>> define (Σ . args) : apply + args
>> [...]>
>> This is to ensure that Wisp are always read as Unicode. Since it uses
>> regular (read) as part of parsing, it must affect (read), too.
>
> OK.  So, Wisp files are supposed to be UTF-8, no matter the locale?
> AFAICT, the SRFI-119 document does not mention this UTF-8 (or UTF-16,
> or ...) requirement anywhere, this seems like an omission in
>  to me.

That’s an omission, yes … but since it was omitted (by me …), you’re
right. Forcing UTF-8 is actually not the way.

> First, I would like to point out the following part of
> ‘(guile)The Top of a Script File’:
>
>• If this source code file is not ASCII or ISO-8859-1 encoded, a
>  coding declaration such as ‘coding: utf-8’ should appear in a
>  comment somewhere in the first five lines of the file: see *note
>  Character Encoding of Source Files::.
…
> (OTOH, (guile)Character Encoding says 'In the absence of any hints,
> UTF-8 is assumed.' 

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

On 04-02-2023 16:46, Dr. Arne Babenhauserheide wrote:

[...]

So I’d like to ask: can we merge Wisp as supported language into Guile?


 From some conversations elsewhere, I got the impression that

(use-modules (foo))

will search for foo.scm and not in foo.w.  I think you'll need to
tweak the loading mechanism to also look for foo.w instead of only
foo.scm, if not done already.


This needs an addition to the extensions via guile -x .w — I wrote that
in the documentation. I didn’t want to do that unconditionally, because
detecting a wisp file as scheme import would cause errors.


If done carefully, I don't think this situations would happen.
More precisely:

  * .w would be in the file extensions list.

  * Instead of a list, it would actually be a map from extensions to
languages:

  .scm -> scheme
  .w -> wisp

With this change, (use-modules (foo)) will load 'foo.scm' as Scheme
and 'foo.w' as Wisp.  (Assuming that foo.go is out-of-date or
doesn't exist.)

(For backwards compatibility, I think %load-extensions needs to
remain a list of strings, but a %extension-language variable could
be defined.)

  * "guile --language=whatever foo" loads foo as whatever, regardless
of the extension of 'foo' (if a specific language is requested,
then the user knows best).

  * "guile foo" without --language will look up the extension of foo in
the extension map. If an entry exists, it would use the
corresponding language.  If no entry exists, it would use
a default language (scheme).

With these changes, I don't think that Wisp code would be detected as 
Scheme or the other way around.



Is there a way to only extend the loading mechanism to detect .w when
language is changed to wisp?


Regardless of whether it's technically possible, that sounds 
insufficient to me.


Suppose someone writes a library 'Foo' in Wisp.
Suppose I write a library 'Bar' in parenthese-y Scheme, that happens to 
use the Foo library as a dependency.


Then when compiling Bar or running its tests, it will be done in the 
Scheme language, and additionally assuming that compiled .go are 
available for Foo, then the language will never be changed to Wisp, and 
hence .w will never be added to %load-extensions.


As such, the Makefile.am or equivalent of Foo would need to be converted 
to Wisp, or '-x w' would need to be added.


I don't care what language the library Foo is written in, and my library 
Bar isn't written in Wisp so it seems unreasonable to have to add -x w. 
(It wouldn't be too much trouble, but still not something that should 
have to be done _in Bar_, as the Wispyness of Foo is just an 
implementation detail of Foo, not Bar.)


Worse, adding the Wispy library Foo of the parenthese-y library Bar 
would be an incompatible change, as parenthese-y dependents of Foo would 
need to add '-x w' in places whereas they didn't to previously.  It's 
easily resolvable, but I think it would be very annoying as well.



readable uses


This sentence appears to be incomplete; I might have misinterpreted it 
below (I don't know what you mean with 'readable' -- its an adjective 
and you are using it as a noun?).



(set! %load-extensions (cons ".sscm" %load-extensions))

Would that be the correct way of doing this?


I assume you meant ".w" instead of ".sscm".  I don't quite see how this 
would be an answer to:


  Is there a way to only extend the loading mechanism to detect .w when
  language is changed to wisp?

More precisely, I'm missing how it addresses 'only ... when the language 
is changed to wisp'.


FWIW, it appears to be an answer to the following unasked question:

  How to make Guile accept "foo.go" when "foo.w" exists and is
  up-to-date.


Also, I think that when foo.go exists, but foo.scm doesn't, then Guile
refuses to load foo.scm, though I'm less sure of that. If this is the
case, I propose removing the requirement that the source code is
available, or alternatively keep the 'source code available'
requirement and also accept 'foo.w', if not done already.


I think accepting any extension supported by any language in Guile would
be better.


This sounds like the second proposal ('alternatively ...'), but the way 
it is written, you appear to proposing it as a third proposal.  Is this 
the case?


(I mean, after this patch, Wisp is a supported language, so it seems 
equivalent to me.)



+; Set locale to something which supports unicode. Required to avoid
using fluids.
+(catch #t


  * Why avoid fluids?


I’m not sure anymore. It has been years since I wrote that code …

I think it was because I did not understand what that would mean for the
program. And I actually still don’t know …

Hoow would I do that instead with fluids?


  * Assuming for sake of argument that fluids are to be avoided,
what is the point of setting the locale to something supporting
Unicode?


I had problems with reading unicode symbols. Things like
define (Σ . args) : apply + args

> [...]>

Re: [PATCH] add language/wisp to Guile?

[Dr. Arne Babenhauserheide]

Thank you for your review!

Maxime Devos  writes:

>> Why add Wisp?
>> For Wisp: it is then available directly wherever Guile is available.
>>   This will make it much easier for people to follow tutorials.
>
> I'm not convinced of this argument, because package managers exist, but ...
>
>> For Guile:
>> - Wisp has proven to be good at enabling people to get an
>>   entrance to Scheme² without pulling them out of the community.
>> - [...]
>
> ... all good points, and the implementation of Wisp is tiny anyway.
> For an additional reason: Wisp is a SRFI (Scheme Requests for
> Implementation) and Guile is a Scheme implementation.

That’s a good point — I should really have written it :-)

>> So I’d like to ask: can we merge Wisp as supported language into Guile?
>
> From some conversations elsewhere, I got the impression that
>
> (use-modules (foo))
>
> will search for foo.scm and not in foo.w.  I think you'll need to
> tweak the loading mechanism to also look for foo.w instead of only
> foo.scm, if not done already.

This needs an addition to the extensions via guile -x .w — I wrote that
in the documentation. I didn’t want to do that unconditionally, because
detecting a wisp file as scheme import would cause errors.

Is there a way to only extend the loading mechanism to detect .w when
language is changed to wisp?

readable uses

(set! %load-extensions (cons ".sscm" %load-extensions))

Would that be the correct way of doing this?

> Also, I think that when foo.go exists, but foo.scm doesn't, then Guile
> refuses to load foo.scm, though I'm less sure of that. If this is the
> case, I propose removing the requirement that the source code is
> available, or alternatively keep the 'source code available'
> requirement and also accept 'foo.w', if not done already.

I think accepting any extension supported by any language in Guile would
be better.

>> +; Set locale to something which supports unicode. Required to avoid
>> using fluids.
>> +(catch #t
>
>  * Why avoid fluids?

I’m not sure anymore. It has been years since I wrote that code …

I think it was because I did not understand what that would mean for the
program. And I actually still don’t know …

Hoow would I do that instead with fluids?

>  * Assuming for sake of argument that fluids are to be avoided,
>what is the point of setting the locale to something supporting
>Unicode?

I had problems with reading unicode symbols. Things like
define (Σ . args) : apply + args

> As-is, it now becomes impossible to use 'gettext' to translate
> software to non-English locales when the software imports (language
> wisp), which seems unfortunate to me.

That is very much not what I want.

> If you elaborate on what your
> goal here is, maybe I have an alternative solution.

This is to ensure that Wisp are always read as Unicode. Since it uses
regular (read) as part of parsing, it must affect (read), too.

>> + ;; allow using "# foo" as #(foo).
>> + (read-hash-extend #\# (λ (chr port) #\#))
>
> That's a rather Wisp-specific extension, but it appears you are
> extending things globally.  Instead, I propose extending it
> temporarily, with the undocumented '%read-hash-procedures' fluid.
>
>> +   (let
>> + (
>> +   (l
>
> Lonely parenthesis.

Thank you! Will be fixed :-)

> + (not (= 0 (line-real-indent (car lines ); -1 is a
> line with a comment
>
> Superfluous space after 'lines'.
>
>> + ; simple recursiive step to the next line
>
> I think the convention is ';;', OTOH there exist multiple conventions.
>
> +(define (wisp-scheme-replace-inline-colons lines)
> + "Replace inline colons by opening parens which close at the
> end of the line"
>
> Too much space; convention is two spaces.

> (Similar styles issues in other places.)
> "guix style" might be useful.

I’ll do that …

>> +(define (wisp-replace-paren-quotation-repr code)
>> + "Replace lists starting with a quotation symbol by
>> + quoted lists."
>> + (match code
>> + (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...)
>> +(list 'quote (map wisp-replace-paren-quotation-repr a)))
>> [...]
>> +(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd")
>> +; define an intermediate dot replacement with UUID to avoid clashes.
>> +(define repr-dot ; .
>> +   (string->symbol (string-append "REPR-DOT-" wisp-uuid)))
>
> There is a risk of collision -- e.g., suppose that someone translates
> your implementation of Wisp into Wisp.  I imagine there might be a
> risk of misinterpreting the 'REPR-QUOTE-...' in
> wisp-replace-parent-quotation-repr, though I haven't tried it out.

This is actually auto-translated from wisp via wisp2lisp :-)

> As such, assuming this actually works, I propose using uninterned
> symbols instead, e.g.:
>
> (define repr-dot (make-symbol "REPR-DOT")).

That looks better — does uninterned symbol mean it can’t be
mis-interpreted?

Can I 

Re: [PATCH] add language/wisp to Guile?

[Maxime Devos]

Why add Wisp?

For Wisp: it is then available directly wherever Guile is available.
  This will make it much easier for people to follow tutorials.


I'm not convinced of this argument, because package managers exist, but ...


For Guile:

- Wisp has proven to be good at enabling people to get an
  entrance to Scheme² without pulling them out of the community.

> - [...]

... all good points, and the implementation of Wisp is tiny anyway.  For 
an additional reason: Wisp is a SRFI (Scheme Requests for 
Implementation) and Guile is a Scheme implementation.



So I’d like to ask: can we merge Wisp as supported language into Guile?


From some conversations elsewhere, I got the impression that

(use-modules (foo))

will search for foo.scm and not in foo.w.  I think you'll need to tweak 
the loading mechanism to also look for foo.w instead of only foo.scm, if 
not done already.


Also, I think that when foo.go exists, but foo.scm doesn't, then Guile 
refuses to load foo.scm, though I'm less sure of that.  If this is the 
case, I propose removing the requirement that the source code is 
available, or alternatively keep the 'source code available' requirement 
and also accept 'foo.w', if not done already.


> +; Set locale to something which supports unicode. Required to avoid
> using fluids.
> +(catch #t

 * Why avoid fluids?
 * Assuming for sake of argument that fluids are to be avoided,
   what is the point of setting the locale to something supporting
   Unicode?

As-is, it now becomes impossible to use 'gettext' to translate software 
to non-English locales when the software imports (language wisp), which 
seems unfortunate to me.  If you elaborate on what your goal here is, 
maybe I have an alternative solution.



+ ;; allow using "# foo" as #(foo).
+ (read-hash-extend #\# (λ (chr port) #\#))


That's a rather Wisp-specific extension, but it appears you are 
extending things globally.  Instead, I propose extending it temporarily, 
with the undocumented '%read-hash-procedures' fluid.



+   (let
+ (
+   (l


Lonely parenthesis.

+ (not (= 0 (line-real-indent (car lines ); -1 is a line 
with a comment


Superfluous space after 'lines'.


+ ; simple recursiive step to the next line


I think the convention is ';;', OTOH there exist multiple conventions.

+(define (wisp-scheme-replace-inline-colons lines)
+ "Replace inline colons by opening parens which close at the 
end of the line"


Too much space; convention is two spaces.

(Similar styles issues in other places.)
"guix style" might be useful.


+(define (wisp-replace-paren-quotation-repr code)
+ "Replace lists starting with a quotation symbol by
+ quoted lists."
+ (match code
+ (('REPR-QUOTE-e749c73d-c826-47e2-a798-c16c13cb89dd a ...)
+(list 'quote (map wisp-replace-paren-quotation-repr a)))
[...]
+(define wisp-uuid "e749c73d-c826-47e2-a798-c16c13cb89dd")
+; define an intermediate dot replacement with UUID to avoid clashes.
+(define repr-dot ; .
+   (string->symbol (string-append "REPR-DOT-" wisp-uuid)))


There is a risk of collision -- e.g., suppose that someone translates 
your implementation of Wisp into Wisp.  I imagine there might be a risk 
of misinterpreting the 'REPR-QUOTE-...' in 
wisp-replace-parent-quotation-repr, though I haven't tried it out.


As such, assuming this actually works, I propose using uninterned 
symbols instead, e.g.:


(define repr-dot (make-symbol "REPR-DOT")).

If this change is done, you might need to replace

+ ;; literal array as start of a line: # (a b) c -> (#(a b) c)
+ ((#\# a ...)
+   (with-input-from-string ;; hack to defer to read
+   (string-append "#"
+   (with-output-to-string
+   (λ ()
+ (write (map 
wisp-replace-paren-quotation-repr a)

+ (current-output-port)
+   read))


(unverified -- I think removing this is unneeded but I don't understand 
this REPR-... stuff well enough).


Also, I wonder if you could just do something like

  (apply vector (map wisp-replace-paren-quotation-repr a))

instead of this 'hack to defer to read' thing.  This seems simpler to me 
and equivalent.


(AFAIK, these REPR-... symbols are never written to a port or turned 
into syntax, so I think that uninterned symbols would work here.)


(Aside from the REPR-... thing, I'm assuming (language wisp) is alright 
-- the SRFI is in 'final' status and it has been stable for years now, 
after all.)


Greetings,
Maxime


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature