Loading R6RS Libraries
Hello all, I encountered some behavior I didn't expect, and I'm not sure if it's intended or not. The following code works fine: (use-modules (rnrs base)) However, the next two lines both give errors: (use-modules (rnrs base (6))) (use-modules (rnrs base 6)) I didn't expect this because the manual refers to an '(rnrs base (6))' library. Is this what's supposed to be happening? Thanks, Noah
Re: Merging R6RS libraries?
Hello! Julian Graham jool...@gmail.com writes: Thanks to Andy's heroic work on the expander over the past week or so, I've just been able to merge `wip-r6rs-libraries' into `master' and push it! Cool, congratulations to both of you! As you'll notice from running `make check', there are still a few issues to be addressed -- specifically, the implementation of `div' and `mod' [0]. Anyone have any thoughts? Not quite, but I trust you’ll find out. ;-) Thanks, Ludo’.
Re: Merging R6RS libraries?
Hi all, Thanks to Andy's heroic work on the expander over the past week or so, I've just been able to merge `wip-r6rs-libraries' into `master' and push it! As you'll notice from running `make check', there are still a few issues to be addressed -- specifically, the implementation of `div' and `mod' [0]. Anyone have any thoughts? I guess I'll start digging into some number theory. Regards, Julian [0] - http://www.mail-archive.com/guile-devel@gnu.org/msg05278.html
Re: Merging R6RS libraries?
Hi Ludo, I've got some status updates: As I said, there aren't that many libraries left to do. Off the top of my head, the still-missing ones are (rnrs eval), (rnrs arithmetic fixnums), (rnrs arithmetic flonums), and the composite library, (rnrs rnrs). I also want to move the library form definitions out of boot-9, as you suggested. The above are done and pushed. At the moment, though, there are some broken bits related to the arithmetic libraries -- specifically, all of the procedures related to R6RS `div' and `mod'. They just weren't fully enough specified to implement: R6RS-lib says they implement number-theoretic integer division [0]. (The main R6RS spec says even less.) I peeked at PLT's implementation, which uses, I think, a stanard number-theoretic division algorithm and tweaks it to match the examples in R6RS-lib. We could poach theirs or write our own, but I'm not enough of a mathematician to know what the Right Thing to do here is. Hmmm. None that are so serious that they're currently blocking me -- I've been developing and testing the libraries using the library/module integration, so I'm quite sensitive to problems when they arise. :-) At the moment, the only thing that's bugging me is an issue with tests that use syntax exported from libraries fail when run via 'make check' but pass when loaded from the REPL. This also seems to be happening with plain old `make'. I've been trying to figure out what's different between running: $ ./meta/uninstalled-env guile-tools compile module/6/rnrs.scm ...and executing scheme@(guile-user) (compile-file module/6/rnrs.scm) ...from inside an instance of ./meta/guile (which works), but so far I've come up empty. This is pretty much the only thing I'd like to fix before telling Andy to pull the merge trigger. Any ideas? Regards, Julian [0] - http://www.r6rs.org/final/html/r6rs-lib/r6rs-lib-Z-H-1.html#node_toc_node_sec_11.2
Re: Merging R6RS libraries?
Ludo wrote: - Do you plan to work on the remaining parts, in particular the dreaded (to me) ‘(rnrs io ports)’? :-) I've been thinking a little about how (rnrs io ports) might be accomplished. If no one has been working on it, I could lay out some of my opinions and something of a plan. -Mike
Re: Merging R6RS libraries?
Hi Mike, Mike Gran spk...@yahoo.com writes: Ludo wrote: - Do you plan to work on the remaining parts, in particular the dreaded (to me) ‘(rnrs io ports)’? :-) I've been thinking a little about how (rnrs io ports) might be accomplished. If no one has been working on it, I could lay out some of my opinions and something of a plan. Yes, please! Thanks, Ludo’.
Merging R6RS libraries?
Hello! Looking at messages on guile-commits, I’m really amazed by the amount of work that Julian has been doing on the R6RS front. Now that all this code is here, I don’t see any reason not to include it in 2.0. Here’s a random list of things that I think should be considered. - What’s the status of your work wrt. to R6RS-lib? - Do you plan to work on the remaining parts, in particular the dreaded (to me) ‘(rnrs io ports)’? :-) - Does the code borrow snippets from reference implementations or some such? - How much code do unit tests cover? It’d be nice to try the test suite of the PLT folks, http://groups.google.com/group/comp.lang.scheme/browse_thread/thread/a7d691b5ca87b94f/6bb6e09f1d39d828. - Are there any known problems with the library/module integration, or pitfalls that ought to be documented? - In terms of documentation, we definitely don’t want to duplicate R6RS. However, it would be nice to have at least a node listing the available libraries and what they’re about. For those that provide features not available elsewhere in Guile, it’d be nice to have more detailed documentation (we’ve done that for bytevectors and I/O ports). Julian: would you like review on specific parts of your work? (I personally don’t have much time to dedicate to it these days but I could look at things you think are important.) What do you think? Is it reasonable to add it to the to-do list of 2.0? Thanks, Ludo’.
Re: Merging R6RS libraries?
Hi Ludovic, Looking at messages on guile-commits, I’m really amazed by the amount of work that Julian has been doing on the R6RS front. Now that all this code is here, I don’t see any reason not to include it in 2.0. Thanks! I was going to send a status update on this once I finished the few remaining libraries and test cases, but now's as good a time as any. - What’s the status of your work wrt. to R6RS-lib? As I said, there aren't that many libraries left to do. Off the top of my head, the still-missing ones are (rnrs eval), (rnrs arithmetic fixnums), (rnrs arithmetic flonums), and the composite library, (rnrs rnrs). I also want to move the library form definitions out of boot-9, as you suggested. There's also some subtler, scarier stuff that's related to R6RS-lib but that I haven't taken on. In particular, pairs and strings now have a notion of mutability that doesn't currently exist (I don't think) in the core; a lot of the core functions (i.e. ones that I've merely repackaged for R6RS-lib) now have more specific error behaviors that I've decided to punt on for the moment. - Do you plan to work on the remaining parts, in particular the dreaded (to me) ‘(rnrs io ports)’? :-) I uh... wasn't planning to, but that's just because I figured they were already done. :-) I dread them a little bit as well, since they seem more likely to require changes to the core. - Does the code borrow snippets from reference implementations or some such? Yes -- in cases in which R6RS-libs makes a suggestion for the implementation of a particular procedure or syntactic form, I've taken it. The rest (I think) of the code is by me, for better or for worse. - How much code do unit tests cover? It’d be nice to try the test suite of the PLT folks, http://groups.google.com/group/comp.lang.scheme/browse_thread/thread/a7d691b5ca87b94f/6bb6e09f1d39d828. The tests cover the cases where the complexity or newness of the code seemed to warrant it. For example, I don't have any unit tests for libraries that are simply repackagings of existing Guile functionality. Let's definitely run against PLT's suite. - Are there any known problems with the library/module integration, or pitfalls that ought to be documented? Hmmm. None that are so serious that they're currently blocking me -- I've been developing and testing the libraries using the library/module integration, so I'm quite sensitive to problems when they arise. :-) At the moment, the only thing that's bugging me is an issue with tests that use syntax exported from libraries fail when run via 'make check' but pass when loaded from the REPL. - In terms of documentation, we definitely don’t want to duplicate R6RS. However, it would be nice to have at least a node listing the available libraries and what they’re about. For those that provide features not available elsewhere in Guile, it’d be nice to have more detailed documentation (we’ve done that for bytevectors and I/O ports). Sure. The records, conditions, exceptions, and enums bear more detailed documentation. And I'd add that the library/module integration itself should probably get a node or two, since aspects of its behavior are non-obvious. Julian: would you like review on specific parts of your work? (I personally don’t have much time to dedicate to it these days but I could look at things you think are important.) Sure! I wouldn't mind a review of what I think are some of the more complex (and central) bits, which, to my mind, are (rnrs records *), (rnrs conditions), and (rnrs exceptions). Is it reasonable to add it to the to-do list of 2.0? I don't know. 2.0's coming out pretty soon, right? The hours at my day job can be sort of unpredictable, and I wouldn't want to hold up the release because I'm strapped for time. Regards, Julian
Re: r6rs libraries, round three
Hello, Neil Jerram n...@ossau.uklinux.net writes: Julian Graham jool...@gmail.com writes: On a related note, I assume Emacs' `scheme-mode' has been adding unwanted tabs? What should I do to correct this in my local environment? (setq indent-tabs-mode nil) We should put that in ‘.dir-locals.el’, once for all. Thanks, Ludo’.
Re: r6rs libraries, round three
Hi all, Find attached a revised and polished version of the `(ice-9 r6rs-libraries)' module I submitted a couple of months ago. This version includes the following changes: * The library transformer code's been cleaned up and compacted (by more than 30%) and now uses a `defmacro' form similar to the one used by `use-modules' and `define-module' instead of the messy syncase-based transformer it was using originally. * I've re-organized the code to more closely resemble the structure of `use-modules' and 'define-module' -- the macros delegate syntax parsing to a set of processing functions. In addition to making the macros simpler, this should make it easier to unify the module and library systems in the future, if desired. * I've added an `import' macro as specified by R6RS 8.1 Top Level Program Syntax [0]. * The module also supports the convention, specified by SRFI-97, that SRFIs can be loaded as R6RS libraries by importing them as `(srfi :[n])' -- my implementation transforms library names of that form to the form used by Guile, `(srfi srfi-[n])'. In case anyone missed the earlier emails on this topic, this module contains macros that transform the R6RS `library' and 'import' forms into Guile's native `define-module' and 'use-modules' forms. In concert with the version and binding export patches that were pushed last week, this means that Guile now supports R6RS 7 Libraries [1], and can thus share code (unmodified!) with any other conforming Scheme implementation. I consider this version of the code to be tentatively complete. You can try it out by dropping r6rs-libraries.scm into module/ice-9 and then loading it in the REPL or including it as a dependency of a normal Guile module. I'm very interested in any feedback people might have, particularly when it comes to the name of the module and where it belongs / when it should be loaded (always? Not in the REPL?). If no one objects, I'll add some documentation and push it. As I mentioned to Andy on IRC, I'm working on a first pass at a set of implementations of the R6RS Standard Libraries (minus the work already done by Ludovic et al on bytevectors, etc.), as much as possible as wrappers around Guile's existing functionality. I should have some status on that soon. Regards, Julian [0] - http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-11.html#node_sec_8.1 [1] - http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-10.html#node_chap_7 ;;; r6rs-libraries.scm --- Support for R6RS `library' and `import' forms ;; Copyright (C) 2009 Free Software Foundation, Inc. ;; ;; This library is free software; you can redistribute it and/or ;; modify it under the terms of the GNU Lesser General Public ;; License as published by the Free Software Foundation; either ;; version 3 of the License, or (at your option) any later version. ;; ;; This library is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU ;; Lesser General Public License for more details. ;; ;; You should have received a copy of the GNU Lesser General Public ;; License along with this library; if not, write to the Free Software ;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA ^L (define-module (ice-9 r6rs-libraries) #:use-module (ice-9 optargs) #:use-module (ice-9 regex) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) #:use-module (srfi srfi-11) #:export-syntax (library import)) (define (name-and-version lst) (let-values (((head tail) (split-at lst (- (length lst) 1 (if (pair? (car tail)) (values head (car tail)) (values lst '() (define srfi-regex (make-regexp ^\\:([0-9]+)$)) (define* (process-import args #:optional import-map) (define (flatten im) (define (load-library library-ref) (define (transform-library-name name) (define (make-srfi m) (cons 'srfi (list (string-symbol (string-append srfi- (match:substring m 1)) (or (and (= (length name) 2) (eq? (car name) 'srfi) (and= (regexp-exec srfi-regex (symbol-string (cadr name))) make-srfi)) name)) (let-values (((name version) (name-and-version library-ref))) (resolve-interface (transform-library-name name) #:version version))) (define (exeq? x y) (if (list? y) (eq? x (cadr y)) (eq? x y))) (if (or (not (list? im))) (error)) (let* ((op (car im)) (l (case op ((only except prefix rename) (flatten (cadr im))) ((library) (load-library (cadr im))) (else (load-library im) (case op ((library) (cons l (module-map (lambda (sym var) sym) l))) ((only) (cons (car l) (lset-intersection exeq? (cdr l) (cddr im ((except) (cons (car l) (lset-difference exeq? (cdr l) (cddr im ((prefix) (let ((p (symbol-prefix-proc (caddr im (cons (car l) (map (lambda (x) (if (list? x) (cons (car x) (p (cadr x))) (cons x (p x (cdr l
Re: r6rs libraries, round three
On Sun 13 Dec 2009 04:24, Julian Graham jool...@gmail.com writes: Find attached updated versions of the patches that provide support for R6RS-compatible versions and renaming bindings on export (the two core modifications required to support the libraries implementation). They've been rebased against the current HEAD and some reasonably comprehensive documentation has been added. Thanks for all. I wrapped your commit messages, detabbed the files, and did some minor editing. Please look at the diffs relative to your patches. But other than that, thanks very much! :-)) Andy -- http://wingolog.org/
Re: r6rs libraries, round three
Julian Graham jool...@gmail.com writes: On a related note, I assume Emacs' `scheme-mode' has been adding unwanted tabs? What should I do to correct this in my local environment? (setq indent-tabs-mode nil) Neil
Re: r6rs libraries, round three
Hi all,
Find attached updated versions of the patches that provide support for
R6RS-compatible versions and renaming bindings on export (the two core
modifications required to support the libraries implementation).
They've been rebased against the current HEAD and some reasonably
comprehensive documentation has been added.
Questions? Comments?
Regards,
Julian
From bb83bbd13263aca6a1e8b246fd68ce96f5dcdb43 Mon Sep 17 00:00:00 2001
From: Julian Graham julian.gra...@aya.yale.edu
Date: Thu, 10 Dec 2009 00:29:11 -0500
Subject: [PATCH 1/2] Support for renaming bindings on module export.
* module/ice-9/boot-9.scm (module-export!, module-replace!, module-re-export!):
Allow members of export list to be pairs, mapping internal names to external ones.
* doc/ref/api-modules.texi (Creating Guile Modules): Update documentation for `#:export', `#:export-syntax', `#:replace', `#:re-export', `#:re-export-syntax', `export', and `re-export' to reflect new format for arguments.
---
doc/ref/api-modules.texi | 50 +
module/ice-9/boot-9.scm | 24 +
2 files changed, 43 insertions(+), 31 deletions(-)
diff --git a/doc/ref/api-modules.texi b/doc/ref/api-modules.texi
index 1c9ab23..65a3564 100644
--- a/doc/ref/api-modules.texi
+++ b/doc/ref/api-modules.texi
@@ -421,40 +421,42 @@ the module is used.
@item #:export @var{list}
@cindex export
-Export all identifiers in @var{list} which must be a list of symbols.
-This is equivalent to @code{(export @var{list})} in the module body.
+Export all identifiers in @var{list} which must be a list of symbols
+or pairs of symbols. This is equivalent to @code{(export @var{list})}
+in the module body.
@item #:re-export @var{list}
@cindex re-export
Re-export all identifiers in @var{list} which must be a list of
-symbols. The symbols in @var{list} must be imported by the current
-module from other modules. This is equivalent to @code{re-export}
-below.
+symbols or pairs of symbols. The symbols in @var{list} must be
+imported by the current module from other modules. This is equivalent
+to @code{re-export} below.
@item #:export-syntax @var{list}
@cindex export-syntax
-Export all identifiers in @var{list} which must be a list of symbols.
-The identifiers in @var{list} must refer to macros (@pxref{Macros})
-defined in the current module. This is equivalent to
-...@code{(export-syntax @var{list})} in the module body.
+Export all identifiers in @var{list} which must be a list of symbols
+or pairs of symbols. The identifiers in @var{list} must refer to
+macros (@pxref{Macros}) defined in the current module. This is
+equivalent to @code{(export-syntax @var{list})} in the module body.
@item #:re-export-syntax @var{list}
@cindex re-export-syntax
Re-export all identifiers in @var{list} which must be a list of
-symbols. The symbols in @var{list} must refer to macros imported by
-the current module from other modules. This is equivalent to
-...@code{(re-export-syntax @var{list})} in the module body.
+symbols or pairs of symbols. The symbols in @var{list} must refer to
+macros imported by the current module from other modules. This is
+equivalent to @code{(re-export-syntax @var{list})} in the module body.
@item #:replace @var{list}
@cindex replace
@cindex replacing binding
@cindex overriding binding
@cindex duplicate binding
-Export all identifiers in @var{list} (a list of symbols) and mark them
-as @dfn{replacing bindings}. In the module user's name space, this
-will have the effect of replacing any binding with the same name that
-is not also ``replacing''. Normally a replacement results in an
-``override'' warning message, @code{#:replace} avoids that.
+Export all identifiers in @var{list} (a list of symbols or pairs of
+symbols) and mark them as @dfn{replacing bindings}. In the module
+user's name space, this will have the effect of replacing any binding
+with the same name that is not also ``replacing''. Normally a
+replacement results in an ``override'' warning message,
+...@code{#:replace} avoids that.
This is useful for modules that export bindings that have the same
name as core bindings. @code{#:replace}, in a sense, lets Guile know
@@ -562,8 +564,11 @@ do not know anything about dangerous procedures.
@c end
@deffn syntax export variable @dots{}
-Add all @var{variable}s (which must be symbols) to the list of exported
-bindings of the current module.
+Add all @var{variable}s (which must be symbols or pairs of symbols) to
+the list of exported bindings of the current module. If @var{variable}
+is a pair, its @code{car} gives the name of the variable as seen by the
+current module and its @code{cdr} specifies a name for the binding in
+the current module's public interface.
@end deffn
@c begin (scm-doc-string boot-9.scm define-public)
@@ -573,9 +578,10 @@ Equivalent to @code{(begin (define foo ...) (export foo))}.
@c end
@deffn syntax re-export variable @dots{}
Re: r6rs libraries, round three
Hi Andy, Note that quasisyntax is now merged. You can do things without quasisyntax using with-syntax. Of course -- our version of quasisyntax is implemented in terms of with-syntax! I was just being lazy. Your code is remarkably short. That is my initial impression, positive :-) But I need to get to writing the NEWS now for today's release. I'll take a look at these within the next week hopefully. Please poke if you don't get another response in the next week. I probably should have said rough prototype instead of working prototype -- the actual macro that transforms library definitions into module definitions is kind of gross and uses datum-syntax a fair amount where it probably doesn't need to / shouldn't. I'm no syncase wizard. But I'm pretty sure it works for conventional libraries that import and export macros and regular bindings. (What I worry about are some of the hairier use cases of the whole phased import mechanism -- like a binding that's imported at `meta' level 2 or higher sharing a name with definition imported for use at runtime.) What I'm mostly interested in is whether you guys think the version and export patches are worth merging in some form or another -- my assumption has been these are features we actually want for Guile's module system. Thanks, Julian
Re: r6rs libraries, round three
Andy Wingo wi...@pobox.com writes: Hi Julian! On Sun 01 Nov 2009 20:26, Julian Graham jool...@gmail.com writes: Find attached a working prototype of R6RS library support I think I missed this one, it was threaded above the end of guile-devel that I read :-) Sorry about that. Note that quasisyntax is now merged. You can do things without quasisyntax using with-syntax. I haven't actually had the pleasure yet of using quasisyntax :P Speaking of psyntax: have you had a look at my tail patterns patch[0] yet? If it is deemed, ok, I can add ChangeLog and NEWS entries, and update the documentation (anything else?). [0] http://article.gmane.org/gmane.lisp.guile.devel/9605/match=tail+pattern Regards, Rotty -- Andreas Rottmann -- http://rotty.yi.org/
Re: r6rs libraries, round three
Julian Graham jool...@gmail.com writes: I probably should have said rough prototype instead of working prototype -- the actual macro that transforms library definitions into module definitions is kind of gross and uses datum-syntax a fair amount where it probably doesn't need to / shouldn't. I'm no syncase wizard. But I'm pretty sure it works for conventional libraries that import and export macros and regular bindings. (What I worry about are some of the hairier use cases of the whole phased import mechanism -- like a binding that's imported at `meta' level 2 or higher sharing a name with definition imported for use at runtime.) IIRC, R6RS doesn't /require/ that implementations are able to differentiate bindings from different phases -- e.g. Ikarus essentially ignores phase specifications (implicit phasing -- there were some discussions about that on ikarus-users, which I can't find ATM, but [0] should sum the issue up nicely). [0] http://www.phyast.pitt.edu/~micheles/scheme/scheme21.html What I'm mostly interested in is whether you guys think the version and export patches are worth merging in some form or another -- my assumption has been these are features we actually want for Guile's module system. Are you aware of SRFI-103? It got recently revised to leave out versions; not supporting them is an option, I guess. Quoting from R6RS: , | When more than one library is identified by a library reference, the | choice of libraries is determined in some implementation-dependent | manner. | | To avoid problems such as incompatible types and replicated state, | implementations should prohibit the two libraries whose library names | consist of the same sequence of identifiers but whose versions do not | match to co-exist in the same program. ` This makes me wonder if versions can be used (or rather be relied on) sensibly in portable libraries at all... Regards, Rotty -- Andreas Rottmann -- http://rotty.yi.org/
Re: r6rs libraries, round three
Hi Andreas, IIRC, R6RS doesn't /require/ that implementations are able to differentiate bindings from different phases -- e.g. Ikarus essentially ignores phase specifications (implicit phasing -- there were some discussions about that on ikarus-users, which I can't find ATM, but [0] should sum the issue up nicely). You're right, it doesn't -- at least, it's not required that an implementation prevent you from referencing an identifier at a phase other than the one it was imported for. I was reading that part of the spec in terms of non-macro definitions, but, come to think of it, it's got to apply to macros as well. So importing everything at once sounds like it'll work just fine. Are you aware of SRFI-103? It got recently revised to leave out versions; not supporting them is an option, I guess. Quoting from R6RS: I was tracking SRFI-103 for a while back when it was (I think) SRFI-100. I'm interested to see how it pans out, but I'm not sure I agree with its rationale -- it seems mostly useful for implementations that don't currently have their own library search mechanism. The bit about distributing and using library files in a portable way seems a bit hand-wavy to me. This makes me wonder if versions can be used (or rather be relied on) sensibly in portable libraries at all... Yes, it's a bit thorny. We discussed the limitations in a thread [1] a while back. The implementation I did reflects the outcome of that thread, which was that the version of a library that gets loaded is a function of the import statements, the available libraries, and the set of already-loaded libraries -- which means that it's not a fully predictable process from the point of view of library authors, but that in practice, collisions aren't likely for a variety of reasons. Regards, Julian [1] - http://www.mail-archive.com/guile-devel@gnu.org/msg03673.html
Re: r6rs libraries, round three
As I said earlier, I'm happy to provide full documentation for all of this code if the consensus is that I'm on the right track. Any feeling either way on those patches? I'm happy to create a remote tracking branch if that'd make it easier for people to review.
Re: r6rs libraries, round three
Hi all, Find attached a working prototype of R6RS library support, in the form of a Guile module called `(r6rs-libraries)'. The module depends on the two attached patches, which add, respectively, support for the `#:version' keyword [1] and support for renaming bindings on export [2]. It works by transforming the R6RS `library' form into Guile's native `define-module' form. Because it's implemented as a macro, it's only required at expansion time -- the resulting compiled module has no dependencies on anything besides other Guile modules. Andreas Rottmann's quasisyntax implementation is included as part of `(r6rs-libraries)' since it's not yet in master and I was finding it difficult to model some things without `unsyntax-splicing'. Also attached are a minimal set of R6RS libraries (as `r6rs-libs.tar.gz') needed to bootstrap the examples from chapter 7 of the R6RS spec (attached as `r6rs-examples.tar.gz'). If you place the r6rs-libraries.scm and the contents of these tarballs somwhere in your `%load-path', you can run the balloon party example as follows: scheme@(guile-user) (use-modules (r6rs-libraries)) scheme@(guile-user) (use-modules (main)) Boom 108 Boom 24 ...and the let-div example as follows: scheme@(guile-user) (use-modules (r6rs-libraries)) scheme@(guile-user) (use-modules (let-div)) scheme@(guile-user) (let-div 5 2 (q r) (display q: ) (display q) (display r: ) (display r) (newline)) q: 2 r: 1 There are certainly some aspects of this implementation that require review -- in particular, I've added infrastructure to distinguish between imports targeted for different phases (i.e., `run', `expand' ... (meta n)), but at the moment, all imports are currently included via #:use-module, which means they're visible at every point from expansion to runtime. R6RS seems to explicitly allow this, though, and, quite frankly, it's much easier to implement. As I said earlier, I'm happy to provide full documentation for all of this code if the consensus is that I'm on the right track. Regards, Julian [1] - http://www.mail-archive.com/guile-devel@gnu.org/msg04506.html [2] - http://www.mail-archive.com/guile-devel@gnu.org/msg04660.html From adcbc77ca4ca68f26da05a204154d826a832a7b7 Mon Sep 17 00:00:00 2001 From: Julian Graham julian.gra...@aya.yale.edu Date: Sun, 25 Oct 2009 13:17:40 -0400 Subject: [PATCH] Complete support for version information in Guile's `module' form. * module/ice-9/boot-9.scm (try-load-module, try-module-autoload): Check for version argument and use `find-versioned-module' if present. * module/ice-9/boot-9.scm (find-versioned-module, version-matches?, module-version, set-module-version!, version-matches?): New functions. * module/ice-9/boot-9.scm (module-type, make-module, resolve-module, try-load-module, process-define-module, make-autoload-interface, compile-interface-spec): Add awareness and checking of version information. --- module/ice-9/boot-9.scm | 149 ++- 1 files changed, 133 insertions(+), 16 deletions(-) diff --git a/module/ice-9/boot-9.scm b/module/ice-9/boot-9.scm index 5852477..3d92fad 100644 --- a/module/ice-9/boot-9.scm +++ b/module/ice-9/boot-9.scm @@ -1333,7 +1333,7 @@ (make-record-type 'module '(obarray uses binder eval-closure transformer name kind duplicates-handlers import-obarray - observers weak-observers) + observers weak-observers version) %print-module)) ;; make-module opt size uses binder @@ -1374,7 +1374,7 @@ #f #f #f (make-hash-table %default-import-size) '() - (make-weak-key-hash-table 31 + (make-weak-key-hash-table 31) #f))) ;; We can't pass this as an argument to module-constructor, ;; because we need it to close over a pointer to the module @@ -1396,6 +1396,8 @@ (define module-transformer (record-accessor module-type 'transformer)) (define set-module-transformer! (record-modifier module-type 'transformer)) +(define module-version (record-accessor module-type 'version)) +(define set-module-version! (record-modifier module-type 'version)) ;; (define module-name (record-accessor module-type 'name)) wait until mods are booted (define set-module-name! (record-modifier module-type 'name)) (define module-kind (record-accessor module-type 'kind)) @@ -2001,6 +2003,7 @@ (eq? interface module)) (let ((interface (make-module 31))) (set-module-name! interface (module-name module)) + (set-module-version! interface (module-version module)) (set-module-kind! interface 'interface) (set-module-public-interface! module interface (if (and (not (memq the-scm-module (module-uses module))) @@ -2008,6 +2011,101 @@ ;; Import the default set of bindings (from the SCM module) in MODULE. (module-use! module the-scm-module))) +(define (version-matches? version-ref target) + (define (any prec lst) +(and (not (null? lst
Re: r6rs libraries, round three
Hi Andy, It should work now, though with hacks -- if you manipulate the module-public-interface directly. But perhaps some more baked in support would be useful. Oh, certainly -- as I've learned over these many months, you can do some very interesting things by working with the lower-level module API. And indeed, that was how I did things in my initial implementation back in March. But... Can you explain a use case a bit more? I think having trouble grasping why you would want to do this :) I'm trying to write a macro to convert `library' forms into `define-module' forms. All of the contortions you can put your imported symbols through in R6RS can be flattened into a form that maps quite neatly onto define-module's #:select, but #:export and #:reexport aren't as flexible. Specifically, the use case is implementing the (rename (identifier1 identifier2) ...) form for R6RS library export-specs. Like you said, you can manipulate the public interface directly -- I could, say, insert the code to do this as part of transforming the library body -- but it would be nice if I could leave the management of the interface entirely up to `define-module'. Regards, Julian
Re: r6rs libraries, round three
Hi all, Besides version, another thing that would be very useful to have native Guile support for is being able to export bindings with names other than the ones given to them within the module -- that is, to be able to rename variables exported as part of the module's public interface in `define-module', similar to what the `#:select' keyword allows you to do for imported bindings. In fact, it could even work the same the way: a given element in the list passed with the `#:export' keyword could be either a symbol or a pair in which the car is the module-local name and the cdr is the name to use in the module's public interface. What do people think? Regards, Julian
Re: r6rs libraries, round three
Hi Guilers, Okay, after poking around in the manual and the code, it looks like `load-module' does what I need. Find attached two patches that, combined, add full support for R6RS-style version information to Guile's module system. I've done a bit of testing and believe that this code does the Right Thing in a variety of situations -- e.g., it always attempts to select the highest version number but can recover from situations in which paths corresponding to higher-numbered versions don't contain actual module implementations. Questions, comments? If it seems like this code is on the right track, I'll add documentation to the appropriate locations. boot-9.scm is getting a little bit crowded, though -- I don't suppose it makes sense to move some of the module handling code to an auxiliary file? Regards, Julian From a1d49c00cd6cc144bf526481e5ba7da6aefa0822 Mon Sep 17 00:00:00 2001 From: Julian Graham julian.gra...@aya.yale.edu Date: Sat, 26 Sep 2009 14:52:56 -0400 Subject: [PATCH] Initial support for version information in Guile's `module' form. * module/ice-9/boot-9.scm (module-version, set-module-version!, version-matches?): New functions. * module/ice-9/boot-9.scm (module-type, make-module, resolve-module, try-load-module, process-define-module, make-autoload-interface, compile-interface-spec): Add awareness and checking of version information. --- module/ice-9/boot-9.scm | 42 ++ 1 files changed, 30 insertions(+), 12 deletions(-) diff --git a/module/ice-9/boot-9.scm b/module/ice-9/boot-9.scm index a1537d1..b49f799 100644 --- a/module/ice-9/boot-9.scm +++ b/module/ice-9/boot-9.scm @@ -1332,8 +1332,8 @@ (define module-type (make-record-type 'module '(obarray uses binder eval-closure transformer name kind - duplicates-handlers import-obarray - observers weak-observers) + duplicates-handlers import-obarray observers + weak-observers version) %print-module)) ;; make-module opt size uses binder @@ -1374,13 +1374,12 @@ #f #f #f (make-hash-table %default-import-size) '() - (make-weak-key-hash-table 31 + (make-weak-key-hash-table 31) #f))) ;; We can't pass this as an argument to module-constructor, ;; because we need it to close over a pointer to the module ;; itself. (set-module-eval-closure! module (standard-eval-closure module)) - module (define module-constructor (record-constructor module-type)) @@ -1396,6 +1395,8 @@ (define module-transformer (record-accessor module-type 'transformer)) (define set-module-transformer! (record-modifier module-type 'transformer)) +(define module-version (record-accessor module-type 'version)) +(define set-module-version! (record-modifier module-type 'version)) ;; (define module-name (record-accessor module-type 'name)) wait until mods are booted (define set-module-name! (record-modifier module-type 'name)) (define module-kind (record-accessor module-type 'kind)) @@ -2008,24 +2009,32 @@ ;; Import the default set of bindings (from the SCM module) in MODULE. (module-use! module the-scm-module))) +;; Temporary kludge before implementing full version matching. +(define version-matches? equal?) + ;; NOTE: This binding is used in libguile/modules.c. ;; (define resolve-module (let ((the-root-module the-root-module)) -(lambda (name . maybe-autoload) +(lambda (name . args) (if (equal? name '(guile)) the-root-module (let ((full-name (append '(%app modules) name))) -(let ((already (nested-ref the-root-module full-name)) - (autoload (or (null? maybe-autoload) (car maybe-autoload +(let* ((already (nested-ref the-root-module full-name)) + (numargs (length args)) + (autoload (or (= numargs 0) (car args))) + (version (and ( numargs 1) (cadr args (cond ((and already (module? already) (or (not autoload) (module-public-interface already))) ;; A hit, a palpable hit. + (and version + (not (version-matches? version (module-version already))) + (error incompatible module version already loaded name)) already) (autoload ;; Try to autoload the module, and recurse. -(try-load-module name) +(try-load-module name version) (resolve-module name #f)) (else ;; A module is not bound (but maybe something else is), @@ -2071,7 +2080,7 @@ ;; (define-special-value '(%app modules new-ws) (lambda () (make-scm-module))) -(define (try-load-module name) +(define (try-load-module name version) (try-module-autoload name)) (define (purify-module! module) @@ -2132,7 +2141,8 @@ (let ((prefix (get-keyword-arg args #:prefix #f))) (and prefix
r6rs libraries, round three
Hi Guilers, Having been motivated by an extended discussion with Andy over pints in Brooklyn last weekend, I've resolved to return to the issue of R6RS library support once more. As discussed the last time we took this on, I think the first step is getting support for version information into the modules system. Find attached a patch that adds trivial support for versions to boot-9.scm. Applying this patch gives you the ability to specify an R6RS-compatible (i.e., `(x y z...)' where x, y, and z are whole numbers) version, via a #:version keyword argument, for both the `define-module' and `use-modules' forms. Specifying a version in your `use-modules' that doesn't match the version of an already-loaded module with that name will raise an error. This patch is trivial because version matching at the moment is done via `equal?' and thus doesn't support the full range of matching behavior outlined by R6RS. More importantly, though, it's missing support for matching versions on disk. I got most of the way through an initial implementation of that when I found myself in some trickiness. To recap, what I think we decided about storing versioned modules was: Module version numbers can be represented in a directory hierarchy: [dir-hint] / x / y / z / module.scm, e.g. ice-9/0/1/2/readline.scm. This is approach has the benefit that it can co-exist with the traditional directory structure for Guile modules, since numbers can't be confused with module name components [1]. But this also means that the path searching performed by `primitive-load-path' won't work for locating these modules, and Andy's symlink solution [2] doesn't really help that much, since R6RS version matching sometimes requires that we select a number based on a set of constraints, not just a straight-up wildcard. The solution I'm working on does the following: 1. Combine every entry in `%load-path' with the dir hint to produce a list of root paths to search. 2. For every component of the version reference, for every root path, find all subdirectories with names that match the reference. 3. Sort the results in numerically descending order; these are the new root paths. 4. Loop back to step 2 until all components of the version reference have been matched and a module file has been found. The problem I ran into is that once I've finished this procedure, I've got an absolute path to the module, and I want to load it by performing the same autocompilation heuristics that `primitive-load-path' provides -- but that function only works on relative paths. How come this magic hasn't been added to `primitive-load' (or some other function that operates on absolute paths)? Regards, Julian [1] - http://www.mail-archive.com/guile-devel@gnu.org/msg03259.html [2] - http://article.gmane.org/gmane.lisp.guile.devel/8585 From a1d49c00cd6cc144bf526481e5ba7da6aefa0822 Mon Sep 17 00:00:00 2001 From: Julian Graham julian.gra...@aya.yale.edu Date: Sat, 26 Sep 2009 14:52:56 -0400 Subject: [PATCH] Initial support for version information in Guile's `module' form. * module/ice-9/boot-9.scm (module-version, set-module-version!, version-matches?): New functions. * module/ice-9/boot-9.scm (module-type, make-module, resolve-module, try-load-module, process-define-module, make-autoload-interface, compile-interface-spec): Add awareness and checking of version information. --- module/ice-9/boot-9.scm | 42 ++ 1 files changed, 30 insertions(+), 12 deletions(-) diff --git a/module/ice-9/boot-9.scm b/module/ice-9/boot-9.scm index a1537d1..b49f799 100644 --- a/module/ice-9/boot-9.scm +++ b/module/ice-9/boot-9.scm @@ -1332,8 +1332,8 @@ (define module-type (make-record-type 'module '(obarray uses binder eval-closure transformer name kind - duplicates-handlers import-obarray - observers weak-observers) + duplicates-handlers import-obarray observers + weak-observers version) %print-module)) ;; make-module opt size uses binder @@ -1374,13 +1374,12 @@ #f #f #f (make-hash-table %default-import-size) '() - (make-weak-key-hash-table 31 + (make-weak-key-hash-table 31) #f))) ;; We can't pass this as an argument to module-constructor, ;; because we need it to close over a pointer to the module ;; itself. (set-module-eval-closure! module (standard-eval-closure module)) - module (define module-constructor (record-constructor module-type)) @@ -1396,6 +1395,8 @@ (define module-transformer (record-accessor module-type 'transformer)) (define set-module-transformer! (record-modifier module-type 'transformer)) +(define module-version (record-accessor module-type 'version)) +(define set-module-version! (record-modifier module-type 'version)) ;; (define module-name (record-accessor module-type 'name)) wait until mods are booted (define set-module-name! (record-modifier module-type 'name))
Re: r6rs libraries, round two
Hi Andy, Back to your question though, what did you think about my symlink solution[1]? [1] http://article.gmane.org/gmane.lisp.guile.devel/8585 It's fine, although any length suffix of a version specifier can be omitted in the case of a match -- that is, you can do without the version altogether, or you can just partially specify a version; in both cases, the system needs to find a matching version for you. Would the symlinking strategy you describe happen at every node in the directory tree of versions for a particular library? E.g., let's say you've got version 1.2.3 of R6RS library `(foo bar)' installed. Does the installation directory look like: foo/1/2/3/bar.scm foo/1/2/bar.default - foo/1/2/3/bar.scm foo/1/bar.default - foo/1/2/3/bar.scm foo/bar.default - foo/1/2/3/bar.scm Regards, Julian
Re: r6rs libraries, round two
Hi Andy, Your solution of doing whole-program analysis is very much in the spirit of R6RS, but it is not in the spirit of Lisp, in my opinion at least. Well, to be fair, it's not whole-program analysis -- as Neil pointed out, we only need to analyze the library and program headers. But, sure, point taken. I actually think that doing a whole-program analysis is actively harmful to the future of Scheme. That's a bit hyperbolic, but it is my opinion. To me, Lisp is about living, adaptable systems -- not about static programs. There's no need to let mistakes on the part of the R6RS editors take us farther down the static path. Fortunately, we can comply to the letter of the standard, and not the spirit. Back to your question though, what did you think about my symlink solution[1]? It's a fine way of handling version-less dependencies, but I don't see how it solves the determinism issue. Let's say my program (or code stream or script or whatever you want to call it) uses libraries from two different authors. Author A trusts his dependencies to remain stable and doesn't use version specifiers in his library references; author B does use them. If my code imports author A's library before author B's, and there's a version clash in their dependencies, what's the take-away for me, the user? Does this mean my code is not well-formed until I reverse the order of my imports? Should I not depend on author A's code at all? On author B's code? Do I have to patch my libraries before importing them? I'm approaching this project in my capacity as a developer with a fair number of dependencies (that I want manage as abstractly as I can) wanting to distribute and test my code on as many platforms as possible. From my point of view, the fact that there hasn't been a unified module format for Scheme is pretty frustrating. To the extent that R6RS has some traction among Scheme implementations, its library spec is attractive -- although, I recognize, for political more than technical reasons. At the same time, after months of poring over the minutiae of R6RS, I've come around to the position that it's got some serious defects. Is there some middle ground possible? The intentions of the R6RS editors with regard to libraries and programs being static seems pretty clear. And for certain use cases, that may be appropriate. But that doesn't mean that Guile modules need to behave that way. Would people be amenable to having two different ways of working with libraries / programs? I.e., the mode that Guile would be in when you ran an R6RS program would be different than when you were in the REPL. Maybe it'd even be a different language in the VM. And we could establish some prescribed mechanisms by which non-R6RS Guile code could use R6RS libraries (with caveats attached), and just not support the converse. I don't know -- this problem seems to get thornier the more we discuss it, but I still think it's worth solving. Sorry if I've missed the point on any of your comments. Regards, Julian
Re: r6rs libraries, round two
Hello! Julian Graham jool...@gmail.com writes: It's a fine way of handling version-less dependencies, but I don't see how it solves the determinism issue. Let's say my program (or code stream or script or whatever you want to call it) uses libraries from two different authors. Author A trusts his dependencies to remain stable and doesn't use version specifiers in his library references; author B does use them. IMO this example is not realistic. Libraries with a well defined API stability policy provide ad hoc versioning mechanisms at the source level, e.g., (use-modules (gnome-2)). Likewise, applications distributed as a source package and that aim to build on a wide range of platforms are likely to explicitly check for the version of their dependencies. By saying Guile should comply to the letter of the standard, I think Andy meant to say it should do the minimum required to be standard-compliant. Thanks, Ludo'.
Re: r6rs libraries, round two
Julian Graham jool...@gmail.com writes: Hi Guilers, With the 1.9 series launched, I wanted to start thinking about R6RS libraries again, since it would be awesome to have some semblance of an implementation ready by October. Indeed, yes. I assume the objective here is to allow a Guile program or module to use a portable R6RS library; i.e., specifically, allowing `(use-modules ...)' or `#:use-module (...)' to resolve to an R6RS library. Is that correct? (In addition, I guess we could support `import' directly in contexts other than R6RS library code, i.e. in a top level program or Guile module. But it's not clear to me if that would provide any extra benefit.) I think such issues should lead us to have a `:version' option that does just what's needed for R6RS, and does not try to do anything smart (which could seem to be non-deterministic). We could even go as far as discouraging its use in the manual. (I'm assuming this means a :version option to use-modules or #:use-module.) Alright -- so, to summarize what (I think) has been decided: 1. Having more than one version of a module loaded at a time in a single Guile process / VM is not going to be supported. Yes, at least as far as R6RS library modules are concerned. I've reviewed the relevant discussions on this now and am happy that there's no desire to support multiple live versions of R6RS libraries. I'd prefer not to rule this out, though, for any future versioning that we might add to (define-module ...). Is that feasible? 2. Mixing load calls that specify version with load calls that do not should produce deterministic behavior independent of the set of already-loaded modules. (As described here: [1].) I think that's only possible if we impose some restrictions on how R6RS libraries can be imported... R6RS says that the mechanism by which implementations match version-less library references is implementation-dependent, but, in order to avoid breaking rule number two, I think there's really only one way to do it: The expander has to recursively examine all of the imports, starting with the top-level program, and keep track of the version specifiers it finds. That sounds like it requires us to `read' all of the program code, and all imported modules and libraries, before deciding which versions to use. Possible problems with that would be (i) reader macros that have non-trivial side effects; (ii) different possible versions of a library that themselves import different versions of another library, and so on. But on the other hand: - Once we're into R6RS library code, we're OK, because the R6RS layout requires all of the imports to be declared upfront; so we don't need to read the rest of the library code. - In a Guile module, we could specify that versioned imports can only be done by #:use-module expressions as part of the (define-module ...) form, and not support versioned imports by (use-modules ...). Then we'd only have to read the (define-module ...) form. - That just leaves the main program. For that we could invent a new form, like define-module but for the main program, and say that it has to cover all versioned imports. Hmm.. it seems this boils down to saying that we would partially deprecate `(use-modules ...)'. In order for a program to be consistent with regard to version, all of the version specifiers for a particular library name must be prefixes of the most fully-specified version of that library. If this is true, then once the most fully-specified version is resolved, wildcard version specifiers can be resolved to that version. Unfortunately, depending on the order in which imports are processed, this mechanism might need to examine the headers of several versions of a particular library. So it's the most correct, but it may be non-optimal. A lazier but potentially less load-thrashy alternative would be to have wildcard versions match the latest available version of a library OR an already-loaded version of that library if present. I can imagine a less clever approach, in which each import is considered as we come to it, and - if there's already a live module with that name - if it's version is compatible with the new import, it's used - if it's version isn't compatible, error - else load a live module as indicated by the new import (somehow - e.g. Andy's symlink approach). Now obviously this could render some programs unstartable. But that could be OK, so long as there was an easy fix for the program author, such as adding (use-modules ((the troublesome module) #:version (the right version))) near the top of the main program. It would be even more OK if we could provide a tool to help work out (the right version). But even with that tool, this less clever approach is a lot less nice than something that Just Works - which is what your suggestion above is. So I'd prefer your suggestion if we can make it work. Regards, Neil
Re: r6rs libraries, round two
Hi Neil, I assume the objective here is to allow a Guile program or module to use a portable R6RS library; i.e., specifically, allowing `(use-modules ...)' or `#:use-module (...)' to resolve to an R6RS library. Is that correct? Actually, my immediate-term goal was to add versioning info to Guile's native modules in an R6RS-compatible way. Once that's done, the level of compatibility you're describing would be feasible, although I think it'd be up for discussion as to whether it's something we want. I kind of assumed that in the first iteration there'd be an explicit bridge into R6RS library territory, e.g., `(rnrs-import ...)' or something like that. Yes, at least as far as R6RS library modules are concerned. I've reviewed the relevant discussions on this now and am happy that there's no desire to support multiple live versions of R6RS libraries. I'd prefer not to rule this out, though, for any future versioning that we might add to (define-module ...). Is that feasible? Not sure I understand -- the impression I got from earlier messages in this thread was that we would be implementing R6RS libraries on top of Guile modules, after first extending the module form to include version metadata. - Once we're into R6RS library code, we're OK, because the R6RS layout requires all of the imports to be declared upfront; so we don't need to read the rest of the library code. Yes -- from what I can tell, R6RS effectively prohibits dynamic access to the import system. That is, an import call always results directly from a previous import, never from the evaluation of an expression. (This strikes me as a little weird, since it means that building an application with a plugin architecture would require you to roll your own import system, but I suppose that's not too far afield from what languages like C# make you do in that regard.) - In a Guile module, we could specify that versioned imports can only be done by #:use-module expressions as part of the (define-module ...) form, and not support versioned imports by (use-modules ...). Then we'd only have to read the (define-module ...) form. So what would the semantics of the version-less (use-modules ...) be in the context of versioned modules already loaded via #:use-module expressions? To minimize version-clash, I'm guessing we'd defer to already-loaded modules when possible -- although this wouldn't address the case in which a module loaded at runtime via (use-modules ...) introduces a #:use-module dependency that conflicts with an already-loaded module. Hmm.. it seems this boils down to saying that we would partially deprecate `(use-modules ...)'. Or at least point out that its use prevents Guile from being able to guarantee conflict-less execution. I can imagine a less clever approach, in which each import is considered as we come to it, and [snip] I think this is what I was trying to suggest, stated more clearly. And unless I'm misunderstanding, I think this makes sense as behavior for (use-modules ...), with the caveats mentioned above. Regards, Julian
Re: r6rs libraries, round two
Hi Julian, On Sun 28 Jun 2009 02:20, Julian Graham jool...@gmail.com writes: With the 1.9 series launched, I wanted to start thinking about R6RS libraries again, since it would be awesome to have some semblance of an implementation ready by October. Yes! I think such issues should lead us to have a `:version' option that does just what's needed for R6RS, and does not try to do anything smart (which could seem to be non-deterministic). We could even go as far as discouraging its use in the manual. Alright -- so, to summarize what (I think) has been decided: 1. Having more than one version of a module loaded at a time in a single Guile process / VM is not going to be supported. I agree. 2. Mixing load calls that specify version with load calls that do not should produce deterministic behavior independent of the set of already-loaded modules. (As described here: [1].) I disagree. I don't think that you can do (2) without (1). Your solution of doing whole-program analysis is very much in the spirit of R6RS, but it is not in the spirit of Lisp, in my opinion at least. Larry Wall was right. Lisp is like oatmeal: of a uniform, mushy consistency. A Lisp system is composed of a procedures and data, permeating each other. You can add (or remove!) procedures and data, and it's still basically the same system. Of course we have modules to keep things manageable, but Guile's take is that modules are mutable entities. You can enter modules and make new definitions, or even remove old ones. Indeed you may have a long-running process that you hack for months, mutating the system over time in response to needs that crop up over time. Of course, with so much mutation -- definition is mutation, after all, in a live system -- you just have to have good practices to keep things under control. Any program that imports unqualified module names (without versions) is inherently not future-portable anyway, for the reasons that Ludovic and others discussed on r6rs-discuss[0]. So /we don't even need to worry about it/. We have to let good practices take care of us here. [0] http://lists.r6rs.org/pipermail/r6rs-discuss/2007-May/002332.html. The only fully specified program is one that has all of the versions declared, but even that program may not have deterministic results -- if an incompatible library is already loaded in the Guile session, the r6rs program or library will fail to load. I actually think that doing a whole-program analysis is actively harmful to the future of Scheme. That's a bit hyperbolic, but it is my opinion. To me, Lisp is about living, adaptable systems -- not about static programs. There's no need to let mistakes on the part of the R6RS editors take us farther down the static path. Fortunately, we can comply to the letter of the standard, and not the spirit. Back to your question though, what did you think about my symlink solution[1]? [1] http://article.gmane.org/gmane.lisp.guile.devel/8585 Cheers, Andy -- http://wingolog.org/
Re: r6rs libraries, round two
Hi Guilers, With the 1.9 series launched, I wanted to start thinking about R6RS libraries again, since it would be awesome to have some semblance of an implementation ready by October. I think such issues should lead us to have a `:version' option that does just what's needed for R6RS, and does not try to do anything smart (which could seem to be non-deterministic). We could even go as far as discouraging its use in the manual. Alright -- so, to summarize what (I think) has been decided: 1. Having more than one version of a module loaded at a time in a single Guile process / VM is not going to be supported. 2. Mixing load calls that specify version with load calls that do not should produce deterministic behavior independent of the set of already-loaded modules. (As described here: [1].) R6RS says that the mechanism by which implementations match version-less library references is implementation-dependent, but, in order to avoid breaking rule number two, I think there's really only one way to do it: The expander has to recursively examine all of the imports, starting with the top-level program, and keep track of the version specifiers it finds. In order for a program to be consistent with regard to version, all of the version specifiers for a particular library name must be prefixes of the most fully-specified version of that library. If this is true, then once the most fully-specified version is resolved, wildcard version specifiers can be resolved to that version. Unfortunately, depending on the order in which imports are processed, this mechanism might need to examine the headers of several versions of a particular library. So it's the most correct, but it may be non-optimal. A lazier but potentially less load-thrashy alternative would be to have wildcard versions match the latest available version of a library OR an already-loaded version of that library if present. What do people think? Regards, Julian [1] - http://www.mail-archive.com/guile-devel@gnu.org/msg03648.html
Re: r6rs libraries, round two
l...@gnu.org (Ludovic Courtès) writes: This is IMO a terrific part of version handling in R6 modules. What will it mean for a 2009 program to import `(rnrs base)' when R27RS is released? By `terrific' do you mean good or bad? Normally terrific means good, and `terrible' means bad, and I suspect you meant `terrible'. Neil
Re: r6rs libraries, round two
Hey Julian, On Sun 31 May 2009 01:22, Neil Jerram n...@ossau.uklinux.net writes: Julian Graham jool...@gmail.com writes: 1. Add an optional `version' field to the module record type Sounds good. Agreed. * What's a good format here? We could mirror the requirements of R6RS here (i.e., (v1 v2 ...) where vx is a whole number) or be more flexible. Given that your objective is to get R6RS library support in, I'd say just stick to the R6RS format for now. It sounds like it will be fairly easy to extend this in future, if we want to. Also agreed. * Should we establish some rules for what you get when you don't specify a version? Yes! The latest available? I don't know. To me this could be a distro decision, like Debian's alternatives system. It would be nice to minimize the number of `stat' calls it takes to load a library -- which would fall out nicely if when asking for a module without specifying a version, like `(foo bar)', we give foo/bar.scm. In the presence of multiple versions, installation rules could handle making the symlink so there is a default version -- typically the version that was installed most recently. Given what we've decided about loading multiple versions of a library (i.e., you can't) I didn't follow why we decided that, but it feels wrong to me. (It seems to me that Guile should be able to handle loading ((foo) v1) and ((foo) v2) simultaneously as easily as it could handle loading ((foo-v1)) and ((foo-v2)) simultaneously.) I guess I should look up the previous thread, please let me know if you have a convenient reference. I agree it would be nice, but as I said in the thread that Julian referenced, that would take some more thought -- more than the R6RS editors were willing to give the problem. And for us, I suspect we would need some changes to our hierarchical namespaces. We probably shouldn't let this be a sticking point for Guile's R6RS libraries support. Regards, Andy -- http://wingolog.org/
Re: r6rs libraries, round two
Hello! Andy Wingo wi...@pobox.com writes: * Should we establish some rules for what you get when you don't specify a version? Yes! The latest available? I don't know. This is IMO a terrific part of version handling in R6 modules. What will it mean for a 2009 program to import `(rnrs base)' when R27RS is released? This was discussed and questioned back then: http://lists.r6rs.org/pipermail/r6rs-discuss/2007-May/002332.html . I think such issues should lead us to have a `:version' option that does just what's needed for R6RS, and does not try to do anything smart (which could seem to be non-deterministic). We could even go as far as discouraging its use in the manual. Thanks, Ludo'.
Re: r6rs libraries, round two
Hi Neil, I didn't follow why we decided that, but it feels wrong to me. (It seems to me that Guile should be able to handle loading ((foo) v1) and ((foo) v2) simultaneously as easily as it could handle loading ((foo-v1)) and ((foo-v2)) simultaneously.) I guess I should look up the previous thread, please let me know if you have a convenient reference. Here's a link [1]. It's possible I misinterpreted Andy's comment and Ludo's follow-up. Regards, Julian [1] - http://www.mail-archive.com/guile-devel@gnu.org/msg03541.html
r6rs libraries, round two
Hi Guilers, I'd like to take another stab at getting R6RS library support in, this time by extending the capabilities of the module system. Here's what I've got in mind to start with: 1. Add an optional `version' field to the module record type * What's a good format here? We could mirror the requirements of R6RS here (i.e., (v1 v2 ...) where vx is a whole number) or be more flexible. For example, Apache Maven (I've got Java on the brain from being at work) lets you specify the version of your project however you want, but if you follow the convention its docs set out, it can do things choose the latest version or match a version within a range. We could do likewise. 2. Add support for optional version arguments to `use-modules', `resolve-module', etc. * Again, do we want to adhere strictly to R6RS-format version references here or extend their syntax? * Should we establish some rules for what you get when you don't specify a version? Given what we've decided about loading multiple versions of a library (i.e., you can't) and the existing behavior of the module-loading functions (you get an already-loaded module if one's available), conflicts seem possible: E.g., if not specifying a version equates to getting the first module in the search path matching the name, then subsequent calls to those functions that *do* specify a version reference may succeed or fail based on the result of a prior call. Regards, Julian
Re: r6rs libraries
Hmm, yes, but how about `foo/bar/baz-6.scm'? Is there a reason to reject it? Well, the part of an R6RS library name that comes before its version is only restricted in that that it must be an identifier -- so a system that relied on filenames to locate libraries could have trouble determining whether a file called ice-9-9.scm contained a library named `(ice-9 (9))' or one named `(ice (9 9))'. And this also doesn't address the problem of doing version matching as part of hybrid module / library load process. After all, I don't think there are any restrictions in R6RS on where libraries need to be stored or how they can be located / loaded, so one thing we could do is ditch the idea that version needs to go into the filename. Guile could require that all installed versions of a particular library would have to reside in the same file, the name of which would be constructed of the non-version parts of the library name: foo/bar/baz.scm would contain the code for: `(foo bar baz (6))' `(foo bar baz (6 1))' and `(foo bar baz (7))' Using the module system to load that file, a la `(use-modules (foo bar baz))' would cause, via a hook in the body of that file, all the library expressions in that file to be stored in a data structure accessible by the library system for the purposes of subsequent import and expansion. Regards, Julian
Re: r6rs libraries
Hi Ludovic, But dot files are traditionally hidden on Unices. Why not go with `foo/bar/6/baz.scm'? Because my initial plan was to make it possible to have R6RS libraries coexist with Guile modules, using the same directory layout and load system -- and, as per our earlier discussion, numbers can't be part of module names. The `r6rs-libraries' part of the module name is just a matter of load path. The important thing here is that both `import' and `use-modules' do the right thing. Then I would not force R6RS module writers to store them under `ice-9/r6rs-libraries' as this is inconvenient and doesn't provide any advantage (AFAICS). Well, it depends on what we plan to support. Here's what I've got implemented so far: `import' (and the `import' specification in the header of a library expression) can locate Guile modules by delegating to Guile's module system. If this fails, the system will attempt to fulfill the import requirement by loading library expressions from disk as per the directory layout I described earlier. What's not clear to me is: * Should `use-modules' be able to locate and load (and evaluate) R6RS library expressions? If so, where should these modules be stores and how should they be named -- specifically with regard to version specifiers? * If both `use-modules' and `import' can load the same R6RS library, the one of them will need to be extended to understand the syntax of the other -- i.e., `use-modules' would need to be able to understand the `library' form or `import' would need to be able to strip off any module wrapper we'd add to a library to make it loadable via `use-modules'. Regards, Julian
Re: r6rs libraries
Hi Rotty, All R6RS implementations I'm somewhat familiar with (Ikarus, PLT, Ypsilon) do some kind of name mangling when library names contain special characters; e.g. (srfi :6 and-let*) maps to srfi/%3a2/and-let%2a.sls, at least for Ikarus (%-escaped hex of the utf-8 encoding, IIRC). Hey, wait, I'm dumb -- why does :6 map to UTF-8 0x3A2? On a sort of related note, though, I was experimenting with acceptable symbol characters / filenames, and .6 seems to work as either. Any reason Guile couldn't map the R6RS library reference (a b c (1 2 3)) to Guile module (a b .1 .2 .3 c)? Regards, Julian Nice to see Guile is on the way to R6RS interoperability! Regards, Rotty
Re: r6rs libraries
Hi Guilers, Just wanted to give an update on the status of the library system I've been working on: * As per the discussion above, my implementation uses an implicit phasing approach (i.e., it doesn't do multiple instantiation) along with `(ice-9 syncase)' to handle expand-time evaluation -- specifically, by way of evaluation environments based on on-the-fly creation of custom module interfaces. I wasn't able to find Abdulaziz's ACM paper, but his formal comment on the subject of R6RS phasing [1] was informative. (In fact, a lot of the formal comments were handy: [2].) I haven't yet delved into how severe the breakage is where there are differences between Guile's `syntax-case' and R6RS's, but I'm hoping it's, you know, mild. * Given the above, my current implementation is hovering at about 200 lines, whereas the current reference implementations, because they don't presume an existing macro or module system, are several orders of magnitude larger. Thanks are due to Tom Lord (and everyone else who designed Guile's module / interface system) for making this possible, if not downright easy. * At the moment, R6RS libraries can import Guile modules directly, but the reverse is not true, partly because I can't figure out a good way to map version information onto the filesystem in a way that Guile's module system can understand. My initial thought was to do it such that a library's version would map to its module name, a la: (library (foo bar baz (6)) would live in /foo/bar/6/baz.scm ...but Guile doesn't like numbers in the module name. Right now I'm trying to think of way to do a transformation on a library expression such that multiple versions of the same library could live in the same Guile module and be accessed individually by some kind of manifest at use time, but I can't figure out what that module name and the extra, associated API should look like. * It would be cool if the module system had hooks of some sort to allow it to receive hints about locating modules. * The party balloons example from the R6Rs is working, at least intermittently. The quotient-and-remainder example is almost there. Regards, Julian [1] - http://www.r6rs.org/formal-comments/comment-123.txt [2] - http://www.r6rs.org/formal-comments/comment-92.txt
Re: r6rs libraries
Julian Graham jool...@gmail.com writes: * At the moment, R6RS libraries can import Guile modules directly, but the reverse is not true, partly because I can't figure out a good way to map version information onto the filesystem in a way that Guile's module system can understand. My initial thought was to do it such that a library's version would map to its module name, a la: (library (foo bar baz (6)) would live in /foo/bar/6/baz.scm ...but Guile doesn't like numbers in the module name. All R6RS implementations I'm somewhat familiar with (Ikarus, PLT, Ypsilon) do some kind of name mangling when library names contain special characters; e.g. (srfi :6 and-let*) maps to srfi/%3a2/and-let%2a.sls, at least for Ikarus (%-escaped hex of the utf-8 encoding, IIRC). Nice to see Guile is on the way to R6RS interoperability! Regards, Rotty
Re: r6rs libraries
Hi Julian, On Mon 26 Jan 2009 01:27, Julian Graham jool...@gmail.com writes: Maybe some more advanced Schemers than I can shed some light on the following: Well, that's not me, but I'll join you in fumbling for a solution :-) The levels system is simply a numerical way of encapsulating this information, but the proper order of evaluation can also be inferred by inspecting the import- and export-specs of the libraries being loaded I think you're right, yes. I think that the approach that you describe has been called Implicit phasing by Ghuloum and Dybvig. They have a paper about it, Implicit phasing in R6RS libraries -- but I haven't been able to find it freely on the web. ACM fail. * R6RS says that a library's imports need to be visited/instantiated at the time the bindings they export are referenced. Why? As above, why can't they be visited/instantiated at the time the imports for the importing library are processed? I could not find the quote that you referred to here -- I think what I can tell (from 7.2): If any of a library’s definitions are referenced at phase 0 in the expanded form of a program, then an instance of the referenced library is created for phase 0 before the program’s definitions and expressions are evaluated. This rule applies transitively: if the expanded form of one library references at phase 0 an identifier from another library, then before the referencing library is instantiated at phase n, the referenced library must be instantiated at phase n. When an identifier is referenced at any phase n greater than 0, in contrast, then the defining library is instantiated at phase n at some unspecified time before the reference is evaluated. Similarly, when a macro keyword is referenced at phase n during the expansion of a library, then the defining library is visited at phase n at some unspecified time before the reference is evaluated. So what this says to me is that: (1) At phase 0, libraries that you need to run a /program/ are instantiated before the program is run. (2) At phase n 0, we do not specify when libraries are imported. Is there any noticeable difference to the user? Dunno, to me it sounds like a concession, that side effects from loading libraries occur before side effects from running a program; but that for meta-levels things are left unspecified. Or do you guys read R6RS 7.2 to mean that the side-effects of top-level expressions absolutely need to happen at a time determined by the import level? No. So, for some of your other questions here's section 7.5 of the rationale: 7.5. Instantiation and initialization Opinions vary on how libraries should be instantiated and initialized during the expansion and execution of library bodies, whether library instances should be distinguished across phases, and whether levels should be declared so that they constrain identifier uses to particular phases. As I read this, it means that at least PLT wanted the separate instantiation model, and Chez wanted single-instantiation, implicit phasing. This report therefore leaves considerable latitude to implementations, while attempting to provide enough guarantees to make portable libraries feasible. So from 7.2 of R6RS itself: An implementation may distinguish instances/visits of a library for different phases or to use an instance/visit at any phase as an instance/visit at any other phase. Which is to say, we allow single instantiation -- as Guile modules are. An implementation may further expand each library form with distinct visits of libraries in any phase and/or instances of libraries in phases above 0. Which is to say, we also allow the PLT model, explicitly. An implementation may create instances/visits of more libraries at more phases than required to satisfy references. This is an odd one. I suppose what it means is that if you need a macro from library A to expand library B, but you don't need library A at runtime, the spec allows library A to be /instantiated/ at runtime. When an identifier appears as an expression in a phase that is inconsistent with the identifier’s level, then an implementation may raise an exception either at expand time or run time, or it may allow the reference. So, furthermore, it seems that not only may library A be instantiated at runtime, /it may be in library B's import list as well/. This is what happens with Guile's current module semantics. Thus, a library whose meaning depends on whether the instances of a library are distinguished or shared across phases or library expansions may be unportable. Indeed, indeed. I understand that the authors of the reference implementation re-created a lot of machinery
Re: r6rs libraries
Hi everyone, (Switching this conversation to guile-devel from guile-user, since it seems more appropriate to this list...) Alright, so I've been studying the van Tonder and Dybvig-Ghuloum implementations and banging my head against chapter 7 of R6RS, all with an eye towards mapping them onto Guile's module system, and I can't for the life of me figure out why the existing implementations are as complicated as they are. Maybe some more advanced Schemers than I can shed some light on the following: * Import and export levels seem to be a fancy way of notifying the library system of the time at which a library needs to be loaded/evaluated -- that is, if you import something from [library foo] for the expand phase of [library bar] you've got to evaluate (i.e., convert to a Guile module) the S-exp for [library foo] before you can evaluate the S-exp for [library bar]. The levels system is simply a numerical way of encapsulating this information, but the proper order of evaluation can also be inferred by inspecting the import- and export-specs of the libraries being loaded -- i.e., if the header of [library bar] specifies an import of anything from [library foo], no matter at what level, it's a safe move to evaluate [library foo] (if you haven't already done so) before finishing the evaluation of [library bar]. Is that right? * R6RS says that a library's imports need to be visited/instantiated at the time the bindings they export are referenced. Why? As above, why can't they be visited/instantiated at the time the imports for the importing library are processed? Is there any noticeable difference to the user? Or do you guys read R6RS 7.2 to mean that the side-effects of top-level expressions absolutely need to happen at a time determined by the import level? * R6RS also says that implementations are free to visit/instantiate libraries more or less often than is required by the import-export graph. Why would you want to visit/instantiate a library more than once? Why not just do it once, turn it into a module, and cache it? Andy Wingo noted that some implementations do a fresh visit for every phase (and that it's problematic), but I can't even see why you'd want to if the spec lets you off the hook for it. I understand that the authors of the reference implementation re-created a lot of machinery out of whole cloth since they were avoiding assumptions about features of their target Scheme platforms, but, man, both van Tonder and Dybvig-Ghuloum look like overkill for Guile. Am I missing a major piece of understanding here? Regards, Julian
R6RS Libraries
Hi, FWIW, I started an implementation of R6RS' bytevector API. It is available in my Arch archive [0], in the branch named `guile-r6rs-libs--devo--0'. Just letting you know so that we avoid duplicating this (tedious) work. ;-) Thanks, Ludovic. [0] http://www.laas.fr/~lcourtes/software/arch-2006/ ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
