Re: a plan for native compilation

[Ludovic Courtès]

Howdy!

Andy Wingo  writes:

> So, my thought is to extend procedures with an additional pointer, a
> pointer to a "native" code structure.

(So your point is what should we do now to allow for such experiments
eventually, right?)

Adding an extra work to programs seems like a good idea, yes.

> Now, what technology to choose for the compiler itself? Dunno. For a
> JIT, it would be useful to use something portable, and perhaps do the
> JIT compilation on the bytecode itself, without more source information.
> It would not produce the fastest code, but it would run fast.

Yes, that’s what I had in mind, using GNU lightning (see
.)  It /seems/ to
be doable, with milestones to do it incrementally, starting from a dumb
version.

> I think we can produce better native code ahead-of-time coming from the
> tree-il layer directly. I feel like eventually we'll need to replace
> GLIL with something else, but I don't really know; we'll find out in the
> future I guess. But I do want to do ahead-of-time compilation, because
> I want Guile programs to start up very quickly and not consume much
> memory.

Sure.

lightning does x86, x86_64, sparc, and powerpc (PLT uses it) while Sassy
does only x86, so it may be that both could play a role.

Anyway, not for now.  :-)

Thanks,
Ludo’.

Re: ffi docs

[Neil Jerram]

Andy Wingo  writes:

> Hi,
>
> Thanks for the feedback!

Thanks for the quick responses.  As far as most of them are concerned,
I'll read the section on foreign functions first before offering more
thoughts, as you suggest.  Also I agree that it will make sense for me
to handle these doc changes.

On a couple of specific points

> On Fri 16 Apr 2010 00:36, Neil Jerram  writes:
>
>>>But yet we as programmers live in both worlds, and Guile itself is
>>> half implemented in C. So it is that Guile's living half pays respect
>>> to its dead counterpart, via a spectrum of interfaces to C ranging from
>>> dynamic loading of Scheme primitives to dynamic binding of stock C
>>> library prodedures.
>>
>> c ---^
>
> What does this mean?

I'm sorry!  It was supposed to indicate a typo, and that there should be
a "c" at the place the caret points - i.e. "procedures" rather than
"prodedures".  Anyway, I'll handle this.

>>>We titled this section "foreign libraries" because although the name
>>> "foreign" doesn't leak into the API, the world of C really is foreign
>>> to Scheme - and that estrangement extends to components of foreign
>>> libraries as well, as we see in future sections.
>>
>> I'm not sure what the message is here.
>
> Probably me being to cutesy, I would imagine.

I didn't think that!

> The facility is typically
> called a "foreign function interface", but that name doesn't appear in
> e.g. "dynamic-link", so I was trying to explain.

Ah yes, I see now.  In that case I think it's just the last clause that
doesn't quite work for me.  I would say that the _immediately_ following
text is about "components of foreign libraries", so why say "as we see
in future sections"?

Maybe: "... really is foreign to Scheme.  Foreign function and data
pointers, obtained via `dynamic-func' and `dynamic-pointer', or as
return values from a foreign function call, are inherently untyped, and
depend on the Scheme programmer using them in a way that is consistent
with the library's documented interface.  Any other usage is unsafe and
can easily cause the containing Scheme program to crash, and the Guile
low-level FFI cannot protect against this.  (This is quite different
from computations on Scheme values; Scheme values are typed, and so
operations on them can check in advance that the value types are as
expected.)"

>> Code below implies that library can be omitted, and that this -
>> i.e. '(dynamic-link)' - means to return an object representing libguile
>> itself.  Should that be mentioned in the following doc?
>>
>>>  Find the shared library denoted by LIBRARY (a string) and link it
>>>  into the running Guile application.  When everything works out,
>>>  return a Scheme object suitable for representing the linked object
>>>  file.  Otherwise an error is thrown.  How object files are
>>>  searched is system dependent.
>>>
>>>  Normally, LIBRARY is just the name of some shared library file
>>>  that will be searched for in the places where shared libraries
>>>  usually reside, such as in `/usr/lib' and `/usr/local/lib'.
>>>
>>>  When LIBRARY is omitted, a "global symbol handle" is returned.
>>>  This handle provides access to the symbols available to the
>>>  program at run-time, including those exported by the program
>>>  itself and the shared libraries already loaded.
>
> I think it is mentioned, no? Is there a way that it can be more clear?

I'm sorry.  I can't believe I missed that.  I looked for it so
carefully!

>> - isn't it actually much more to do with the ELF binary format, rather
>>   than with C?  If libguile could read and parse C, it would be able to
>>   infer the type of any variable that the Scheme layer might request.
>>   The problem is precisely that what we are linking with is *not* C
>>   anymore...  It's just untyped pointers.
>
> I guess you're right, this is confusing. C doesn't really exist at
> runtime, and this API is all about accessing runtime values.

On further reflection, I think I'm not completely right.  Functions
assume that they will be called according to well known C calling
conventions.  So I guess there are vestiges of C.

Out of interest, do other languages that compile to library format use
different calling conventions, and if so can dlopen/dlsym and FFIs work
with them?

More tomorrow...

Regards,
  Neil

Re: [Guile-commits] GNU Guile branch, wip-manual-2, updated. release_1-9-9-85-g0a864be

[Neil Jerram]

Andy Wingo  writes:

> Hi Neil,

Hi Andy,

> I also think that wip-manual-2 is looking great! Your commit granularity
> is also really nice.

Thanks.

>>From my perspective each commit is ready! Why not just merge now? It's
> getting monotonically better :)

OK, I'll do that; thanks for the suggestion.  For the sake of
explanation: I guess when I started wip-manual-2 I wasn't totally sure
that my direction was correct, or that it would meet with approval.  But
given that it does, I agree that it makes sense now to continue in
master.

> That way when someone writes new docs, they have a better idea about how
> that fits in. For example I need to write something about "nil", and
> I'll look at your branch and see where you're going, but if it were
> merged already I could just look at the state of master and see where
> the right place would be.

Yes, absolutely.

> I still think it's the right statement to make :) Guile _is_ and _has_
> other things as well, but fundamentally it's a Scheme. IMO at least :)

Agreed.  Historically and philosophically that's clearly true.  And from
a practical point of view, however many other languages we support,
Scheme will always remain special because the compiler itself is written
in Scheme.

Regards,
  Neil

Re: a plan for native compilation

[No Itisnt]

One option I am really starting to like is LLVM. I know what you're
thinking, huge memory consumption, giant dependency, etc, but it's so
cool! It supports every desktop architecture too.

tracing and traping?

[stefan]

Hi,

I learned to like to trace functions when 
debugging and found out that guiles version needs work.
 so has anybody steped up to fix this?

/Stefan

Re: our nil story

[Andy Wingo]

Hi Daniel,

On Fri 16 Apr 2010 14:35, Daniel Kraft  writes:

> IIRC there's still some info about implementation details and other
> stuff that may be interesting in the module/languages/elisp/README
> file. You probably are aware of it anyway, though ;)

I had forgotten about it, actually. Good stuff!

Andy
-- 
http://wingolog.org/

Re: our nil story

[Andy Wingo]

Hi!

Thanks for the comments, I've folded them in. [In the future though,
patches will be just as acceptable ;-)]

On Fri 16 Apr 2010 14:12, l...@gnu.org (Ludovic Courtès) writes:

>>Guile can warn when compiling code that has equality comparisons with
>> `#f', `'()', or `nil'. *Note Compilation::, for details.
>
> Hmm, not yet, right?  :-)

Indeed, but that means we just have to fix the implementation and not
the docs ;-)

> Heh heh.  What’s your take on ES support for 2.0?
>
> Mine is that “we” should make the basics work, as they used to back when
> you implemented it, and fill in ecmascript.test.  IOW, nothing fancy,
> but just basic compilation.

Probably so. "Let's" get at it, then ;-)

Andy
-- 
http://wingolog.org/

Re: [Guile-commits] GNU Guile branch, wip-manual-2, updated. release_1-9-9-85-g0a864be

[Andy Wingo]

Hi Neil,

I also think that wip-manual-2 is looking great! Your commit granularity
is also really nice.

On Mon 12 Apr 2010 02:15, Neil Jerram  writes:

> at a point in the not too distant future, the manual in wip-manual-2
> will be ready for pulling back into master, and publishing.

>From my perspective each commit is ready! Why not just merge now? It's
getting monotonically better :)

That way when someone writes new docs, they have a better idea about how
that fits in. For example I need to write something about "nil", and
I'll look at your branch and see where you're going, but if it were
merged already I could just look at the state of master and see where
the right place would be.

> Now that multiple language support is really starting to take off,
> last night I was thinking that the first sentence "Guile is an
> implementation of the Scheme programming language." is probably not
> the right first sentence any more...
>
> 

I still think it's the right statement to make :) Guile _is_ and _has_
other things as well, but fundamentally it's a Scheme. IMO at least :)

Happy hacking!

Andy
-- 
http://wingolog.org/

Re: our nil story

[Daniel Kraft]

Hi Andy,

sounds good!  With regards to some elisp details (like dynamic binding 
or others), IIRC there's still some info about implementation details 
and other stuff that may be interesting in the 
module/languages/elisp/README file.  You probably are aware of it 
anyway, though ;)


Yours,
Daniel

Re: our nil story

[Ludovic Courtès]

Hi!

Andy Wingo  writes:

> Just updated some docs on other languages and nil. It's in master, but
> I'm pasting here for comments.

Cool!

> 0.1 Support for Other Languages
> ===
>
> In addition to Scheme, a user may write a Guile program in an increasing
> number of other languages. Currently supported languages include Emacs
> Lisp and ECMAScript.
>
>Guile is still fundamentally a Scheme, but it tries to support a wide
> variety of language building-blocks, so that a user can implement other
> languages on top of Guile.

... and so that users can write or extend applications in languages
other than Scheme, too.

>The easiest way to deal with syntax-heavy language is with files, via
> `compile-file' and friends. However it is possible to invoke a
> language's reader on a port, and then compile the resulting expression
> (which is a datum at that point). For more information, *Note
> Compilation::.

Perhaps mention ‘read-and-compile’?

>On a low level, the bit representations for `#f', `#t', `nil', and
> `'()' are made in such a way that the differ by only one bit, and so a

Typo: “that _they_ differ”.

>Both of these examples are correct standard Scheme, but, depending on
> what they really want to do, they are not correct Guile Scheme.
> Correctly written, they would test the _properties_ of falsehood or
> nullity, not the individual members of that set. That is to say, they
> should use `not' or `null?' to test for falsehood or nullity, not `eq?'
> or `memv' or the like.

Excellent!  I think the examples and explanations in this section make
it perfectly clear how things interoperate and what the pitfalls are.

>Guile can warn when compiling code that has equality comparisons with
> `#f', `'()', or `nil'. *Note Compilation::, for details.

Hmm, not yet, right?  :-)

> 0.1.2.3 Dynamic Binding
> ...

@xref{Fluids}?

> 0.1.3 ECMAScript
> 
>
> ECMAScript was not the first non-Schemey language implemented by Guile,

@url{http://.../the-spec.html, ECMAScript} maybe.

> but it was the first implemented for Guile's bytecode compiler. The goal
> was to support ECMAScript version 3.1, a relatively small language, but
> the implementor was completely irresponsible and got distracted by other
> things before finishing the standard library, and even some bits of the
> syntax. So, ECMAScript does deserve a mention in the manual, but it
> doesn't deserve an endorsement until its implementation is completed,
> perhaps by some more responsible hacker.
>
>In the meantime, the charitable user might investigate such
> invocations as `,L ecmascript' and `cat
> test-suite/tests/ecmascript.test'.

Heh heh.  What’s your take on ES support for 2.0?

Mine is that “we” should make the basics work, as they used to back when
you implemented it, and fill in ecmascript.test.  IOW, nothing fancy,
but just basic compilation.

Thanks!

Ludo’.

a plan for native compilation

[Andy Wingo]

Hi,

I've been thinking some about native compilation, and how to do
ahead-of-time and just-in-time compilation, and not totally explode
Guile's mental footprint -- the number of cases and things that one has
to keep in mind when debugging Guile code.

Currently, Guile has a compiler to a custom virtual machine, and the
associated toolchain: assemblers and disassemblers, stack walkers, the
debugger, etc. One can get the source location of a particular
instruction pointer, for example. It's all mostly understandable, and
mostly uniform across platforms (modulo word size and byte order, and
perhaps we should make that uniform as well, especially if we can do
native compilation.)

So, my thought is to extend procedures with an additional pointer, a
pointer to a "native" code structure. The native code could be written
out ahead-of-time, or compiled at runtime. But procedures would still
have bytecode, for various purposes for example to enable code coverage
via the next-instruction hook, and in the JIT case, because only some
procedures will be native-compiled.

We keep the same stack representation, so stack walkers and the debugger
still work. Some local variables can be allocated into registers, but
procedure args are still passed and returned on the stack. Though the
procedure's arity and other metadata would be the same, the local
variable allocations and source locations would differ, so we would need
some additional debugger support, but we can work on that when the time
comes.

All Scheme procedures, bytecode and native, will run inside the VM. If a
bytecode procedure calls a native procedure, the machine registers are
saved, and some machine-specific stub transfers control to the native
code. Native code calling native code uses the same stack as the VM,
though it has its own conventions over what registers to save; and
native code calling bytecode prepares the Scheme stack, then restores
the VM-saved machine registers.

In this way we can do incremental experimentation with compiled
procedures, compiling at runtime, or ahead-of-time.

Now, what technology to choose for the compiler itself? Dunno. For a
JIT, it would be useful to use something portable, and perhaps do the
JIT compilation on the bytecode itself, without more source information.
It would not produce the fastest code, but it would run fast.

AIUI the hotspot compiler actually does an SSA transformation of Java
bytecode, then works on that. I'm not particularly interested in
something like that; I'm more interested in something direct and fast,
and obviously correct and understandable by our debugging
infrastructure.

I think we can produce better native code ahead-of-time coming from the
tree-il layer directly. I feel like eventually we'll need to replace
GLIL with something else, but I don't really know; we'll find out in the
future I guess. But I do want to do ahead-of-time compilation, because
I want Guile programs to start up very quickly and not consume much
memory.

Anyway, just some thoughts here. I'm not going to focus on native
compilation in the coming months, as there are other things to do, but
this is how I think it should be done :-)

Cheers,

Andy
-- 
http://wingolog.org/

Re: ffi docs

[Andy Wingo]

Hi,

Thanks for the feedback!

On Fri 16 Apr 2010 00:36, Neil Jerram  writes:

>>But yet we as programmers live in both worlds, and Guile itself is
>> half implemented in C. So it is that Guile's living half pays respect
>> to its dead counterpart, via a spectrum of interfaces to C ranging from
>> dynamic loading of Scheme primitives to dynamic binding of stock C
>> library prodedures.
>
> c ---^

What does this mean?

>>We titled this section "foreign libraries" because although the name
>> "foreign" doesn't leak into the API, the world of C really is foreign
>> to Scheme - and that estrangement extends to components of foreign
>> libraries as well, as we see in future sections.
>
> I'm not sure what the message is here.

Probably me being to cutesy, I would imagine. The facility is typically
called a "foreign function interface", but that name doesn't appear in
e.g. "dynamic-link", so I was trying to explain.

Beyond that I guess I meant to say that "native" depends on where you're
coming from; that Scheme calls are native to Scheme, and C calls are
foreign to Scheme.

>>  -- Scheme Procedure: dynamic-link [library]
>>  -- C Function: scm_dynamic_link (library)
>
> Code below implies that library can be omitted, and that this -
> i.e. '(dynamic-link)' - means to return an object representing libguile
> itself.  Should that be mentioned in the following doc?
>
>>  Find the shared library denoted by LIBRARY (a string) and link it
>>  into the running Guile application.  When everything works out,
>>  return a Scheme object suitable for representing the linked object
>>  file.  Otherwise an error is thrown.  How object files are
>>  searched is system dependent.
>>
>>  Normally, LIBRARY is just the name of some shared library file
>>  that will be searched for in the places where shared libraries
>>  usually reside, such as in `/usr/lib' and `/usr/local/lib'.
>>
>>  When LIBRARY is omitted, a "global symbol handle" is returned.
>>  This handle provides access to the symbols available to the
>>  program at run-time, including those exported by the program
>>  itself and the shared libraries already loaded.

I think it is mentioned, no? Is there a way that it can be more clear?

>>Given some set of C extensions to Guile, the next logical step is to
>> integrate these glue libraries into the module system of Guile so that
>> you can load new primitives into a running system just as you can load
>> new Scheme code.
>>
>>  -- Scheme Procedure: load-extension lib init
>>  -- C Function: scm_load_extension (lib, init)
>>  Load and initialize the extension designated by LIB and INIT.
>>  When there is no pre-registered function for LIB/INIT, this is
>>  equivalent to
>>
>>   (dynamic-call INIT (dynamic-link LIB))
>>
>>  When there is a pre-registered function, that function is called
>>  instead.
>>
>>  Normally, there is no pre-registered function.  This option exists
>>  only for situations where dynamic linking is unavailable or
>>  unwanted.  In that case, you would statically link your program
>>  with the desired library, and register its init function right
>>  after Guile has been initialized.
>
> Should there be a reference from here to wherever the registration API
> is covered?

Probably. Is it documented somewhere? :) I think no. I would doc it
here, fwiw...

>>  LIB should be a string denoting a shared library without any file
>>  type suffix such as ".so".  The suffix is provided automatically.
>>  It should also not contain any directory components.  Libraries
>>  that implement Guile Extensions should be put into the normal
>>  locations for shared libraries.  We recommend to use the naming
>>  convention libguile-bla-blum for a extension related to a module
>>  `(bla blum)'.
>
> I believe this will shortly be out of date, won't it? - given our desire
> to support parallel installs.

Hm, good point; though if it is installed into the extensionsdir as
suggested below, we do work around this issue.

>>A compiled module should have a specially named "module init
>> function".  Guile knows about this special name and will call that
>> function automatically after having linked in the shared library.  For
>> our example, we replace `init_math_bessel' with the following code in
>> `bessel.c':
>>
>>  void
>>  init_math_bessel (void *unused)
>>  {
>>scm_c_define_gsubr ("j0", 1, 0, 0, j0_wrapper);
>>scm_c_export ("j0", NULL);
>>  }
>>
>>  void
>>  scm_init_math_bessel_module ()
>>  {
>>scm_c_define_module ("math bessel", init_math_bessel, NULL);
>>  }
>>
>>The general pattern for the name of a module init function is:
>> `scm_init_', followed by the name of the module where the individual
>> hierarchical components are concatenated with underscores, followed by
>> `_module'.
>
> Is this still correct?  I

our nil story

[Andy Wingo]

Hi all,

Just updated some docs on other languages and nil. It's in master, but
I'm pasting here for comments.

0.1 Support for Other Languages
===

In addition to Scheme, a user may write a Guile program in an increasing
number of other languages. Currently supported languages include Emacs
Lisp and ECMAScript.

   Guile is still fundamentally a Scheme, but it tries to support a wide
variety of language building-blocks, so that a user can implement other
languages on top of Guile. This section describes the languages that
have been implemented.

   (For details on how to implement a language, *Note Compiling to the
Virtual Machine::.)

0.1.1 Using Other Languages
---

There are currently only two ways to access other languages from within
Guile: at the REPL, and via `compile' or `compile-file'.

   The REPL is Guile's command prompt (*note Using Guile
Interactively::).  The REPL has a concept of the "current language",
which defaults to Scheme. The user may change that language, via the
meta-command `,language'.

   For example, the following meta-command enables Emacs Lisp input:

 scheme@(guile-user)> ,language elisp
 Happy hacking with Emacs Lisp!  To switch back, type `,L scheme'.
 elisp@(guile-user)> (eq 1 2)
 $1 = #nil

   Each language has its short name: for example, `elisp', for Elisp.
The same short name may be used to compile source code programmatically,
via `compile':

 elisp@(guile-user)> ,L scheme
 Happy hacking with Guile Scheme!  To switch back, type `,L elisp'.
 scheme@(guile-user)> (compile '(eq 1 2) #:from 'elisp)
 $2 = #nil

   Granted, as the input to `compile' is a datum, this works best for
Lispy languages, which have a straightforward datum representation.
Other languages that need more parsing are better dealt with as strings.

   The easiest way to deal with syntax-heavy language is with files, via
`compile-file' and friends. However it is possible to invoke a
language's reader on a port, and then compile the resulting expression
(which is a datum at that point). For more information, *Note
Compilation::.

   For more details on introspecting aspects of different languages,
*Note Compiler Tower::.

0.1.2 Emacs Lisp


Emacs Lisp (Elisp) is a dynamically-scoped Lisp dialect used in the
Emacs editor. *Note Overview: (elisp)top, for more information on Emacs
Lisp.

   We hope that eventually Guile's implementation of Elisp will be good
enough to replace Emacs' own implementation of Elisp. For that reason,
we have thought long and hard about how to support the various features
of Elisp in a performant and compatible manner.

   Readers familiar with Emacs Lisp might be curious about how exactly
these various Elisp features are supported in Guile. The rest of this
section focuses on addressing these concerns of the Elisp elect.

0.1.2.1 Nil
...

`nil' in ELisp is an amalgam of Scheme's `#f' and `'()'.  It is false,
and it is the end-of-list; thus it is a boolean, and a list as well.

   Guile has chosen to support `nil' as a separate value, distinct from
`#f' and `'()'. This allows existing Scheme and Elisp code to maintain
their current semantics. `nil', which in Elisp would just be written
and read as `nil', in Scheme has the external representation `#nil'.

   This decision to have `nil' as a low-level distinct value
facilitates interoperability between the two languages. Guile has chosen
to have Scheme deal with `nil' as follows:

 (boolean? #nil) => #t
 (not #nil) => #t
 (null? #nil) => #t

   And in C, one has:

 scm_is_bool (SCM_ELISP_NIL) => 1
 scm_is_false (SCM_ELISP_NIL) => 1
 scm_is_null (SCM_ELISP_NIL) => 1

   In this way, a version of `fold' written in Scheme can correctly
fold a function written in Elisp (or in fact any other language) over a
nil-terminated list, as Elisp makes. The converse holds as well; a
version of `fold' written in Elisp can fold over a `'()'-terminated
list, as made by Scheme.

   On a low level, the bit representations for `#f', `#t', `nil', and
`'()' are made in such a way that the differ by only one bit, and so a
test for, for example, `#f'-or-`nil' may be made very efficiently. See
`libguile/boolean.h', for more information.

0.1.2.2 Equality


Since Scheme's `equal?' must be transitive, and `'()' is not `equal?'
to `#f', to Scheme `nil' is not `equal?' to `#f' or `'()'.

 (eq? #f '()) => #f
 (eq? #nil '()) => #f
 (eq? #nil #f) => #f
 (eqv? #f '()) => #f
 (eqv? #nil '()) => #f
 (eqv? #nil #f) => #f
 (equal? #f '()

Re: ffi docs

[Ludovic Courtès]

Hi Neil,

A few answers/comments and I’ll leave the rest to Andy.  ;-)

Neil Jerram  writes:

> Andy Wingo  writes:

>>  -- Scheme Procedure: dynamic-link [library]
>>  -- C Function: scm_dynamic_link (library)
>
> Code below implies that library can be omitted, and that this -
> i.e. '(dynamic-link)' - means to return an object representing libguile
> itself.  Should that be mentioned in the following doc?

This is actually documented:

>>  When LIBRARY is omitted, a "global symbol handle" is returned.
>>  This handle provides access to the symbols available to the
>>  program at run-time, including those exported by the program
>>  itself and the shared libraries already loaded.

[...]

>>A compiled module should have a specially named "module init
>> function".  Guile knows about this special name and will call that
>> function automatically after having linked in the shared library.  For
>> our example, we replace `init_math_bessel' with the following code in
>> `bessel.c':
>>
>>  void
>>  init_math_bessel (void *unused)
>>  {
>>scm_c_define_gsubr ("j0", 1, 0, 0, j0_wrapper);
>>scm_c_export ("j0", NULL);
>>  }
>>
>>  void
>>  scm_init_math_bessel_module ()
>>  {
>>scm_c_define_module ("math bessel", init_math_bessel, NULL);
>>  }
>>
>>The general pattern for the name of a module init function is:
>> `scm_init_', followed by the name of the module where the individual
>> hierarchical components are concatenated with underscores, followed by
>> `_module'.
>
> Is this still correct?

The bit that says “Guile knows about this special name” has been
incorrect since 1.8 AFAIK, because the name of the init function has to
be explicitly given to ‘load-extension’.  So this part can be removed.

The bit about the “general pattern for the name” is correct, but it
should probably made clear that it’s just a convention.

>>So when accessing a C value through a Scheme pointer, we must give
>> the type of the pointed-to value explicitly, as a parameter to any
>> Scheme procedure that accesses the value.
>
> This confused me at first.  I think I understand the point now, but
>
> - isn't it actually much more to do with the ELF binary format, rather
>   than with C?  If libguile could read and parse C, it would be able to
>   infer the type of any variable that the Scheme layer might request.
>   The problem is precisely that what we are linking with is *not* C
>   anymore...  It's just untyped pointers.

C itself is very weakly typed: about anything can be cast to anything
else.

> - I think "give the type ... as a parameter to any Scheme procedure that
>   accesses the value" is misleading, because we don't do that!  Rather,
>   we construct a box that includes both the pointer and the type, and
>   then pass the box around.

Agreed.

[...]

>>As an example, `(dynamic-pointer "foo" void bar-lib)' links in the
>> FOO symbol in the BAR-LIB library as a pointer to `void': a `void*'.
>>
>>Void pointers may be accessed as bytevectors.
>>
>>  -- Scheme Procedure: foreign->bytevector foreign [uvec_type [offset
>>   [len]]]
>>  -- C Function: scm_foreign_to_bytevector foreign uvec_type offset len
>>  Return a bytevector aliasing the memory pointed to by FOREIGN.
>>
>>  FOREIGN must be a void pointer, a foreign whose type is VOID. By
>>  default, the resulting bytevector will alias all of the memory
>>  pointed to by FOREIGN, from beginning to end, treated as a `vu8'
>>  array.
>
> It feels like we're missing a unification trick here.

What do you mean?

> Thought #1: if we have, e.g., an int8 pointer ip, why not just use
> (foreign-ref ip n) to interpret the pointer as pointing to an array, and
> get its nth element?
>
> Thought #2: but if we do that we'll be duplicating the bytevector API.
> So instead, shouldn't the fundamental operation be (foreign->bytevector
> NAME TYPE LIBRARY [LEN]), and get/set then done using the bytevector
> API?

What would be LIBRARY here?

FWIW I’m fine with this procedure.

Thanks,
Ludo’.