Re: GNU Guile 2.2.2 released

[Andy Wingo]

On Sat 13 May 2017 15:05, Matt Wette  writes:

>> On Apr 21, 2017, at 7:41 AM, Andy Wingo  wrote:
>> 
>> We sheepishly announce GNU Guile release 2.2.2, a quick bug-fix to the
>> recent 2.2.1 release.
>
> The FreeBSD section in README is not quite correct IMO.   Here are suggested 
> changes:
>  
>  FreeBSD 11.0:
> -  For a build supporting threads, please `pkg install' the following
> +  Please `pkg install' the following:
>  - pkgconf : provides pkg-config
> +- texinfo : provides makeinfo

Not needed for tarball builds, right?

>  - gmake : /usr/bin/make does not work
> -- boehm-gc-threaded : needed for threaded support

This is correct, no?

Not sure which part of this is wrong :)

Andy

Re: Cygwin port of Guile 2.2

[Andy Wingo]

Greets,

On Fri 12 May 2017 16:13, Derek Upham  writes:

> Andy Wingo  writes:
>
>> scm_join_thread isn't actually implemented in terms of
>> scm_i_pthread_join any more.  Probably that's what's going wrong here --
>> and probably that should be fixed to ensure that we actually join the
>> thread.  (Otherwise it would be a memory leak too AFAIU.)  Bcc'ing
>> bug-guile to create a bug for that.
>
> I noticed that scm_join_thread was calling back into Scheme-land.  Are these 
> statements all correct?
>
> - We are using call-with-new-thread underneath the hood.

Underneath the hood of what? :)

> - call-with-new-thread is documented to return a Scheme object from a
> thunk/handler.  Any underlying pthreads should be implementation
> details.

Correct.  In practice call-with-new-thread will create a pthread but I
can imagine circumstances in which it might (in the future) spawn an
auxiliary pthread for some reason, and I wouldn't want to rule that out.

> - The spawned thread sends the Scheme object to the condition variable
> as soon as the user thunk exits.  Any number of operations can happen
> afterwards; the thread is still running in Scheme-land at this point,
> in call-with-new-thread’s wrapping thunk.
> - join-thread waits on the condition variable only.

These are implementation details.  They are correct but probably the
implementation should change to do the scm_i_pthread_join and we should
guarantee that after the join, the thread is really dead.  This is bug
26858.

> So at the end of join-thread we need to add a call to
> scm_i_pthread_join (which we implement in threads.c) to ensure that
> the pthread is completely gone before that join-thread returns.  Is
> that accurate?

Well... yes, but we have to ensure that we call scm_i_pthread_join at
most once.  I think calling pthread_join twice on a thread is
undefined.  So there are some gnarlies here.  Need to fix this.

> Unfortunately, I think the GC threads are going to end up being
> immovable objects in the path to full process-form support.

You can disable marker threads with the GC_MARKERS environment variable,
and the finalization thread should come and go as needed.  Probably this
is not a blocker from your POV.  Signal handling is probably the most
serious issue; perhaps we can avoid the thread somehow, since we handle
signals asynchronously anyway..

Andy

Re: 2.2 git users should switch to stable-2.2 branch

[Andy Wingo]

On Thu 11 May 2017 21:54, Christopher Allan Webber  
writes:

>> Incidentally if you use Guile commercially and would like to support my
>> work on it, let me know.
>
> I don't know what the right answer is to it (maybe eventually there
> would be a company making money off of Guix deployment that can manage
> to pay for full-time Guile work) but obviously this would be nice to see
> happen.

I think there's room for lots of organizations :) Commercially I see the
value as being in niches that can apply technology from the Guile
universe to problems people have.  We can (and should!) spread the
gospel of Scheme but that only gets you so far; my feeling is that
successful guile-hacking orgs will have a touch with some market need
beyond consultancy on Guile itself.  In that regard too that niche
relationship can exist in many different Guile-hacking orgs; a central
"GuileCo" doesn't seem likely to me.  Dunno tho :)

I personally don't see the "finish line" of a full-time Guile gig in
sight (yet?), so I am not going to put an extraordinary sprint of effort
into that side of things right now; Guile is still a side gig for me at
this point.  It's one which I'd be happy to expand of course but not
something on which to gamble the rent :)

Happy hacking,

Andy

Re: Including sjson (formerly (ice-9 json)) and fash.scm in guile proper?

[Christopher Allan Webber]

Mark H Weaver writes:

> I wrote:
>> Most of the modifications you've made are good, but I'm very
>> uncomfortable with the use of #nil in this API.  [...]
>
> Christopher Allan Webber  writes:
>> Oh!  No you got it backwards, the library *was* using #nil initially,
>> and I modified it to use 'null now instead. :)
>
> Ah, my mistake.  Excellent!
>
> Having now looked more closely, I'm mostly happy with the API, except
> for one issue: I don't like the way fash support was hacked in, with the
> 'use-fash' flag and the (if use-fash [fash-code] [alist-code]) sprinkled
> around.  If this truly needs to be done within the json library itself
> (which I wonder), then I'd prefer to generalize it to support any
> dictionary data structure, and thereby remove the dependency on fashes.

I agree that it's pretty hacky.  Allowing other dictionary structures is
fine by me.

> My main concern about fashes, besides the fact that Andy hasn't yet
> proposed adding them to Guile himself, is that the implementation is
> very complex, and I'd like to achieve some degree of confidence in its
> correctness before adding it.  I'd also tend to favor adding a simpler,
> truly immutable dictionary data structure based on Phil Bagwell's HAMTs
> (Hash Array Mapped Tries) to eliminate the need for thread
> synchronization, but I'm open to suggestions.

I don't really understand enough of the field to really know what the
right direction is.  I do know that I need something that's not O(n) for
json-ld processing, though I guess one option always would have been to
read in the sexp structure and transform it before doing all that
processing.   I've long wanted a better immutable dictionary
structure in Guile though, but am open to what it would be.

> Anyway, since writing my previous message in this thread, I've started
> carefully reviewing the code, making modifications as I go.  At this
> point, my proposed modifications have become quite extensive.  So far,
> I've reworked the code to greatly reduce heap allocations, support
> arbitrary dictionary types (removing the fash dependency, while still
> allowing its use), and fix various bugs (e.g. relying on unspecified
> evaluation order, failure to handle 12-character hex escapes properly,
> producing and accepting invalid JSON in some cases, etc).
>
> I'll followup with another message when I've completed my proposed
> revisions.  Feel free to ping me if it takes more than a week.

Wow, exciting!

>>> Otherwise, I'm generally in favor of incorporating this library into
>>> Guile, after we make sure that it is robust against malicious inputs.
>>
>> Okay, cool!  The other thing is to add more specific error messages, as
>> discussed.
>
> Indeed, better error messages would be a good thing.
>
>> What examples of malicious inputs should we test against?
>
> I'm mostly trying to address that by careful code review.

Yay!  Thank you for doing it.

Re: Including sjson (formerly (ice-9 json)) and fash.scm in guile proper?

[Mark H Weaver]

I wrote:
> Most of the modifications you've made are good, but I'm very
> uncomfortable with the use of #nil in this API.  [...]

Christopher Allan Webber  writes:
> Oh!  No you got it backwards, the library *was* using #nil initially,
> and I modified it to use 'null now instead. :)

Ah, my mistake.  Excellent!

Having now looked more closely, I'm mostly happy with the API, except
for one issue: I don't like the way fash support was hacked in, with the
'use-fash' flag and the (if use-fash [fash-code] [alist-code]) sprinkled
around.  If this truly needs to be done within the json library itself
(which I wonder), then I'd prefer to generalize it to support any
dictionary data structure, and thereby remove the dependency on fashes.

My main concern about fashes, besides the fact that Andy hasn't yet
proposed adding them to Guile himself, is that the implementation is
very complex, and I'd like to achieve some degree of confidence in its
correctness before adding it.  I'd also tend to favor adding a simpler,
truly immutable dictionary data structure based on Phil Bagwell's HAMTs
(Hash Array Mapped Tries) to eliminate the need for thread
synchronization, but I'm open to suggestions.

Anyway, since writing my previous message in this thread, I've started
carefully reviewing the code, making modifications as I go.  At this
point, my proposed modifications have become quite extensive.  So far,
I've reworked the code to greatly reduce heap allocations, support
arbitrary dictionary types (removing the fash dependency, while still
allowing its use), and fix various bugs (e.g. relying on unspecified
evaluation order, failure to handle 12-character hex escapes properly,
producing and accepting invalid JSON in some cases, etc).

I'll followup with another message when I've completed my proposed
revisions.  Feel free to ping me if it takes more than a week.

>> Otherwise, I'm generally in favor of incorporating this library into
>> Guile, after we make sure that it is robust against malicious inputs.
>
> Okay, cool!  The other thing is to add more specific error messages, as
> discussed.

Indeed, better error messages would be a good thing.

> What examples of malicious inputs should we test against?

I'm mostly trying to address that by careful code review.

 Regards,
   Mark