Re: [racket-dev] Quick feedback on first planet2 experience

[Jay McCarthy]

Thanks

On Mon, Dec 3, 2012 at 9:39 PM, David Van Horn  wrote:
> - It's not clear what the red asterisk means here:
>   https://plt-etc.byu.edu:9004/search
>   (Sam tells me it means "recently updated".)

Yup

> - There's a typo ("User-spcific") in the output of raco pkg show.

Fixed

> - The "details" link on the package upload page is broken:
>
> http://pre.racket-lang.org/docs/html/planet2/Planet_2_Concepts.html#(tech._package._source)

Fixed

> - Searching for "planet2" should also return a link to the top of
>   "Package Management in Racket".
>
> - Does the documentation contain a link to:
>   https://plt-etc.byu.edu:9004/search
>   (I missed it if it does.)

Yes, it says: "PLT supports two package name services, which are
enabled by default: https://plt-etc.byu.edu:9004 for new packages and
https://plt-etc.byu.edu:9003 for automatically generated packages for
old PLaneT packages."

> - "dependencies on thar packages" in Package Metadata docs.

Fixed

> - "This categories will" in Future Plans docs.

Fixed

Jay

--
Jay McCarthy 
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

[racket-dev] Quick feedback on first planet2 experience

[David Van Horn]

- It's not clear what the red asterisk means here:
  https://plt-etc.byu.edu:9004/search
  (Sam tells me it means "recently updated".)

- There's a typo ("User-spcific") in the output of raco pkg show.

- The "details" link on the package upload page is broken:

http://pre.racket-lang.org/docs/html/planet2/Planet_2_Concepts.html#(tech._package._source)

- Searching for "planet2" should also return a link to the top of
  "Package Management in Racket".

- Does the documentation contain a link to:
  https://plt-etc.byu.edu:9004/search
  (I missed it if it does.)

- "dependencies on thar packages" in Package Metadata docs.

- "This categories will" in Future Plans docs.

David
_
 Racket Developers list:
 http://lists.racket-lang.org/dev

Re: [racket-dev] DrRacket automatic parentheses mode update

[Robby Findler]

Thanks, Nadeem!

Robby

On Mon, Dec 3, 2012 at 9:30 PM, Nadeem Abdul Hamid  wrote:
> Some improvements to DrRacket's "automatic parentheses" mode are now
> available in the nightly build version and git repository. If you have
> previous tried auto-parens and abandoned it, or if you have never tried it,
> please do try it now!
>
> In auto-parens mode, typing a closing parenthesis will skip over an existing
> one if the cursor is right in front of it. You can also type M+)  (meta key
> + close parens), whether in auto-parens mode or not, to skip right past the
> closing parentheses of the enclosing expression, or, if the expression is
> not well-balanced, just forward to the next closing parentheses. This
> behavior should also work as expected with other closing parenthesis-like
> symbols, double quotes, and | ... | pairs. There are also some tweaks to the
> auto-parens mode so that block comments #| ... |# are inserted correctly.
> Also auto-parens mode has no effect when inside a string literal or comment
> (you can always use M+( to force a parens pair if you really want one) or
> when typing a character literal.
>
> While 'undo' works properly now for just-inserted () pairs in auto-parens
> mode, I did not mess at all with delete key behavior. I've left the
> Paredit-like functionality for another time/person! :-)
>
> Have fun,
>
> nadeem
>
>
> _
>   Racket Developers list:
>   http://lists.racket-lang.org/dev
>
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

[racket-dev] DrRacket automatic parentheses mode update

[Nadeem Abdul Hamid]

Some improvements to DrRacket's "automatic parentheses" mode are now
available in the nightly build version and git repository. If you have
previous tried auto-parens and abandoned it, or if you have never tried it,
please do try it now!

In auto-parens mode, typing a closing parenthesis will skip over an
existing one if the cursor is right in front of it. You can also type M+)
 (meta key + close parens), whether in auto-parens mode or not, to skip
right past the closing parentheses of the enclosing expression, or, if the
expression is not well-balanced, just forward to the next closing
parentheses. This behavior should also work as expected with other closing
parenthesis-like symbols, double quotes, and | ... | pairs. There are also
some tweaks to the auto-parens mode so that block comments #| ... |# are
inserted correctly. Also auto-parens mode has no effect when inside a
string literal or comment (you can always use M+( to force a parens pair if
you really want one) or when typing a character literal.

While 'undo' works properly now for just-inserted () pairs in auto-parens
mode, I did not mess at all with delete key behavior. I've left the
Paredit-like functionality for another time/person! :-)

Have fun,

nadeem
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] [plt] Push #25831: master branch updated

[Robby Findler]

Right, thanks. Somehow I thought it was just in a top-level collection
(I didn't read carefully enough).

Sorry,
Robby

On Mon, Dec 3, 2012 at 8:06 PM, Sam Tobin-Hochstadt  wrote:
> We make no commitments about compatibility for 'unstable', so keeping the
> collection around shouldn't even be necessary.
>
> Sam
>
> On Dec 3, 2012 8:54 PM, "Ryan Culpepper"  wrote:
>>
>> I updated the references in the racket git repo, and I left the
>> unstable/lazy-require module in place with a re-export of lazy-require. I
>> did remove a feature that didn't seem to be in use (unquote), though, so if
>> someone is using lazy-require with unquote outside of the racket git repo,
>> that will break.
>>
>> Ryan
>>
>>
>> On 12/03/2012 07:57 PM, Robby Findler wrote:
>>>
>>> Does the lazy-require move break code?
>>>
>>> Robby
>>>
>>> On Mon, Dec 3, 2012 at 6:40 PM,   wrote:

 ryanc has updated `master' from 0252207e38 to 33f3574f7e.
http://git.racket-lang.org/plt/0252207e38..33f3574f7e

 =[ 4 Commits ]==
 Directory summary:
10.7% collects/racket/
39.3% collects/unstable/scribblings/
44.7% collects/unstable/
 5.1% collects/

 ~~
 [...]
>>
>>
>> _
>>  Racket Developers list:
>>  http://lists.racket-lang.org/dev
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] [plt] Push #25831: master branch updated

[Sam Tobin-Hochstadt]

We make no commitments about compatibility for 'unstable', so keeping the
collection around shouldn't even be necessary.

Sam
On Dec 3, 2012 8:54 PM, "Ryan Culpepper"  wrote:

> I updated the references in the racket git repo, and I left the
> unstable/lazy-require module in place with a re-export of lazy-require. I
> did remove a feature that didn't seem to be in use (unquote), though, so if
> someone is using lazy-require with unquote outside of the racket git repo,
> that will break.
>
> Ryan
>
>
> On 12/03/2012 07:57 PM, Robby Findler wrote:
>
>> Does the lazy-require move break code?
>>
>> Robby
>>
>> On Mon, Dec 3, 2012 at 6:40 PM,   wrote:
>>
>>> ryanc has updated `master' from 0252207e38 to 33f3574f7e.
>>>
>>> http://git.racket-lang.org/**plt/0252207e38..33f3574f7e
>>>
>>> =[ 4 Commits ]=**
>>> =
>>> Directory summary:
>>>10.7% collects/racket/
>>>39.3% collects/unstable/scribblings/
>>>44.7% collects/unstable/
>>> 5.1% collects/
>>>
>>> ~~
>>> [...]
>>>
>>
> _
>  Racket Developers list:
>  http://lists.racket-lang.org/**dev 
>
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] [plt] Push #25831: master branch updated

[Ryan Culpepper]

I updated the references in the racket git repo, and I left the 
unstable/lazy-require module in place with a re-export of lazy-require. 
I did remove a feature that didn't seem to be in use (unquote), though, 
so if someone is using lazy-require with unquote outside of the racket 
git repo, that will break.


Ryan


On 12/03/2012 07:57 PM, Robby Findler wrote:

Does the lazy-require move break code?

Robby

On Mon, Dec 3, 2012 at 6:40 PM,   wrote:

ryanc has updated `master' from 0252207e38 to 33f3574f7e.
   http://git.racket-lang.org/plt/0252207e38..33f3574f7e

=[ 4 Commits ]==
Directory summary:
   10.7% collects/racket/
   39.3% collects/unstable/scribblings/
   44.7% collects/unstable/
5.1% collects/

~~
[...]


_
 Racket Developers list:
 http://lists.racket-lang.org/dev

Re: [racket-dev] define #lang and use in same file (+ some funny behavior)

[Stephen Chang]

Ah, thanks! I somehow missed the existence of the submod form.



On Mon, Dec 3, 2012 at 4:44 PM, Carl Eastlund  wrote:
> You can't define the reader part of a #lang in the same file and then use
> it, but you can definitely define the bindings, from things like
> #%module-begin and #%app on up, and use it in the same file.  The following
> works fine for me:
>
> #lang racket
>
> (module greeting racket
>   (provide #%module-begin)
>   (require (for-syntax syntax/parse))
>   (define-syntax #%module-begin
> (syntax-parser
>   [(_ one:str two:str)
>#'(#%plain-module-begin
>(printf "Hello, ~a!\n" 'one)
>(printf "Salutations, ~a!\n" 'two))])))
>
> (module main (submod ".." greeting)
>   "Santa Claus"
>   "Easter Bunny")
>
>
> Carl Eastlund
>
>
> On Mon, Dec 3, 2012 at 3:46 PM, Stephen Chang  wrote:
>>
>> When doing quick experiments, I've often wanted the ability to define
>> a new language and then use it in the same file. I thought submodules
>> might allow me to do this, but I've realized that I still can't.
>> However, in trying to do so I ran into some interesting behavior:
>>
>>
>> Here is a file named test.rkt that is saved to disk:
>>
>> #lang racket
>>
>> (define-syntax-rule (app x ...) (begin (displayln "hello") (#%app x
>> ...)))
>>
>> (provide (except-out (all-from-out racket) #%app)
>>  (rename-out [app #%app]))
>>
>>
>> If I then add to the same file:
>>
>> (module test "test.rkt" (+ 1 2))
>>
>> but *don't* save, pressing run in drracket displays:
>>
>> Welcome to DrRacket, version 5.3.1.1 [3m].
>> Language: racket [custom].
>> hello
>> 3
>> >
>>
>>
>> Then if I save and run again, I get an error:
>>
>> Welcome to DrRacket, version 5.3.1.1 [3m].
>> Language: racket [custom].
>> . . ../plt/collects/compiler/cm.rkt:350:0:
>> standard-module-name-resolver: cycle in loading
>>   at path: /home/stchang/tmp/test.rkt
>>   paths:
>>/home/stchang/tmp/test.rkt
>> >
>>
>>
>> Is this the intended behavior? I guess this is what I would expect but
>> it feels funny since an identical buffer produces different output
>> depending on it's saved status.
>> _
>>   Racket Developers list:
>>   http://lists.racket-lang.org/dev
>>
>
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] define #lang and use in same file (+ some funny behavior)

[Carl Eastlund]

You can't define the reader part of a #lang in the same file and then use
it, but you can definitely define the bindings, from things like
#%module-begin and #%app on up, and use it in the same file.  The following
works fine for me:

#lang racket

(module greeting racket
  (provide #%module-begin)
  (require (for-syntax syntax/parse))
  (define-syntax #%module-begin
(syntax-parser
  [(_ one:str two:str)
   #'(#%plain-module-begin
   (printf "Hello, ~a!\n" 'one)
   (printf "Salutations, ~a!\n" 'two))])))

(module main (submod ".." greeting)
  "Santa Claus"
  "Easter Bunny")


Carl Eastlund


On Mon, Dec 3, 2012 at 3:46 PM, Stephen Chang  wrote:

> When doing quick experiments, I've often wanted the ability to define
> a new language and then use it in the same file. I thought submodules
> might allow me to do this, but I've realized that I still can't.
> However, in trying to do so I ran into some interesting behavior:
>
>
> Here is a file named test.rkt that is saved to disk:
>
> #lang racket
>
> (define-syntax-rule (app x ...) (begin (displayln "hello") (#%app x
> ...)))
>
> (provide (except-out (all-from-out racket) #%app)
>  (rename-out [app #%app]))
>
>
> If I then add to the same file:
>
> (module test "test.rkt" (+ 1 2))
>
> but *don't* save, pressing run in drracket displays:
>
> Welcome to DrRacket, version 5.3.1.1 [3m].
> Language: racket [custom].
> hello
> 3
> >
>
>
> Then if I save and run again, I get an error:
>
> Welcome to DrRacket, version 5.3.1.1 [3m].
> Language: racket [custom].
> . . ../plt/collects/compiler/cm.rkt:350:0:
> standard-module-name-resolver: cycle in loading
>   at path: /home/stchang/tmp/test.rkt
>   paths:
>/home/stchang/tmp/test.rkt
> >
>
>
> Is this the intended behavior? I guess this is what I would expect but
> it feels funny since an identical buffer produces different output
> depending on it's saved status.
> _
>   Racket Developers list:
>   http://lists.racket-lang.org/dev
>
>
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] Planet 2 package names

[Danny Yoo]

> Planet 1 packages and Racket collections also don't use A-Z (with one or
> two exceptions on Planet).  In theory, we could disallow those characters
> and gain compatibility with case-insensitive mediums such as the default
> Mac OS X filesystem.  I'm not particularly attached to this second
> proposal, but since it leads to potential errors I thought it was worth
> bringing up.  If we leave both cases in, we at least need to make sure
> programs that run on Windows and Unix don't break on Mac due to package
> names conflicting on the filesystem.
>
>
One of those exceptions is probably UPPERCASE.plt,


http://planet.racket-lang.org/display.ss?package=UPPERCASE.plt&owner=dyoo

but as it's intended only as a joke, please ignore it.  :)
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

[racket-dev] define #lang and use in same file (+ some funny behavior)

[Stephen Chang]

When doing quick experiments, I've often wanted the ability to define
a new language and then use it in the same file. I thought submodules
might allow me to do this, but I've realized that I still can't.
However, in trying to do so I ran into some interesting behavior:


Here is a file named test.rkt that is saved to disk:

#lang racket

(define-syntax-rule (app x ...) (begin (displayln "hello") (#%app x ...)))

(provide (except-out (all-from-out racket) #%app)
 (rename-out [app #%app]))


If I then add to the same file:

(module test "test.rkt" (+ 1 2))

but *don't* save, pressing run in drracket displays:

Welcome to DrRacket, version 5.3.1.1 [3m].
Language: racket [custom].
hello
3
>


Then if I save and run again, I get an error:

Welcome to DrRacket, version 5.3.1.1 [3m].
Language: racket [custom].
. . ../plt/collects/compiler/cm.rkt:350:0:
standard-module-name-resolver: cycle in loading
  at path: /home/stchang/tmp/test.rkt
  paths:
   /home/stchang/tmp/test.rkt
>


Is this the intended behavior? I guess this is what I would expect but
it feels funny since an identical buffer produces different output
depending on it's saved status.
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] Planet 2 package names

[Eli Barzilay]

A few minutes ago, Jay McCarthy wrote:
> On the second point, I think it may be a good idea to disallow upper
> case.
> 
> I presume that Eli and Matthew would have good recommendations about
> this?

I'd defer to whatever happens with relative string paths (and I prefer
seeing the explanation phrased in those terms to make these kind of
confusions less problematic).  But one thing to note is that Windows
is even more likely to be just case-preserving (not -sensitive) than
OSX, so this is a real problem...

-- 
  ((lambda (x) (x x)) (lambda (x) (x x)))  Eli Barzilay:
http://barzilay.org/   Maze is Life!
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] Planet 2 package names

[Jay McCarthy]

On the second point, I think it may be a good idea to disallow upper case.

I presume that Eli and Matthew would have good recommendations about this?

Jay

On Mon, Dec 3, 2012 at 1:13 PM, Carl Eastlund  wrote:
> Yay!
>
> Carl Eastlund
>
>
>
> On Mon, Dec 3, 2012 at 3:10 PM, Jay McCarthy  wrote:
>>
>> That's a typo in the docs. Numbers are allowed.
>>
>> Jay
>>
>> On Mon, Dec 3, 2012 at 1:03 PM, Carl Eastlund  wrote:
>> > Currently (the latest nightly build), the names of Planet 2 packages are
>> > restricted to use only the characters a-z, A-Z, _, and -.  The current
>> > set
>> > of Racket collections, on the other hand, uses a-z, 0-9, and -.  Planet
>> > 1
>> > also has several packages using 0-9.  We need to add digits to the set
>> > of
>> > valid names if we want 2htdp, algol60, planet2 (itself!), r5rs, and r6rs
>> > to
>> > be "package-friendly".  I know if I were to release a package named
>> > after
>> > ACL2 somehow, I'd be hard-pressed to figure out what to do with the "2".
>> >
>> > Planet 1 packages and Racket collections also don't use A-Z (with one or
>> > two
>> > exceptions on Planet).  In theory, we could disallow those characters
>> > and
>> > gain compatibility with case-insensitive mediums such as the default Mac
>> > OS
>> > X filesystem.  I'm not particularly attached to this second proposal,
>> > but
>> > since it leads to potential errors I thought it was worth bringing up.
>> > If
>> > we leave both cases in, we at least need to make sure programs that run
>> > on
>> > Windows and Unix don't break on Mac due to package names conflicting on
>> > the
>> > filesystem.
>> >
>> > Carl Eastlund
>> >
>> > _
>> >   Racket Developers list:
>> >   http://lists.racket-lang.org/dev
>> >
>>
>>
>>
>> --
>> Jay McCarthy 
>> Assistant Professor / Brigham Young University
>> http://faculty.cs.byu.edu/~jay
>>
>> "The glory of God is Intelligence" - D&C 93
>>
>



-- 
Jay McCarthy 
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] Planet 2 package names

[Carl Eastlund]

Yay!

Carl Eastlund


On Mon, Dec 3, 2012 at 3:10 PM, Jay McCarthy  wrote:

> That's a typo in the docs. Numbers are allowed.
>
> Jay
>
> On Mon, Dec 3, 2012 at 1:03 PM, Carl Eastlund  wrote:
> > Currently (the latest nightly build), the names of Planet 2 packages are
> > restricted to use only the characters a-z, A-Z, _, and -.  The current
> set
> > of Racket collections, on the other hand, uses a-z, 0-9, and -.  Planet 1
> > also has several packages using 0-9.  We need to add digits to the set of
> > valid names if we want 2htdp, algol60, planet2 (itself!), r5rs, and r6rs
> to
> > be "package-friendly".  I know if I were to release a package named after
> > ACL2 somehow, I'd be hard-pressed to figure out what to do with the "2".
> >
> > Planet 1 packages and Racket collections also don't use A-Z (with one or
> two
> > exceptions on Planet).  In theory, we could disallow those characters and
> > gain compatibility with case-insensitive mediums such as the default Mac
> OS
> > X filesystem.  I'm not particularly attached to this second proposal, but
> > since it leads to potential errors I thought it was worth bringing up.
>  If
> > we leave both cases in, we at least need to make sure programs that run
> on
> > Windows and Unix don't break on Mac due to package names conflicting on
> the
> > filesystem.
> >
> > Carl Eastlund
> >
> > _
> >   Racket Developers list:
> >   http://lists.racket-lang.org/dev
> >
>
>
>
> --
> Jay McCarthy 
> Assistant Professor / Brigham Young University
> http://faculty.cs.byu.edu/~jay
>
> "The glory of God is Intelligence" - D&C 93
>
>
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] Planet 2 package names

[Jay McCarthy]

That's a typo in the docs. Numbers are allowed.

Jay

On Mon, Dec 3, 2012 at 1:03 PM, Carl Eastlund  wrote:
> Currently (the latest nightly build), the names of Planet 2 packages are
> restricted to use only the characters a-z, A-Z, _, and -.  The current set
> of Racket collections, on the other hand, uses a-z, 0-9, and -.  Planet 1
> also has several packages using 0-9.  We need to add digits to the set of
> valid names if we want 2htdp, algol60, planet2 (itself!), r5rs, and r6rs to
> be "package-friendly".  I know if I were to release a package named after
> ACL2 somehow, I'd be hard-pressed to figure out what to do with the "2".
>
> Planet 1 packages and Racket collections also don't use A-Z (with one or two
> exceptions on Planet).  In theory, we could disallow those characters and
> gain compatibility with case-insensitive mediums such as the default Mac OS
> X filesystem.  I'm not particularly attached to this second proposal, but
> since it leads to potential errors I thought it was worth bringing up.  If
> we leave both cases in, we at least need to make sure programs that run on
> Windows and Unix don't break on Mac due to package names conflicting on the
> filesystem.
>
> Carl Eastlund
>
> _
>   Racket Developers list:
>   http://lists.racket-lang.org/dev
>



-- 
Jay McCarthy 
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

[racket-dev] Planet 2 package names

[Carl Eastlund]

Currently (the latest nightly build), the names of Planet 2 packages are
restricted to use only the characters a-z, A-Z, _, and -.  The current set
of Racket collections, on the other hand, uses a-z, 0-9, and -.  Planet 1
also has several packages using 0-9.  We need to add digits to the set of
valid names if we want 2htdp, algol60, planet2 (itself!), r5rs, and r6rs to
be "package-friendly".  I know if I were to release a package named after
ACL2 somehow, I'd be hard-pressed to figure out what to do with the "2".

Planet 1 packages and Racket collections also don't use A-Z (with one or
two exceptions on Planet).  In theory, we could disallow those characters
and gain compatibility with case-insensitive mediums such as the default
Mac OS X filesystem.  I'm not particularly attached to this second
proposal, but since it leads to potential errors I thought it was worth
bringing up.  If we leave both cases in, we at least need to make sure
programs that run on Windows and Unix don't break on Mac due to package
names conflicting on the filesystem.

Carl Eastlund
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] [plt] Push #25821: master branch updated

[Neil Toronto]

If `build-array' gets #f, it'll raise a contract error. I could get 
around that by keeping another reference to the procedure that I set to 
#f later, but I'm going to try Matthew's suggestion first.


On 12/03/2012 09:47 AM, Robby Findler wrote:

In the test, if you get unlucky and there is a gc between the
definition of bx and unboxing of it then this test will pass in
correctly, I think (unless build-array complains if it gets #f?)

On Sun, Dec 2, 2012 at 11:44 PM,   wrote:

ntoronto has updated `master' from 325600b0cf to 8f17913d55.
   http://git.racket-lang.org/plt/325600b0cf..8f17913d55

=[ One Commit ]=
Directory summary:
   74.0% collects/math/private/array/
   25.9% collects/math/tests/

~~

8f17913 Neil Toronto  2012-12-02 19:02
:
| Fixed memory leak in making arrays strict: doing so wouldn't clear
| the reference to the original procedure, which itself could hold on
| to a lot of memory
:
   A collects/math/tests/strictness-memory-leak-test.rkt
   M .../math/private/array/typed-array-struct.rkt | 28 

=[ Overall Diff ]===

collects/math/private/array/typed-array-struct.rkt
~~
--- OLD/collects/math/private/array/typed-array-struct.rkt
+++ NEW/collects/math/private/array/typed-array-struct.rkt
@@ -76,17 +76,23 @@

  (: unsafe-build-array (All (A) (Indexes (Indexes -> A) -> (Array A
  (define (unsafe-build-array ds f)
-  (define size (check-array-shape-size 'unsafe-build-array ds))
-  (define: data : (U #f (Vectorof A)) #f)
-  (define (strict!)
-(set! data (inline-build-array-data ds (λ (js j) (f js)) A)))
-  (define unsafe-proc
-(λ: ([js : Indexes])
-  (let ([data data])
-(if data
-(unsafe-vector-ref data (unsafe-array-index->value-index ds js))
-(f js)
-  (Array ds size ((inst box Boolean) #f) strict! unsafe-proc))
+  ;; This box's contents get replaced when the array we're constructing is 
made strict, so that
+  ;; the array stops referencing f. If we didn't do this, long chains of array 
computations would
+  ;; keep hold of references to all the intermediate procs, which is a memory 
leak.
+  (let ([f  (box f)])
+(define size (check-array-shape-size 'unsafe-build-array ds))
+;; Sharp readers might notice that strict! doesn't check to see whether 
the array is already
+;; strict; that's okay - array-strict! does it instead, which makes the 
"once strict, always
+;; strict" invariant easier to ensure in subtypes, which we don't always 
have control over
+(define (strict!)
+  (let* ([old-f  (unbox f)]
+ [vs (inline-build-array-data ds (λ (js j) (old-f js)) A)])
+;; Make a new f that just indexes into vs
+(set-box! f (λ: ([js : Indexes])
+  (unsafe-vector-ref vs (unsafe-array-index->value-index 
ds js))
+(define unsafe-proc
+  (λ: ([js : Indexes]) ((unbox f) js)))
+(Array ds size ((inst box Boolean) #f) strict! unsafe-proc)))

  (: unsafe-build-strict-array (All (A) (Indexes (Indexes -> A) -> (Array A
  (define (unsafe-build-strict-array ds f)

collects/math/tests/strictness-memory-leak-test.rkt
~~~
--- /dev/null
+++ NEW/collects/math/tests/strictness-memory-leak-test.rkt
@@ -0,0 +1,16 @@
+#lang racket
+
+(require math/array
+ rackunit)
+
+;; Make a procedure that returns a random value to keep the optimizer from 
converting it to a
+;; top-level, non-closure; if that happens, the module keeps a reference to 
it, which makes this
+;; test always fail
+(define bx (make-weak-box (let ([v  (random)]) (λ (js) v
+
+(define arr (build-array #() (weak-box-value bx)))
+
+;; Making `arr' strict should release the only remaining reference to the 
contents of `bx'
+(array-strict! arr)
+(collect-garbage)
+(check-false (weak-box-value bx))


_
 Racket Developers list:
 http://lists.racket-lang.org/dev

Re: [racket-dev] FFI: pointer to an array in a C struct type

[Neil Toronto]

On 12/03/2012 12:10 PM, Matthew Flatt wrote:

At Mon, 03 Dec 2012 11:05:10 -0700, Neil Toronto wrote:

On 12/03/2012 07:31 AM, Matthew Flatt wrote:

Neil, can you say more about how `_mpz' instances are used with foreign
functions?


They represent GMP's bignums, and they're used as both input and output
arguments.

When used as input arguments, GMP's functions never mutate them. It
should be safe to pass an `mpz' that contains a pointer to memory that
Racket manages, as long as it doesn't get relocated during the function
call. In this case, I need to be able to set the `limbs' array's
elements in Racket code.

When used as output arguments, GMP expects to be able to free the array
pointed at by the `limbs' field, allocate a new array for the result,
and set the `limbs' field to point to it. I use the GMP function
"mpz_init" to initialize the fields of an `mpz' instance before using it
as an output argument. In this case, I need to be able to read the
`limbs' array's elements in Racket code.


Overall, it sounds to me like you should `malloc' a `limbs' array in
'raw mode, use `free' to free it, use `_pointer' as the ctype for the
`limbs' field, and so on. If you need something like finalization, use
`ffi/unsafe/alloc' to make sure that a freeing function for a `_mpz' is
paired with an `_mpz' allocation.


That's what I've had in the past, except I used 'atomic-interior instead 
of 'raw. I also had the `_mpz' instances allocated using 
'atomic-interior. IIRC, the limbs didn't survive a GC. Now that I think 
about it, that was probably because the `_mpz' instances were allocated 
using "atomic"...


I've been avoiding 'raw generally, in case the foreign libraries have 
their own allocation schemes. MPFR has its own special malloc and free 
for character strings, for example. It's weird.


But I've just discovered mpz_init2, which allocates enough limbs to 
store an n-bit number. Between that and mpz_clear, I've basically got 
GMP's malloc and free. This is my code now:



(define-cstruct _mpz ([alloc _int] [size _int] [limbs _pointer]))

(define mpz-init
 (get-gmp-fun '__gmpz_init (_fun _mpz-pointer -> _void)))
(define mpz-init2
 (get-gmp-fun '__gmpz_init2 (_fun _mpz-pointer _ulong -> _void)))
(define mpz-clear
 (get-gmp-fun '__gmpz_clear (_fun _mpz-pointer -> _void)))

(define (new-mpz [bits 0])
  (define z (ptr-ref (malloc _mpz 'atomic-interior) _mpz))
  (if (= bits 0) (mpz-init z) (mpz-init2 z bits))
  (register-finalizer z mpz-clear)
  z)


I think that return values from `new-mpz' won't be relocated (interior) 
and won't be traced (atomic). I've verified that the limbs are freed 
when their `_mpz' instances are collected, by allocating one with 200MB 
of limbs, filling the limbs to page them in, getting the `_mpz' instance 
collected, and watching Racket's memory usage drop.


The downside to this is that Racket doesn't know about the memory GMP 
manages. The 200MB of limbs only showed up in my system monitor, not in 
DrRacket's memory indicator. I'm only using them for temporary values, 
though.


(FWIW, I've managed to get bigfloats' limbs managed by Racket, using 
(_gcable _cvector) and malloc with 'atomic-interior. I never need to set 
or ref those limbs in Racket code.)



Just to me clear, even if it somehow worked to use `_gcpointer' as the
ctype of a field in an `_mpz' that you allocate, that would be the
wrong ctype for an `_mpz' that was filled in by `mpz_init' or updated
as a result.


Gotcha. If I wanted to have Racket manage an `_mpz' instance's limbs, 
then, I'd need a different ctype for that.


Neil ⊥

_
 Racket Developers list:
 http://lists.racket-lang.org/dev

Re: [racket-dev] FFI: pointer to an array in a C struct type

[Matthew Flatt]

At Mon, 03 Dec 2012 11:05:10 -0700, Neil Toronto wrote:
> On 12/03/2012 07:31 AM, Matthew Flatt wrote:
> > At Mon, 3 Dec 2012 12:31:37 +0100, Tobias Hammer wrote:
> >> On Mon, 03 Dec 2012 11:45:08 +0100, Neil Toronto 
> >> wrote:
> >>
> >>> This error seems wrong:
> >>>
> >>>
> >>> #lang racket
> >>>
> >>> (require ffi/unsafe
> >>>ffi/unsafe/cvector)
> >>>
> >>> (define-cstruct _mpz ([alloc _int]
> >>> [size _int]
> >>> [limbs (_gcable _cvector)]))
> >>>
> >>> (define z (make-mpz 1 1 (list->cvector '(1) _long)))
> >>> (mpz-limbs z)
> >>>
> >>>   >>> _cvector: cannot automatically convert a C pointer to a cvector
> >>>
> >>>
> >>> The error might be correct, though late and confusing, if a cvector's
> >>> "base representation" isn't a pointer type. Is it?
> >>
> >> The base representation should be a pointer but the length and type
> >> information to convert it back to a cvector is missing. Whats stored
> >> inside the struct is only a plain pointer, without the information you
> >> supplied to make-mpz, and because of this _cvector only has a
> >> racket->c-conversion (see ffi/unsafe/cvector.rkt).
> >>
> >>> If that error is correct, then how are FFI users meant to define a C
> >>> struct that has a "long *" field that points to an array? I've not yet
> >>> managed to define one whose instances survive a garbage collection cycle
> >>> without using (_gcable _cvector). Here's one of my desperate attempts:
> >>>
> >>>
> >>> #lang racket
> >>>
> >>> (require ffi/unsafe
> >>>ffi/unsafe/cvector)
> >>>
> >>> (define-cstruct _mpz ([alloc _int] [size _int] [limbs _gcpointer]))
> >>>
> >>> (define z (make-mpz 1 1 (cvector-ptr (list->cvector '(1) _long
> >>>
> >>>   > (ptr-ref (mpz-limbs z) _long 0)
> >>> 1
> >>>   > (collect-garbage)
> >>>   > (ptr-ref (mpz-limbs z) _long 0)
> >>> 139856348920568
> >>>
> >>>
> >>> I mean to be complain-y this time. This shouldn't be this hard to figure
> >>> out.
> >>
> >> I guess the problem here is, that the gc only knows memory with only
> >> pointers (to gcable memory) inside and (atomic) memory without any
> >> pointers it should follow. You try to mix them up. make-mpz should malloc
> >> (see ffi docs) atomic memory and you put a pointer managed by the gc into
> >> it. The pointer gets relocated on gc, but the ref inside the struct is not
> >> known and not updated.
> >>
> >> I unfortunately don't know a simple, gc- and typesafe way to put pointers
> >> inside structs. My attempts to get around this were pretty hacking (using
> >> a raw malloc'd pointer, embed a struct with plain values as pointer inside
> >> an interor-malloc'd struct together with all others pointers, etc).
> >> I would be really interested in a way to accomplish this, too.
> >
> > Yes. I've thought about this, but for the libraries I've written, the
> > cases where the GC can really solve the problem automatically seem
> > relatively rare, so I haven't been sufficiently motivated for my own
> > uses.
> >
> > In the `_mpz' example above, the presence of both `alloc' and `size'
> > suggests that a callee might be responsible for malloc()ing `limbs' if
> > there's not already enough room. If so, `_gcpointer' isn't the right
> > type. Maybe this example doesn't actually go that way, but it's a
> > typical problem.
> >
> > Neil, can you say more about how `_mpz' instances are used with foreign
> > functions?
> 
> They represent GMP's bignums, and they're used as both input and output 
> arguments.
> 
> When used as input arguments, GMP's functions never mutate them. It 
> should be safe to pass an `mpz' that contains a pointer to memory that 
> Racket manages, as long as it doesn't get relocated during the function 
> call. In this case, I need to be able to set the `limbs' array's 
> elements in Racket code.
> 
> When used as output arguments, GMP expects to be able to free the array 
> pointed at by the `limbs' field, allocate a new array for the result, 
> and set the `limbs' field to point to it. I use the GMP function 
> "mpz_init" to initialize the fields of an `mpz' instance before using it 
> as an output argument. In this case, I need to be able to read the 
> `limbs' array's elements in Racket code.

Overall, it sounds to me like you should `malloc' a `limbs' array in
'raw mode, use `free' to free it, use `_pointer' as the ctype for the
`limbs' field, and so on. If you need something like finalization, use
`ffi/unsafe/alloc' to make sure that a freeing function for a `_mpz' is
paired with an `_mpz' allocation.

Just to me clear, even if it somehow worked to use `_gcpointer' as the
ctype of a field in an `_mpz' that you allocate, that would be the
wrong ctype for an `_mpz' that was filled in by `mpz_init' or updated
as a result.

_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] [plt] Push #25824: master branch updated

[Neil Toronto]

On 12/03/2012 10:42 AM, mfl...@racket-lang.org wrote:


492167c Kevin Tew  2012-11-29 05:27
:
| read and write support for fxvectors and flvectors


+10

Neil ⊥

_
 Racket Developers list:
 http://lists.racket-lang.org/dev

Re: [racket-dev] FFI: pointer to an array in a C struct type

[Neil Toronto]

On 12/03/2012 07:31 AM, Matthew Flatt wrote:

At Mon, 3 Dec 2012 12:31:37 +0100, Tobias Hammer wrote:

On Mon, 03 Dec 2012 11:45:08 +0100, Neil Toronto 
wrote:


This error seems wrong:


#lang racket

(require ffi/unsafe
   ffi/unsafe/cvector)

(define-cstruct _mpz ([alloc _int]
[size _int]
[limbs (_gcable _cvector)]))

(define z (make-mpz 1 1 (list->cvector '(1) _long)))
(mpz-limbs z)

  >>> _cvector: cannot automatically convert a C pointer to a cvector


The error might be correct, though late and confusing, if a cvector's
"base representation" isn't a pointer type. Is it?


The base representation should be a pointer but the length and type
information to convert it back to a cvector is missing. Whats stored
inside the struct is only a plain pointer, without the information you
supplied to make-mpz, and because of this _cvector only has a
racket->c-conversion (see ffi/unsafe/cvector.rkt).


If that error is correct, then how are FFI users meant to define a C
struct that has a "long *" field that points to an array? I've not yet
managed to define one whose instances survive a garbage collection cycle
without using (_gcable _cvector). Here's one of my desperate attempts:


#lang racket

(require ffi/unsafe
   ffi/unsafe/cvector)

(define-cstruct _mpz ([alloc _int] [size _int] [limbs _gcpointer]))

(define z (make-mpz 1 1 (cvector-ptr (list->cvector '(1) _long

  > (ptr-ref (mpz-limbs z) _long 0)
1
  > (collect-garbage)
  > (ptr-ref (mpz-limbs z) _long 0)
139856348920568


I mean to be complain-y this time. This shouldn't be this hard to figure
out.


I guess the problem here is, that the gc only knows memory with only
pointers (to gcable memory) inside and (atomic) memory without any
pointers it should follow. You try to mix them up. make-mpz should malloc
(see ffi docs) atomic memory and you put a pointer managed by the gc into
it. The pointer gets relocated on gc, but the ref inside the struct is not
known and not updated.

I unfortunately don't know a simple, gc- and typesafe way to put pointers
inside structs. My attempts to get around this were pretty hacking (using
a raw malloc'd pointer, embed a struct with plain values as pointer inside
an interor-malloc'd struct together with all others pointers, etc).
I would be really interested in a way to accomplish this, too.


Yes. I've thought about this, but for the libraries I've written, the
cases where the GC can really solve the problem automatically seem
relatively rare, so I haven't been sufficiently motivated for my own
uses.

In the `_mpz' example above, the presence of both `alloc' and `size'
suggests that a callee might be responsible for malloc()ing `limbs' if
there's not already enough room. If so, `_gcpointer' isn't the right
type. Maybe this example doesn't actually go that way, but it's a
typical problem.

Neil, can you say more about how `_mpz' instances are used with foreign
functions?


They represent GMP's bignums, and they're used as both input and output 
arguments.


When used as input arguments, GMP's functions never mutate them. It 
should be safe to pass an `mpz' that contains a pointer to memory that 
Racket manages, as long as it doesn't get relocated during the function 
call. In this case, I need to be able to set the `limbs' array's 
elements in Racket code.


When used as output arguments, GMP expects to be able to free the array 
pointed at by the `limbs' field, allocate a new array for the result, 
and set the `limbs' field to point to it. I use the GMP function 
"mpz_init" to initialize the fields of an `mpz' instance before using it 
as an output argument. In this case, I need to be able to read the 
`limbs' array's elements in Racket code.


I think that's everything.

Neil ⊥

_
 Racket Developers list:
 http://lists.racket-lang.org/dev

Re: [racket-dev] [plt] Push #25821: master branch updated

[Robby Findler]

In the test, if you get unlucky and there is a gc between the
definition of bx and unboxing of it then this test will pass in
correctly, I think (unless build-array complains if it gets #f?)

On Sun, Dec 2, 2012 at 11:44 PM,   wrote:
> ntoronto has updated `master' from 325600b0cf to 8f17913d55.
>   http://git.racket-lang.org/plt/325600b0cf..8f17913d55
>
> =[ One Commit ]=
> Directory summary:
>   74.0% collects/math/private/array/
>   25.9% collects/math/tests/
>
> ~~
>
> 8f17913 Neil Toronto  2012-12-02 19:02
> :
> | Fixed memory leak in making arrays strict: doing so wouldn't clear
> | the reference to the original procedure, which itself could hold on
> | to a lot of memory
> :
>   A collects/math/tests/strictness-memory-leak-test.rkt
>   M .../math/private/array/typed-array-struct.rkt | 28 
> 
>
> =[ Overall Diff ]===
>
> collects/math/private/array/typed-array-struct.rkt
> ~~
> --- OLD/collects/math/private/array/typed-array-struct.rkt
> +++ NEW/collects/math/private/array/typed-array-struct.rkt
> @@ -76,17 +76,23 @@
>
>  (: unsafe-build-array (All (A) (Indexes (Indexes -> A) -> (Array A
>  (define (unsafe-build-array ds f)
> -  (define size (check-array-shape-size 'unsafe-build-array ds))
> -  (define: data : (U #f (Vectorof A)) #f)
> -  (define (strict!)
> -(set! data (inline-build-array-data ds (λ (js j) (f js)) A)))
> -  (define unsafe-proc
> -(λ: ([js : Indexes])
> -  (let ([data data])
> -(if data
> -(unsafe-vector-ref data (unsafe-array-index->value-index ds js))
> -(f js)
> -  (Array ds size ((inst box Boolean) #f) strict! unsafe-proc))
> +  ;; This box's contents get replaced when the array we're constructing is 
> made strict, so that
> +  ;; the array stops referencing f. If we didn't do this, long chains of 
> array computations would
> +  ;; keep hold of references to all the intermediate procs, which is a 
> memory leak.
> +  (let ([f  (box f)])
> +(define size (check-array-shape-size 'unsafe-build-array ds))
> +;; Sharp readers might notice that strict! doesn't check to see whether 
> the array is already
> +;; strict; that's okay - array-strict! does it instead, which makes the 
> "once strict, always
> +;; strict" invariant easier to ensure in subtypes, which we don't always 
> have control over
> +(define (strict!)
> +  (let* ([old-f  (unbox f)]
> + [vs (inline-build-array-data ds (λ (js j) (old-f js)) A)])
> +;; Make a new f that just indexes into vs
> +(set-box! f (λ: ([js : Indexes])
> +  (unsafe-vector-ref vs (unsafe-array-index->value-index 
> ds js))
> +(define unsafe-proc
> +  (λ: ([js : Indexes]) ((unbox f) js)))
> +(Array ds size ((inst box Boolean) #f) strict! unsafe-proc)))
>
>  (: unsafe-build-strict-array (All (A) (Indexes (Indexes -> A) -> (Array A
>  (define (unsafe-build-strict-array ds f)
>
> collects/math/tests/strictness-memory-leak-test.rkt
> ~~~
> --- /dev/null
> +++ NEW/collects/math/tests/strictness-memory-leak-test.rkt
> @@ -0,0 +1,16 @@
> +#lang racket
> +
> +(require math/array
> + rackunit)
> +
> +;; Make a procedure that returns a random value to keep the optimizer from 
> converting it to a
> +;; top-level, non-closure; if that happens, the module keeps a reference to 
> it, which makes this
> +;; test always fail
> +(define bx (make-weak-box (let ([v  (random)]) (λ (js) v
> +
> +(define arr (build-array #() (weak-box-value bx)))
> +
> +;; Making `arr' strict should release the only remaining reference to the 
> contents of `bx'
> +(array-strict! arr)
> +(collect-garbage)
> +(check-false (weak-box-value bx))

_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] Testing whether a procedure gets collected

[Robby Findler]

Thanks. Is the N suggestion a future-proofing kind of a thing, or is
there something today that could cause such a test to pass where a
single one might fail?

Robby

On Mon, Dec 3, 2012 at 10:42 AM, Matthew Flatt  wrote:
> This guide material (as opposed to language specification and
> guarantees) looks pretty good to me. I'll edit and add the suggestion
> of N allocations.
>
> At Mon, 3 Dec 2012 10:39:09 -0600, Robby Findler wrote:
>> Let me also say that I think it is important to give advice on how to
>> test so I think we need to say something.
>>
>> Robby
>>
>> On Mon, Dec 3, 2012 at 10:37 AM, Robby Findler
>>  wrote:
>> > On Mon, Dec 3, 2012 at 7:47 AM, Matthew Flatt  wrote:
>> >> At Mon, 3 Dec 2012 08:04:15 -0500, Sam Tobin-Hochstadt wrote:
>> >>> On Mon, Dec 3, 2012 at 7:54 AM, Robby Findler
>> >>>  wrote:
>> >>> >
>> >>> > I agree that when something is collected is a pretty intentional
>> >>> > property but I think it is possible to say a little bit more since
>> >>> > there is a pretty stable core idea there (namely that if something
>> >>> > isn't reachable and you call collect-garbage you can be pretty sure
>> >>> > it'll be gone -- this was not the case back in the boehm gc days).
>> >>>
>> >>> Do we really want to commit to this?  I agree that it's true for a
>> >>> simple 2-generation GC, but something more complex like the Sun
>> >>> garbage-first collector or Felix Klock's regional collector wouldn't
>> >>> have this property, let alone something more complex like stack
>> >>> allocation for non-escaping data.
>> >>
>> >> Yes, I think that a guarantee about individual objects with respect to
>> >> `collect-garbage' is going to be too strong, because the garbage
>> >> collector is meant to provide asymptotic guarantees instead of
>> >> individual-object guarantees.
>> >>
>> >> I think tests like Neil's are best written to allocate N things and
>> >> check that a good fraction of the N are released by a garbage
>> >> collection. In Neil's specific example, I'd probably write the test to
>> >> allocate 100 or 1000 of them, each with its own weak box, and make sure
>> >> that at most of the weak boxes are empty after a forced collection.
>> >
>> > Ah, whoops. I went and wrote a section to try to explain this
>> > (assuming the current tech) before reading this message.
>> >
>> > It is here:
>> >
>> >
>> http://git.racket-lang.org/plt/blob/280d924349c2af595af307d38ffdb24940ede80a:/co
>> llects/scribblings/guide/performance.scrbl#l417
>> >
>> > My initial reaction to the above (perhaps colored by the fact that I
>> > just wrote that section) is to still explain the current state, and to
>> > revise this explanation when things improve (and, perhaps, explain in
>> > the current version that things might improve in the future).
>> >
>> > Or maybe does someone else want to try to edit my prose to take this
>> > into account?
>> >
>> > Robby
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] Testing whether a procedure gets collected

[Matthew Flatt]

This guide material (as opposed to language specification and
guarantees) looks pretty good to me. I'll edit and add the suggestion
of N allocations.

At Mon, 3 Dec 2012 10:39:09 -0600, Robby Findler wrote:
> Let me also say that I think it is important to give advice on how to
> test so I think we need to say something.
> 
> Robby
> 
> On Mon, Dec 3, 2012 at 10:37 AM, Robby Findler
>  wrote:
> > On Mon, Dec 3, 2012 at 7:47 AM, Matthew Flatt  wrote:
> >> At Mon, 3 Dec 2012 08:04:15 -0500, Sam Tobin-Hochstadt wrote:
> >>> On Mon, Dec 3, 2012 at 7:54 AM, Robby Findler
> >>>  wrote:
> >>> >
> >>> > I agree that when something is collected is a pretty intentional
> >>> > property but I think it is possible to say a little bit more since
> >>> > there is a pretty stable core idea there (namely that if something
> >>> > isn't reachable and you call collect-garbage you can be pretty sure
> >>> > it'll be gone -- this was not the case back in the boehm gc days).
> >>>
> >>> Do we really want to commit to this?  I agree that it's true for a
> >>> simple 2-generation GC, but something more complex like the Sun
> >>> garbage-first collector or Felix Klock's regional collector wouldn't
> >>> have this property, let alone something more complex like stack
> >>> allocation for non-escaping data.
> >>
> >> Yes, I think that a guarantee about individual objects with respect to
> >> `collect-garbage' is going to be too strong, because the garbage
> >> collector is meant to provide asymptotic guarantees instead of
> >> individual-object guarantees.
> >>
> >> I think tests like Neil's are best written to allocate N things and
> >> check that a good fraction of the N are released by a garbage
> >> collection. In Neil's specific example, I'd probably write the test to
> >> allocate 100 or 1000 of them, each with its own weak box, and make sure
> >> that at most of the weak boxes are empty after a forced collection.
> >
> > Ah, whoops. I went and wrote a section to try to explain this
> > (assuming the current tech) before reading this message.
> >
> > It is here:
> >
> > 
> http://git.racket-lang.org/plt/blob/280d924349c2af595af307d38ffdb24940ede80a:/co
> llects/scribblings/guide/performance.scrbl#l417
> >
> > My initial reaction to the above (perhaps colored by the fact that I
> > just wrote that section) is to still explain the current state, and to
> > revise this explanation when things improve (and, perhaps, explain in
> > the current version that things might improve in the future).
> >
> > Or maybe does someone else want to try to edit my prose to take this
> > into account?
> >
> > Robby
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] Testing whether a procedure gets collected

[Robby Findler]

Let me also say that I think it is important to give advice on how to
test so I think we need to say something.

Robby

On Mon, Dec 3, 2012 at 10:37 AM, Robby Findler
 wrote:
> On Mon, Dec 3, 2012 at 7:47 AM, Matthew Flatt  wrote:
>> At Mon, 3 Dec 2012 08:04:15 -0500, Sam Tobin-Hochstadt wrote:
>>> On Mon, Dec 3, 2012 at 7:54 AM, Robby Findler
>>>  wrote:
>>> >
>>> > I agree that when something is collected is a pretty intentional
>>> > property but I think it is possible to say a little bit more since
>>> > there is a pretty stable core idea there (namely that if something
>>> > isn't reachable and you call collect-garbage you can be pretty sure
>>> > it'll be gone -- this was not the case back in the boehm gc days).
>>>
>>> Do we really want to commit to this?  I agree that it's true for a
>>> simple 2-generation GC, but something more complex like the Sun
>>> garbage-first collector or Felix Klock's regional collector wouldn't
>>> have this property, let alone something more complex like stack
>>> allocation for non-escaping data.
>>
>> Yes, I think that a guarantee about individual objects with respect to
>> `collect-garbage' is going to be too strong, because the garbage
>> collector is meant to provide asymptotic guarantees instead of
>> individual-object guarantees.
>>
>> I think tests like Neil's are best written to allocate N things and
>> check that a good fraction of the N are released by a garbage
>> collection. In Neil's specific example, I'd probably write the test to
>> allocate 100 or 1000 of them, each with its own weak box, and make sure
>> that at most of the weak boxes are empty after a forced collection.
>
> Ah, whoops. I went and wrote a section to try to explain this
> (assuming the current tech) before reading this message.
>
> It is here:
>
> http://git.racket-lang.org/plt/blob/280d924349c2af595af307d38ffdb24940ede80a:/collects/scribblings/guide/performance.scrbl#l417
>
> My initial reaction to the above (perhaps colored by the fact that I
> just wrote that section) is to still explain the current state, and to
> revise this explanation when things improve (and, perhaps, explain in
> the current version that things might improve in the future).
>
> Or maybe does someone else want to try to edit my prose to take this
> into account?
>
> Robby
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] Testing whether a procedure gets collected

[Robby Findler]

On Mon, Dec 3, 2012 at 7:47 AM, Matthew Flatt  wrote:
> At Mon, 3 Dec 2012 08:04:15 -0500, Sam Tobin-Hochstadt wrote:
>> On Mon, Dec 3, 2012 at 7:54 AM, Robby Findler
>>  wrote:
>> >
>> > I agree that when something is collected is a pretty intentional
>> > property but I think it is possible to say a little bit more since
>> > there is a pretty stable core idea there (namely that if something
>> > isn't reachable and you call collect-garbage you can be pretty sure
>> > it'll be gone -- this was not the case back in the boehm gc days).
>>
>> Do we really want to commit to this?  I agree that it's true for a
>> simple 2-generation GC, but something more complex like the Sun
>> garbage-first collector or Felix Klock's regional collector wouldn't
>> have this property, let alone something more complex like stack
>> allocation for non-escaping data.
>
> Yes, I think that a guarantee about individual objects with respect to
> `collect-garbage' is going to be too strong, because the garbage
> collector is meant to provide asymptotic guarantees instead of
> individual-object guarantees.
>
> I think tests like Neil's are best written to allocate N things and
> check that a good fraction of the N are released by a garbage
> collection. In Neil's specific example, I'd probably write the test to
> allocate 100 or 1000 of them, each with its own weak box, and make sure
> that at most of the weak boxes are empty after a forced collection.

Ah, whoops. I went and wrote a section to try to explain this
(assuming the current tech) before reading this message.

It is here:

http://git.racket-lang.org/plt/blob/280d924349c2af595af307d38ffdb24940ede80a:/collects/scribblings/guide/performance.scrbl#l417

My initial reaction to the above (perhaps colored by the fact that I
just wrote that section) is to still explain the current state, and to
revise this explanation when things improve (and, perhaps, explain in
the current version that things might improve in the future).

Or maybe does someone else want to try to edit my prose to take this
into account?

Robby
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] FFI: pointer to an array in a C struct type

[Matthew Flatt]

At Mon, 3 Dec 2012 12:31:37 +0100, Tobias Hammer wrote:
> On Mon, 03 Dec 2012 11:45:08 +0100, Neil Toronto   
> wrote:
> 
> > This error seems wrong:
> >
> >
> > #lang racket
> >
> > (require ffi/unsafe
> >   ffi/unsafe/cvector)
> >
> > (define-cstruct _mpz ([alloc _int]
> >[size _int]
> >[limbs (_gcable _cvector)]))
> >
> > (define z (make-mpz 1 1 (list->cvector '(1) _long)))
> > (mpz-limbs z)
> >
> >  >>> _cvector: cannot automatically convert a C pointer to a cvector
> >
> >
> > The error might be correct, though late and confusing, if a cvector's  
> > "base representation" isn't a pointer type. Is it?
> 
> The base representation should be a pointer but the length and type  
> information to convert it back to a cvector is missing. Whats stored  
> inside the struct is only a plain pointer, without the information you  
> supplied to make-mpz, and because of this _cvector only has a  
> racket->c-conversion (see ffi/unsafe/cvector.rkt).
> 
> > If that error is correct, then how are FFI users meant to define a C  
> > struct that has a "long *" field that points to an array? I've not yet  
> > managed to define one whose instances survive a garbage collection cycle  
> > without using (_gcable _cvector). Here's one of my desperate attempts:
> >
> >
> > #lang racket
> >
> > (require ffi/unsafe
> >   ffi/unsafe/cvector)
> >
> > (define-cstruct _mpz ([alloc _int] [size _int] [limbs _gcpointer]))
> >
> > (define z (make-mpz 1 1 (cvector-ptr (list->cvector '(1) _long
> >
> >  > (ptr-ref (mpz-limbs z) _long 0)
> > 1
> >  > (collect-garbage)
> >  > (ptr-ref (mpz-limbs z) _long 0)
> > 139856348920568
> >
> >
> > I mean to be complain-y this time. This shouldn't be this hard to figure  
> > out.
> 
> I guess the problem here is, that the gc only knows memory with only  
> pointers (to gcable memory) inside and (atomic) memory without any  
> pointers it should follow. You try to mix them up. make-mpz should malloc  
> (see ffi docs) atomic memory and you put a pointer managed by the gc into  
> it. The pointer gets relocated on gc, but the ref inside the struct is not  
> known and not updated.
> 
> I unfortunately don't know a simple, gc- and typesafe way to put pointers  
> inside structs. My attempts to get around this were pretty hacking (using  
> a raw malloc'd pointer, embed a struct with plain values as pointer inside  
> an interor-malloc'd struct together with all others pointers, etc).
> I would be really interested in a way to accomplish this, too.

Yes. I've thought about this, but for the libraries I've written, the
cases where the GC can really solve the problem automatically seem
relatively rare, so I haven't been sufficiently motivated for my own
uses.

In the `_mpz' example above, the presence of both `alloc' and `size'
suggests that a callee might be responsible for malloc()ing `limbs' if
there's not already enough room. If so, `_gcpointer' isn't the right
type. Maybe this example doesn't actually go that way, but it's a
typical problem.

Neil, can you say more about how `_mpz' instances are used with foreign
functions?

_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] Testing whether a procedure gets collected

[Matthew Flatt]

At Mon, 3 Dec 2012 08:04:15 -0500, Sam Tobin-Hochstadt wrote:
> On Mon, Dec 3, 2012 at 7:54 AM, Robby Findler
>  wrote:
> >
> > I agree that when something is collected is a pretty intentional
> > property but I think it is possible to say a little bit more since
> > there is a pretty stable core idea there (namely that if something
> > isn't reachable and you call collect-garbage you can be pretty sure
> > it'll be gone -- this was not the case back in the boehm gc days).
> 
> Do we really want to commit to this?  I agree that it's true for a
> simple 2-generation GC, but something more complex like the Sun
> garbage-first collector or Felix Klock's regional collector wouldn't
> have this property, let alone something more complex like stack
> allocation for non-escaping data.

Yes, I think that a guarantee about individual objects with respect to
`collect-garbage' is going to be too strong, because the garbage
collector is meant to provide asymptotic guarantees instead of
individual-object guarantees.

I think tests like Neil's are best written to allocate N things and
check that a good fraction of the N are released by a garbage
collection. In Neil's specific example, I'd probably write the test to
allocate 100 or 1000 of them, each with its own weak box, and make sure
that at most of the weak boxes are empty after a forced collection.

_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] Testing whether a procedure gets collected

[Sam Tobin-Hochstadt]

On Mon, Dec 3, 2012 at 7:54 AM, Robby Findler
 wrote:
>
> I agree that when something is collected is a pretty intentional
> property but I think it is possible to say a little bit more since
> there is a pretty stable core idea there (namely that if something
> isn't reachable and you call collect-garbage you can be pretty sure
> it'll be gone -- this was not the case back in the boehm gc days).

Do we really want to commit to this?  I agree that it's true for a
simple 2-generation GC, but something more complex like the Sun
garbage-first collector or Felix Klock's regional collector wouldn't
have this property, let alone something more complex like stack
allocation for non-escaping data.
--
sam th
sa...@ccs.neu.edu
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] Testing whether a procedure gets collected

[Robby Findler]

On Sun, Dec 2, 2012 at 11:40 PM, Neil Toronto  wrote:
> On 12/02/2012 12:10 PM, Robby Findler wrote:
>>
>> On Sun, Dec 2, 2012 at 11:43 AM, Matthias Felleisen
>>  wrote:
>>>
>>>
>>> On Dec 1, 2012, at 9:23 PM, Robby Findler wrote:
>>>
 I think the high-level answer is that you have to understand something
 about details that aren't currently specified but nevertheless are how
 things currently work and then make a test that will work when you
 make those additional assumptions (and then keep it running in drdr so
 you can tell when the assumptions get broken).
>>>
>>>
>>>
>>> Doesn't this suggest deep down that Neil is trying to 'beat'
>>> Racket and its implementation with his program? I think the
>>> entire discussion (I didn't follow every detail) points to
>>> something lacking about the language. -- Matthias
>>
>>
>> I think that Neil wants to formulate a test case that checks that his
>> datastructure doesn't leak and is complaining about the lack of
>> specification for what "If the garbage collector has proven" (quote
>> from the docs) means.
>
>
> I didn't mean to sound complain-y... :p

Oh no. "complain" here was not meant to be a negative. Sorry. You
"expressed dissatisfaction" and I think it is reasonable to do so.

>> My experience with similar testing is that the level of specification
>> is actually okay. I can't speak for finalizers (they are more complex)
>> but weak boxes seem fine.
>
>
> And that helps a lot, so thanks!
>
> You're both right, of course. I'm essentially trying to test something that
> is unobservable. The fact that I can observe it is an implementation
> accident. The specific accident, that a weak box is cleared as soon as
> possible, happens to be common and generally expected.
>
> I don't know whether it's worth it in this case to tighten the
> specification. I don't want anyone to spend time on it unless it's actually
> worth researching.

I don't know that there is a paper to be found, but I agree that an
informal description that explains a little more about when one can
intuitively expect a weak box to be cleared seems useful.

I agree that when something is collected is a pretty intentional
property but I think it is possible to say a little bit more since
there is a pretty stable core idea there (namely that if something
isn't reachable and you call collect-garbage you can be pretty sure
it'll be gone -- this was not the case back in the boehm gc days).

Hm. I'll think more about this and see if I can improve the docs.

Robby
_
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket-dev] FFI: pointer to an array in a C struct type

[Tobias Hammer]

On Mon, 03 Dec 2012 11:45:08 +0100, Neil Toronto   
wrote:



This error seems wrong:


#lang racket

(require ffi/unsafe
  ffi/unsafe/cvector)

(define-cstruct _mpz ([alloc _int]
   [size _int]
   [limbs (_gcable _cvector)]))

(define z (make-mpz 1 1 (list->cvector '(1) _long)))
(mpz-limbs z)

 >>> _cvector: cannot automatically convert a C pointer to a cvector


The error might be correct, though late and confusing, if a cvector's  
"base representation" isn't a pointer type. Is it?


The base representation should be a pointer but the length and type  
information to convert it back to a cvector is missing. Whats stored  
inside the struct is only a plain pointer, without the information you  
supplied to make-mpz, and because of this _cvector only has a  
racket->c-conversion (see ffi/unsafe/cvector.rkt).


If that error is correct, then how are FFI users meant to define a C  
struct that has a "long *" field that points to an array? I've not yet  
managed to define one whose instances survive a garbage collection cycle  
without using (_gcable _cvector). Here's one of my desperate attempts:



#lang racket

(require ffi/unsafe
  ffi/unsafe/cvector)

(define-cstruct _mpz ([alloc _int] [size _int] [limbs _gcpointer]))

(define z (make-mpz 1 1 (cvector-ptr (list->cvector '(1) _long

 > (ptr-ref (mpz-limbs z) _long 0)
1
 > (collect-garbage)
 > (ptr-ref (mpz-limbs z) _long 0)
139856348920568


I mean to be complain-y this time. This shouldn't be this hard to figure  
out.


I guess the problem here is, that the gc only knows memory with only  
pointers (to gcable memory) inside and (atomic) memory without any  
pointers it should follow. You try to mix them up. make-mpz should malloc  
(see ffi docs) atomic memory and you put a pointer managed by the gc into  
it. The pointer gets relocated on gc, but the ref inside the struct is not  
known and not updated.


I unfortunately don't know a simple, gc- and typesafe way to put pointers  
inside structs. My attempts to get around this were pretty hacking (using  
a raw malloc'd pointer, embed a struct with plain values as pointer inside  
an interor-malloc'd struct together with all others pointers, etc).

I would be really interested in a way to accomplish this, too.

Tobias




--
-
Tobias Hammer
DLR / Institute of Robotics and Mechatronics
Muenchner Str. 20, D-82234 Wessling
Tel.: 08153/28-1487
Mail: tobias.ham...@dlr.de
_
 Racket Developers list:
 http://lists.racket-lang.org/dev

[racket-dev] FFI: pointer to an array in a C struct type

[Neil Toronto]

This error seems wrong:


#lang racket

(require ffi/unsafe
 ffi/unsafe/cvector)

(define-cstruct _mpz ([alloc _int]
  [size _int]
  [limbs (_gcable _cvector)]))

(define z (make-mpz 1 1 (list->cvector '(1) _long)))
(mpz-limbs z)

>>> _cvector: cannot automatically convert a C pointer to a cvector


The error might be correct, though late and confusing, if a cvector's 
"base representation" isn't a pointer type. Is it?


If that error is correct, then how are FFI users meant to define a C 
struct that has a "long *" field that points to an array? I've not yet 
managed to define one whose instances survive a garbage collection cycle 
without using (_gcable _cvector). Here's one of my desperate attempts:



#lang racket

(require ffi/unsafe
 ffi/unsafe/cvector)

(define-cstruct _mpz ([alloc _int] [size _int] [limbs _gcpointer]))

(define z (make-mpz 1 1 (cvector-ptr (list->cvector '(1) _long

> (ptr-ref (mpz-limbs z) _long 0)
1
> (collect-garbage)
> (ptr-ref (mpz-limbs z) _long 0)
139856348920568


I mean to be complain-y this time. This shouldn't be this hard to figure 
out.


Neil ⊥
_
 Racket Developers list:
 http://lists.racket-lang.org/dev