Re: [racket-dev] finding data files in packages during development

2016-03-04 Thread Matthew Flatt 
At Fri, 4 Mar 2016 19:35:15 -0500, Jon Zeppieri wrote:
> On Fri, Mar 4, 2016 at 6:51 PM, Jon Zeppieri  wrote:
> 
> > One question about this, Matthew: should I ensure that all of the data
> > files have distinct collection-relative paths, instead of making them all
> > cldr2/data/json.zip?
> >
> 
> Oh, never mind -- obviously I need to give them different paths. -J

I would have said that using the same collection-relative path is be
fine for non-module files. After all, the `find-relevant-directories`
function will give you directories, not collection paths, and you can
resolve any path in the an "info.rkt" relative to its directory.

But maybe I'm missing some other constraint that requires different
collection-relative paths.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-dev/56da3646.0c01620a.9d2b4.c3b0SMTPIN_ADDED_MISSING%40gmr-mx.google.com.
For more options, visit https://groups.google.com/d/optout.

Re: [racket-dev] finding data files in packages during development

2016-03-04 Thread Jon Zeppieri 
On Fri, Mar 4, 2016 at 6:51 PM, Jon Zeppieri  wrote:

> One question about this, Matthew: should I ensure that all of the data
> files have distinct collection-relative paths, instead of making them all
> cldr2/data/json.zip?
>

Oh, never mind -- obviously I need to give them different paths. -J

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-dev/CAKfDxxyiKZxAEfoxPDqTELrXaUWEHiHDtAdkqRuWmzGX_jY-nw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [racket-dev] finding data files in packages during development

2016-03-04 Thread Jon Zeppieri 
One question about this, Matthew: should I ensure that all of the data
files have distinct collection-relative paths, instead of making them all
cldr2/data/json.zip?


On Fri, Mar 4, 2016 at 6:33 PM, Jon Zeppieri  wrote:

> This sounds like a good approach. Thanks, Matthew! -J
>
>
> On Fri, Mar 4, 2016 at 6:19 PM, Matthew Flatt  wrote:
>
>> Things that you want to access from a program should be based on
>> collection, based on packages. In principle, packages don't exist at
>> all at run-time --- and they really don't exist at run-time for a
>> program bundled with `raco exe`.
>>
>> A good way to register extensions via the collection layer is to use a
>> new "info.rkt" field. Each package can supply a "cldr2/data" collection
>> directory, with an "info.rkt" file defining a field that lists the
>> provided data files. Then you can use `find-relevant-directories` to
>> find all the relevant directories (i.e., all the "info.rkt" files that
>> define your new field).
>>
>> Another possibility is to use `copy-shared-files` in "info.rkt" to
>> instruct `raco setup` (and `raco pkg install`) to install files in the
>> "share" directory. In this case, I think the strategy that uses a new
>> "info.rkt" field is probably better.
>>
>> One more piece of the puzzle: in the code that accesses the data files,
>> use `define-runtime-path-list` to build a list of all the currently
>> installed files. That way, `raco exe` will know to perform that
>> computation at build time and pull along the relevant files.
>>
>> At Fri, 4 Mar 2016 16:58:15 -0500, Jon Zeppieri wrote:
>> > I'm working on a new version of the CLDR (localization data) packages,
>> with
>> > the goal of reducing the amount of data that most installations will
>> need.
>> >
>> > The packages are broken up into:
>> > cldr2: provides functions for accessing the data and resolving locale
>> names
>> > cldr2-data-core: provides the data from
>> > https://github.com/unicode-cldr/cldr-core
>> > cldr2-data_: one per locale, provides data from the rest of
>> > the unicode-cldr repos, but only for the locale in the package name
>> >
>> > The cldr2 package depends on cldr2-data-core, but a user is free to
>> install
>> > as many or as few locale-specific packages as desired.
>> >
>> > The problem I've run into is how, at runtime, to find the data files I
>> need
>> > in a way that works during development.
>> >
>> > Each of the cldr2-data-* packages has a data archive at
>> cldr2/data/json.zip
>> > from the package root directory. So, if someone wants data from the core
>> > package, we could do:
>> >
>> > ```
>> > (define dir (pkg-directory "cldr2-data-core" #:cache PKG-CACHE))
>> > ... [raise an exception if the package isn't installed] ...
>> > (define zip-path (build-path dir "cldr2" "data" "json.zip"))
>> > ... [open the archive and extract the relevant data] ...
>> > ```
>> >
>> > And that's fine, except that it doesn't work in development when using
>> raco
>> > link, because raco link manages collections, not packages. And I don't
>> > think that collection-file-path is useful here either, since all of
>> these
>> > json.zip files have exactly the same collection-relative path.
>> >
>> > What's the best way to handle this? Should I just give the zip files
>> > distinct names and use collection-file-path? Or is there a better way to
>> > handle this situation? (I'm a bit reluctant to use collection-file-path,
>> > since I think it searches the file system and so would be a bit
>> expensive.
>> > pkg-directory needs to parse the package catalog, but it allows the
>> results
>> > of that parse to be cached.)
>> >
>> > -Jon
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups
>> > "Racket Developers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an
>> > email to [email protected].
>> > To post to this group, send email to [email protected].
>> > To view this discussion on the web visit
>> >
>> https://groups.google.com/d/msgid/racket-dev/CAKfDxxxywSUuR0j%3DM1sgU3foezz_jt3
>> > %2BAChNEqCO3SuqYtffRA%40mail.gmail.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-dev/CAKfDxxxvT4ah_kq4g088_bGz7D%2BWUSDrhiSY1eqbr_%3D44kA%2B0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [racket-dev] finding data files in packages during development

2016-03-04 Thread Jon Zeppieri 
This sounds like a good approach. Thanks, Matthew! -J


On Fri, Mar 4, 2016 at 6:19 PM, Matthew Flatt  wrote:

> Things that you want to access from a program should be based on
> collection, based on packages. In principle, packages don't exist at
> all at run-time --- and they really don't exist at run-time for a
> program bundled with `raco exe`.
>
> A good way to register extensions via the collection layer is to use a
> new "info.rkt" field. Each package can supply a "cldr2/data" collection
> directory, with an "info.rkt" file defining a field that lists the
> provided data files. Then you can use `find-relevant-directories` to
> find all the relevant directories (i.e., all the "info.rkt" files that
> define your new field).
>
> Another possibility is to use `copy-shared-files` in "info.rkt" to
> instruct `raco setup` (and `raco pkg install`) to install files in the
> "share" directory. In this case, I think the strategy that uses a new
> "info.rkt" field is probably better.
>
> One more piece of the puzzle: in the code that accesses the data files,
> use `define-runtime-path-list` to build a list of all the currently
> installed files. That way, `raco exe` will know to perform that
> computation at build time and pull along the relevant files.
>
> At Fri, 4 Mar 2016 16:58:15 -0500, Jon Zeppieri wrote:
> > I'm working on a new version of the CLDR (localization data) packages,
> with
> > the goal of reducing the amount of data that most installations will
> need.
> >
> > The packages are broken up into:
> > cldr2: provides functions for accessing the data and resolving locale
> names
> > cldr2-data-core: provides the data from
> > https://github.com/unicode-cldr/cldr-core
> > cldr2-data_: one per locale, provides data from the rest of
> > the unicode-cldr repos, but only for the locale in the package name
> >
> > The cldr2 package depends on cldr2-data-core, but a user is free to
> install
> > as many or as few locale-specific packages as desired.
> >
> > The problem I've run into is how, at runtime, to find the data files I
> need
> > in a way that works during development.
> >
> > Each of the cldr2-data-* packages has a data archive at
> cldr2/data/json.zip
> > from the package root directory. So, if someone wants data from the core
> > package, we could do:
> >
> > ```
> > (define dir (pkg-directory "cldr2-data-core" #:cache PKG-CACHE))
> > ... [raise an exception if the package isn't installed] ...
> > (define zip-path (build-path dir "cldr2" "data" "json.zip"))
> > ... [open the archive and extract the relevant data] ...
> > ```
> >
> > And that's fine, except that it doesn't work in development when using
> raco
> > link, because raco link manages collections, not packages. And I don't
> > think that collection-file-path is useful here either, since all of these
> > json.zip files have exactly the same collection-relative path.
> >
> > What's the best way to handle this? Should I just give the zip files
> > distinct names and use collection-file-path? Or is there a better way to
> > handle this situation? (I'm a bit reluctant to use collection-file-path,
> > since I think it searches the file system and so would be a bit
> expensive.
> > pkg-directory needs to parse the package catalog, but it allows the
> results
> > of that parse to be cached.)
> >
> > -Jon
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Racket Developers" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [email protected].
> > To post to this group, send email to [email protected].
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/racket-dev/CAKfDxxxywSUuR0j%3DM1sgU3foezz_jt3
> > %2BAChNEqCO3SuqYtffRA%40mail.gmail.com.
> > For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-dev/CAKfDxxy4C5m4%2BNE950%2B42HE7MLKDCdHoJKWfREHrAp87Lj%2BSuQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [racket-dev] finding data files in packages during development

2016-03-04 Thread Matthew Flatt 
Things that you want to access from a program should be based on
collection, based on packages. In principle, packages don't exist at
all at run-time --- and they really don't exist at run-time for a
program bundled with `raco exe`.

A good way to register extensions via the collection layer is to use a
new "info.rkt" field. Each package can supply a "cldr2/data" collection
directory, with an "info.rkt" file defining a field that lists the
provided data files. Then you can use `find-relevant-directories` to
find all the relevant directories (i.e., all the "info.rkt" files that
define your new field).

Another possibility is to use `copy-shared-files` in "info.rkt" to
instruct `raco setup` (and `raco pkg install`) to install files in the
"share" directory. In this case, I think the strategy that uses a new
"info.rkt" field is probably better.

One more piece of the puzzle: in the code that accesses the data files,
use `define-runtime-path-list` to build a list of all the currently
installed files. That way, `raco exe` will know to perform that
computation at build time and pull along the relevant files.

At Fri, 4 Mar 2016 16:58:15 -0500, Jon Zeppieri wrote:
> I'm working on a new version of the CLDR (localization data) packages, with
> the goal of reducing the amount of data that most installations will need.
> 
> The packages are broken up into:
> cldr2: provides functions for accessing the data and resolving locale names
> cldr2-data-core: provides the data from
> https://github.com/unicode-cldr/cldr-core
> cldr2-data_: one per locale, provides data from the rest of
> the unicode-cldr repos, but only for the locale in the package name
> 
> The cldr2 package depends on cldr2-data-core, but a user is free to install
> as many or as few locale-specific packages as desired.
> 
> The problem I've run into is how, at runtime, to find the data files I need
> in a way that works during development.
> 
> Each of the cldr2-data-* packages has a data archive at cldr2/data/json.zip
> from the package root directory. So, if someone wants data from the core
> package, we could do:
> 
> ```
> (define dir (pkg-directory "cldr2-data-core" #:cache PKG-CACHE))
> ... [raise an exception if the package isn't installed] ...
> (define zip-path (build-path dir "cldr2" "data" "json.zip"))
> ... [open the archive and extract the relevant data] ...
> ```
> 
> And that's fine, except that it doesn't work in development when using raco
> link, because raco link manages collections, not packages. And I don't
> think that collection-file-path is useful here either, since all of these
> json.zip files have exactly the same collection-relative path.
> 
> What's the best way to handle this? Should I just give the zip files
> distinct names and use collection-file-path? Or is there a better way to
> handle this situation? (I'm a bit reluctant to use collection-file-path,
> since I think it searches the file system and so would be a bit expensive.
> pkg-directory needs to parse the package catalog, but it allows the results
> of that parse to be cached.)
> 
> -Jon
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/racket-dev/CAKfDxxxywSUuR0j%3DM1sgU3foezz_jt3
> %2BAChNEqCO3SuqYtffRA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-dev/20160304231920.3D47F65006F%40mail-svr1.cs.utah.edu.
For more options, visit https://groups.google.com/d/optout.

Re: [racket-dev] finding data files in packages during development

2016-03-04 Thread Jon Zeppieri 
On Fri, Mar 4, 2016 at 6:03 PM, Robby Findler 
wrote:

> I think that files in collections is the way to go. You don't have to use
> collection-file-path, you could search the collections yourself, but I'm
> not sure that it would be faster. Are you seeing something specific that
> has bad performance?
>
> Robby
>

No, I haven't actually implemented this yet. It's just that I imagine it
has to work something like this:
- find all of the possible directories that represent the "cldr2" collection
- look in each for the given path until you find it

If you had hundreds of locales installed on your machine, this would seem
to be a pretty expensive way to look for a single file, especially since it
can only be provided by a single package. I could default to looking for
the package directory and then fall back on the collection search, but I'd
still need to verify that both actually work. Tests would use different
code paths in dev and in prod, which isn't great.

Or, I guess I could keep the collection search small by using different
sub-collections for each locale (and a special one for the core data). I'll
need to look at the implementation of collection-file-path to see if that
would make a difference.

-Jon




>
>
> On Fri, Mar 4, 2016 at 3:58 PM, Jon Zeppieri  wrote:
>
>> I'm working on a new version of the CLDR (localization data) packages,
>> with the goal of reducing the amount of data that most installations will
>> need.
>>
>> The packages are broken up into:
>> cldr2: provides functions for accessing the data and resolving locale
>> names
>> cldr2-data-core: provides the data from
>> https://github.com/unicode-cldr/cldr-core
>> cldr2-data_: one per locale, provides data from the rest of
>> the unicode-cldr repos, but only for the locale in the package name
>>
>> The cldr2 package depends on cldr2-data-core, but a user is free to
>> install as many or as few locale-specific packages as desired.
>>
>> The problem I've run into is how, at runtime, to find the data files I
>> need in a way that works during development.
>>
>> Each of the cldr2-data-* packages has a data archive at
>> cldr2/data/json.zip from the package root directory. So, if someone wants
>> data from the core package, we could do:
>>
>> ```
>> (define dir (pkg-directory "cldr2-data-core" #:cache PKG-CACHE))
>> ... [raise an exception if the package isn't installed] ...
>> (define zip-path (build-path dir "cldr2" "data" "json.zip"))
>> ... [open the archive and extract the relevant data] ...
>> ```
>>
>> And that's fine, except that it doesn't work in development when using
>> raco link, because raco link manages collections, not packages. And I don't
>> think that collection-file-path is useful here either, since all of these
>> json.zip files have exactly the same collection-relative path.
>>
>> What's the best way to handle this? Should I just give the zip files
>> distinct names and use collection-file-path? Or is there a better way to
>> handle this situation? (I'm a bit reluctant to use collection-file-path,
>> since I think it searches the file system and so would be a bit expensive.
>> pkg-directory needs to parse the package catalog, but it allows the results
>> of that parse to be cached.)
>>
>> -Jon
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Racket Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/racket-dev/CAKfDxxxywSUuR0j%3DM1sgU3foezz_jt3%2BAChNEqCO3SuqYtffRA%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-dev/CAKfDxxzbf2ew0mtKReKV6sLto%3Df0WvNEcDV6BxUM6T1BH-n1hQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [racket-dev] finding data files in packages during development

2016-03-04 Thread Robby Findler 
I think that files in collections is the way to go. You don't have to use
collection-file-path, you could search the collections yourself, but I'm
not sure that it would be faster. Are you seeing something specific that
has bad performance?

Robby


On Fri, Mar 4, 2016 at 3:58 PM, Jon Zeppieri  wrote:

> I'm working on a new version of the CLDR (localization data) packages,
> with the goal of reducing the amount of data that most installations will
> need.
>
> The packages are broken up into:
> cldr2: provides functions for accessing the data and resolving locale names
> cldr2-data-core: provides the data from
> https://github.com/unicode-cldr/cldr-core
> cldr2-data_: one per locale, provides data from the rest of
> the unicode-cldr repos, but only for the locale in the package name
>
> The cldr2 package depends on cldr2-data-core, but a user is free to
> install as many or as few locale-specific packages as desired.
>
> The problem I've run into is how, at runtime, to find the data files I
> need in a way that works during development.
>
> Each of the cldr2-data-* packages has a data archive at
> cldr2/data/json.zip from the package root directory. So, if someone wants
> data from the core package, we could do:
>
> ```
> (define dir (pkg-directory "cldr2-data-core" #:cache PKG-CACHE))
> ... [raise an exception if the package isn't installed] ...
> (define zip-path (build-path dir "cldr2" "data" "json.zip"))
> ... [open the archive and extract the relevant data] ...
> ```
>
> And that's fine, except that it doesn't work in development when using
> raco link, because raco link manages collections, not packages. And I don't
> think that collection-file-path is useful here either, since all of these
> json.zip files have exactly the same collection-relative path.
>
> What's the best way to handle this? Should I just give the zip files
> distinct names and use collection-file-path? Or is there a better way to
> handle this situation? (I'm a bit reluctant to use collection-file-path,
> since I think it searches the file system and so would be a bit expensive.
> pkg-directory needs to parse the package catalog, but it allows the results
> of that parse to be cached.)
>
> -Jon
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-dev/CAKfDxxxywSUuR0j%3DM1sgU3foezz_jt3%2BAChNEqCO3SuqYtffRA%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-dev/CAL3TdOOZ71T8NDBx2%2BRJx1dh4PnzA9%3DH9L42LGA8b%3DQhAYHs6w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.