Re: Data structure for (package) options
Am Donnerstag, den 30.01.2020, 09:50 +0100 schrieb Han-Wen Nienhuys: > On Tue, Jan 28, 2020 at 2:48 PM David Kastrup wrote: > > > > I think we should aim to avoid textual inclusion as a mechanism > > > that > > > powers packaging. > > > > At the implementation side, "textual inclusion" will be what has to > > power systems where one can "plug in" packages and have them work > > by > > being present. Even Guile's use-modules mechanism works at that > > level. > > But of course that doesn't mean that we want \include as a user- > > level > > access method in the long haul. > > By textual inclusion, I mean a mechanism that must insert all its > data > in the global namespace. Library systems usually must have means to > be > explicit about external APIs and internal implementation details. I > am > advocating that there is an easy to use mechanism such that OLL > packages can shield some of the implementation from other packages. > My idea for packages is: * packages *always* have .ly files as entry points (even if their "real" work is exclusively Scheme). That makes it easier to handle (no need to load packages in two different forms) and gives regular users a lower threshold of moving library code to packages. * they explicitly define their public interface. See below for my ideas about that. * A new command \usepackage will either load the whole package or only a subset of its interface (like the "from X load Y" syntax in Python). Maybe a clean way of making an interface explicit would be not to do myFunction = #(define-music-function ... but having a new (set of?) command(s) that allow \export myFunction #(define-music-function When parsing the package (i.e. loading it for the first time `myFunction` is stored as the package's interface (in an internal structure storing the interfaces of all loaded packages, along with their defined options). [This is something *I* can do with the current Scheme possibilities] When using a package that has already been loaded (\usepackage checks that first) the stored Scheme values (which may equally be variables or procedures) are made visible in the caller's scope. The question I have is: If \usepackage makes the names available to everything else later (like a regular \include of a LilyPond file) I can do this with what we already have in Scheme. If \usepackage should make the names available only within the current input file (like (use-modules) does) I think \usepackage would have to be implemented at the parser level. Urs
Re: Data structure for (package) options
On Tue, Jan 28, 2020 at 2:48 PM David Kastrup wrote: > > I think we should aim to avoid textual inclusion as a mechanism that > > powers packaging. > > At the implementation side, "textual inclusion" will be what has to > power systems where one can "plug in" packages and have them work by > being present. Even Guile's use-modules mechanism works at that level. > But of course that doesn't mean that we want \include as a user-level > access method in the long haul. By textual inclusion, I mean a mechanism that must insert all its data in the global namespace. Library systems usually must have means to be explicit about external APIs and internal implementation details. I am advocating that there is an easy to use mechanism such that OLL packages can shield some of the implementation from other packages. > It's likely that a typical more complex package may consist of both .ly > and/or .scm components and that is an implementation detail that a > package user should not need to bother about, and that hopefully should > not significantly complicate things for people wanting to provide > packages with either or both .ly and/or .scm components. > > -- > David Kastrup -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Data structure for (package) options
Am 28. Januar 2020 14:48:54 MEZ schrieb David Kastrup :
>Han-Wen Nienhuys writes:
>
>> On Mon, Jan 27, 2020 at 11:39 PM Urs Liska
>wrote:
>>>
>>> I didn't have time to really think about much (about LilyPond) the
>past
>>> week, just enjoyed seeing so much constructive discussion.
>>> [..]
>>
>> I haven't read your messages in detail, but I'd like to throw out one
>> thought to consider: we use GUILE modules as a mechanism for
>> identifier namespacing (\paper, \header all create modules). I think
>> it might be useful if we could provide a LilyPond native mechanism
>for
>> packaging that declares a namespace implicitly, eg.
>>
>> \module "edition" {
>> internal = ...
>> addTweak = ...
>> }
>>
>> \import edition.addTweak
>>
>> that would let you define constructs in a package that doesn't
>pollute
>> the global namespace, and let .ly packages control explicitly what
>> symbols they want to use.
>>
>> I think we should aim to avoid textual inclusion as a mechanism that
>> powers packaging.
>
>At the implementation side, "textual inclusion" will be what has to
>power systems where one can "plug in" packages and have them work by
>being present. Even Guile's use-modules mechanism works at that level.
>But of course that doesn't mean that we want \include as a user-level
>access method in the long haul.
>
>It's likely that a typical more complex package may consist of both .ly
>and/or .scm components and that is an implementation detail that a
>package user should not need to bother about, and that hopefully should
>not significantly complicate things for people wanting to provide
>packages with either or both .ly and/or .scm components.
From my experience with OLL it would sesm reasonable to have all packages use
.ly files as entry point. From there they can use Scheme modules if they need
to. OLL does some work to add its root to the guile path, so there's the way to
use modules relstive to the package.
Urs
--
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.
Re: Data structure for (package) options
Han-Wen Nienhuys writes:
> On Mon, Jan 27, 2020 at 11:39 PM Urs Liska wrote:
>>
>> I didn't have time to really think about much (about LilyPond) the past
>> week, just enjoyed seeing so much constructive discussion.
>> [..]
>
> I haven't read your messages in detail, but I'd like to throw out one
> thought to consider: we use GUILE modules as a mechanism for
> identifier namespacing (\paper, \header all create modules). I think
> it might be useful if we could provide a LilyPond native mechanism for
> packaging that declares a namespace implicitly, eg.
>
> \module "edition" {
> internal = ...
> addTweak = ...
> }
>
> \import edition.addTweak
>
> that would let you define constructs in a package that doesn't pollute
> the global namespace, and let .ly packages control explicitly what
> symbols they want to use.
>
> I think we should aim to avoid textual inclusion as a mechanism that
> powers packaging.
At the implementation side, "textual inclusion" will be what has to
power systems where one can "plug in" packages and have them work by
being present. Even Guile's use-modules mechanism works at that level.
But of course that doesn't mean that we want \include as a user-level
access method in the long haul.
It's likely that a typical more complex package may consist of both .ly
and/or .scm components and that is an implementation detail that a
package user should not need to bother about, and that hopefully should
not significantly complicate things for people wanting to provide
packages with either or both .ly and/or .scm components.
--
David Kastrup
Re: Data structure for (package) options
Urs Liska writes: > Am Dienstag, den 28.01.2020, 00:34 +0100 schrieb David Kastrup: >> Urs Liska writes: >> >> So basically there is long-term potential for efficiency to mostly >> become a non-issue. > > I'm sorry, but I don't fully understand what you are trying to tell me > here. Are you saying that you already had some internal changes in mind > and that storing the package options in an alist is something I > shouldn't really worry about? a) the storage format is something you should not worry about with regard to efficiency b) xxx.xxx.xxx syntax for organising material should not be shied away from if convenient since it is comparatively straightforward to stop alists from being a mandatory underlying data structure c) don't choose between tree or flat organisation for reasons of efficiency: only take user convenience into account > BTW: I mistook my own example (shouldn't have started writing late in > the evening ...). What I meant is storing the options in objects, not > assigning symbols: > > (define package-options >`((edition-engraver . ,ee-options) > (scholarly . ,scholarly-options) > ... > )) > > with ee-options and scholarly-options having been bound to tree objects > before. This prescribes a particular data structure. package-options.edition-engraver = #ee-options package-options.scholarly = #scholarly-options in contrast will at the _current_ point of time use a particular data structure but we can do a switcheroo underneath in some near future. >> > It seems reasonable to not store a package's options as a nested >> > alist, though. I'd rather consider using a tree implementation in a >> > class, which would for example make it cleaner to encapsulate the >> > behaviour and e.g. implement type checking in that class rather >> > than in the accessor functions, or enable alternative tree >> > implementations if that should become interesting for custom data >> > storage. We have a tree implementation in oll-core at >> > https://github.com/openlilylib/oll-core/blob/master/scheme/oll-core/tree.scm >> > Would that be something to use here? >> >> Whatever we do, it should be an implementation detail the user _and_ >> the package programmers do not really need to know about. There >> should be accessors hiding the organisation. > > Yes, that's what I intended. It should be a tree implementation with a > given characteristic that basically noone should bother about. There > should be the *possibility* to drop-in a different tree implementation > (with the same interface) if for whatever reason a package might want > to do so (I could imagine that for some algorithmic use of the data it > might be reasonable to consider some sort of self-organizing tree to > optimize access times - but definitely not for the bread-and-butter > use case of package options). I think that our basic data structures should be accessible at the LilyPond syntax level without even going down into Scheme. -- David Kastrup
Re: Data structure for (package) options
Am Dienstag, den 28.01.2020, 09:26 +0100 schrieb Han-Wen Nienhuys:
> On Mon, Jan 27, 2020 at 11:39 PM Urs Liska
> wrote:
> > I didn't have time to really think about much (about LilyPond) the
> > past
> > week, just enjoyed seeing so much constructive discussion.
> > [..]
>
> I haven't read your messages in detail, but I'd like to throw out one
> thought to consider: we use GUILE modules as a mechanism for
> identifier namespacing (\paper, \header all create modules). I think
> it might be useful if we could provide a LilyPond native mechanism
> for
> packaging that declares a namespace implicitly,
That sound similar to what David suggested in
https://lists.gnu.org/archive/html/lilypond-devel/2020-01/msg00349.html
> > Ok. One thing to think about is that we want package files to be
> > contributed by "ordinary" users. But something like
> >
> > \exportSymbols transposeSequence,instrumentGroup,scratchMyBack
> >
> > would be perfectly feasible syntactical sugar.
and following conversation on that thread.
I'm all for such an encapsulation strategy for packages.
> eg.
>
> \module "edition" {
> internal = ...
> addTweak = ...
> }
I think this concrete suggestion *looks* nice but would be awkward to
handle and impose restriction on how package authors can express
themselves.
I'd prefer (IISC more to David's suggestion) being able to write a
package as a regular .ly or .scm file with commands to explicitly
export/expose selected functions and variables.
>
> \import edition.addTweak
In analogy to Python one could have a choice between
\import edition
for making all exported names visible or
\import \with {
addTweak.foo.bar
} edition
for selectively importing three names.
(the \with being my current abuse of ly:context-mod?)
I have no strong opinion, and \import seems like a natural choice with
a programmer's eye, but \usepackage might be more expressive from a
document author's perspective.
>
> that would let you define constructs in a package that doesn't
> pollute
> the global namespace, and let .ly packages control explicitly what
> symbols they want to use.
>
> I think we should aim to avoid textual inclusion as a mechanism that
> powers packaging.
>
That's a good point but will definitely be beyond what *I* can do.
Urs
Re: Data structure for (package) options
On Mon, Jan 27, 2020 at 11:39 PM Urs Liska wrote:
>
> I didn't have time to really think about much (about LilyPond) the past
> week, just enjoyed seeing so much constructive discussion.
> [..]
I haven't read your messages in detail, but I'd like to throw out one
thought to consider: we use GUILE modules as a mechanism for
identifier namespacing (\paper, \header all create modules). I think
it might be useful if we could provide a LilyPond native mechanism for
packaging that declares a namespace implicitly, eg.
\module "edition" {
internal = ...
addTweak = ...
}
\import edition.addTweak
that would let you define constructs in a package that doesn't pollute
the global namespace, and let .ly packages control explicitly what
symbols they want to use.
I think we should aim to avoid textual inclusion as a mechanism that
powers packaging.
--
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Data structure for (package) options
Am Dienstag, den 28.01.2020, 00:34 +0100 schrieb David Kastrup: > Urs Liska writes: > > > I didn't have time to really think about much (about LilyPond) the > > past > > week, just enjoyed seeing so much constructive discussion. > > > > I think the first thing I'd like to go for with the > > package/extension > > mechanism is storing and handling (package) options. > > > > There are three use cases which are differently closely related to > > the > > actual package mechanism but should be dealt with together: > > > > * package options > > * specified, typed and preset with default values by a package > > * handled by getter and setter functions > > * custom options > > * behave like package options > > * not associated with a package but explicitly called for in > > user > > code > > * custom data > > * use the option syntax/infrastructure to store arbitrary data > > * for example a music tree as described in Jan-Peter's > > templating > > > > I think package options should be handled in one association list, > > hidden in a Scheme module and only accessed by specific accessor > > code. > > The list elements should have the package name as car and an option > > *object* as cdr: > > > > (define package-options > >'((edition-engraver . ee-options) > > (scholarly . scholarly-options) > > ... > > )) > > > > I expect the number of packages loaded in a document ranging from > > "a > > few" to "a few dozens", so probably a simple alist should be the > > right > > data structure? > > Before we devolve into an efficiency discussion here, let me sketch > one > of my "should make sense" projects": our access of alists mostly is > through our own accessor function ly:assoc-get . If the elements on > an > alist could not just be a key-value pair but a hashtable (for looking > up > key-value, of course) or a vector (I think I'd prefer it being 1- > based > in spite of making the Scheme indexing less obvious), our lookup > routines should be able to deal with that. Some alists could be > compressed on use, making things like drumtables and note alists > efficient behind the scenes. > > Such a replacement would work transparently for things like > > package-options.edition-engraver = #'ee-options > > or similar. So basically there is long-term potential for efficiency > to > mostly become a non-issue. I'm sorry, but I don't fully understand what you are trying to tell me here. Are you saying that you already had some internal changes in mind and that storing the package options in an alist is something I shouldn't really worry about? BTW: I mistook my own example (shouldn't have started writing late in the evening ...). What I meant is storing the options in objects, not assigning symbols: (define package-options `((edition-engraver . ,ee-options) (scholarly . ,scholarly-options) ... )) with ee-options and scholarly-options having been bound to tree objects before. > > > It seems reasonable to not store a package's options as a nested > > alist, > > though. I'd rather consider using a tree implementation in a class, > > which would for example make it cleaner to encapsulate the > > behaviour > > and e.g. implement type checking in that class rather than in the > > accessor functions, or enable alternative tree implementations if > > that > > should become interesting for custom data storage. > > We have a tree implementation in oll-core at > > https://github.com/openlilylib/oll-core/blob/master/scheme/oll-core/tree.scm > > Would that be something to use here? > > Whatever we do, it should be an implementation detail the user _and_ > the > package programmers do not really need to know about. There should > be > accessors hiding the organisation. > Yes, that's what I intended. It should be a tree implementation with a given characteristic that basically noone should bother about. There should be the *possibility* to drop-in a different tree implementation (with the same interface) if for whatever reason a package might want to do so (I could imagine that for some algorithmic use of the data it might be reasonable to consider some sort of self-organizing tree to optimize access times - but definitely not for the bread-and-butter use case of package options). Urs
Re: Data structure for (package) options
Urs Liska writes: > I didn't have time to really think about much (about LilyPond) the past > week, just enjoyed seeing so much constructive discussion. > > I think the first thing I'd like to go for with the package/extension > mechanism is storing and handling (package) options. > > There are three use cases which are differently closely related to the > actual package mechanism but should be dealt with together: > > * package options > * specified, typed and preset with default values by a package > * handled by getter and setter functions > * custom options > * behave like package options > * not associated with a package but explicitly called for in user > code > * custom data > * use the option syntax/infrastructure to store arbitrary data > * for example a music tree as described in Jan-Peter's templating > > I think package options should be handled in one association list, > hidden in a Scheme module and only accessed by specific accessor code. > The list elements should have the package name as car and an option > *object* as cdr: > > (define package-options >'((edition-engraver . ee-options) > (scholarly . scholarly-options) > ... > )) > > I expect the number of packages loaded in a document ranging from "a > few" to "a few dozens", so probably a simple alist should be the right > data structure? Before we devolve into an efficiency discussion here, let me sketch one of my "should make sense" projects": our access of alists mostly is through our own accessor function ly:assoc-get . If the elements on an alist could not just be a key-value pair but a hashtable (for looking up key-value, of course) or a vector (I think I'd prefer it being 1-based in spite of making the Scheme indexing less obvious), our lookup routines should be able to deal with that. Some alists could be compressed on use, making things like drumtables and note alists efficient behind the scenes. Such a replacement would work transparently for things like package-options.edition-engraver = #'ee-options or similar. So basically there is long-term potential for efficiency to mostly become a non-issue. > It seems reasonable to not store a package's options as a nested alist, > though. I'd rather consider using a tree implementation in a class, > which would for example make it cleaner to encapsulate the behaviour > and e.g. implement type checking in that class rather than in the > accessor functions, or enable alternative tree implementations if that > should become interesting for custom data storage. > We have a tree implementation in oll-core at > https://github.com/openlilylib/oll-core/blob/master/scheme/oll-core/tree.scm > Would that be something to use here? Whatever we do, it should be an implementation detail the user _and_ the package programmers do not really need to know about. There should be accessors hiding the organisation. -- David Kastrup
Data structure for (package) options
I didn't have time to really think about much (about LilyPond) the past week, just enjoyed seeing so much constructive discussion. I think the first thing I'd like to go for with the package/extension mechanism is storing and handling (package) options. There are three use cases which are differently closely related to the actual package mechanism but should be dealt with together: * package options * specified, typed and preset with default values by a package * handled by getter and setter functions * custom options * behave like package options * not associated with a package but explicitly called for in user code * custom data * use the option syntax/infrastructure to store arbitrary data * for example a music tree as described in Jan-Peter's templating I think package options should be handled in one association list, hidden in a Scheme module and only accessed by specific accessor code. The list elements should have the package name as car and an option *object* as cdr: (define package-options '((edition-engraver . ee-options) (scholarly . scholarly-options) ... )) I expect the number of packages loaded in a document ranging from "a few" to "a few dozens", so probably a simple alist should be the right data structure? Package and custom config options are organized as a tree. From my experience so far the nodes in that tree have 1-10 children each, while I can imagine this to be 20 to 30 or even 50 in rare cases, but surely not more. It seems reasonable to not store a package's options as a nested alist, though. I'd rather consider using a tree implementation in a class, which would for example make it cleaner to encapsulate the behaviour and e.g. implement type checking in that class rather than in the accessor functions, or enable alternative tree implementations if that should become interesting for custom data storage. We have a tree implementation in oll-core at https://github.com/openlilylib/oll-core/blob/master/scheme/oll-core/tree.scm Would that be something to use here? My idea is to put that in an .scm file, say, scm/options.scm, and the accessor code in a file ly/options.ly. Additionally there's a file ly/packages.ly that will define a \usepackage command. \usepackage will check if there's already an element in the options alist for the given package name, and if not it will create one with a new tree object. The package file is then free to register options which will implicitly be stored in the tree object. Does that look like a viable approach? Urs
