Re: [racket-dev] experiment reorganizing the repo into packages
Yesterday, Matthew Flatt wrote: > At Mon, 3 Jun 2013 10:36:51 -0400, Eli Barzilay wrote: > > (BTW, a possible source of confusion: I'm assuming that > > distribution must be done via archives and not via repository > > specs, since there should be some way to put the compiled files in > > there. > > I don't think that's the right assumption. We want it to have > packages that combine source with built entities, but I don't think > we want to require it. Requiring it would break the way GitHub > repositories are meant to be used as packages. But they are required for all non-source things anyway, right? And given that the plt packages have binary distributions, then they should be someplace else anyway. If the sources *do* come from the repository, how does it get a specific version that corresponds to whatever was distributed in binary form? If I have v1.2.3 installed, and it came with web server as precompiled zo files, and I install the source -- how do I get that specific version of the source? The only thing I can think of is relying on tags but that's a bad assumption since it assumes that package owners keep and maintain the proper tags, and the release should come with adding tags in all of these repos -- and that contradicts letting people do whatever they want to do. This is going further off-current-topic, but when I talked to Ryan about possibilities for revising the release process, I thought that the thing that makes most sense is that at the branch creation time (which still happens as usual in the core package repo), package owners hand over a sha1 that corresponds to their recent version. That gets used by the build (possibly via a meta-repo that has these sha1s as submodule specifications) script which gets these versions to build the packages. The equivalent of me making a "merge this commit" request is now turning into asking Ryan to change my sha1 to a newer one, and Ryan can now use a diff between the two sha1s to review the changes that I made. The main point here is that it makes the package repositories be completely independent: package maintainers can keep a `release' branch and release tags if they want, or they can avoid the whole thing. And in fact I think that the much more common case would be to not having a release branch etc. But going back to the on-topic point: this whole plan assumes that the sha1 is what gets used to pull the content and build zipped package archives. > > You've mentioned at some point a "binary catalog" which seems > > unnecessary to me, but maybe there's something I don't get here.) > > Yes, binary packages are another way to address the underlying > issue. > > I'm going to try to synthesize the progress and discussion so far in > a new message, and hopefully I'll managed to explain binary packages > this time around. It would be good to take the above into account, as another reason for doing this which is independent of actual binary files. -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
At Mon, 3 Jun 2013 10:36:51 -0400, Eli Barzilay wrote: > (BTW, a possible source of confusion: I'm assuming that distribution > must be done via archives and not via repository specs, since there > should be some way to put the compiled files in there. I don't think that's the right assumption. We want it to have packages that combine source with built entities, but I don't think we want to require it. Requiring it would break the way GitHub repositories are meant to be used as packages. > You've > mentioned at some point a "binary catalog" which seems unnecessary to > me, but maybe there's something I don't get here.) Yes, binary packages are another way to address the underlying issue. I'm going to try to synthesize the progress and discussion so far in a new message, and hopefully I'll managed to explain binary packages this time around. _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
50 minutes ago, Matthew Flatt wrote: > At Mon, 3 Jun 2013 08:27:19 -0400, Eli Barzilay wrote: > > > > A very cheap way to do this is to use these sub-package > > specifications only for creating packages for distribution. With > > the obvious resulting package file names, this means that there is > > no change at all that is needed on the consumer side. > > I don't understand the suggestion. As a concrete example, can you > sketch our how a user installs the "web-server" package without > documentation, and how the package manager later knows to upgrade > the "web-server" package to include documentation when requested by > the user? On the building side, you'd run some command/script/whatever that goes over the distribution kinds and generates a package for each -- so if you have (define distributions '([lib "*.rkt"] [doc "*.scrbl" "scribblings"])) the result will be "web-server-lib.zip", "web-server-doc.zip", and "web-server.zip" where the last one contains dependencies on the other two. (So this last one is the only one that is special since it's completely made up.) Then all of these files will get distributed in the same way they would be with the directory/repo split. (BTW, a possible source of confusion: I'm assuming that distribution must be done via archives and not via repository specs, since there should be some way to put the compiled files in there. You've mentioned at some point a "binary catalog" which seems unnecessary to me, but maybe there's something I don't get here.) Updates would happen in a similar way, with new zip files. -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
At Mon, 3 Jun 2013 08:27:19 -0400, Eli Barzilay wrote: > On Thursday, Matthew Flatt wrote: > > > > You've sketched out the producer side, and I'm not sure of some > > about some of those details. The consumer side seems even more > > complex to me. It seems like the package system would have to keep > > track of which subpackages are installed for a package, provide an > > interface to the user (e.g., in the GUI) for subpackages, and be > > able to update a package with new subpackages --- all while tracking > > dependencies at the level of subpackages. I'm sure it can all be > > done, but I'm not sure how difficult or important it will be > > relative to everything else that still needs to be done, and I'm > > pretty sure it can't all be done right now. > > A very cheap way to do this is to use these sub-package specifications > only for creating packages for distribution. With the obvious > resulting package file names, this means that there is no change at > all that is needed on the consumer side. I don't understand the suggestion. As a concrete example, can you sketch our how a user installs the "web-server" package without documentation, and how the package manager later knows to upgrade the "web-server" package to include documentation when requested by the user? _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
On Thursday, Matthew Flatt wrote: > > You've sketched out the producer side, and I'm not sure of some > about some of those details. The consumer side seems even more > complex to me. It seems like the package system would have to keep > track of which subpackages are installed for a package, provide an > interface to the user (e.g., in the GUI) for subpackages, and be > able to update a package with new subpackages --- all while tracking > dependencies at the level of subpackages. I'm sure it can all be > done, but I'm not sure how difficult or important it will be > relative to everything else that still needs to be done, and I'm > pretty sure it can't all be done right now. A very cheap way to do this is to use these sub-package specifications only for creating packages for distribution. With the obvious resulting package file names, this means that there is no change at all that is needed on the consumer side. And I think that while it looks like it reduces the whole thing considerably, it's still worth it for some important advantages that are still kept: * No need to move files around in a distribution-oriented layout in code repositories (which is important since people really maintain the freedom of puting files in a way that fits their needs logically). * No restriction on a few known "-foo" names -- keeping it open for new kinds of "package kinds". (Related note: I spoke to Ryan about having db in the minimal core and the possible reason for this being due to sqlite -- it would be nice, and according to Ryan not too difficult, to split the db package into several "db-foo" backends, which hooks back into my point of some splittings that make sense in just one package.) * Minimize code hacks that compensate for the lack of such sub-packages. There will still be some hacks -- for example, it will be convenient in the beginning to just show available package names and the end user will need to figure out what they mean, and when later on you want to grouping back packages that really came from a single source for the UI, instead of some hack that is based on names, it can use this information instead. All of these are important IMO, but the last one is particularly good, since hacking something based on naming conventions means that these hacks will stay around for a very long time -- which will lead to yet more hacks. In the spirit of making things usable, I also worry about accumulation of command-line flags, a problem that plagued .plt file generation and installation (making it into black magic that not many people could handle). > Instead of trying to create subpackages, we might stick with a > naming convention. Even with a naming convention, it might make > sense to extend the package-manager GUI so that it recognizes the > convention and groups related packages together in a list view (but > all the machinery remains at the package level). (This is an example for the kind of hacks where some of them can be avoided with the above, and with others I think that it will be easier to "recover" proper code after a transition period with such hacks.) > Also, a package implementor is free to develop "subpackages" along > the lines you suggest, but splitting out the subpackages into > packages before handling them to a catalog. (And having scripts to do such splitting by developers would not be needed at all.) > Or, we could just not try to slice packages as finely for now, so > that the "-lib" and "-docs" packages of the experimental split are > merged together --- leaving a future split possibly to subpackages. > That will most likely result in a coarse layering that is close to > "build-deps" up to the "drracket" layer, but still provide more > separation for things that are outside the layer needed by DrRacket > itself; for example, "games", "redex", "htdp", "plai", > "picturing-programs", and "plot" can still be separate packages, I > think. FWIW, I'd view this (avoid fine slicing) as a possible way to avoid growing hacks, but I really like the idea of smaller packages and the benefits it comes with. -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
As I understand things, we should certainly try to align divisions of responsibility with divisions of code (including tests and docs) among packages. My initial experiment probably doesn't align them very well, and I'd expect a better alignment to emerge from everyone's efforts to improve some initial cut. At Thu, 30 May 2013 11:02:16 -0400, Matthias Felleisen wrote: > > Matthew, > > I have seen 'personal responsibility' mentioned only in passing. > > Does existence of package imply any personal responsibilities? > The current organization doesn't seem to say so. > > -- Matthias > > _ > Racket Developers list: > http://lists.racket-lang.org/dev _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
Matthew, I have seen 'personal responsibility' mentioned only in passing. Does existence of package imply any personal responsibilities? The current organization doesn't seem to say so. -- Matthias _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
At Thu, 30 May 2013 10:02:29 -0400, Sam Tobin-Hochstadt wrote: > One question, though: how does this work with the package manager. > Right now, only repositories can be specified as the source when > creating a package at pkg.racket-lang.org. There must be something > I'm missing about how this will work. A repository spec is github://github.com where the can be a subdirectory name. So, if the Scribble repo is github://github.com/plt/scribble, then scribble-lib => github://github.com/plt/scribble/master/scribble-lib scribble-docs => github://github.com/plt/scribble/master/scribble-docs scribble => github://github.com/plt/scribble/master/scribble should work. That does add yet another directory layer if we try to make the "scribble" repo a submodule of another repo, though. _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
On Thu, May 30, 2013 at 9:07 AM, Matthew Flatt wrote: > At Thu, 30 May 2013 09:01:16 -0400, Sam Tobin-Hochstadt wrote: >> I also really don't want to have Typed Racket's documentation outside >> of Typed Racket's code repository, and I think it would be a mistake >> to do that for other parts of Racket. An on-point comment on this >> from just yesterday in a different context: >> https://twitter.com/domenic/status/339848565838983168 and I think the >> same reasons that's true would make splitting repositories for code >> and documentation a big mistake. > > Just to make sure that everyone is on the same page: different packages > do not imply different repositories. > > When packages are in a single repository, though, things work best if > each package has its own subdirectory. I would imagine, for example, > that "scribble-lib", "scribble-doc", and "scribble" would all be > directories in a single repository (referenced by three packages), for > example. > > It may even make sense to have certain different packages in a single > repository whether or not we split "-lib" and "-doc" into packages. Ok, that's reassuring. In that case, I'm ambivalent between 2 & 3. One question, though: how does this work with the package manager. Right now, only repositories can be specified as the source when creating a package at pkg.racket-lang.org. There must be something I'm missing about how this will work. Sam _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
At Thu, 30 May 2013 09:01:16 -0400, Sam Tobin-Hochstadt wrote: > I also really don't want to have Typed Racket's documentation outside > of Typed Racket's code repository, and I think it would be a mistake > to do that for other parts of Racket. An on-point comment on this > from just yesterday in a different context: > https://twitter.com/domenic/status/339848565838983168 and I think the > same reasons that's true would make splitting repositories for code > and documentation a big mistake. Just to make sure that everyone is on the same page: different packages do not imply different repositories. When packages are in a single repository, though, things work best if each package has its own subdirectory. I would imagine, for example, that "scribble-lib", "scribble-doc", and "scribble" would all be directories in a single repository (referenced by three packages), for example. It may even make sense to have certain different packages in a single repository whether or not we split "-lib" and "-doc" into packages. _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
On Thu, May 30, 2013 at 8:50 AM, Matthew Flatt wrote:
>
> To summarize, I see our options as
>
> 1. Suspend the effort to reorganize our repository into packages and
> instead start experimenting with subpackages.
>
> 2. Use a naming convention and keep the same kind of split as in the
> experimental reorganization --- and decide on some initial
> conventions ("-lib", "-doc", "-typed", and "-exe"?).
>
> 3. Reorganize our repository into packages, but don't try to split out
> things that conceptually belong together but have different kinds
> of dependencies (such as run-time code versus documentation).
>
> My preference in order is 2, 3, 1.
I don't think we should think of 1 as an option at this point.
I also really don't want to have Typed Racket's documentation outside
of Typed Racket's code repository, and I think it would be a mistake
to do that for other parts of Racket. An on-point comment on this
from just yesterday in a different context:
https://twitter.com/domenic/status/339848565838983168 and I think the
same reasons that's true would make splitting repositories for code
and documentation a big mistake.
That doesn't mean that "-typed" versions of some packages doesn't make
sense, though (gui-typed, for example).
I would significantly prefer the binary packages idea to splitting the
repositories.
Sam
_
Racket Developers list:
http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
At Wed, 29 May 2013 15:51:30 -0400, Eli Barzilay wrote:
> 20 minutes ago, Matthew Flatt wrote:
> > > [..."package distribution kinds"...]
> >
> > Well, I agree with all these thoughts, but what's the conclusion?
> >
> > There's no requirement that packages be in multiple repositories,
> > but different packages in a single repository currently have to be
> > different subdirectories in the repository. Do we need (or do we
> > really want) a notion of "subpackages" to support some general form
> > of splitting within a package?
>
> I think so.
I've been trying to think through the details, and I don't see this as
the right direction for now.
You've sketched out the producer side, and I'm not sure of some about
some of those details. The consumer side seems even more complex to me.
It seems like the package system would have to keep track of which
subpackages are installed for a package, provide an interface to the
user (e.g., in the GUI) for subpackages, and be able to update a
package with new subpackages --- all while tracking dependencies at the
level of subpackages. I'm sure it can all be done, but I'm not sure how
difficult or important it will be relative to everything else that
still needs to be done, and I'm pretty sure it can't all be done right
now.
I'm inclined to think that the analogy to submodules is apt at the
level of the design process: we should try plain packages, first, and
when we understand them better try to improve with an extension to
subpackages.
Instead of trying to create subpackages, we might stick with a naming
convention. Even with a naming convention, it might make sense to
extend the package-manager GUI so that it recognizes the convention and
groups related packages together in a list view (but all the machinery
remains at the package level). Also, a package implementor is free to
develop "subpackages" along the lines you suggest, but splitting out
the subpackages into packages before handling them to a catalog.
Or, we could just not try to slice packages as finely for now, so that
the "-lib" and "-docs" packages of the experimental split are merged
together --- leaving a future split possibly to subpackages. That will
most likely result in a coarse layering that is close to "build-deps"
up to the "drracket" layer, but still provide more separation for
things that are outside the layer needed by DrRacket itself; for
example, "games", "redex", "htdp", "plai", "picturing-programs", and
"plot" can still be separate packages, I think.
To summarize, I see our options as
1. Suspend the effort to reorganize our repository into packages and
instead start experimenting with subpackages.
2. Use a naming convention and keep the same kind of split as in the
experimental reorganization --- and decide on some initial
conventions ("-lib", "-doc", "-typed", and "-exe"?).
3. Reorganize our repository into packages, but don't try to split out
things that conceptually belong together but have different kinds
of dependencies (such as run-time code versus documentation).
My preference in order is 2, 3, 1.
_
Racket Developers list:
http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
[shuffled] 20 minutes ago, Matthew Flatt wrote: > Just to be clear, I agree that we want to minimize cycles. > > I don't think that actually prohibiting cycles is going to be workable, > though, especially at the level of documentation. > > Cycles at the level of "-lib" suggest to me that the organization > should be improved. > > Cycles at the level of "-doc" suggest to me that the documentation is > thorough and helpful. I completely agree with the first two paragraphs, and with the two conclusions. Specifically, it seems to me that allowing all cycles first, discouraging code dependencies next, and trying to get to no code cycles last and possibly finally prohibiting them is a very practical approach to dealing with the dependency mess. (And here "code" is source code except documentation source.) Trying to do the right thing now is probably hard enough to make it not actually happen, making it impractical. I'm also very happy (and positively surprised) to hear you say that most -lib packages are not cyclic. 30 minutes ago, Matthew Flatt wrote: > [You had *lots* of comments about specific packages --- which are > certainly welcome, but let's tackle the big picture, first.] Absolutely. Like I said, much of it is a laundry list to keep in some todo list, which I did in an attempt to help organizing it and since I'm instinctively looking for these kind of issues... (BTW, I see now another possible problem: both the xrepl packages have an info file, with a scribblings entry in one of them.) Now, about this higher level issue: > > * I usually like the fine split to packages like -docs and -lib. > > However, minor subpoints: > > > > - Shouldn't the "-docs" suffix be "-doc" to be in-line with the > > directory name and also symmetric to "-lib"? (BTW, looking at > > the Fedora installations that I have, there are both > > conventions, but "-doc" is much more common than "-docs" , and > > "-libs" is a little more common than "-lib".) (This is a technicality.) > > - I think that a -lib package is for cases where there is some > > use for it by itself, and in some cases there's no need for > > that. Things that I've seen: > > - "xrepl-lib" -- there's no library functionality in xrepl > > that is useful by itself, and if there is (or if something > > does become useful), then it should move out of it. > > - "at-exp-lib" -- the "lib" in the name seems confusing > > (because there is no "at-exp", and I don't think that it's > > right to have a "-lib"-only package...) > > But I was confused: These items make sense after I saw the rest of it -- and pending a nicer solution (to avoid the multiple repositories/directories thing). > > [..."package distribution kinds"...] > > Well, I agree with all these thoughts, but what's the conclusion? > > There's no requirement that packages be in multiple repositories, > but different packages in a single repository currently have to be > different subdirectories in the repository. Do we need (or do we > really want) a notion of "subpackages" to support some general form > of splitting within a package? I think so. I have several reasons for this. First, I don't see these kinds as a closed set -- it's easy to imagine new kinds of distributions (devel vs application, compiled & optimized vs debuggable, textual/command-line vs gui, drracket tool vs emacs or vim plugin, etc etc). Worse, it's easy to imagine such kinds that are applicable in contexts that are specific to just a few or even one package (like the UI language for the string constants or for a language analysis tool, or different foo themes for the foo application). Second, given that they're open-ended, I can see people revising their packages by adding such kinds later. An easy example: take a random collection, and add a small TR interface -- probably no change -- extend it to have the full functionality available for TR, and that will cause changes. Similarly, make lazy into a more serious language, and now many packages will want such changes. And even if it's a directory-per-kind, the thought of moving files around because of the different distribution modes seems wrong. The first thing makes me want to have user-definable distribution kinds. The second thing makes me want to do that via a level of indirection. Taking both, I'd like to see something like this: (define name "eXtended REPL") (define distributions '([lib "*.rkt"] [doc "*.scrbl" "scribblings"] [he "hebrew-commands"] [binary-debugging "libs/portable-gdb.so"])) And I'd like that to be used for distribution: either to create packages (of different kinds), or more easily, pack up the whole thing and have the client decide which parts to actually install (as with some windows installers). There's a bunch of technical questions here. Like the specification of file subsets to use: filters, unions/intersections, etc. Given that
Re: [racket-dev] experiment reorganizing the repo into packages
Just to be clear, I agree that we want to minimize cycles. I don't think that actually prohibiting cycles is going to be workable, though, especially at the level of documentation. Cycles at the level of "-lib" suggest to me that the organization should be improved. Cycles at the level of "-doc" suggest to me that the documentation is thorough and helpful. At Wed, 29 May 2013 14:31:08 -0400, Eli Barzilay wrote: > 20 minutes ago, Matthew Flatt wrote: > > > > Yes. Package dependencies can be cyclic (unlike module > > dependencies), and packages can have cyclic build dependencies > > without cyclic run dependencies. > > (*sigh*) So the idea of acyclic package graph as module containers is > dead? > > But to explain my sigh -- I'm not surprised given my other comment > (sent prematurely); I just hope to see it still as a desirable goal. > Maybe some quick package-level graph to show cycles and try to reduce > them at easy points? > > (I think that still having lots of cycles will make things more > difficult with packages, so that will be one form of discouragement, > but there will also be problems that are harder to deal with.) > > -- > ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: > http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
At Wed, 29 May 2013 14:14:11 -0400, Eli Barzilay wrote: > * Does this split actually work wrt having no circular dependencies? It's mostly non-circular, but there are some exceptions. I think there are no circularities among the "-lib" packages, but I'm not 100% certain. The existing "-docs" packages are probably all ultimately mutually dependent. That's not all documentation, though, and there are still some unresolved links; maybe we want to add more dependencies, or maybe we want to use the experimental "indirect" options for section and module-name references. I should just write a program to draw the graph. > * I usually like the fine split to packages like -docs and -lib. > However, minor subpoints: > > - Shouldn't the "-docs" suffix be "-doc" to be in-line with the > directory name and also symmetric to "-lib"? (BTW, looking at the > Fedora installations that I have, there are both conventions, but > "-doc" is much more common than "-docs" , and "-libs" is a little > more common than "-lib".) > - I think that a -lib package is for cases where there is some use > for it by itself, and in some cases there's no need for that. > Things that I've seen: > - "xrepl-lib" -- there's no library functionality in xrepl that is > useful by itself, and if there is (or if something does become > useful), then it should move out of it. > > - "at-exp-lib" -- the "lib" in the name seems confusing (because > there is no "at-exp", and I don't think that it's right to have > a "-lib"-only package...) > But I was confused: > > - Also, I'm looking at the "draw" package (and now that I see it, > "xrepl" too), and it looks like a hack that is compensating for > an inability to specify some "doc of" relation between packages. > (It's useful to have meta-packages, of course, it just looks like > the wrong level to specify such relationships.) The fact that I > was confused about it seems to support this... > > - I'm also worried about the repo-implications of such splittage: > would there actually be three repositories for "foo", "foo-doc", > and "foo-lib"? If so, then the price of not having in-package > specification for these things seem much higher; I will certainly > not want to see three such repos for xrepl... > > - There are also some "-lib" packages (like string-constants-lib) > that have no other forms. (Especially in the string-constant > thing, having it be a -lib seems wrong to me.) > > - Similar issues are going to be relevant for other "package kinds" > like "-typed". (The following sentence written after I've seen > much more:) So the more I see these things, the more I think that > it's bad to rely or encourage on this kind of post-pended suffix > as an indication of the package type -- I'd rather see these > things more formal and hopefully all distributable from within a > single per-real-package repository instead of an extra > meta-package for each. Well, I agree with all these thoughts, but what's the conclusion? There's no requirement that packages be in multiple repositories, but different packages in a single repository currently have to be different subdirectories in the repository. Do we need (or do we really want) a notion of "subpackages" to support some general form of splitting within a package? [You had *lots* of comments about specific packages --- which are certainly welcome, but let's tackle the big picture, first.] _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
20 minutes ago, Matthew Flatt wrote: > > Yes. Package dependencies can be cyclic (unlike module > dependencies), and packages can have cyclic build dependencies > without cyclic run dependencies. (*sigh*) So the idea of acyclic package graph as module containers is dead? But to explain my sigh -- I'm not surprised given my other comment (sent prematurely); I just hope to see it still as a desirable goal. Maybe some quick package-level graph to show cycles and try to reduce them at easy points? (I think that still having lots of cycles will make things more difficult with packages, so that will be one form of discouragement, but there will also be problems that are harder to deal with.) -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
An hour ago, Neil Toronto wrote: > > I expect `plot' to depend on `math' in the near-ish future. Right > now, `math' already depends on `plot' to build its docs. IOW, I > expect the sources to be interdependent, but as binaries, `plot' > will depend on `math' but `math' won't depend on `plot'. Can the > package system handle that, or will I have to merge the two or > factor out a common parent? I'm guessing that you won't -- I don't know enough details, but one thing that I can guarantee is that this kind of situation is nothing compared to some of the existing dependency spaghetti... -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
Yesterday, Robby Findler wrote: > This looks great to me! > > I don't have a good sense of what level of granularity is the right > one, but I naturally would have gone even finer grained with > drracket: the macro-debugger, pkg/gui, and maybe even the > gui-debugger I would have separated out. (Probably you were focused > on lower-level things first, tho.) > > Also for the macro-debugger: I think it is useful by itself, without > drracket (I would like to use it for files that in drracket's > implementation, for example). So I think there should be two > packages there: macro-debugger-standalone with the "give me a file > and show the macro debugger's result on it" interface and > macro-debugger-plugin pkg that fits into drracket. Besides the usable-by-itself point, I think that there's a strong point for code owners... In this case the owners are very different (and my guess is that any contributions to the code you do/might make are a kind of a suggestion), so it's better to split them; and also there is no sharing of code + maintenance (which is why this point doesn't apply to core stuff). -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
Yesterday, Sam Tobin-Hochstadt wrote: > > One question -- a bunch of unstable seems to be in the > typed-racket-lib package. Why is that? Since these things are used in TR only, I think that they should move back. Ie, you get a trimmed TR repo with unstable/stuff which you move to a better place on your own... (Or do so now and save some splitting work.) -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
At Wed, 29 May 2013 11:11:18 -0600, Neil Toronto wrote: > On 05/28/2013 03:44 PM, Sam Tobin-Hochstadt wrote: > > On Tue, May 28, 2013 at 5:14 PM, Matthew Flatt wrote: > >> > >> I don't know whether the "-lib"/"-docs" split is worthwhile, but it's > >> part of erring on the side of breaking things apart. Maybe it makes > >> more sense to keep things together and rely on binary packaging to > >> reduce dependencies. > > Would using binary packaging for that be more complicated? Relying on binary packaging is simpler for an author, since it means not splitting into "-lib" and "-docs" directories. The drawback of binary packaging is for users: someone who installs the package does not get source. > >> So, how does this split correspond to what you expected? (My guess is > >> that this far too fine-grained for some of us, while others will want > >> exactly this kind of flexibility.) > > I expect `plot' to depend on `math' in the near-ish future. Right now, > `math' already depends on `plot' to build its docs. IOW, I expect the > sources to be interdependent, but as binaries, `plot' will depend on > `math' but `math' won't depend on `plot'. Can the package system handle > that[...]? Yes. Package dependencies can be cyclic (unlike module dependencies), and packages can have cyclic build dependencies without cyclic run dependencies. _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
Yesterday, Matthew Flatt wrote:
> Here's a first experiment at moving collections around into packages:
>
> https://github.com/mflatt/racket/tree/pkg
> [...]
Comments in no particular order, mostly about the actual file
selections. Many are kind of small todo-style laundry items, some are
repeating obvious things -- I missed starting with the readme file...
In any case, it should now be obvious that this is going to be long.
* This is *impressive*! [=> The rest of these items are not too
strong (at least not ATM), since it looks good enough to be good
anyway.]
* Does this split actually work wrt having no circular dependencies?
(It's pretty amazing if it does -- it's what I've been worried about
for a long time...)
* I usually like the fine split to packages like -docs and -lib.
However, minor subpoints:
- Shouldn't the "-docs" suffix be "-doc" to be in-line with the
directory name and also symmetric to "-lib"? (BTW, looking at the
Fedora installations that I have, there are both conventions, but
"-doc" is much more common than "-docs" , and "-libs" is a little
more common than "-lib".)
- I think that a -lib package is for cases where there is some use
for it by itself, and in some cases there's no need for that.
Things that I've seen:
- "xrepl-lib" -- there's no library functionality in xrepl that is
useful by itself, and if there is (or if something does become
useful), then it should move out of it.
- "at-exp-lib" -- the "lib" in the name seems confusing (because
there is no "at-exp", and I don't think that it's right to have
a "-lib"-only package...)
But I was confused:
- Also, I'm looking at the "draw" package (and now that I see it,
"xrepl" too), and it looks like a hack that is compensating for
an inability to specify some "doc of" relation between packages.
(It's useful to have meta-packages, of course, it just looks like
the wrong level to specify such relationships.) The fact that I
was confused about it seems to support this...
- I'm also worried about the repo-implications of such splittage:
would there actually be three repositories for "foo", "foo-doc",
and "foo-lib"? If so, then the price of not having in-package
specification for these things seem much higher; I will certainly
not want to see three such repos for xrepl...
- There are also some "-lib" packages (like string-constants-lib)
that have no other forms. (Especially in the string-constant
thing, having it be a -lib seems wrong to me.)
- Similar issues are going to be relevant for other "package kinds"
like "-typed". (The following sentence written after I've seen
much more:) So the more I see these things, the more I think that
it's bad to rely or encourage on this kind of post-pended suffix
as an indication of the package type -- I'd rather see these
things more formal and hopefully all distributable from within a
single per-real-package repository instead of an extra
meta-package for each.
* I think that "docs-index" would be better named as "core-docs" or
something similar ("core" in the sense of making up the main doc
pages, not in the sense of a documentation of the core).
* Given that there are a *lot* of packages, it'll probably make things
convenient to have some meta-repository with submodules after the
split. Just as a convenience thing to get the whole set (or some
useful subset) without listing them. But not as something that
would be used for actual stuff. (With submodules once you commit a
subrepo you usually want to commit the toplevel which has a sha1
pointer to each sub-repo -- the idea is that you can commit work
separately from what you get for the toplevel directory, but I'm
talking about something simple like some script that will always
update the pointer, or ignore the pointers and always use the master
of the sub-repos.)
* As a random idea, maybe it would be a good time to move things like
"drscheme" or the "net/*-{unit,sig}.rkt" files into some -compat
packages? Or maybe that's better done later on by the package
maintainers. (Including things like "scheme", eventually.)
* I think that the contents of drracket/rackunit should move into some
"rackunit-gui" package, or maybe "rackunit-tool" is better. Same
goes for drracket/scribble (=> "scribble-tool").
* Is "browser" really needed? From a random grep, it looks like the
only thing that is needed there is `browser/external', which is
actually a wrapper around `net/send-url' so maybe leave just that
there?
* Similarly, "help/bug-report*" are obviously better moved into
drracket... (Obviously to me, at least.) And also "pkg/gui" (which
is even worse, since it looks like a weird name from a bad past
hierarchy...). And a few others (slideshow, texpict).
* The drracket/macro-debugger directory looks like something that
should be i
Re: [racket-dev] experiment reorganizing the repo into packages
On 05/28/2013 03:44 PM, Sam Tobin-Hochstadt wrote: On Tue, May 28, 2013 at 5:14 PM, Matthew Flatt wrote: I don't know whether the "-lib"/"-docs" split is worthwhile, but it's part of erring on the side of breaking things apart. Maybe it makes more sense to keep things together and rely on binary packaging to reduce dependencies. Would using binary packaging for that be more complicated? It seems that not splitting the source would keep things simpler for most developers, so I'd lean that way. But it's not a strong opinion. For Typed Racket, it seems worth it in one place, and isn't needed in another. For the core of TR, making it available without docs is useful. I'm less sure about the contents of `typed/*`. I suppose that will depend on whether those packages themselves ship with documentation. For the core of TR, could the uses you're thinking about benefit just as much from a binary-only core TR? (I imagine you mean other packages like `math', which just use the TR language, but I'm not sure.) Also worth noting is that the "unstable" collection does not work nicely as a package --- as should be expected. I created a few "unstable-" packages and imagine many more, but I'm not sure that's the right way to go. I think we should just make all of `unstable` go away as part of this transition. It was mostly a way to work around the monolithic nature of the collections tree, and thus has outlived its purpose. That sounds right to me. So, how does this split correspond to what you expected? (My guess is that this far too fine-grained for some of us, while others will want exactly this kind of flexibility.) I expect `plot' to depend on `math' in the near-ish future. Right now, `math' already depends on `plot' to build its docs. IOW, I expect the sources to be interdependent, but as binaries, `plot' will depend on `math' but `math' won't depend on `plot'. Can the package system handle that, or will I have to merge the two or factor out a common parent? If the package system won't handle it, I'd rather have both the `plot' and the `math' collections in a single `math' package. Neil ⊥ _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
On Tue, May 28, 2013 at 8:11 PM, Matthew Flatt wrote: > At Tue, 28 May 2013 19:36:14 -0500, Robby Findler wrote: > > I don't have a good sense of what level of granularity is the right one, > > but I naturally would have gone even finer grained with drracket: the > > macro-debugger, pkg/gui, and maybe even the gui-debugger I would have > > separated out. (Probably you were focused on lower-level things first, > > tho.) > > Sounds good. > > > Also for the macro-debugger: I think it is useful by itself, without > > drracket (I would like to use it for files that in drracket's > > implementation, for example). So I think there should be two packages > > there: macro-debugger-standalone with the "give me a file and show the > > macro debugger's result on it" interface and macro-debugger-plugin pkg > that > > fits into drracket. > > I think it's close to that in the current experiment, actually. The > "macro-debugger" collection is spread across "macro-debugger-text-lib" > (which is used by "xrepl-lib"), "macro-debugger" (probably close to a > stand-alone application, if not there already), and "drracket" (which > just has "tool.rkt"). > > Oh! I see that now. Sorry. Robby _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
At Tue, 28 May 2013 19:36:14 -0500, Robby Findler wrote: > I don't have a good sense of what level of granularity is the right one, > but I naturally would have gone even finer grained with drracket: the > macro-debugger, pkg/gui, and maybe even the gui-debugger I would have > separated out. (Probably you were focused on lower-level things first, > tho.) Sounds good. > Also for the macro-debugger: I think it is useful by itself, without > drracket (I would like to use it for files that in drracket's > implementation, for example). So I think there should be two packages > there: macro-debugger-standalone with the "give me a file and show the > macro debugger's result on it" interface and macro-debugger-plugin pkg that > fits into drracket. I think it's close to that in the current experiment, actually. The "macro-debugger" collection is spread across "macro-debugger-text-lib" (which is used by "xrepl-lib"), "macro-debugger" (probably close to a stand-alone application, if not there already), and "drracket" (which just has "tool.rkt"). _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
At Tue, 28 May 2013 17:44:08 -0400, Sam Tobin-Hochstadt wrote: > One question -- a bunch of unstable seems to be in the > typed-racket-lib package. Why is that? Just a shortcut. Typed Racket seemed to be the only user of the libraries in the repo, so I put the libraries there for now. > > So, how does this split correspond to what you expected? (My guess is > > that this far too fine-grained for some of us, while others will want > > exactly this kind of flexibility.) > > For me, I'd expect the `typed-racket-more` package to be smaller, and > probably to be distributed to some of its dependencies. Or perhaps > split out into things like `rackunit-typed` and `gui-typed`. That makes sense to me. _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
This looks great to me! I don't have a good sense of what level of granularity is the right one, but I naturally would have gone even finer grained with drracket: the macro-debugger, pkg/gui, and maybe even the gui-debugger I would have separated out. (Probably you were focused on lower-level things first, tho.) Also for the macro-debugger: I think it is useful by itself, without drracket (I would like to use it for files that in drracket's implementation, for example). So I think there should be two packages there: macro-debugger-standalone with the "give me a file and show the macro debugger's result on it" interface and macro-debugger-plugin pkg that fits into drracket. Robby On Tue, May 28, 2013 at 4:14 PM, Matthew Flatt wrote: > Here's a first experiment at moving collections around into packages: > > https://github.com/mflatt/racket/tree/pkg > > That repository really is meant as an experiment and a point of > discussion --- not a conclusion! > > In particular, this experiment is about what it might look like to > break up the Racket sources. It's about what the packages will actually > be and how finely we want to layer the packages. > > To get the discussion started, I've tried to err on the side of > breaking things apart (i.e., lots of layers). So, you'll see lots of > packages. > > > The "racket" directory in the repo is pretty much the same as the > `min-racket' experiment, while each other directory is a package (or a > sketch of a package, because I didn't get to everything, yet). > > If you build in the "racket" directory, install suitable native-library > packages (see "README.txt") and then run > > racket/bin/raco pkg install -i --link --deps link-auto drracket > > then you should get a Racket variant that includes DrRacket, but not > the teaching languages, Redex, or lots of other things. > > If you instead try > > racket/bin/raco pkg install -i --link --deps link-auto gui-lib > > then you should get a Racket that can run GUI programs, but no DrRacket > or even documentation. > > As the "README.txt" notes, there are many useful packages that you can > install and still keep things relatively small. For example, you can > get a Scribble (without documentation!), Typed Racket, or a web server > with relatively few dependencies. > > > Meanwhile, part of this experiment is defining "-lib" packages that do > not provide documentation, which means they can have fewer build-time > dependencies than they would need for documentation. The "-docs" > packages provide the corresponding documentation. (I'm not sure about > the naming convention; I just had to pick something for now.) > Naturally, the "X" package pulls in both the "X-lib" and "X-doc" > packages. > > I don't know whether the "-lib"/"-docs" split is worthwhile, but it's > part of erring on the side of breaking things apart. Maybe it makes > more sense to keep things together and rely on binary packaging to > reduce dependencies. > > > Also worth noting is that the "unstable" collection does not work > nicely as a package --- as should be expected. I created a few > "unstable-" packages and imagine many more, but I'm not sure that's the > right way to go. > > > So, how does this split correspond to what you expected? (My guess is > that this far too fine-grained for some of us, while others will want > exactly this kind of flexibility.) > > _ > Racket Developers list: > http://lists.racket-lang.org/dev > _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] experiment reorganizing the repo into packages
On Tue, May 28, 2013 at 5:14 PM, Matthew Flatt wrote: > > I don't know whether the "-lib"/"-docs" split is worthwhile, but it's > part of erring on the side of breaking things apart. Maybe it makes > more sense to keep things together and rely on binary packaging to > reduce dependencies. For Typed Racket, it seems worth it in one place, and isn't needed in another. For the core of TR, making it available without docs is useful. I'm less sure about the contents of `typed/*`. I suppose that will depend on whether those packages themselves ship with documentation. > Also worth noting is that the "unstable" collection does not work > nicely as a package --- as should be expected. I created a few > "unstable-" packages and imagine many more, but I'm not sure that's the > right way to go. I think we should just make all of `unstable` go away as part of this transition. It was mostly a way to work around the monolithic nature of the collections tree, and thus has outlived its purpose. One question -- a bunch of unstable seems to be in the typed-racket-lib package. Why is that? > So, how does this split correspond to what you expected? (My guess is > that this far too fine-grained for some of us, while others will want > exactly this kind of flexibility.) For me, I'd expect the `typed-racket-more` package to be smaller, and probably to be distributed to some of its dependencies. Or perhaps split out into things like `rackunit-typed` and `gui-typed`. Other than that, the split makes sense to me for the code I maintain. Sam _ Racket Developers list: http://lists.racket-lang.org/dev
[racket-dev] experiment reorganizing the repo into packages
Here's a first experiment at moving collections around into packages: https://github.com/mflatt/racket/tree/pkg That repository really is meant as an experiment and a point of discussion --- not a conclusion! In particular, this experiment is about what it might look like to break up the Racket sources. It's about what the packages will actually be and how finely we want to layer the packages. To get the discussion started, I've tried to err on the side of breaking things apart (i.e., lots of layers). So, you'll see lots of packages. The "racket" directory in the repo is pretty much the same as the `min-racket' experiment, while each other directory is a package (or a sketch of a package, because I didn't get to everything, yet). If you build in the "racket" directory, install suitable native-library packages (see "README.txt") and then run racket/bin/raco pkg install -i --link --deps link-auto drracket then you should get a Racket variant that includes DrRacket, but not the teaching languages, Redex, or lots of other things. If you instead try racket/bin/raco pkg install -i --link --deps link-auto gui-lib then you should get a Racket that can run GUI programs, but no DrRacket or even documentation. As the "README.txt" notes, there are many useful packages that you can install and still keep things relatively small. For example, you can get a Scribble (without documentation!), Typed Racket, or a web server with relatively few dependencies. Meanwhile, part of this experiment is defining "-lib" packages that do not provide documentation, which means they can have fewer build-time dependencies than they would need for documentation. The "-docs" packages provide the corresponding documentation. (I'm not sure about the naming convention; I just had to pick something for now.) Naturally, the "X" package pulls in both the "X-lib" and "X-doc" packages. I don't know whether the "-lib"/"-docs" split is worthwhile, but it's part of erring on the side of breaking things apart. Maybe it makes more sense to keep things together and rely on binary packaging to reduce dependencies. Also worth noting is that the "unstable" collection does not work nicely as a package --- as should be expected. I created a few "unstable-" packages and imagine many more, but I'm not sure that's the right way to go. So, how does this split correspond to what you expected? (My guess is that this far too fine-grained for some of us, while others will want exactly this kind of flexibility.) _ Racket Developers list: http://lists.racket-lang.org/dev
