Re: Fix #2529: explicit protection package #3651
On 8/26/14, 9:46 AM, Dicebot wrote: ...and it has just been merged! ^_^ Thanks Walter! Nice! This felt like a glaring hole, and you've filled it!
Re: Fix #2529: explicit protection package #3651
..and it has just been merged! ^_^ Thanks Walter!
Re: Fix #2529: explicit protection package #3651
On Tue, 26 Aug 2014 16:46:19 + Dicebot via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: ..and it has just been merged! ^_^ WOW! that's great! Thanks Walter! second that! signature.asc Description: PGP signature
Re: Fix #2529: explicit protection package #3651
On 8/26/14, 9:46 AM, Dicebot wrote: ...and it has just been merged! ^_^ Thanks Walter! Congratulations for a job well done. -- Andrei
Re: Fix #2529: explicit protection package #3651
On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote: On 8/19/2014 7:01 AM, Dicebot wrote: Walter, now that release is out can you please state your opinion about https://github.com/D-Programming-Language/dmd/pull/3651 ? It is blocking Phobos module split and decoupling. I keep thinking there's gotta be a way to do this without language changes. In the meanwhile I have updated https://github.com/D-Programming-Language/dmd/pull/3651 to address latest Walter comments and be based on recent master
Re: Fix #2529: explicit protection package #3651
On Sunday, 24 August 2014 at 14:15:25 UTC, Kagamin wrote: On Sunday, 24 August 2014 at 02:22:41 UTC, Dicebot wrote: Well difference is that internal substring in the fully qualified name that is much more likely to tell user he is better to not touch it. However, original Kagamin proposal of embedding it into module names themselves does address that if you are ok with resulting uglyness. Both ways it's a convention, and I don't see, why such convention should exist in the first place, member protection has nothing to do with module name, which reflects module's functionality. Because we have no other means of enforcing this protection - this is why my PR event exists. On Sunday, 24 August 2014 at 02:34:01 UTC, Dicebot wrote: It can be a philosophical matter, but in my experience grouping by functionality is genuine, and access is an afterthought, so grouping by access doesn't work in the long term, as code naturally (spontaneously) groups by functionality. That does contradict your statement that any stuff used in parent packages must go to up the hierarchy which is exactly grouping by access :) Access there means protection, usage reflects functionality. And how that makes your statement less contradictive? I can probably give a more practical example ... code duplication. 1. Making wrapper protected will preclude writing new serializer. 2. Using wrapper methods can be meaningless without serializer. 3. Serializer may just not expose wrapper, then user will have no way to access it. 4. .net has quite a lot of things like http://msdn.microsoft.com/en-us/library/system.runtime.compilerservices.configuredtaskawaitable%28v=vs.110%29.aspx and nothing explodes even though .net programmers are believed to be really stupid and evil. It's a virtue of Stackoverflow Driven Development. I am afraid I don't really understand what your point here is - that my design is wrong and must be banned or that package concept is flawed and must be reworked? Neither is really an acceptable argument. There is a simple and strict requirement - no one must be able to extend serializers but few developers that routinely maintain that package (and all sub-packages). Never in user code. This idea is not a subject to reconsideration - if language says such design is illegal I am going to call that language bullshit. I'm afraid, hierarchies make things harder to find. If you don't know what is where, flat organization will present you with everything readily available right before your eyes. With deep hierarchy you're left with abstract or poorly chosen categories at every hierarchy level, so effectively you have to walk the entire hierarchy, which is much more tedious than scroll a flat list of modules viewing ten modules per scroll. Badly named list of modules (like what we have now in phobos) scales well up to 100, well-named list is much more manageable: if you need xml, you already know it's near the end of the list - it's easy to find even among 1000 files - you don't ever need to scroll entire list. If it's not there, where do you go? There's no obvious answer. So even shallow hierarchy is more troublesome than a flat list of 1000 modules. I don't believe hierarchy will magically solve navigation problems just because it has folders. You are speaking about _finding_ things. I am speaking about _discovering_ things. See the difference? There is no way you can quickly tell aha, so this data formats are supported out of the box when looking at plain flat list of 1000 files. With solid hierarchy it becomes trivial. And System.Xml is not System.Text.Xml, System.Net is not System.IO.Net - where is deep hierarchy? If you want System.Xml, you type `s.x` and it doesn't matter, how many namespaces are there, you're presented with System.Xml. If it's not there, where to find it, in System.Text, System.Formats, System.Parsers, System.Lang, System.Markup, System.Sgml? You pick few that make most sense to you and ignore the rest. If routinely needed module is not exposed via obvious path it is an issue worth filing in the bug tracker. More rarely used / obscure modules can have more surprising paths. I don't see a problem here. Again, searching for a known package is never a problem both ways - you can just, well, run the search query. Discoverability of uknowns is the key.
Re: Fix #2529: explicit protection package #3651
On Sunday, 24 August 2014 at 02:22:41 UTC, Dicebot wrote: Well difference is that internal substring in the fully qualified name that is much more likely to tell user he is better to not touch it. However, original Kagamin proposal of embedding it into module names themselves does address that if you are ok with resulting uglyness. Both ways it's a convention, and I don't see, why such convention should exist in the first place, member protection has nothing to do with module name, which reflects module's functionality. On Sunday, 24 August 2014 at 02:34:01 UTC, Dicebot wrote: It can be a philosophical matter, but in my experience grouping by functionality is genuine, and access is an afterthought, so grouping by access doesn't work in the long term, as code naturally (spontaneously) groups by functionality. That does contradict your statement that any stuff used in parent packages must go to up the hierarchy which is exactly grouping by access :) Access there means protection, usage reflects functionality. I can probably give a more practical example I have just recently encountered when re-designing one of our internal library packages. It was a serialization package and sub-packages defined different (de)serialization models, some of those defining special wrapper types for resulting data that enforce certain semantics specific to that serialization model via the type system. It is not surprising that most internal details of such wrappers were kept package protected as exposing those to user could have easily violated all assumptions (but was needed to implement (de)serialization efficiently). It has become more complicated when meta-serializers have been added to parent package - templated stuff that took any of sub-package implementations and added some functionality (like versioning support) on top. Being generic thing it reside in higher level serialization package but to be implemented efficiently it needs access to those internal wrapper fields. Moving wrapper modules to parent package is not an option here because those are closely related to specific serialization model. Exposing stuff as public is not an option because anyone not deeply familiar with package implementation can easily break all type system guarantees that way. Moving meta-serializers to sub-packages is quite a code duplication. Right now I keep that stuff public and say in docs please don't use this which is hardly a good solution. 1. Making wrapper protected will preclude writing new serializer. 2. Using wrapper methods can be meaningless without serializer. 3. Serializer may just not expose wrapper, then user will have no way to access it. 4. .net has quite a lot of things like http://msdn.microsoft.com/en-us/library/system.runtime.compilerservices.configuredtaskawaitable%28v=vs.110%29.aspx and nothing explodes even though .net programmers are believed to be really stupid and evil. It's a virtue of Stackoverflow Driven Development. On Sunday, 24 August 2014 at 02:39:40 UTC, Dicebot wrote: On Saturday, 23 August 2014 at 09:00:30 UTC, Kagamin wrote: What is difficult to find? With flat structure you have all files right before your eyes. If you need std.datetime.systime module, you open std/datetime/systime.d file - that's the reason of needlessly restricting code structure with modules as if one size fits all. It is the same reasoning as with deep filesystem hierarchies and, well, any data hierarchies - once the element (module / file) count becomes bigger than ~dozen you only really notice things you know to look for. Contrary to that deeply nested categorized hierarchies are easy to casually to search through if you don't know exact module name - just iteratively pick whatever package fits the theme until you find what you want. I'm afraid, hierarchies make things harder to find. If you don't know what is where, flat organization will present you with everything readily available right before your eyes. With deep hierarchy you're left with abstract or poorly chosen categories at every hierarchy level, so effectively you have to walk the entire hierarchy, which is much more tedious than scroll a flat list of modules viewing ten modules per scroll. Badly named list of modules (like what we have now in phobos) scales well up to 100, well-named list is much more manageable: if you need xml, you already know it's near the end of the list - it's easy to find even among 1000 files - you don't ever need to scroll entire list. If it's not there, where do you go? There's no obvious answer. So even shallow hierarchy is more troublesome than a flat list of 1000 modules. I don't believe hierarchy will magically solve navigation problems just because it has folders. I remember coding a bit in C#/.NET platform ages ago - it was totally possible to find relevant modules without even looking in docs, just using auto-complete through suggested package
Re: Fix #2529: explicit protection package #3651
On Friday, 22 August 2014 at 15:06:14 UTC, Dmitry Olshansky wrote: Example structure: mod/package.d mod/internal/bar.d mod/... I actually face it in Phobos right now, my blocked pull: https://github.com/D-Programming-Language/phobos/pull/2412 I think, it should have structure std.regex.package; std.regex.backtracking; std.regex.ir; What is the reason to have regex stuff in a remote directory? That would make things hard to find and navigate.
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 14:40:34 UTC, Dicebot wrote: On Wednesday, 20 August 2014 at 14:36:59 UTC, Kagamin wrote: As I see on the realistic example of datetime, which is BIG, we only need to split it into a flat set of files without an overly deep package hierarchy. We _may_ split it into flat set files (solving only part of the problem) but it is desirable to have a deeper package hierarchy. Package hierarchy is not just an encapsulation tool, it is also a great way to simplify navigation and finding needed modules - something that Phobos is current terrible at exactly because of flat hierarchies. What is difficult to find? With flat structure you have all files right before your eyes. If you need std.datetime.systime module, you open std/datetime/systime.d file - that's the reason of needlessly restricting code structure with modules as if one size fits all.
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 14:40:34 UTC, Dicebot wrote: On Wednesday, 20 August 2014 at 14:36:59 UTC, Kagamin wrote: As I see on the realistic example of datetime, which is BIG, we only need to split it into a flat set of files without an overly deep package hierarchy. We _may_ split it into flat set files (solving only part of the problem) but it is desirable to have a deeper package hierarchy. Package hierarchy is not just an encapsulation tool, it is also a great way to simplify navigation and finding needed modules - something that Phobos is current terrible at exactly because of flat hierarchies. And with an excessively deep folder structure it will be much more tedious to walk the whole thing. It's deep nesting, which will make things hard to find.
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 20:39:06 UTC, Chris Nicholson-Sauls wrote: module foo.bar.one; module foo.bar.internals; // package-protected utilities module foo.bar.subpkg.two; module foo.bar.subpkg.internals; // package-protected utilities and module 'two' cannot access the 'bar' utilities. Really? Looks like a bug. You'd better check if it's intended to work that way. This is needlessly limiting, forcing design choices that should not be dictated by the ability/inability to separate public and private API. It also precludes many valid and good uses of nested package.d modules. If some thing is used in entire foo.bar package, then it's foo.bar's internal, not foo.bar.subpkg's internal. I think, it's natural when the wider the thing is used, the higher in hierarchy it sits. I really don't see any equally strong counter-argument. But then, I've wanted this exact fix for literally years now. I don't really see, what it blocks. It also is not limited to internal utility modules. It can be useful for systems that select at compile time from one of a number of system-specific implementations of a given interface/api. It can be useful for granting privileged access to certain api's and/or resources to specific module(s). An example being: grant access to unsafe but versitile data manipulators solely to the subpackage containing the loaders. If those manipulators are used solely in loaders subpackage, they should belong to that package. Though I don't really see a problem here: there was an implementation of xml document, which allowed unsafe (but fast) specialization, which was believed to be useful for public usage. It fits well into D philosophy that safe things should be available by default and unsafe/fast things should be possible.
Re: Fix #2529: explicit protection package #3651
23-Aug-2014 12:57, Kagamin пишет: On Friday, 22 August 2014 at 15:06:14 UTC, Dmitry Olshansky wrote: Example structure: mod/package.d mod/internal/bar.d mod/... I actually face it in Phobos right now, my blocked pull: https://github.com/D-Programming-Language/phobos/pull/2412 I think, it should have structure std.regex.package; std.regex.backtracking; std.regex.ir; What is the reason to have regex stuff in a remote directory? That would make things hard to find and navigate. Yes, that was my initial pull. The stuff inside was marked as `package`, as only package.d currently has the same public API. Then Dicebot suggested that since all modules are for the moment internal they have to be moved somewhere until public API is fleshed out (as it may change module name etc.). 2 options that both do NOT currently work: std.regex.internal std.internal.regex -- Dmitry Olshansky
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 14:44:21 UTC, Rory McGuire via Digitalmars-d-announce wrote: On Wed, Aug 20, 2014 at 4:33 PM, Kagamin via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: Do we need a hierarchy of internals, is the problem this big? Why mybiglib.wisdom is not good? It gets really unwieldy when you want to put a whole bunch of things into one project and give control of different packages to different teams. Yes you could use multiple libraries in a lot of cases but if you're wanting it in a single setup / package (like phobos), especially one that is internal to another package you need sub-packages. Yes, public interfaces of packages can form a hierarchy (which still shouldn't be more complex than necessary), but internals? Examples that would use this are: - kitchen sink libraries like phobos I proposed an example of datetime module, that I don't see a problem with internals there. Can you point to an exact line in source, which suffers from this?
Re: Fix #2529: explicit protection package #3651
On 2014-08-23 11:12, Kagamin wrote: If some thing is used in entire foo.bar package, then it's foo.bar's internal, not foo.bar.subpkg's internal. I think, it's natural when the wider the thing is used, the higher in hierarchy it sits. A symbol declared package can only be accessed within its own package. It cannot be accessed from sub packages or super packages. -- /Jacob Carlborg
Re: Fix #2529: explicit protection package #3651
On Saturday, 23 August 2014 at 09:14:13 UTC, Dmitry Olshansky wrote: Yes, that was my initial pull. The stuff inside was marked as `package`, as only package.d currently has the same public API. Then Dicebot suggested that since all modules are for the moment internal they have to be moved somewhere until public API is fleshed out (as it may change module name etc.). 2 options that both do NOT currently work: std.regex.internal std.internal.regex If the user relies that a symbol is found in std.internal.regex.backtracking, moving it will break the code too. So what's the difference?
Re: Fix #2529: explicit protection package #3651
On Saturday, 23 August 2014 at 09:24:04 UTC, Jacob Carlborg wrote: A symbol declared package can only be accessed within its own package. It cannot be accessed from sub packages or super packages. I didn't know about that. Should it really work that way?
Re: Fix #2529: explicit protection package #3651
23-Aug-2014 13:33, Kagamin пишет: On Saturday, 23 August 2014 at 09:14:13 UTC, Dmitry Olshansky wrote: Yes, that was my initial pull. The stuff inside was marked as `package`, as only package.d currently has the same public API. Then Dicebot suggested that since all modules are for the moment internal they have to be moved somewhere until public API is fleshed out (as it may change module name etc.). 2 options that both do NOT currently work: std.regex.internal std.internal.regex If the user relies that a symbol is found in std.internal.regex.backtracking, moving it will break the code too. So what's the difference? I don't see the difference. In any case user should see public aliases and never know where they lead to. Dicebot - any objections? However since my pull hang there for months, I decide to appease puristic requests and move things to internals, only to find out that it doesn't work. For what its worth I'd _prefer_ std.regex.internal as in my opinion internals got to be well contained. But as discussed, `package` doesn't work this way. -- Dmitry Olshansky
Re: Fix #2529: explicit protection package #3651
On Saturday, 23 August 2014 at 10:04:40 UTC, Dmitry Olshansky wrote: For what its worth I'd _prefer_ std.regex.internal as in my opinion internals got to be well contained. But as discussed, `package` doesn't work this way. It can be a philosophical matter, but in my experience grouping by functionality is genuine, and access is an afterthought, so grouping by access doesn't work in the long term, as code naturally (spontaneously) groups by functionality. The reason is when you maintain code, you traverse it according to functionality, because you work with implementation, and protection is interface, so protection is ignored. If you add a public functionality to internal module (ranges often end up like that), you can either just qualify it with public access, as it's done in normal modules, or refactor and move it to public module, so you end up with logically grouped functionality separated between two or more modules, which try to enforce grouping by access, while in normal modules you have logically grouped functionality in the same module even if they have different access. Also you can end up with public functionality in internal module, if the member proved to be stable, robust and use case for it arises.
Re: Fix #2529: explicit protection package #3651
On Saturday, 23 August 2014 at 11:04:35 UTC, Jacob Carlborg wrote: That's what we're trying to fix with this change. The proposal is to make internals visible to super packages, I only wonder whether internals visible to some package should be hidden from its subpackages.
Re: Fix #2529: explicit protection package #3651
On Saturday, 23 August 2014 at 10:04:40 UTC, Dmitry Olshansky wrote: If the user relies that a symbol is found in std.internal.regex.backtracking, moving it will break the code too. So what's the difference? I don't see the difference. In any case user should see public aliases and never know where they lead to. Dicebot - any objections? Well difference is that internal substring in the fully qualified name that is much more likely to tell user he is better to not touch it. However, original Kagamin proposal of embedding it into module names themselves does address that if you are ok with resulting uglyness.
Re: Fix #2529: explicit protection package #3651
On Saturday, 23 August 2014 at 11:12:34 UTC, Kagamin wrote: On Saturday, 23 August 2014 at 11:04:35 UTC, Jacob Carlborg wrote: That's what we're trying to fix with this change. The proposal is to make internals visible to super packages, I only wonder whether internals visible to some package should be hidden from its subpackages. It addresses both - explicitly supplied package identifiers imply access to all subpackages. Current implementation is very striaghtforward and naive in that regard.
Re: Fix #2529: explicit protection package #3651
On Saturday, 23 August 2014 at 11:08:31 UTC, Kagamin wrote: On Saturday, 23 August 2014 at 10:04:40 UTC, Dmitry Olshansky wrote: For what its worth I'd _prefer_ std.regex.internal as in my opinion internals got to be well contained. But as discussed, `package` doesn't work this way. It can be a philosophical matter, but in my experience grouping by functionality is genuine, and access is an afterthought, so grouping by access doesn't work in the long term, as code naturally (spontaneously) groups by functionality. That does contradict your statement that any stuff used in parent packages must go to up the hierarchy which is exactly grouping by access :) I can probably give a more practical example I have just recently encountered when re-designing one of our internal library packages. It was a serialization package and sub-packages defined different (de)serialization models, some of those defining special wrapper types for resulting data that enforce certain semantics specific to that serialization model via the type system. It is not surprising that most internal details of such wrappers were kept package protected as exposing those to user could have easily violated all assumptions (but was needed to implement (de)serialization efficiently). It has become more complicated when meta-serializers have been added to parent package - templated stuff that took any of sub-package implementations and added some functionality (like versioning support) on top. Being generic thing it reside in higher level serialization package but to be implemented efficiently it needs access to those internal wrapper fields. Moving wrapper modules to parent package is not an option here because those are closely related to specific serialization model. Exposing stuff as public is not an option because anyone not deeply familiar with package implementation can easily break all type system guarantees that way. Moving meta-serializers to sub-packages is quite a code duplication. Right now I keep that stuff public and say in docs please don't use this which is hardly a good solution.
Re: Fix #2529: explicit protection package #3651
On Saturday, 23 August 2014 at 09:00:30 UTC, Kagamin wrote: What is difficult to find? With flat structure you have all files right before your eyes. If you need std.datetime.systime module, you open std/datetime/systime.d file - that's the reason of needlessly restricting code structure with modules as if one size fits all. It is the same reasoning as with deep filesystem hierarchies and, well, any data hierarchies - once the element (module / file) count becomes bigger than ~dozen you only really notice things you know to look for. Contrary to that deeply nested categorized hierarchies are easy to casually to search through if you don't know exact module name - just iteratively pick whatever package fits the theme until you find what you want. I remember coding a bit in C#/.NET platform ages ago - it was totally possible to find relevant modules without even looking in docs, just using auto-complete through suggested package names for import. It was really positive experience for a newbie I was. At the same time a lot of people have no idea how many cool things Phobos actually has.
Re: Fix #2529: explicit protection package #3651
On Sun, 24 Aug 2014 02:39:39 + Dicebot via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: It is the same reasoning as with deep filesystem hierarchies and, well, any data hierarchies i already told that, but without any effects. signature.asc Description: PGP signature
Re: Fix #2529: explicit protection package #3651
On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote: On 8/19/2014 7:01 AM, Dicebot wrote: Walter, now that release is out can you please state your opinion about https://github.com/D-Programming-Language/dmd/pull/3651 ? It is blocking Phobos module split and decoupling. I keep thinking there's gotta be a way to do this without language changes. For what it's worth I don't care how things turn out w.r.t. syntax, language extension or convention but the problem must be solved. The way I see the problem: There must be nice and clean way to have internal stuff that is invisible for user. Preferably this stuff must not be monolithic and have form of package. Example structure: mod/package.d mod/internal/bar.d mod/... I actually face it in Phobos right now, my blocked pull: https://github.com/D-Programming-Language/phobos/pull/2412 Options: 1. Use current package attribute. Disadvantages: a) Must retain internals in the same package, user may still import them no problem at all. At least it does not do much harm as nothing is accessible. b) Can't use std.mod.internal.bar 'package' stuff in std.mod (upper one). That makes it next to unusable. Basically it means we have to push internals up, instead of pushing them down. 2. Use separate public internal package. Main disadvantage: a) Now user may import and actually USE internal stuff. b) Not only that but also internal methods of equally internal structs and classes leak out. In other words: a struct defined in internal module may never be exposed without some kind of wrapping. Ugly and unnecessary complication. I'd rather have a language feature, but whatever you guys invent as 3rd option that doesn't suck will have my vote. --- Dmitry Olshansky
Re: Fix #2529: explicit protection package #3651
All I want is that whatever decision Walter makes to happen sooner than in few years from now.
Re: Fix #2529: explicit protection package #3651
Dicebot: All I want is that whatever decision Walter makes to happen sooner than in few years from now. There are other pending patches, like the support for the nice [$] syntax by Kenji. I keep thinking there's gotta be a way to do this without language changes. or I keep thinking there must be a better way than [$]. are the kind of standard answers Walter gives when he doesn't have time to think about some design (because he has time he usually needs only hours or very few days to implement features, like when he made UDAs or C++ namespeces). So I guess Walter is currently busy with something (like finishing the 2.066 release and its regressions, or developing Warp for Facebook, or something else), so you have to wait some time for him to come back to the language development side, to receive a good answer and a good solution. Bye, bearophile
Re: Fix #2529: explicit protection package #3651
On Fri, 22 Aug 2014 20:48:22 + bearophile via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: or (smart plan!) just apply the necessary PRs, build dmd from sources and start using the features. if authors of popular libraries will do this, there will be no choice and maintainers will be forced to apply this patches. this is called 'driven by the community'. ;-) ok-ok-ok, you got me, this was a silly joke! or maybe not?.. signature.asc Description: PGP signature
Re: Fix #2529: explicit protection package #3651
On Sat, 23 Aug 2014 04:50:36 +0300 ketmar via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: to be honest, i myself is a happy user of Kenji's $ patch and package(...) patch. yes, it costs some efforts to keep 'em up-to-date, but hey, i have no life anyway. signature.asc Description: PGP signature
Re: Fix #2529: explicit protection package #3651
On Tuesday, 19 August 2014 at 17:11:19 UTC, Dicebot wrote:
Any specific ideas? I can't imagine any clean solution - and
proposed language extensions fits naturally into existing
system without introducing any new concepts. It is also
somewhat frequently asked about in NG.
---
module std.internal.mod1;
package(std) void foo() {}
module std.mod2;
import std.internal.mod2;
void bar() { foo(); }
---
Isn't it equivalent to
---
module std.internal_mod1;
package void foo() {}
module std.mod2;
import std.internal_mod1;
void bar() { foo(); }
---
Re: Fix #2529: explicit protection package #3651
On Wed, 20 Aug 2014 07:25:43 + Kagamin via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: but it requires to dump all 'subpackages' into one flat directory and with ugly names. proposed extension will not break any existing code, yet will allow much nicer hierarchy. signature.asc Description: PGP signature
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 07:47:36 UTC, ketmar via Digitalmars-d-announce wrote: but it requires to dump all 'subpackages' into one flat directory and with ugly names. Huh? If std.datetime and std.regex are subpackages of `std` package, it usually means, they should reside in `std` directory, no? And how explicit package protection can change that?
Re: Fix #2529: explicit protection package #3651
On Wed, 20 Aug 2014 09:26:36 + Kagamin via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: Huh? If std.datetime and std.regex are subpackages of `std` package, it usually means, they should reside in `std` directory, no? And how explicit package protection can change that? imagine mybiglib, mybiglib.internal, mybiglib.other, and each of them not single module, but package itself. so we have module mybiglib.other.thingy which wants to use functions from module mybiglib.internal.wisdom, which is part of mybiglib.internal package. signature.asc Description: PGP signature
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 09:26:37 UTC, Kagamin wrote: On Wednesday, 20 August 2014 at 07:47:36 UTC, ketmar via Digitalmars-d-announce wrote: but it requires to dump all 'subpackages' into one flat directory and with ugly names. Huh? If std.datetime and std.regex are subpackages of `std` package, it usually means, they should reside in `std` directory, no? And how explicit package protection can change that? Your proposal makes such module layout illegal: std.package1.module1 std.package1.module2 std.package1.subpackage.module1 std.package1.subpackage.module2 (can't have symbols in subpackage that are also available to package1 but not whole std / public, must have `subpackage_module1` instead, `package_1_subpackage_module1` if it must be available for whole std) It severely limits expressive power of package system for code organization and encourages flat hierarchies with minimal nesting. It also looks ugly - anyone seeing module name with package name mangled in it will feel that there is something wrong with the language if it needs to resort to such hacks.
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 13:05:54 UTC, Dicebot wrote: Your proposal makes such module layout illegal: std.package1.module1 std.package1.module2 std.package1.subpackage.module1 std.package1.subpackage.module2 (can't have symbols in subpackage that are also available to package1 but not whole std / public, must have `subpackage_module1` instead, `package_1_subpackage_module1` if it must be available for whole std) If some utility is internal to std.package1 an used in the entire std.package1, shouldn't it sit in std.package1? Why push it to subpackage? std.package1.module1 std.package1.module2 std.package1.internal - package1's internals go here std.package1.subpackage.module1 std.package1.subpackage.module2
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 09:40:37 UTC, ketmar via Digitalmars-d-announce wrote: imagine mybiglib, mybiglib.internal, mybiglib.other, and each of them not single module, but package itself. so we have module mybiglib.other.thingy which wants to use functions from module mybiglib.internal.wisdom, which is part of mybiglib.internal package. Do we need a hierarchy of internals, is the problem this big? Why mybiglib.wisdom is not good?
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 14:25:59 UTC, Kagamin wrote: If some utility is internal to std.package1 an used in the entire std.package1, shouldn't it sit in std.package1? Why push it to subpackage? std.package1.module1 std.package1.module2 std.package1.internal - package1's internals go here std.package1.subpackage.module1 std.package1.subpackage.module2 It may semantically belong to subpackage but still needs to be available to package1, something not uncommon in templated code (subpackage is generic implementation, package1 is specialization that still needs access to non-public functions). Also you seem to imply that internal is always small tightly coupled module which is hardly true for any higher level functionality. So in practice it will be more like this: std.package1.module1 std.package1.module2 std.package1.internal1 std.package1.internal2 ... std.package1.internal20 Not really encouraging.. What you propose is effectively prohibiting to use packages to organize your code and requiring to design your module hierarchy based on desired protection relations, not other way around. I think it is conceptually wrong approach and unnecessarily restrictive compared to overall D design principles (no single true way of doing things)
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 14:33:53 UTC, Kagamin wrote: On Wednesday, 20 August 2014 at 09:40:37 UTC, ketmar via Digitalmars-d-announce wrote: imagine mybiglib, mybiglib.internal, mybiglib.other, and each of them not single module, but package itself. so we have module mybiglib.other.thingy which wants to use functions from module mybiglib.internal.wisdom, which is part of mybiglib.internal package. Do we need a hierarchy of internals, is the problem this big? Why mybiglib.wisdom is not good? As I see on the realistic example of datetime, which is BIG, we only need to split it into a flat set of files without an overly deep package hierarchy.
Re: Fix #2529: explicit protection package #3651
On Wed, Aug 20, 2014 at 4:33 PM, Kagamin via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: Do we need a hierarchy of internals, is the problem this big? Why mybiglib.wisdom is not good? It gets really unwieldy when you want to put a whole bunch of things into one project and give control of different packages to different teams. Yes you could use multiple libraries in a lot of cases but if you're wanting it in a single setup / package (like phobos), especially one that is internal to another package you need sub-packages. Examples that would use this are: - kitchen sink libraries like phobos - frameworks such as vibe.d - libraries with lots of internal code such as Ogre3d if it were made in D Well that seems to be good reasons to me.
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 14:36:59 UTC, Kagamin wrote: As I see on the realistic example of datetime, which is BIG, we only need to split it into a flat set of files without an overly deep package hierarchy. We _may_ split it into flat set files (solving only part of the problem) but it is desirable to have a deeper package hierarchy. Package hierarchy is not just an encapsulation tool, it is also a great way to simplify navigation and finding needed modules - something that Phobos is current terrible at exactly because of flat hierarchies.
Re: Fix #2529: explicit protection package #3651
On Wed, 20 Aug 2014 14:33:52 + Kagamin via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: Do we need a hierarchy of internals, is the problem this big? Why mybiglib.wisdom is not good? ah, why we need such things as subdirectories at all? CP/M was fine without concept of subdirectories! signature.asc Description: PGP signature
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 14:35:31 UTC, Dicebot wrote: It may semantically belong to subpackage but still needs to be available to package1, something not uncommon in templated code (subpackage is generic implementation, package1 is specialization that still needs access to non-public functions). If generic implementation is designed to be customized, it probably means, it should be widely available for customization, like xml document and base64 encoder. Also it's probably a logical error if base type is less visible than the derived type. What you propose is effectively prohibiting to use packages to organize your code and requiring to design your module hierarchy based on desired protection relations, not other way around. I think it is conceptually wrong approach and unnecessarily restrictive compared to overall D design principles (no single true way of doing things) It's ok for packages to exchange public interface, but internals? If a subpackage has internals, they are for its usage, and consumable functionality should be provided through public interface.
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 15:13:06 UTC, Kagamin wrote: On Wednesday, 20 August 2014 at 14:35:31 UTC, Dicebot wrote: It may semantically belong to subpackage but still needs to be available to package1, something not uncommon in templated code (subpackage is generic implementation, package1 is specialization that still needs access to non-public functions). If generic implementation is designed to be customized, it probably means, it should be widely available for customization, like xml document and base64 encoder. Also it's probably a logical error if base type is less visible than the derived type. This is just your design preference and hardly a good one size fits them all decision. I also don't speak about inheritance but about composition - customization may be implemented via unsafe field exposure and unsuitable for general public usage but useful for simplifying internal implementation maintenance. We should not force users into single good package structure based on certain design beliefs. What you propose is effectively prohibiting to use packages to organize your code and requiring to design your module hierarchy based on desired protection relations, not other way around. I think it is conceptually wrong approach and unnecessarily restrictive compared to overall D design principles (no single true way of doing things) It's ok for packages to exchange public interface, but internals? If a subpackage has internals, they are for its usage, and consumable functionality should be provided through public interface. Same here. As library size grows you completely lose the distinction between public and internal, certain parts of the library may become hidden from other parts and public but available for others. It is pretty much a necessity to keep up with maintenance when there is a large team working on it simultaneously. Right now people mostly rely on convention, I have seem quite some comments like should have been private but needs to be accessed from module X, don't ever touch it (public-but-undocumented functions of Phobos sometimes fall into the same category). Telling people that they design applications in a wrong way and need to change their habits is a good approach to alienate them against the language.
Re: Fix #2529: explicit protection package #3651
On 8/20/14, 7:49 AM, ketmar via Digitalmars-d-announce wrote: On Wed, 20 Aug 2014 14:33:52 + Kagamin via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: Do we need a hierarchy of internals, is the problem this big? Why mybiglib.wisdom is not good? ah, why we need such things as subdirectories at all? CP/M was fine without concept of subdirectories! No need to demean the question. It is valid. -- Andrei
Re: Fix #2529: explicit protection package #3651
On Wednesday, 20 August 2014 at 17:19:58 UTC, Andrei Alexandrescu wrote: On 8/20/14, 7:49 AM, ketmar via Digitalmars-d-announce wrote: On Wed, 20 Aug 2014 14:33:52 + Kagamin via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: Do we need a hierarchy of internals, is the problem this big? Why mybiglib.wisdom is not good? ah, why we need such things as subdirectories at all? CP/M was fine without concept of subdirectories! No need to demean the question. It is valid. -- Andrei Originally flat Phobos hierarchy was considered good enough. Now we can see that such approach doesn't scale well - features are often missed because of non-intuitive module placement, compile times suffer because of many cross-module dependencies (flat hierarchy encourages big modules). Is there any reason to think that same logic applied to sub-packages will scale any better as Phobos size grows? I doubt so. In fact I feel it is already beyond the size where it is convenient and only reason why even smaller deeply nested modules are not an option is exactly because it is too hard to keep both existing protection attribute relation and provide easy to navigate hierarchy at the same time. The fact that some of people who have actually tried to use package.d support this language change is pretty good anecdotal evidence that there is a problem to be solved.
Re: Fix #2529: explicit protection package #3651
module foo.bar.one; module foo.bar.internals; // package-protected utilities module foo.bar.subpkg.two; module foo.bar.subpkg.internals; // package-protected utilities Current situation: module 'one' cannot access the 'bar.subpkg' utilities, and module 'two' cannot access the 'bar' utilities. This is needlessly limiting, forcing design choices that should not be dictated by the ability/inability to separate public and private API. It also precludes many valid and good uses of nested package.d modules. I really don't see any equally strong counter-argument. But then, I've wanted this exact fix for literally years now. It also is not limited to internal utility modules. It can be useful for systems that select at compile time from one of a number of system-specific implementations of a given interface/api. It can be useful for granting privileged access to certain api's and/or resources to specific module(s). An example being: grant access to unsafe but versitile data manipulators solely to the subpackage containing the loaders. Currently, there is no absolutely enforceable way of doing such things; so one ends up falling back on convention... and conventions, sadly, never hold in the real world for long.
Re: Fix #2529: explicit protection package #3651
On Wed, 20 Aug 2014 10:19:59 -0700 Andrei Alexandrescu via Digitalmars-d-announce digitalmars-d-announce@puremagic.com wrote: No need to demean the question. It is valid. -- Andrei sorry, i don't mean to insult anyone, just trying to make people see analogies. hierarchies are everywhere, it's convient way to store alot of things. modules itself is a great example of hierarchies -- we can dump all source code in one file instead of using modules. i.e. file system hierarchy is handy what there are many files. module hierarchy is handy when there are many modules in library. the ability to have fine-grained visibility control in module hierarchy is handy. this was written numerous times in this thread, yet he continues to ask the same question again and again as if he just don't bother to read any answers. that's why i asked about subdirs. signature.asc Description: PGP signature
Re: Fix #2529: explicit protection package #3651
On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote: On 8/19/2014 7:01 AM, Dicebot wrote: Walter, now that release is out can you please state your opinion about https://github.com/D-Programming-Language/dmd/pull/3651 ? It is blocking Phobos module split and decoupling. I keep thinking there's gotta be a way to do this without language changes. Any specific ideas? I can't imagine any clean solution - and proposed language extensions fits naturally into existing system without introducing any new concepts. It is also somewhat frequently asked about in NG.
Re: Fix #2529: explicit protection package #3651
On Tuesday, 19 August 2014 at 17:11:19 UTC, Dicebot wrote: On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote: On 8/19/2014 7:01 AM, Dicebot wrote: Walter, now that release is out can you please state your opinion about https://github.com/D-Programming-Language/dmd/pull/3651 ? It is blocking Phobos module split and decoupling. I keep thinking there's gotta be a way to do this without language changes. Any specific ideas? I can't imagine any clean solution - and proposed language extensions fits naturally into existing system without introducing any new concepts. It is also somewhat frequently asked about in NG. Yeah, I don't see how this could be done without a language change. Currently, modules in sub-packages are treated no differently from modules in completely different packages, so anything you did to give access to a module in a super-package to one in a sub-package would give access to any module. - Jonathan M Davis
Re: Fix #2529: explicit protection package #3651
On Tuesday, 19 August 2014 at 17:11:19 UTC, Dicebot wrote: On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote: On 8/19/2014 7:01 AM, Dicebot wrote: Walter, now that release is out can you please state your opinion about https://github.com/D-Programming-Language/dmd/pull/3651 ? It is blocking Phobos module split and decoupling. I keep thinking there's gotta be a way to do this without language changes. Any specific ideas? I can't imagine any clean solution - and proposed language extensions fits naturally into existing system without introducing any new concepts. It is also somewhat frequently asked about in NG. To put it differently - this issue is a very real blocker for some of planned Phobos changes (pretty much anything that is going to use package.d) and it has been hanging around for a long time. If you are going to propose alternative solution I'd be very glad to see it sooner than later as everyone else who has commented on that topic seems to be satisfied with the proposal.
