Re: ModuleInfo, factories, and unittesting
On 2016-12-18 17:53, Andrei Alexandrescu wrote: Compulsive ModuleInfo generation seems to be a drag. I'm not very familiar with the particulars but my understanding is ModuleInfo is needed for (a) Object.factory and (b) finding and running unittests. Walter and I think Object.factory was a mistake and we'd like to make it opt-in long term (either by means of a build flag or a class attribute). Until then, a very nice step forward is to NOT generate ModuleInfo if a module introduces no class. For unittests, I figure things like static introspection should make it easier to enumerate and run unittests without a need for ModuleInfo. What other issues/opportunities do you see related to ModuleInfo? Looking in the compiler, I see references to: * Coverage * Asserts * Arrays * Objective-C class info. If this is not generated Objective-C class will not work I'm not entirely sure what the above do. -- /Jacob Carlborg
Re: ModuleInfo, factories, and unittesting
On 2016-12-18 17:53, Andrei Alexandrescu wrote: For unittests, I figure things like static introspection should make it easier to enumerate and run unittests without a need for ModuleInfo. We currently have no good way to collect unittest between modules. It's easy to collect them for a single specific module, but not globally. -- /Jacob Carlborg
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On Sunday, 18 December 2016 at 18:42:36 UTC, Andrei Alexandrescu wrote: On 12/18/16 1:03 PM, Joakim wrote: I largely agree with Dmitry. Ilya refactored several Phobos modules to use scoped, selective imports much more, and I pitched in for some remaining imports in the largest modules, so that only these module-level imports remain, ie those necessary for symbols imported in template constraints: std.datetime - https://github.com/dlang/phobos/pull/4373/files std.uni - https://github.com/dlang/phobos/pull/4365/files std.string and std.traits - https://github.com/dlang/phobos/pull/4370/files Yah, there's been a lot of good work (Jack Stouffer did a lot as well IIRC) on pushing imports inside. The following searches should be relevant: git grep '^\(private \)\?import' | wc -l yields about 426 top-level import declarations. The number of indented imports is 4605: git grep ' *import\W' | wc -l So we're looking at 10% of imports being problematic. Sadly, they turn out to make things quite difficult for Phobos. Last time I looked at the module dependency graph it wasn't a lot better than it used to before scoped imports. (I don't have it handy... could anyone please produce it?) Why do you care _so_ much about the module dependency graph? To make this question concrete, let's look at an example, the std.array module you keep mentioning. This is what it looked like before Ilya scoped as many imports as he could: https://github.com/dlang/phobos/commit/3fcf723aa498b96de165361b5abb9d3450fdc069#diff-54cf8402b22024ae667d4048a5126f0e That was a mess, similar to opaque C/C++ code, 13 modules imported at module-scope were whittled down to 4. You just made those more specific in a commit related to this DIP, by listing the actual symbols selectively imported from those four modules: https://github.com/dlang/phobos/commit/e064d5664f92c4b2f0866c08f6d0290ba66825ed#diff-54cf8402b22024ae667d4048a5126f0e If I'm looking at the template constraints for any particular function and see a couple symbols I don't recognize, I don't think it's a big deal to find the symbols in that list at the top. In other words, D already allows you to scope most imports. I don't consider the dozen or two remaining symbols from templaint constraints and function arguments to provide much overhead. Rather, I consider the weight of this additional syntax, ie the cognitive overhead from having to remember and parse more syntax in my head, to be worse than the remaining dependency reasoning problem you're trying to solve: the cost outweights the benefit. Perhaps that's subjective and others may disagree. Now, there's also the question of purely technical benefits, like compilation speed or executable bloat. I looked at the latter a little last summer, after Ilya had cleaned up a lot of the standard library: http://forum.dlang.org/thread/gmjqfjoemwtvgqrtd...@forum.dlang.org I found that commenting out a single scoped, selective import of "std.string: format" in std.utf led to a 5% decrease in executable size for "hello world." This is a problem with how dmd compiles or appends these module dependencies and would presumably still be there after this DIP, as you would not remove the dependency. I think scoped, selective imports have been great at hacking away at the module dependency graph, as you lay out. It is not clear what technical costs you see from the remaining few dependencies and if this DIP is the best way to remove them. I think you should explain why you want to untangle the remaining dependency graph, and consider if this DIP is really doing that much. When I first saw this DIP, like Dmitry I was happy that we could get rid of those too, but the more I see these horrible syntax suggestions for what is really a minor convenience, I changed my mind. std.datetime, the 35k line (17 kloc according to Dscanner) beast of phobos, only needs 20 or so symbols at module-scope. std.uni- 10k lines, 4.2 kloc- only needs 17 symbols, all from the three modules Dmitry mentioned. I don't think his workaround of splitting up modules is even needed for such a low amount of module-level imports. This paragraph is a good example of a couple of counterarguments that I think point directly to flaws in the DIP: (1) The DIP uses Phobos as an example, so it is applicable mostly to Phobos. In fact, Phobos is among the systems that would benefit least from the DIP: it has only druntime as dependency, and is distributed in its entirety. Many projects out there list multiple dependencies and may have various building and distribution policies. It is not clear how those alternate dependency, building and distribution policies change the picture. Perhaps you should cite one of those as an example. The converse is to believe that working around a problem in Phobos would render the DIP less useful in general. The argument is not that Phobos has "worked around" the
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 19:28:24 UTC, Andrei Alexandrescu wrote: On 12/18/2016 01:49 PM, Ilya Yaroshenko wrote: On Sunday, 18 December 2016 at 17:41:31 UTC, Walter Bright wrote: On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote: We already have better `cpuid` and better `random` packages. Great! Please PR them for Phobos. cpuid is used in Mir GLAS and it should be a betterC library. It's this kind of imaginary dialog that I don't quite grok: Ilya Yaroshenko: "I can't use druntime's cpuid so I defined my own." Andrei Alexandrescu: "OK, so what's different?" IY: "Mine has a few engineering improvements." AA: "Cool, why don't you merge them into core.cpuid?" IY: "Well mine doesn't have a shared static constructor, so it doesn't need a runtime to automatically call that library initialization function. User need to explicitly call an init function before using it." AA: "I understand. Great, so how about this - we add your code to core.cpuid_v2 to druntime. Then we make all in core.cpuid to forward to it so there's no duplication. It all works out!" IY: "No, I don't want to do that. It's still in druntime and I don't want druntime. I want betterC." AA: "But it will be compilable with betterC and we can add unittests to make that happen. YOU HAVE MY SUPPORT. Let's do it." IY: "No, I want to change it often. The deployment schedule of druntime is too slow." AA: "How often do you need to change it? Is it that unstable?" IY: "Um, not too often." AA: "Then what is the matter? Are you worried about the IP of the engineering improvements you are making? Are you licensing this differently?" IY: "No, it's for the most part similar to core.cpuid and the license will be also Boost." AA: "Then what is the matter? Do you want me to wait until you release a stable mir.cpuid and copy it over with credit, per the Boost license, to core.cpuid_v2?" IY: "..." It's this kind of stuff I need to have a better understanding of. Some technical arguments are meaningful, some others point to problems with obvious solutions that are somehow shunned, and yet others are like a hidden Markov model - I see the effects, but I don't see the causes. Andrei I think, the reason of misunderstanding is different skills in a subject. And even different subjects. Some experts are compilers developers, some are library developers. They all in different subject. In c++ community there is a group of compiler developers experts who desides, how to implement this or that feature, and the ability of implementation. They know how to develop compilers. It doesn't mean they are more clever than library developers, just different skills. And when Andrei asking to explain the reason of some proposal, he waits arguments in _terminology_ of comliler developer. _terminology_ understanding (in wide) means _experience_ (compiler developing in this case). If no experience in the subject, there is a way to get it, and the other way to ask a recommendation of experienced colleague.
Re: A betterC modular standard library?
On Monday, 19 December 2016 at 05:22:10 UTC, Ilya Yaroshenko wrote: What exactly do you mean by "betterC"? As far as I know, the `-betterC` switch just removes ModuleInfo generation. Because -betterC was not implemented yet. I'm curious to know what a complete implementation entails. Do you actually mean, a standard library that has fewer dependencies on druntime features? I mean this too I've been wanting to decouple from druntime since I started learning D in late 2013, though probably for different reasons than yours. There have been two recent posts [1] [2] wanting to do the same thing, and now your post envisioning a more modular druntime-less phobos. [1] - https://forum.dlang.org/post/drqdrobcoaanenocy...@forum.dlang.org [2] - https://forum.dlang.org/post/tmofjecvnqdthvete...@forum.dlang.org It's encouraging to see this trend. I am interested in seeing D become more modular and pay-as-you-go, though I'd prefer to see changes in the compiler and druntime (e.g. [3] [4] [5]) before phobos. [3] - http://forum.dlang.org/post/mr71kt$2rgi$1...@digitalmars.com [4] - http://forum.dlang.org/post/eiwalbqlbkipdrmsr...@forum.dlang.org [5] - https://issues.dlang.org/show_bug.cgi?id=11666 Mike
Re: A betterC modular standard library?
On Monday, 19 December 2016 at 01:52:21 UTC, Andrei Alexandrescu wrote: On 12/18/16 7:31 PM, Seb wrote: tl:dr: Phobos is very bloated. How does the D standard library compare in size with some other languages (C++, Rust, Go, Java, Python)? C++ and Rust are the best from this list in terms of low level programming. They have smaller libraries comparing with D. Go, Java, Python --- If I need a language like them I will you them, not D. Go - awesome GC, commercially supported, easy to write your own libs Java - awesome GC, commercially supported, Scala(!!!) Python - awesome scripting, commercially supported D - ??? awesome what ? What do we have better then Scala? Performance? NO. BRandomAccessRanges makes our performance less then C/C++. Clean and clever API in standard library? NO. We are placing our Range API whenever possible including RNGs. We have not a language standard for I/O. Half of Phobos is outdated to language standards. Best generics? NO. Scala is more powerful. -- What do we have awesome? Where D is the best? -- D is awesome for low-level system libraries in betterC mode. (and small text processing routines) This will make D great in other fields. -- Let's do something about it! I've said this several times, and it bears repeating. We want to make D entirely pay-as-you go, and offer full support to anyone working on it. The best "something" anyone can do about it is do good engineering work in the D mainline. National D Foundation Party. That will automatically benefit all users of the D language, instead of balkanizing the community. ... Granted, the lure of the clean slate is strong, and more care is needed to mind backwards compatibility, but that's where the real impact is. Andrei How Mir projects breaks backward compatibility ? How proposed direction breaks backward compatibility? Lets use other prefix for the new libraries. Mir uses `mir.` And allow Phobos to be a sandbox for C++ as it is now. Ilya
Re: Making preconditions better specified and faster
On Friday, 16 December 2016 at 10:28:03 UTC, qznc wrote: On Thursday, 15 December 2016 at 18:48:22 UTC, Andrei Alexandrescu wrote: https://issues.dlang.org/show_bug.cgi?id=16975 Whenever I think about preconditions and speed, I think that they should actually be put into the caller instead of into the function/callee. Not only faster, but this is the right thing to do. Failing precondition is a caller error, not a callee. Beside optimization: - The error will be reported at the right place. - The compilation flags of the caller decide if the contract is checked, not the callee.
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On Sunday, 18 December 2016 at 23:18:27 UTC, Andrei Alexandrescu wrote: Great, thanks. Please take a look at the accuracy of the discussion. I expanded the "Workaround" section and moved it near the top. https://github.com/dlang/DIPs/pull/51 https://github.com/dlang/DIPs/blob/dd46252e820dce66df746540d7ab94e0b00a6505/DIPs/DIP1005.md Andrei What's wrong with the parentheseless version ? The DIP says it looks "out of place" but that doesn't strike me as a very good argument. IMO the version without ";" is the way to go, as it doesn't require to add a new syntax for imports. Identical things looking identical is valuable.
Re: A betterC modular standard library?
On Monday, 19 December 2016 at 01:33:42 UTC, Mike wrote: On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko wrote: Who is interested in betterC _modular_* standard library? What exactly do you mean by "betterC"? As far as I know, the `-betterC` switch just removes ModuleInfo generation. Because -betterC was not implemented yet. Do you actually mean, a standard library that has fewer dependencies on druntime features? I mean this too Ilya
proposal string std.utf:sanitizeUTF(string) which returns an always valid UTF8 string
I keep running into issues due to auto-decoding (arguably a significant
design flaw of phobos) when using strings from external sources (which may
not be 100% valid UTF8) eg see stracktrace [1] on
getSomeExternalString().splitLines,
Could we have something like `sanitizeUTF` in std.utf, to allow for a
simple fix when running into such UTF8 issues see proposal implementation
[2]; the fix would then be:
```
getSomeExternalString().splitLines,
=>
getSomeExternalString().sanitizeUTF.splitLines,
```
[1]
core.exception.AssertError@std/utf.d(2254): Assertion failure
??:? _d_assert [0x4f4e63]
??:? void std.utf.__assert(int) [0x53a304]
??:? pure nothrow @nogc @safe ubyte
std.utf.codeLength!(char).codeLength(dchar) [0xa5d78191]
??:? pure nothrow @nogc @safe int
std.string.stripRight!(immutable(char)[]).stripRight(immutable(char)[]).__foreachbody2(ref
ulong, ref dchar) [0xa5c42bd9]
??:? _aApplyRcd2 [0x4f9bd1]
??:? pure @nogc @safe immutable(char)[]
std.string.stripRight!(immutable(char)[]).stripRight(immutable(char)[])
[0xa5c42b5c]
??:? pure @property @nogc @safe immutable(char)[]
std.algorithm.iteration.stripRight.MapResult.front() [0xa5cda053]
??:? pure @safe immutable(char)[]
std.array.join!(std.algorithm.iteration.stripRight.MapResult,
immutable(char)[]).join(std.algorithm.iteration.stripRight.MapResult,
immutable(char)[]) [0xa5cda39a]
[2] sanitizeUTF proposal:
// TODO: rangeify to make it work in more situations
string sanitizeUTF(string a){
import std.utf;
Appender!string b;
while(a.length){
b~=decodeFront!(Yes.useReplacementDchar)(a);
}
return b.data;
}
Re: A betterC modular standard library?
On 12/18/16 7:31 PM, Seb wrote: tl:dr: Phobos is very bloated. How does the D standard library compare in size with some other languages (C++, Rust, Go, Java, Python)? Let's do something about it! I've said this several times, and it bears repeating. We want to make D entirely pay-as-you go, and offer full support to anyone working on it. The best "something" anyone can do about it is do good engineering work in the D mainline. That will automatically benefit all users of the D language, instead of balkanizing the community. Granted, the lure of the clean slate is strong, and more care is needed to mind backwards compatibility, but that's where the real impact is. Andrei
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko wrote: Who is interested in betterC _modular_* standard library? What exactly do you mean by "betterC"? As far as I know, the `-betterC` switch just removes ModuleInfo generation. Do you actually mean, a standard library that has fewer dependencies on druntime features? Mike
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On Monday, 19 December 2016 at 00:54:13 UTC, Andrei Alexandrescu wrote: On 12/18/16 7:44 PM, pineapple wrote: On Sunday, 18 December 2016 at 23:18:27 UTC, Andrei Alexandrescu wrote: Great, thanks. Please take a look at the accuracy of the discussion. I expanded the "Workaround" section and moved it near the top. I would also like to register that while I respect your argument regarding scalability, I have personally found that a greater number of smaller files is easier to manage than a smaller number of larger files. Including for the 580,000+ line project I work on for a living. Is it publicly available? -- Andrei Sadly not.
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On 12/18/16 7:44 PM, pineapple wrote: On Sunday, 18 December 2016 at 23:18:27 UTC, Andrei Alexandrescu wrote: Great, thanks. Please take a look at the accuracy of the discussion. I expanded the "Workaround" section and moved it near the top. I would also like to register that while I respect your argument regarding scalability, I have personally found that a greater number of smaller files is easier to manage than a smaller number of larger files. Including for the 580,000+ line project I work on for a living. Is it publicly available? -- Andrei
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On Monday, 19 December 2016 at 00:44:14 UTC, pineapple wrote: On Sunday, 18 December 2016 at 23:18:27 UTC, Andrei Alexandrescu wrote: Great, thanks. Please take a look at the accuracy of the discussion. I expanded the "Workaround" section and moved it near the top. I would also like to register that while I respect your argument regarding scalability, I have personally found that a greater number of smaller files is easier to manage than a smaller number of larger files. Including for the 580,000+ line project I work on for a living. (Granted, that old ugly codebase has a lot of problems of its own, and smaller files does not always mean small on the same scale that mach.d's files are small, but I can tell you from experience that once a single ~4,000 line module was broken down into many ~200 line modules, that code became a great deal easier to understand and to reason about and to maintain.)
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On Sunday, 18 December 2016 at 23:18:27 UTC, Andrei Alexandrescu wrote: Great, thanks. Please take a look at the accuracy of the discussion. I expanded the "Workaround" section and moved it near the top. I would also like to register that while I respect your argument regarding scalability, I have personally found that a greater number of smaller files is easier to manage than a smaller number of larger files. Including for the 580,000+ line project I work on for a living.
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On Sunday, 18 December 2016 at 23:18:27 UTC, Andrei Alexandrescu wrote: Great, thanks. Please take a look at the accuracy of the discussion. I expanded the "Workaround" section and moved it near the top. https://github.com/dlang/DIPs/pull/51 https://github.com/dlang/DIPs/blob/dd46252e820dce66df746540d7ab94e0b00a6505/DIPs/DIP1005.md Andrei It's reasonably accurate. The only thing I would point out is that few modules define only one public symbol; the median probably lies around 4 or 5. A few modules, most or all being in the traits package, define considerably more. The mach.traits.primitives module defines 28 public symbols, by my count. (Though that's still small stuff relative to Phobos.) I think it would be most accurate to word the sentence as: It is organized as such that each module contains a small number of related declarations (such as canAdjoin, Adjoin, and AdjoinFlat in module adjoin.d) along with documentation and unit tests.
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 18:49:33 UTC, Ilya Yaroshenko wrote: 1. Modularity: D should provide a very slim library (like std.traits and may be default GC with old core, which can be turned off or replaced). Other parts or Phobos/Druntime should be split into parts and the parts should have their own repositories. They should be dub packages with D Foundation review / control. This would be awesome to have!! tl:dr: Phobos is very bloated. Let's do something about it! I absolutely agree that there's a lot of stuff that doesn't belong into Phobos. Also many parts are outdated or have ugly APIs, but they can only be replaced if a superior community solution has evolved. So why not "dubify" the optional parts of Phobos? Solved Problem: Versioning -- This also solves the versioning problem Andrei discussed last month nicely. For example when std.allocator was merged into Phobos a lot of builds broke because packages did can their dependency from dub to std.x.allocator, but many people were still compiling with an older compiler. Solved Problem: High quality community libraries Moreover, at the last DConf there were many voices that D is lacking good community libraries (aka the gold libraries). So the Phobos collection could be one. Btw one shouldn't forgot that people have already written a lot of replacements of Phobos modules and imho in most cases it made sense! A quick list (this list is incomplete and intended as an example to show that for most parts of Phobos better solutions co-exists) etc.c.sqlite (aka std.database) -> https://github.com/buggins/ddbc (and: https://wiki.dlang.org/Database_Libraries) etc.c.zlib / std.zip -> https://github.com/rcythr/archive std.base64 -> https://code.dlang.org/packages/base-d std.bigint -> https://github.com/andersonpd/eris/blob/master/integer/extended.d std.csv -> https://github.com/eBay/tsv-utils-dlang std.container -> https://github.com/economicmodeling/containers (uses allocators!) std.complex I am pretty sure this is on Ilya's list as well ;-) std.encoding: -> https://github.com/e10s/d-base32 std.getopt -> https://github.com/jasonwhite/darg -> https://github.com/SirTony/commando etc. std.json: -> https://github.com/s-ludwig/std_data_json std.math: -> https://code.dlang.org/packages/ctstdmath -> https://github.com/libmir/mir-math std.net.curl: -> https://github.com/ikod/dlang-requests std.net.isemail: -> https://github.com/anton-dutov/mail -> http://vibed.org/api/vibe.mail.smtp/Mail std.signals: -> https://code.dlang.org/packages/phobosx -> https://code.dlang.org/packages/observe std.socket: -> http://vibed.org/api/vibe.http.websockets/ -> http://vibed.org/api/vibe.core.net/TCPConnection std.stdio: -> https://github.com/jasonwhite/io -> https://github.com/schveiguy/iopipe -> https://github.com/rejectedsoftware/vibe.d/blob/master/stream/vibe/stream/stdio.d std.uri: -> https://github.com/rikkimax/alphaPhobos/blob/master/source/std/experimental/uri.d -> http://vibed.org/api/vibe.inet.url/ -> https://github.com/adamdruppe/arsd/blob/master/http2.d std.variant -> https://github.com/s-ludwig/taggedalgebraic std.xml -> https://github.com/lodo1995/experimental.xml -> https://github.com/jacob-carlborg/orange Also a lot of proposed Phobos modules already exist in the community: std.color https://github.com/TurkeyMan/color std.decimal https://github.com/andersonpd/eris/ https://github.com/jaypha/fixed std.events (planned?) -> https://github.com/etcimon/libasync Of course many of these libraries are one man projects and aren't considered "high-quality", but the reason they exist in the first place shows that a lot of things just shouldn't be "standardized", but more provided with a "practical variant". We should focus solely on modules that can be standardized easily (std.traits, std.ranges, ...) or are required to create a unified experience (e.g. a common allocation/logging interface). Imho there's really no need to ship sth. like SQLite as part of the standard library. Final remarks - - From a technical point of view there shouldn't be a problem to have a unified documentation experience etc. - Thanks to Martin's awesome work, we already test a couple of "selected" DUB packages on every DMD/Druntime/Phobos commit
Re: ModuleInfo, factories, and unittesting
On Sunday, 18 December 2016 at 16:53:24 UTC, Andrei Alexandrescu wrote: What other issues/opportunities do you see related to ModuleInfo? I think it would be better to take a more holistic look at the relationship between the compiler and druntime. It's my understanding that many of the contracts between the compiler and druntime were written at at time when D didn't have things like templates and rich compile-time features. I think there is an opportunity to refactor the ModuleInfo, TypeInfo, and probably many other features, moving them out of the compiler into druntime (See http://forum.dlang.org/post/eiwalbqlbkipdrmsr...@forum.dlang.org for a precise explanation). Then, with druntime's source code containing templates, conditional compilation, static-if, and the like, when the user compiles their code it only generates code that is actually being used. It could even be taken further with something like this (http://forum.dlang.org/post/psssnzurlzeqeneag...@forum.dlang.org) where druntime's implementation is distributed as .di header files for compile-time verification and advantages beyond your original stated goal. Mike
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On Sunday, 18 December 2016 at 22:31:34 UTC, Andrei Alexandrescu wrote: Is there a simple command to e.g. unittest everything in the project? Also, is there a build process or it's all templated? -- Andrei There's no build process. To run tests, I compile the root `package.d` file with rdmd, including the -debug, -unittest, and --main flags, and -I"path/to/repo/mach.d". To compile the mach.sdl package, which is not currently imported by the root `package.d`, you would have to include a directory containing the requisite Derelict dependencies for the -I option.
Re: ModuleInfo, factories, and unittesting
On Monday, 19 December 2016 at 00:00:36 UTC, Andrei Alexandrescu wrote: On 12/18/16 6:48 PM, Nicholas Wilson wrote: [...] Cool cool cool. Thanks. [...] Noice. Wait, the top foreach iterates what? Its a compile time magic loop I think. Both of them. [...] SoA = ? Structure of array. Think of a transpose of an array of structs, or consult the internet. Andrei
Re: ModuleInfo, factories, and unittesting
On 12/18/16 6:48 PM, Nicholas Wilson wrote:
On Sunday, 18 December 2016 at 16:53:24 UTC, Andrei Alexandrescu wrote:
Compulsive ModuleInfo generation seems to be a drag. I'm not very
familiar with the particulars but my understanding is ModuleInfo is
needed for (a) Object.factory and (b) finding and running unittests.
Walter and I think Object.factory was a mistake and we'd like to make
it opt-in long term (either by means of a build flag or a class
attribute).
Don't forget that typeinfos are classes too, which makes the class
attribute approach less attractive.
Until then, a very nice step forward is to NOT generate ModuleInfo if
a module introduces no class.
Module (static) ctors & dtors need MI.
Cool cool cool. Thanks.
For unittests, I figure things like static introspection should make
it easier to enumerate and run unittests without a need for ModuleInfo.
We have __traits(getUnitTests,...) but the way to do DIY unittests is
foreach(m; ModuleInfo)
foreach(test; __traits(getUnitTests,m)
test();
Noice. Wait, the top foreach iterates what?
What other issues/opportunities do you see related to ModuleInfo?
AFAIK a module info is basically
struct ModuleInfo
{
alias F = void function();
ModuleInfo* parent;
string name; // may be static char array
F s_ctor; // Static constructor
F s_dtor;
F ctor;// Thread local constructor
F dtor;
F unittester; //may be an array for each block
void function(Object)[string] factory_mapping; //Foward to argless
ctor, not sure what mapping is actually used.
}
If we optionally remove Object.factory, and move the unittests elsewhere
(where to?) we are left with the ctors and dtors. If we store these as
SoA
SoA = ?
Andrei
Our students could use your reviews!
Obviously all PRs need our attention. Among them, our DLangUPB scholarship recipients are folks committed to work on longer-term projects and could use our help with ramping up to speed. Here's a canned search with all of their PRs: https://github.com/search?utf8=%E2%9C%93&q=user%3Adlang+repo%3Aphobos+state%3Aopen+author%3ARazvanN7+author%3Aedi33416+author%3ADarredevil+author%3Asomzzz&type=Issues&ref=searchresults If you could find the time to give feedback it would be much appreciated. Thanks! Andrei
Re: ModuleInfo, factories, and unittesting
On Sunday, 18 December 2016 at 16:53:24 UTC, Andrei Alexandrescu
wrote:
Compulsive ModuleInfo generation seems to be a drag. I'm not
very familiar with the particulars but my understanding is
ModuleInfo is needed for (a) Object.factory and (b) finding and
running unittests.
Walter and I think Object.factory was a mistake and we'd like
to make it opt-in long term (either by means of a build flag or
a class attribute).
Don't forget that typeinfos are classes too, which makes the
class attribute approach less attractive.
Until then, a very nice step forward is to NOT generate
ModuleInfo if a module introduces no class.
Module (static) ctors & dtors need MI.
For unittests, I figure things like static introspection should
make it easier to enumerate and run unittests without a need
for ModuleInfo.
We have __traits(getUnitTests,...) but the way to do DIY
unittests is
foreach(m; ModuleInfo)
foreach(test; __traits(getUnitTests,m)
test();
What other issues/opportunities do you see related to
ModuleInfo?
AFAIK a module info is basically
struct ModuleInfo
{
alias F = void function();
ModuleInfo* parent;
string name; // may be static char array
F s_ctor; // Static constructor
F s_dtor;
F ctor;// Thread local constructor
F dtor;
F unittester; //may be an array for each block
void function(Object)[string] factory_mapping; //Foward to
argless ctor, not sure what mapping is actually used.
}
If we optionally remove Object.factory, and move the unittests
elsewhere (where to?) we are left with the ctors and dtors. If we
store these as SoA then we minimise space (as few modules define
*tors), but lose the mapping from Module -> *tor, but these are
defined to be run in an implementation defined order anyway so no
big loss. This is only a problem when dealing with multiple
libraries compiled separately and dynamic libraries, but if the
dynamic libs store their own SoA'd *tors and we merge the static
libs Arrays at link time** then we have nothing useful left in
the module info.
If we could get factory_mapping mergeable at link-time then we
would get a pay for what you use
** I don't know if this is possible, I don't mean merge duplicate
symbols, I mean liba.a defines 3 module ctors, so does libb.a ->
link == one ctor symbol 6 entries. Similar to what is done for
C's global constructors, they end up in one big list.
Thanks,
Andrei
Re: ModuleInfo, factories, and unittesting
On 19/12/2016 12:37 PM, bitwise wrote: On Sunday, 18 December 2016 at 22:55:30 UTC, rikki cattermole wrote: - -typeinfo=low/low-min/normal/high/none None is pretty much -betterC Low would be unittests + module constructors Low-min would be like low but with only the fields that is needed Normal is what we have now more or less High of course includes all the goodies like class fields and methods reflection - Full class + struct + union symbol reflection, so fields uda's ext. IMO, this seems like too much. I think none/minimal/full would be easier, where minimal would be equivalent to what's currently available right now. My specific concern is compatibility between compiled objects and knowing what symbols to expect. Also, just keeping the question of which level to use simple. If one object file was compiled without unittests, for example, a project as a whole could pass without error, but actually be broken..couldn't it? Yes, if you don't include unittests via none, then it won't be tested. The only one of my suggestions that actually breaks typeinfo definitions is that of low-min. This is for memory constrained situations like kernels or MCU's but when you still want features such as unittesting and module constructors. All others will happily interlink without error. Overall, when the question is simple, the answer is too. Here the question isn't simple and so my answer isn't either. You've got to take into consideration that bloat depends on use case e.g. normal may very well be considered bloated by game dev standards while full is just about right for web applications.
Re: ModuleInfo, factories, and unittesting
On Sunday, 18 December 2016 at 22:55:30 UTC, rikki cattermole wrote: - -typeinfo=low/low-min/normal/high/none None is pretty much -betterC Low would be unittests + module constructors Low-min would be like low but with only the fields that is needed Normal is what we have now more or less High of course includes all the goodies like class fields and methods reflection - Full class + struct + union symbol reflection, so fields uda's ext. IMO, this seems like too much. I think none/minimal/full would be easier, where minimal would be equivalent to what's currently available right now. My specific concern is compatibility between compiled objects and knowing what symbols to expect. Also, just keeping the question of which level to use simple. If one object file was compiled without unittests, for example, a project as a whole could pass without error, but actually be broken..couldn't it?
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On 12/18/2016 05:18 PM, pineapple wrote: On Sunday, 18 December 2016 at 21:09:46 UTC, Andrei Alexandrescu wrote: On 12/18/2016 10:01 AM, pineapple wrote: On Sunday, 18 December 2016 at 13:31:48 UTC, Andrei Alexandrescu wrote: Is the source code publicly available? https://github.com/pineapplemachine/mach.d The code looks clean, congrats. With your permission I'd like to give this library as an example in the "Workaround: Increasing Granularity of Modules" section. Please advise, thanks. -- Andrei No problem, feel free. Thank you for asking. Great, thanks. Please take a look at the accuracy of the discussion. I expanded the "Workaround" section and moved it near the top. https://github.com/dlang/DIPs/pull/51 https://github.com/dlang/DIPs/blob/dd46252e820dce66df746540d7ab94e0b00a6505/DIPs/DIP1005.md Andrei
Re: ModuleInfo, factories, and unittesting
On 19/12/2016 5:53 AM, Andrei Alexandrescu wrote: Compulsive ModuleInfo generation seems to be a drag. I'm not very familiar with the particulars but my understanding is ModuleInfo is needed for (a) Object.factory and (b) finding and running unittests. Walter and I think Object.factory was a mistake and we'd like to make it opt-in long term (either by means of a build flag or a class attribute). Until then, a very nice step forward is to NOT generate ModuleInfo if a module introduces no class. For unittests, I figure things like static introspection should make it easier to enumerate and run unittests without a need for ModuleInfo. What other issues/opportunities do you see related to ModuleInfo? Thanks, Andrei I've been toying with the idea of completely overhauling TypeInfo. Here is the gist of what I want: - -typeinfo=low/low-min/normal/high/none None is pretty much -betterC Low would be unittests + module constructors Low-min would be like low but with only the fields that is needed Normal is what we have now more or less High of course includes all the goodies like class fields and methods reflection - Full class + struct + union symbol reflection, so fields uda's ext. My feeling is D really is two different languages. An application one and a system one. The difference isn't much. It really comes down to how hooked into druntime it is and so TypeInfo. From there we just need to remove all language features that require druntime or allow a way to you know, clear deallocation path.
Re: The Computer Language Benchmarks Game - no D
On Sunday, 18 December 2016 at 21:29:05 UTC, jmh530 wrote: On Sunday, 18 December 2016 at 21:26:36 UTC, Boston wrote: Some days ago I'd been looking for comparisons between different programming languages, and I found this site: It's been discussed on the forum before. Yeah, many times. We'd all like to see D there but Isaac Gouy (the maintainer of the language shootout) has made it pretty clear D isn't getting on there but he welcomes anyone to use the shootout's infrastructure to run their own benchmarks for D and put it up on their own site. I understand his position because no matter what he does and what he includes there will always be someone unhappy with it. I'd like to see something like the Techempower web framework benchmarks done for general computation language benchmarking like the language shootout, personally.
Re: Making preconditions better specified and faster
On Sunday, 18 December 2016 at 21:58:57 UTC, Andrei Alexandrescu
wrote:
On 12/18/2016 04:24 PM, Caspar Kielwein wrote:
On Sunday, 18 December 2016 at 10:47:42 UTC, Stefan Koch wrote:
On Sunday, 18 December 2016 at 09:32:53 UTC, Caspar Kielwein
wrote:
On Thursday, 15 December 2016 at 18:48:22 UTC, Andrei
Alexandrescu
wrote:
https://issues.dlang.org/show_bug.cgi?id=16975
I'd love if preconditions where available at the caller.
This would
make it possible to use preconditions for input scrubbing,
even in
release builds with disabled asserts.
That is exactly what they are not for.
I know that contracts are for specifying guarantees and
conditions in
regards to correctness of a program.
A simple way to look at this is: a program may build and run
with all contracts disabled. If correct, there should be no
change in semantics. Is a contract therefore a right place to
make sure your files are not corrupt etc? -- Andrei
I'll try clarify what I mean.
I absolutely agree that contracts assert the correctness of the
program and not of user inputs.
When I pass user inputs to functions with preconditions, I need
to validate these inputs to make sure the function call and thus
my program is correct.
The code for this validation is often redundant to the
specification of the precondition in it's logic. It of cause has
a very different effect and often different syntax.
I'd like to write something like:
try{
foo.verified(x);
} catch {
...
}
To manually call the check of the precondition and avoid the
duplication and possibility of missing a case.
Another case (which hits me in my job writing c++) is when I call
performance critical code from (legacy) code with dubious
correctness. I'd like to have all asserts enabled at the caller
side to catch bugs, without the performance hit of enabling
asserts within a numeric kernel.
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On 12/18/2016 05:18 PM, pineapple wrote: On Sunday, 18 December 2016 at 21:09:46 UTC, Andrei Alexandrescu wrote: On 12/18/2016 10:01 AM, pineapple wrote: On Sunday, 18 December 2016 at 13:31:48 UTC, Andrei Alexandrescu wrote: Is the source code publicly available? https://github.com/pineapplemachine/mach.d The code looks clean, congrats. With your permission I'd like to give this library as an example in the "Workaround: Increasing Granularity of Modules" section. Please advise, thanks. -- Andrei No problem, feel free. Thank you for asking. Is there a simple command to e.g. unittest everything in the project? Also, is there a build process or it's all templated? -- Andrei
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On Sunday, 18 December 2016 at 21:09:46 UTC, Andrei Alexandrescu wrote: On 12/18/2016 10:01 AM, pineapple wrote: On Sunday, 18 December 2016 at 13:31:48 UTC, Andrei Alexandrescu wrote: Is the source code publicly available? https://github.com/pineapplemachine/mach.d The code looks clean, congrats. With your permission I'd like to give this library as an example in the "Workaround: Increasing Granularity of Modules" section. Please advise, thanks. -- Andrei No problem, feel free. Thank you for asking.
Re: Making preconditions better specified and faster
On 12/18/2016 04:24 PM, Caspar Kielwein wrote: On Sunday, 18 December 2016 at 10:47:42 UTC, Stefan Koch wrote: On Sunday, 18 December 2016 at 09:32:53 UTC, Caspar Kielwein wrote: On Thursday, 15 December 2016 at 18:48:22 UTC, Andrei Alexandrescu wrote: https://issues.dlang.org/show_bug.cgi?id=16975 I'd love if preconditions where available at the caller. This would make it possible to use preconditions for input scrubbing, even in release builds with disabled asserts. That is exactly what they are not for. I know that contracts are for specifying guarantees and conditions in regards to correctness of a program. A simple way to look at this is: a program may build and run with all contracts disabled. If correct, there should be no change in semantics. Is a contract therefore a right place to make sure your files are not corrupt etc? -- Andrei
The Computer Language Benchmarks Game - no D
Some days ago I'd been looking for comparisons between different programming languages, and I found this site: http://benchmarksgame.alioth.debian.org/ It seems like a good way to compare the performance of different PLs, when fed different tasks to do, but there's a problem - there's no D in it! Which seems very strange, given the inclusion of Clojure (found at the bottom of the top-50 list in the TIOBE index - http://www.tiobe.com/tiobe-index/ ). Is there any reason in particular it isn't included, or do the D masters have no time to spare for such trivial things? :) It would be interesting to see where D fits in the Performance list - http://benchmarksgame.alioth.debian.org/u64q/performance.php?test=nbody - don't you think?
Re: The Computer Language Benchmarks Game - no D
On Sunday, 18 December 2016 at 21:26:36 UTC, Boston wrote: Some days ago I'd been looking for comparisons between different programming languages, and I found this site: It's been discussed on the forum before.
Re: Making preconditions better specified and faster
On Sunday, 18 December 2016 at 10:47:42 UTC, Stefan Koch wrote: On Sunday, 18 December 2016 at 09:32:53 UTC, Caspar Kielwein wrote: On Thursday, 15 December 2016 at 18:48:22 UTC, Andrei Alexandrescu wrote: https://issues.dlang.org/show_bug.cgi?id=16975 I'd love if preconditions where available at the caller. This would make it possible to use preconditions for input scrubbing, even in release builds with disabled asserts. That is exactly what they are not for. I know that contracts are for specifying guarantees and conditions in regards to correctness of a program. Still, at the first point where I call a function with user inputs, I often have to pretty much repeat the precondition of that function (but react different on violation) to scrub the inputs. I think it would be nice to be able to make the conscious choice of using the specified precondition for that.
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On 12/18/2016 10:01 AM, pineapple wrote: On Sunday, 18 December 2016 at 13:31:48 UTC, Andrei Alexandrescu wrote: On 12/17/16 10:21 PM, pineapple wrote: I am developing a general-use library for D that is currently resting at around 50,000 lines. Is the source code publicly available? https://github.com/pineapplemachine/mach.d The code looks clean, congrats. With your permission I'd like to give this library as an example in the "Workaround: Increasing Granularity of Modules" section. Please advise, thanks. -- Andrei
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 20:16:11 UTC, Andrei Alexandrescu wrote: On 12/18/2016 03:10 PM, Ilya Yaroshenko wrote: On Sunday, 18 December 2016 at 19:28:24 UTC, Andrei Alexandrescu wrote: On 12/18/2016 01:49 PM, Ilya Yaroshenko wrote: On Sunday, 18 December 2016 at 17:41:31 UTC, Walter Bright wrote: On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote: We already have better `cpuid` and better `random` packages. Great! Please PR them for Phobos. cpuid is used in Mir GLAS and it should be a betterC library. It's this kind of imaginary dialog that I don't quite grok: Ilya Yaroshenko: "I can't use druntime's cpuid so I defined my own." [...] IY: "..." It's this kind of stuff I need to have a better understanding of. Some technical arguments are meaningful, some others point to problems with obvious solutions that are somehow shunned, and yet others are like a hidden Markov model - I see the effects, but I don't see the causes. Andrei My last argument was: Linking compatibility. Assume A has builded a betterC library AA using GDC version XXX and B has builded a betterC library BB using LDC version YYY. Question: how mister C can use BB and AA together in his C-library if both this libraries depends on different DRuntimes (compiler and version)? My understanding is that if cpuid_v2.d does not pull any dependency, it doesn't matter whether it sits in druntime or elsewhere. Is that the case? DRuntime is large and evaluates fast. So is it moving too slow or too fast? It is too slow as library to contribute to and too fast as standard library to depend on. We have not backward binary compatibility in DRuntime and we have not binary compatibility between different compiler versions. Even if we don't, we can guarantee that core.cpuid_v2 adds no link-time dependency. We may offer it as a pure template library, or as C APIs. Whatever floats your boat. Is this enough for ensuring compatibility? I need to understand how it is can be done because I did not find a valid solution. mir-cpuid has global symbols, they are accessed using extern(C) API. If multiple libraries trying to initialize it than it will be initialized only once. extern C API does not solve a problem with DRuntime because current DRuntime is not a betterC library and it is huge. A huge betterC library is a problem too because it is less portable. Anyway, if you find a proper solution please share it. How it can be implemented as template library? Does different libraries will have their own cpuid instantiations and __gshared symbols? How modularity can break existing Phobos? Why not to allow new packages like mir-cpuid and std.allocator to be a betterC library with their own repository? This will not break existing Phobos and Druntime but this will allow me to write server and numeric software without copy-pasting Phobos codes. I prefer to fork something rather then copypast. My plans to work on: 1. Low-level numeric libraries 2. Low-level fast async I/O adaptive generic event loop, asymmetric green threads, http2 3. Microelectromechanical systems They will require new threads, new timers, new mutexes, new event loop (existing C libs are slow), to be used. Existing one does not satisfy my requirements. I do not earn any money for this work. I am working on this projects because I believe in D and believe that my current and future clients will believe in D much more then now. I know what they need. Ilya
Re: [OT] D in makefiles
On 12/18/16 2:47 PM, Shachar Shemesh wrote: On 17/12/16 10:06, John Colvin wrote: On Saturday, 17 December 2016 at 02:00:34 UTC, Superstar64 wrote: Makefiles allow custom shells. So with a small wrapper script we can use D in makefile. Here's my proof of concept: https://gist.github.com/Superstar64/8b896312ebe1a6e6240b1cba8aed2488. If you define SHELL = rdmd you might not need any wrapper script. Since newlines in make mean new execution, you will find it pretty hard to write a recipe that will be both usable and readable. IIRC there is an option in gmake that allows one to run all commands within the same shell. https://www.gnu.org/software/make/manual/html_node/One-Shell.html. Anyway, it seems such usage of rdmd would be quite extreme. -- Andrei
Re: A betterC modular standard library?
On 12/18/2016 03:10 PM, Ilya Yaroshenko wrote: On Sunday, 18 December 2016 at 19:28:24 UTC, Andrei Alexandrescu wrote: On 12/18/2016 01:49 PM, Ilya Yaroshenko wrote: On Sunday, 18 December 2016 at 17:41:31 UTC, Walter Bright wrote: On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote: We already have better `cpuid` and better `random` packages. Great! Please PR them for Phobos. cpuid is used in Mir GLAS and it should be a betterC library. It's this kind of imaginary dialog that I don't quite grok: Ilya Yaroshenko: "I can't use druntime's cpuid so I defined my own." [...] IY: "..." It's this kind of stuff I need to have a better understanding of. Some technical arguments are meaningful, some others point to problems with obvious solutions that are somehow shunned, and yet others are like a hidden Markov model - I see the effects, but I don't see the causes. Andrei My last argument was: Linking compatibility. Assume A has builded a betterC library AA using GDC version XXX and B has builded a betterC library BB using LDC version YYY. Question: how mister C can use BB and AA together in his C-library if both this libraries depends on different DRuntimes (compiler and version)? My understanding is that if cpuid_v2.d does not pull any dependency, it doesn't matter whether it sits in druntime or elsewhere. Is that the case? DRuntime is large and evaluates fast. So is it moving too slow or too fast? We have not backward binary compatibility in DRuntime and we have not binary compatibility between different compiler versions. Even if we don't, we can guarantee that core.cpuid_v2 adds no link-time dependency. We may offer it as a pure template library, or as C APIs. Whatever floats your boat. Is this enough for ensuring compatibility? Andrei
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 19:28:24 UTC, Andrei Alexandrescu wrote: On 12/18/2016 01:49 PM, Ilya Yaroshenko wrote: On Sunday, 18 December 2016 at 17:41:31 UTC, Walter Bright wrote: On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote: We already have better `cpuid` and better `random` packages. Great! Please PR them for Phobos. cpuid is used in Mir GLAS and it should be a betterC library. It's this kind of imaginary dialog that I don't quite grok: Ilya Yaroshenko: "I can't use druntime's cpuid so I defined my own." [...] IY: "..." It's this kind of stuff I need to have a better understanding of. Some technical arguments are meaningful, some others point to problems with obvious solutions that are somehow shunned, and yet others are like a hidden Markov model - I see the effects, but I don't see the causes. Andrei My last argument was: Linking compatibility. Assume A has builded a betterC library AA using GDC version XXX and B has builded a betterC library BB using LDC version YYY. Question: how mister C can use BB and AA together in his C-library if both this libraries depends on different DRuntimes (compiler and version)? DRuntime is large and evaluates fast. We have not backward binary compatibility in DRuntime and we have not binary compatibility between different compiler versions. mir-glas is a betterC library for all languages. It depends in mir-cpuid. I want mir-glas to replace OpenBLAS. To do it I need the same portability. OK, assume we already have DRuntime for all targets including MIPS, POWER, armv7, aarch64, Loongson. Does this solves problem? No. The reason is very simple. All non-betterC D libraries must be compiled in lockstep: with the same compiler vendor and druntime version! This will just kill GLAS. How this can be packed for Ubuntu for example? It will depends on LDC libphobos == XXX. Another numeric library, say FFT, will depend on libphobos == YYY by GDC. How they can be linked if they require different DRuntime from different compilers? Or different druntime versions for the same compiler? Do you want to constraint D users to do not distribute binary libraries in popular repositories like Debian/Ubuntu? DRuntime incompatibility is a huge constraints for business. Do you want to force all companies to use identical version of compiler and identical vendor? Yes, C++ changes their API, but it few orders of magnitude slower then D. And they support multiple branches with bugs fixes. D supports only the recent one. This is why GLAS/cpuid need to be a betterC library. Ilya
Re: [WIP] A Big O notation and algebra for D
https://en.wikipedia.org/wiki/Lattice_(order)
On Sun, Dec 18, 2016 at 10:46 AM, John Colvin via Digitalmars-d <
digitalmars-d@puremagic.com> wrote:
> On Sunday, 18 December 2016 at 18:26:27 UTC, Andrei Alexandrescu wrote:
>
>> Article: http://erdani.com/d/bigo.html (do not publish; should do it
>> with Mike)
>>
>> Code (preliminary): https://github.com/dlang/phobos/pull/4965
>>
>>
>> Andrei
>>
>
> static assert(
>> complexity!(insertFrontMany!MyC) <= O("n2") * log(O("n1")),
>> "Too high complexity for insertFrontMany.");
>>
>
> You defined insertFrontMany as taking 2 template arguments, but here you
> only give it 1.
>
> as the top of the lattice defined by the partial order ≤.
>>
>
> reads weird to me. s/lattice/ladder ?
>
Re: [OT] D in makefiles
On 17/12/16 10:06, John Colvin wrote: On Saturday, 17 December 2016 at 02:00:34 UTC, Superstar64 wrote: Makefiles allow custom shells. So with a small wrapper script we can use D in makefile. Here's my proof of concept: https://gist.github.com/Superstar64/8b896312ebe1a6e6240b1cba8aed2488. If you define SHELL = rdmd you might not need any wrapper script. Since newlines in make mean new execution, you will find it pretty hard to write a recipe that will be both usable and readable. This is, of course, assuming you overcome the fact that rdmd does not accept the program as a "-c" argument. Shachar
Re: A betterC modular standard library?
On 12/18/2016 01:49 PM, Ilya Yaroshenko wrote: On Sunday, 18 December 2016 at 17:41:31 UTC, Walter Bright wrote: On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote: We already have better `cpuid` and better `random` packages. Great! Please PR them for Phobos. cpuid is used in Mir GLAS and it should be a betterC library. It's this kind of imaginary dialog that I don't quite grok: Ilya Yaroshenko: "I can't use druntime's cpuid so I defined my own." Andrei Alexandrescu: "OK, so what's different?" IY: "Mine has a few engineering improvements." AA: "Cool, why don't you merge them into core.cpuid?" IY: "Well mine doesn't have a shared static constructor, so it doesn't need a runtime to automatically call that library initialization function. User need to explicitly call an init function before using it." AA: "I understand. Great, so how about this - we add your code to core.cpuid_v2 to druntime. Then we make all in core.cpuid to forward to it so there's no duplication. It all works out!" IY: "No, I don't want to do that. It's still in druntime and I don't want druntime. I want betterC." AA: "But it will be compilable with betterC and we can add unittests to make that happen. YOU HAVE MY SUPPORT. Let's do it." IY: "No, I want to change it often. The deployment schedule of druntime is too slow." AA: "How often do you need to change it? Is it that unstable?" IY: "Um, not too often." AA: "Then what is the matter? Are you worried about the IP of the engineering improvements you are making? Are you licensing this differently?" IY: "No, it's for the most part similar to core.cpuid and the license will be also Boost." AA: "Then what is the matter? Do you want me to wait until you release a stable mir.cpuid and copy it over with credit, per the Boost license, to core.cpuid_v2?" IY: "..." It's this kind of stuff I need to have a better understanding of. Some technical arguments are meaningful, some others point to problems with obvious solutions that are somehow shunned, and yet others are like a hidden Markov model - I see the effects, but I don't see the causes. Andrei
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 14:37:55 UTC, Radu wrote: On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko Who is interested in betterC _modular_* standard library? I am planing to make libmir org a community for it. Thought and concerns? I hope this doesn't turn into a new Tango vs. Phobos war. My first thought as well. Probably an irrational fear, but the question will pop up often.
Re: ModuleInfo, factories, and unittesting
On Sunday, 18 December 2016 at 16:53:24 UTC, Andrei Alexandrescu wrote: Compulsive ModuleInfo generation seems to be a drag. I'm not very familiar with the particulars but my understanding is ModuleInfo is needed for (a) Object.factory and (b) finding and running unittests. Walter and I think Object.factory was a mistake and we'd like to make it opt-in long term (either by means of a build flag or a class attribute). Until then, a very nice step forward is to NOT generate ModuleInfo if a module introduces no class. For unittests, I figure things like static introspection should make it easier to enumerate and run unittests without a need for ModuleInfo. What other issues/opportunities do you see related to ModuleInfo? Thanks, Andrei If ModuleInfo were to be opt in, would that mean it could be expanded then, to include things like class/struct fields?
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 17:41:31 UTC, Walter Bright wrote: On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote: We already have better `cpuid` and better `random` packages. Great! Please PR them for Phobos. cpuid is used in Mir GLAS and it should be a betterC library. Without DRuntime dependency because DRuntime has not backward binary compatibility and has not compatibility between different compilers. From my reply to Andrei: Linking compatibility. Assume A has builded a betterC library AA using GDC version XXX and B has builded a betterC library BB using LDC version YYY. Question: how mister C can use BB and AA together in his C-library if both this libraries depends on different DRuntimes (compiler and version)? mir-glas and mir-cpuid are libraries with extern(C) interface for all languages. It is silly to build a BLAS implementation only for one language. The betterC std.range and std.algorithm analogs would be released with new ndslice implementation. Mir's algorithm would be faster then Phobos and will generate less template bloat. Then lightweight threads, multithread GLAS, matrix inversion. Fastest I/O and http2 ... (they all are betterC modular libraries) Propose them for Phobos! First I need to implement them and announce for the community. Then community will say "What he is doing??? C++ iterators, really?" Hehe This is reason why ndslice is deprecated in Phobos. It requires another std.algorithm and std.range. I am not able to implement all this ideas in Phobos PRs: it is requires too much time. In addition, I can not use Phobos for other reasons. Andrei and probably 90% of existing D users don't want Phobos to move this direction. I don't understand that point. What direction? 1. Modularity: D should provide a very slim library (like std.traits and may be default GC with old core, which can be turned off or replaced). Other parts or Phobos/Druntime should be split into parts and the parts should have their own repositories. They should be dub packages with D Foundation review / control. 2. Low-level API first! A high level OOP/GC-oriented API can be provided later or can be provided in DUB libraries. 3. betterC guaranties / No problems with backward binary compatibility for betterC libs. This means that code should be templated / inlined. Some parts, like Exceptions and GC may have this problems, but Exceptions and GC should be optional language features and they are not a part of betterC mode. In other hand I need a commercial attractive D infrastructure for large and heavy system projects. There is no commercial perspective for me to contribute to Phobos because: 1. Phobos version depends on compiler version. Delay with LDC release is too large. It should / can be one day. The language evolves over time, and the standard library must, too. It happens with every language. Agreed. Because that it is more flexible to have a very slim std library with good set of community supported libraries. This is a reason why almost all portable multi-language numeric libraries are written in C. C runtime is slim and evaluates very-very slow. 2. We can not do hot fixes in Phobos without compiling and deploying patched libphobos. Mir's DUB package are much more flexible. 3. Dependencies should be clear. Modularity is a proper way for large std library. In phobos everything integrated with everything. DRuntime -> Phobos abstraction is weird for betterC because system modules can depends universal algorithms, but universal algorithm are more portable if they have not system dependencies. Not sure what you mean. Algorithms in Phobos are not dependent on system idiosyncracies. They are, for example they depends on RangeErrors. See https://github.com/dlang/phobos/pull/4937 4. Phobos does not provide and will not provide betterC guaranties. If something works for betterC mode now it may not work in the future release. I've done a fair amount of work to remove such things. Clearly more can and should be done. Generally, we do have a goal of making Phobos entirely "pay as you go" instead of being so interconnected (which is what betterC is really all about). We can definitely use help in improving that and helping it along. Great. Why not to go to the proposed direction: 1 Modularity, 2 Low-level API first, 3 betterC guaranties? Thank you, Ilya
Re: [WIP] A Big O notation and algebra for D
On Sunday, 18 December 2016 at 18:26:27 UTC, Andrei Alexandrescu
wrote:
Article: http://erdani.com/d/bigo.html (do not publish; should
do it with Mike)
Code (preliminary): https://github.com/dlang/phobos/pull/4965
Andrei
static assert(
complexity!(insertFrontMany!MyC) <= O("n2") * log(O("n1")),
"Too high complexity for insertFrontMany.");
You defined insertFrontMany as taking 2 template arguments, but
here you only give it 1.
as the top of the lattice defined by the partial order ≤.
reads weird to me. s/lattice/ladder ?
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On 12/18/16 1:03 PM, Joakim wrote: I largely agree with Dmitry. Ilya refactored several Phobos modules to use scoped, selective imports much more, and I pitched in for some remaining imports in the largest modules, so that only these module-level imports remain, ie those necessary for symbols imported in template constraints: std.datetime - https://github.com/dlang/phobos/pull/4373/files std.uni - https://github.com/dlang/phobos/pull/4365/files std.string and std.traits - https://github.com/dlang/phobos/pull/4370/files Yah, there's been a lot of good work (Jack Stouffer did a lot as well IIRC) on pushing imports inside. The following searches should be relevant: git grep '^\(private \)\?import' | wc -l yields about 426 top-level import declarations. The number of indented imports is 4605: git grep ' *import\W' | wc -l So we're looking at 10% of imports being problematic. Sadly, they turn out to make things quite difficult for Phobos. Last time I looked at the module dependency graph it wasn't a lot better than it used to before scoped imports. (I don't have it handy... could anyone please produce it?) When I first saw this DIP, like Dmitry I was happy that we could get rid of those too, but the more I see these horrible syntax suggestions for what is really a minor convenience, I changed my mind. std.datetime, the 35k line (17 kloc according to Dscanner) beast of phobos, only needs 20 or so symbols at module-scope. std.uni- 10k lines, 4.2 kloc- only needs 17 symbols, all from the three modules Dmitry mentioned. I don't think his workaround of splitting up modules is even needed for such a low amount of module-level imports. This paragraph is a good example of a couple of counterarguments that I think point directly to flaws in the DIP: (1) The DIP uses Phobos as an example, so it is applicable mostly to Phobos. In fact, Phobos is among the systems that would benefit least from the DIP: it has only druntime as dependency, and is distributed in its entirety. Many projects out there list multiple dependencies and may have various building and distribution policies. The converse is to believe that working around a problem in Phobos would render the DIP less useful in general. (2) "I don't like the syntax, hence I don't like the feature." I see this as a good opportunity for tasteful design, not a downside of the feature. Maybe there are other issues having to do with symbol resolution and dependency encapsulation that are addressed by this DIP, ie the technical performance of the compiler rather than refactoring or code clarity, that I don't fully grasp, but from the first two points of the claimed benefits of DCDs, ie ease of reasoning about dependencies and refactoring code, I don't think this feature will come anywhere close to carrying its own weight. Does the refactoring in https://github.com/dlang/phobos/pull/4962 make dependencies clearer? Are you e.g. clear on what you need in order to use Appender? Would you want to take this (or another) experiment to another module and see how it improves its dependency structure? As for the third benefit having to do with scalable template libraries, I'm not sure I completely understand all the details there, but I wonder if those problems aren't an artifact of the way dmd works now rather than something that can't be fixed without this DIP. The DIP now dedicates an entire section to the pluses and minuses of lazy imports, and concludes that lazy imports would address scalability if carefully used. Andrei
[WIP] A Big O notation and algebra for D
Article: http://erdani.com/d/bigo.html (do not publish; should do it with Mike) Code (preliminary): https://github.com/dlang/phobos/pull/4965 Andrei
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On Thursday, 15 December 2016 at 22:56:42 UTC, Dmitry Olshansky
wrote:
On 12/13/16 11:33 PM, Andrei Alexandrescu wrote:
Destroy.
https://github.com/dlang/DIPs/pull/51/files
Andrei
On first it seems like an awesome idea. That solves ... but
wait what? Thinking more about the problem at hand - I fail to
see what this DIP accomplishes that can't be done better today.
1. The benefit of placing import to each function is based on
the untold assumption that we have:
a) huge modules with many functions
b) that constraints of these functions require different
(large) modules
The reality is far from this picture - if anything 99% of
template constraints are dependent on std.range.primitives and
std.traits with a bit of std.meta from time to time. So we'd
have a boilerplate of
auto foo(R)(R range)
(import std.range.primitives)
if(isInputRange!R){ ... }
everywhere for no noticeable benefit - touch one of functions
and you get full set of imports of these _small_ modules.
2. By itself the mechanism for delaying import even for
constraint until the function is touched is moot as long as the
module in question is huge and is not split in pieces. In other
words:
auto foo(R)(R range)
(import std.range)
if(isInputRange!R){ ... }
Pulls in full std.range the moment foo is touched, compared to
import std.range.primitives;
...
auto foo(R)(R range)
if(isInputRange!R){ ... }
which is because it actually isolates the whole mess of
complete std.range from the user of a template.
All in all my practical response is split the modules at least
in 2 parts: constraints + full functionality, then import the
one with constraints at the top level. Works today and solves
the practical issues unlike the proposal.
---
Dmitry Olshansky
I largely agree with Dmitry. Ilya refactored several Phobos
modules to use scoped, selective imports much more, and I pitched
in for some remaining imports in the largest modules, so that
only these module-level imports remain, ie those necessary for
symbols imported in template constraints:
std.datetime - https://github.com/dlang/phobos/pull/4373/files
std.uni - https://github.com/dlang/phobos/pull/4365/files
std.string and std.traits -
https://github.com/dlang/phobos/pull/4370/files
When I first saw this DIP, like Dmitry I was happy that we could
get rid of those too, but the more I see these horrible syntax
suggestions for what is really a minor convenience, I changed my
mind. std.datetime, the 35k line (17 kloc according to Dscanner)
beast of phobos, only needs 20 or so symbols at module-scope.
std.uni- 10k lines, 4.2 kloc- only needs 17 symbols, all from the
three modules Dmitry mentioned. I don't think his workaround of
splitting up modules is even needed for such a low amount of
module-level imports.
Maybe there are other issues having to do with symbol resolution
and dependency encapsulation that are addressed by this DIP, ie
the technical performance of the compiler rather than refactoring
or code clarity, that I don't fully grasp, but from the first two
points of the claimed benefits of DCDs, ie ease of reasoning
about dependencies and refactoring code, I don't think this
feature will come anywhere close to carrying its own weight.
As for the third benefit having to do with scalable template
libraries, I'm not sure I completely understand all the details
there, but I wonder if those problems aren't an artifact of the
way dmd works now rather than something that can't be fixed
without this DIP.
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 15:18:37 UTC, Andrei Alexandrescu wrote: On 12/18/16 4:26 AM, Ilya Yaroshenko wrote: Hi, Who is interested in betterC _modular_* standard library? I am planing to make libmir org a community for it. Thought and concerns? We already have better `cpuid` and better `random` packages. The betterC std.range and std.algorithm analogs would be released with new ndslice implementation. Mir's algorithm would be faster then Phobos and will generate less template bloat. Then lightweight threads, multithread GLAS, matrix inversion. Fastest I/O and http2 ... (they all are betterC modular libraries) Andrei and probably 90% of existing D users don't want Phobos to move this direction. Please do not misconstrue what I said. You have my support to improve the state of affairs with the D core runtime and standard library, and have had it from day one. With regard to new functionality that supplants existing pieces such as cpuid_v2 or random_v2, definitely. We can and should add that and many other great things to phobos. Go for it. With regard to avoiding to link druntime, we should build the argument better. Is all of druntime bad, or just parts of it? DRuntime is good for applications. But writing multi-language libraries with DRuntime is bad idea. The last paragraph in [1] describes why DRuntime can not be used for GLAS. I never said we should "remove GC" or "remove runtime". We just need to provide a clear way for betterC subset and modularise library. DRuntime and Phobos are well designed hight level libraries. E.g. if something does not link the GC but does use vtables, is it good/bad, and to what extent? Very good point. First, I don't see a real reason why we need DRuntime for vtables (classes and interfaces). LDC allows to use vtables without DRuntime (only with extern(C++), hehe). D classes can be constructed on stack or with allocators. The only problem is vtable initlization, which requires `typeid` (TypeInfo object). I am not a OOP expert, but it looks like we can have extern(C++) classes without DRuntime if we solve problem with initilization. These issues are not entirely clear to me and it would be great if you explained them better. I definitely need to be better educated on this. What I do know is making "don't use druntime" a goal in and of itself is not the most useful way to frame the problem. 0. Linking compatibility. Assume A has builded a betterC library AA using GDC version XXX and B has builded a betterC library BB using LDC version YYY. Question: how mister C can use BB and AA together in his C-library if both this libraries depends on different DRuntimes (compiler and version)? 1. Different mangling names for DMD and LDC. mir-cpuid provides extern(C) interface for all functionality except few very low level struct declarations. The general problem is that we do not have binary compatibility where we may have it. And it is not clear where we already have it. 2. Any function may throw an Error. 3. Any throwable is not a part of betterC. Phobos core team rejected approach to replace RangeErrors with asserts or another function declaration. Possible alternative is __range_assert() or a library function. 4. Threads are GC oriented: 1. They are classes (they can be structs like allocators) 2. They are integrated with GC. This requires additional computation time if I am not wrong. 3. They are not replaceable. Phobos multithread modules can not use something else. 5. Mutexes are GC oriented: ditto 1. and 3. 6. Intrinsics are compiler magic without specification. It is not clear what it is intrinsic, are they inlined, where their bodies are defined, how/if they works in CTFE. LLVM provides a list of intrinsics [3], this is very useful to write a betterC libraries. ... and many other ... They all are not clear for me too. - A solution for mister C: Mister A and mister B should use a runtime which has either a small stable extern(C) interface or all its functions are inlined. All other functions can be a dub packages. - Proof: cpuid: extern(C) interface with aliases. a fork of aligned mallocator for GLAS: all functions are always inlined. With regard to changes that break the entire fabric of Phobos (such as: we must eliminate random-access ranges), this is definitely more like simple survival rather than things I want. I don't want eliminate R-A-R now. But we can change our Range concept and provide alternative module as dub package, which will be announced as a future replacement (say, after 5 years). But duplicated sorting implementation with cursors (lightweight iterators) in the same module is weird. It is better to continue this discussion after the cursor based implementation will be ready for your review. With regard to properly attributing credit for good work, I believe this is important and I commit to do anything reasona
Re: A betterC modular standard library?
On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote: We already have better `cpuid` and better `random` packages. Great! Please PR them for Phobos. The betterC std.range and std.algorithm analogs would be released with new ndslice implementation. Mir's algorithm would be faster then Phobos and will generate less template bloat. Then lightweight threads, multithread GLAS, matrix inversion. Fastest I/O and http2 ... (they all are betterC modular libraries) Propose them for Phobos! Andrei and probably 90% of existing D users don't want Phobos to move this direction. I don't understand that point. What direction? In other hand I need a commercial attractive D infrastructure for large and heavy system projects. There is no commercial perspective for me to contribute to Phobos because: 1. Phobos version depends on compiler version. Delay with LDC release is too large. It should / can be one day. The language evolves over time, and the standard library must, too. It happens with every language. 2. We can not do hot fixes in Phobos without compiling and deploying patched libphobos. Mir's DUB package are much more flexible. 3. Dependencies should be clear. Modularity is a proper way for large std library. In phobos everything integrated with everything. DRuntime -> Phobos abstraction is weird for betterC because system modules can depends universal algorithms, but universal algorithm are more portable if they have not system dependencies. Not sure what you mean. Algorithms in Phobos are not dependent on system idiosyncracies. 4. Phobos does not provide and will not provide betterC guaranties. If something works for betterC mode now it may not work in the future release. I've done a fair amount of work to remove such things. Clearly more can and should be done. Generally, we do have a goal of making Phobos entirely "pay as you go" instead of being so interconnected (which is what betterC is really all about). We can definitely use help in improving that and helping it along.
Re: Linux Kernel in D?
On Sunday, 18 December 2016 at 02:54:10 UTC, Whatsthisnow wrote: What are your thoughts on how I have implemented the strcpy type stuff? It looks like you are trying to wrap D types to interact with C. I think Walter's advice should be considered. Convert the files to match the original as closely as possible, and don't fix bugs or change any algorithm. Once that is complete and passing tests refactoring can take place.
Re: Linux Kernel in D?
On Sunday, 18 December 2016 at 11:22:49 UTC, Whatsthisnow wrote: thousands of programmers that program for Linux, so creating an entire OS in D based on Linux would encourage (hopefully) a lot of Linux programmers to also write for a D Linux. Since its largely familiar, the learning process would be relatively painless. I would put to much faith in Linux developers having an interest in supporting a language change. Now the idea that their is interest in having libc written in D, the project could be of use if the entire Linux kernel could be built and use it, talk about real world testing.
Re: Enabling data-oriented design
On Sunday, 18 December 2016 at 16:06:38 UTC, Joakim wrote: On Sunday, 18 December 2016 at 14:58:53 UTC, Stefan Koch wrote: On Sunday, 18 December 2016 at 14:22:29 UTC, Joakim wrote: [...] Hmm, looks like I pasted a link to the wrong CppCon video, here's the one on data-oriented design: https://youtube.com/watch?v=rX0ItVEVjHc [...] I would not be too sure that this is something the language can help with. It is more a question of the right tools. And In theory features like mixins, introspection and property functions should make it possible to profile your memory-access patterns in a nice way. One of the arguments for DOD in the WP link and elsewhere is that typically OOP-heavy code, as you see in much game programming, is not conducive to this type of design. To the extent someone is using D's OOP features heavily, they will run into this too. I would imagine ranges are very conducive to the DOD approach, so it's not like D doesn't support it. The question is whether we can support this design trend more. This is something I'm interested in, too. If you haven't seen them already, Jonathan Blow has videos showing his development of Jai, a language focused on DOD. I would love to see some support or tutorials of DOD in D.
ModuleInfo, factories, and unittesting
Compulsive ModuleInfo generation seems to be a drag. I'm not very familiar with the particulars but my understanding is ModuleInfo is needed for (a) Object.factory and (b) finding and running unittests. Walter and I think Object.factory was a mistake and we'd like to make it opt-in long term (either by means of a build flag or a class attribute). Until then, a very nice step forward is to NOT generate ModuleInfo if a module introduces no class. For unittests, I figure things like static introspection should make it easier to enumerate and run unittests without a need for ModuleInfo. What other issues/opportunities do you see related to ModuleInfo? Thanks, Andrei
Re: Enabling data-oriented design
On Sunday, 18 December 2016 at 14:58:53 UTC, Stefan Koch wrote: On Sunday, 18 December 2016 at 14:22:29 UTC, Joakim wrote: I was looking at the most popular videos from CppCon and the second-most popular from the last three years is the one on data-oriented design: https://youtube.com/watch?v=1OEu9C51K2A Hmm, looks like I pasted a link to the wrong CppCon video, here's the one on data-oriented design: https://youtube.com/watch?v=rX0ItVEVjHc The wikipedia page has a summary, basically focusing on cache effects more, with links to articles: https://en.wikipedia.org/wiki/Data-oriented_design Obviously the effects of cache are well-known here and other places that focus on performance, but maybe there's more we can do to enable this paradigm in the language. I would not be too sure that this is something the language can help with. It is more a question of the right tools. And In theory features like mixins, introspection and property functions should make it possible to profile your memory-access patterns in a nice way. One of the arguments for DOD in the WP link and elsewhere is that typically OOP-heavy code, as you see in much game programming, is not conducive to this type of design. To the extent someone is using D's OOP features heavily, they will run into this too. I would imagine ranges are very conducive to the DOD approach, so it's not like D doesn't support it. The question is whether we can support this design trend more.
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On 12/18/2016 5:38 AM, Andrei Alexandrescu wrote: On 12/18/16 6:47 AM, Jacob Carlborg wrote: Rubocop, the major linter in the Ruby world, will complain if a class is more than 100 lines of code. Does that include full documentation and unittests? My complaint is more along the lines of aggregates that seem to have scores of member functions. It brings to mind Scott Meyers' article about kitchen sink aggregates: http://www.drdobbs.com/cpp/how-non-member-functions-improve-encapsu/184401197
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 15:20:48 UTC, Radu wrote: 2. In the same time many thing is not possible to create with Phobos/Druntime. The simplest example is numeric libraries. I can create a multithread GLAS with core.thread, but if I do so GLAS will never be accepted as default BLAS implementation for Julia. So, i need to create my own low level thread library. Intel MKL, OpenBLAS have their own thread management: this is common practice. Cheers, Ilya I get point 2, but I should be able to use your new modules without giving up druntime/phobos, right?. They should not be mutual exclusive! Yes, you are able to use any Mir libs with Phobos/DRuntime --Ilya
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 15:20:48 UTC, Radu wrote: I get point 2, but I should be able to use your new modules without giving up druntime/phobos, right?. They should not be mutual exclusive! I think that being able to use them the same as any other C library should be sufficient.
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 15:01:55 UTC, Ilya Yaroshenko wrote: On Sunday, 18 December 2016 at 14:37:55 UTC, Radu wrote: [...] Maybe I will be able to propose them on DConf. [...] Yes/No in the same time. 1. It would be always possible to use Mir and Phobos together, 100% guarantee. Furthermore, some Mir libraries are going to be infrastructure for _all_, not only for D. For example Mir GLAS will target the Julia language after D. 2. In the same time many thing is not possible to create with Phobos/Druntime. The simplest example is numeric libraries. I can create a multithread GLAS with core.thread, but if I do so GLAS will never be accepted as default BLAS implementation for Julia. So, i need to create my own low level thread library. Intel MKL, OpenBLAS have their own thread management: this is common practice. Cheers, Ilya I get point 2, but I should be able to use your new modules without giving up druntime/phobos, right?. They should not be mutual exclusive!
Re: A betterC modular standard library?
On 12/18/16 4:26 AM, Ilya Yaroshenko wrote: Hi, Who is interested in betterC _modular_* standard library? I am planing to make libmir org a community for it. Thought and concerns? We already have better `cpuid` and better `random` packages. The betterC std.range and std.algorithm analogs would be released with new ndslice implementation. Mir's algorithm would be faster then Phobos and will generate less template bloat. Then lightweight threads, multithread GLAS, matrix inversion. Fastest I/O and http2 ... (they all are betterC modular libraries) Andrei and probably 90% of existing D users don't want Phobos to move this direction. Please do not misconstrue what I said. You have my support to improve the state of affairs with the D core runtime and standard library, and have had it from day one. With regard to new functionality that supplants existing pieces such as cpuid_v2 or random_v2, definitely. We can and should add that and many other great things to phobos. Go for it. With regard to avoiding to link druntime, we should build the argument better. Is all of druntime bad, or just parts of it? E.g. if something does not link the GC but does use vtables, is it good/bad, and to what extent? These issues are not entirely clear to me and it would be great if you explained them better. I definitely need to be better educated on this. What I do know is making "don't use druntime" a goal in and of itself is not the most useful way to frame the problem. With regard to changes that break the entire fabric of Phobos (such as: we must eliminate random-access ranges), this is definitely more like simple survival rather than things I want. With regard to properly attributing credit for good work, I believe this is important and I commit to do anything reasonable for making it happen. I kindly advise you to better understand what you are trying to do, and how to do it to maximize impact. It seems you are not entirely clear on what you want to do and what the best route is, as shown by you retiring your std.experimental.ndslice work after you've had full freedom on what should go in it. In other hand I need a commercial attractive D infrastructure for large and heavy system projects. There is no commercial perspective for me to contribute to Phobos because: 1. Phobos version depends on compiler version. Delay with LDC release is too large. It should / can be one day. Then a good way to help this is improve the LDC lag. 2. We can not do hot fixes in Phobos without compiling and deploying patched libphobos. Mir's DUB package are much more flexible. The reverse of that is less stability. The good part about merging your work in Phobos is you maximize impact and visibility. These seem to me artificial arguments. Yes, being preoccupied over quick deployment may make sense in the initial stages when there's no stability and the design may take any direction. That should get less necessary once some abstractions are in place. Organizations that are interested in patching minute fixes may build their own standard library (and sometimes compiler). This is routinely the case with C++ at Facebook, Google, and in all likelihood other large-scale tech shops. It would be tenuous to argue that this should be the default way to go about things for everyone. I don't understand why you consider it an essential matter. 3. Dependencies should be clear. Modularity is a proper way for large std library. In phobos everything integrated with everything. DRuntime -> Phobos abstraction is weird for betterC because system modules can depends universal algorithms, but universal algorithm are more portable if they have not system dependencies. I read this three times and cannot understand it. At any rate: DIP 1005, which is partially motivated by our discussions, should allow us to make dependencies as simple and clean as function-level. 4. Phobos does not provide and will not provide betterC guaranties. If something works for betterC mode now it may not work in the future release. Yes, we can definitely improve this. We (the Foundation) are all for making things better for achieving the things you want to achieve. At this point D needs unity and rallying under a shared vision, not balkanization in several special interest groups. There are so many great things you can do starting literally today to make things better for everybody using D, and why in the world would anyone want to stop you. I strongly believe your participation to the Standard D language and library will be vastly better for you personally and professionally. Andrei
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 14:37:55 UTC, Radu wrote: On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko wrote: Hi, Who is interested in betterC _modular_* standard library? I am planing to make libmir org a community for it. Thought and concerns? We already have better `cpuid` and better `random` packages. The betterC std.range and std.algorithm analogs would be released with new ndslice implementation. Mir's algorithm would be faster then Phobos and will generate less template bloat. Then lightweight threads, multithread GLAS, matrix inversion. Fastest I/O and http2 ... (they all are betterC modular libraries) Andrei and probably 90% of existing D users don't want Phobos to move this direction. In other hand I need a commercial attractive D infrastructure for large and heavy system projects. There is no commercial perspective for me to contribute to Phobos because: 1. Phobos version depends on compiler version. Delay with LDC release is too large. It should / can be one day. 2. We can not do hot fixes in Phobos without compiling and deploying patched libphobos. Mir's DUB package are much more flexible. 3. Dependencies should be clear. Modularity is a proper way for large std library. In phobos everything integrated with everything. DRuntime -> Phobos abstraction is weird for betterC because system modules can depends universal algorithms, but universal algorithm are more portable if they have not system dependencies. 4. Phobos does not provide and will not provide betterC guaranties. If something works for betterC mode now it may not work in the future release. [1] https://github.com/libmir/ * - separate github/DUB libraries with clear dependencies. Best regards, Ilya I hope this doesn't turn into a new Tango vs. Phobos war. If you have ideas on how to improve things without messing the entire community with ego wars and such, then it sounds good. Otherwise thumbs down! Maybe I will be able to propose them on DConf. The first sign of faith will be that your modules will always work with druntime and phobos - i.e. you wont require a special runtime for your stuff to work. Yes/No in the same time. 1. It would be always possible to use Mir and Phobos together, 100% guarantee. Furthermore, some Mir libraries are going to be infrastructure for _all_, not only for D. For example Mir GLAS will target the Julia language after D. 2. In the same time many thing is not possible to create with Phobos/Druntime. The simplest example is numeric libraries. I can create a multithread GLAS with core.thread, but if I do so GLAS will never be accepted as default BLAS implementation for Julia. So, i need to create my own low level thread library. Intel MKL, OpenBLAS have their own thread management: this is common practice. Cheers, Ilya
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On Sunday, 18 December 2016 at 13:31:48 UTC, Andrei Alexandrescu wrote: On 12/17/16 10:21 PM, pineapple wrote: I am developing a general-use library for D that is currently resting at around 50,000 lines. Is the source code publicly available? https://github.com/pineapplemachine/mach.d
Re: Enabling data-oriented design
On Sunday, 18 December 2016 at 14:22:29 UTC, Joakim wrote: I was looking at the most popular videos from CppCon and the second-most popular from the last three years is the one on data-oriented design: https://youtube.com/watch?v=1OEu9C51K2A The wikipedia page has a summary, basically focusing on cache effects more, with links to articles: https://en.wikipedia.org/wiki/Data-oriented_design Obviously the effects of cache are well-known here and other places that focus on performance, but maybe there's more we can do to enable this paradigm in the language. I would not be too sure that this is something the language can help with. It is more a question of the right tools. And In theory features like mixins, introspection and property functions should make it possible to profile your memory-access patterns in a nice way.
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko wrote: Hi, Who is interested in betterC _modular_* standard library? I am planing to make libmir org a community for it. Thought and concerns? We already have better `cpuid` and better `random` packages. The betterC std.range and std.algorithm analogs would be released with new ndslice implementation. Mir's algorithm would be faster then Phobos and will generate less template bloat. Then lightweight threads, multithread GLAS, matrix inversion. Fastest I/O and http2 ... (they all are betterC modular libraries) Andrei and probably 90% of existing D users don't want Phobos to move this direction. In other hand I need a commercial attractive D infrastructure for large and heavy system projects. There is no commercial perspective for me to contribute to Phobos because: 1. Phobos version depends on compiler version. Delay with LDC release is too large. It should / can be one day. 2. We can not do hot fixes in Phobos without compiling and deploying patched libphobos. Mir's DUB package are much more flexible. 3. Dependencies should be clear. Modularity is a proper way for large std library. In phobos everything integrated with everything. DRuntime -> Phobos abstraction is weird for betterC because system modules can depends universal algorithms, but universal algorithm are more portable if they have not system dependencies. 4. Phobos does not provide and will not provide betterC guaranties. If something works for betterC mode now it may not work in the future release. [1] https://github.com/libmir/ * - separate github/DUB libraries with clear dependencies. Best regards, Ilya I hope this doesn't turn into a new Tango vs. Phobos war. If you have ideas on how to improve things without messing the entire community with ego wars and such, then it sounds good. Otherwise thumbs down! The first sign of faith will be that your modules will always work with druntime and phobos - i.e. you wont require a special runtime for your stuff to work. As for Andrei and Walter - I think you need to see why people (talented people!) want to side walk the phobos & druntime normal contribution routes. Something clearly is wrong there!
Enabling data-oriented design
I was looking at the most popular videos from CppCon and the second-most popular from the last three years is the one on data-oriented design: https://youtube.com/watch?v=1OEu9C51K2A The wikipedia page has a summary, basically focusing on cache effects more, with links to articles: https://en.wikipedia.org/wiki/Data-oriented_design Obviously the effects of cache are well-known here and other places that focus on performance, but maybe there's more we can do to enable this paradigm in the language.
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On 12/18/16 6:47 AM, Jacob Carlborg wrote: On 2016-12-18 01:34, Andrei Alexandrescu wrote: Yeah, std/datetime.d is a monster, from what I can tell owing to a rote and redundant way of handling unittesting. I didn't look at its dependencies, but I doubt they are special. I was quite vocal about breaking it up, but I got mellower with time since (a) someone measured its size without unittests and it was something like one order of magnitude smaller, and (b) there was really no more trouble using or maintaining it than with anything else in Phobos. Most other languages don't have inline unit tests, which saves a lot of lines of code. How does the lack of the feature save lines? Doesn't it just move them elsewhere? Not sure if this is the case. But if we have unit tests that are more on the functional/integration side perhaps those should be moved to a separate file structure. I should also add that each large project has a couple of outliers like that. I even recall a switch of a couple thousand lines once :o). Just because another project is worse doesn't mean we're in a good position. The point was just that it's par for the course. Rubocop, the major linter in the Ruby world, will complain if a class is more than 100 lines of code. Does that include full documentation and unittests? I think that is on the extreme side but I think any module with more than 2000 lines of code is too big. Phobos' average file size 2055 is lines and median file size is 903 lines. Do you find these appropriate for your preferences, considering this size includes documentation and unittests? Would you find it helpful to use a tool that collapses these? Andrei
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On 12/17/16 10:21 PM, pineapple wrote: On Sunday, 18 December 2016 at 02:40:59 UTC, Chris Wright wrote: D doesn't have either of those pitfalls, so I haven't seen it cause problems. I'm also a bit skeptical that this will see much use outside phobos. This isn't really an argument against it. I just don't see any argument for it, not that's supported by my own experience. I would like to echo this sentiment. I am developing a general-use library for D that is currently resting at around 50,000 lines. Is the source code publicly available? I have never felt a need for a feature like this, and I can't imagine a reason to begin using it. Dependency management has just never presented an issue. Very nearly all modules in the library are fewer than 1,000 lines long and very nearly all symbols are selectively imported, and the approach has proven to be completely manageable. If it can be added without interfering with the existing patterns, I don't really have an argument against this feature. But I do think that what this DIP is meant to address is not really a problem experienced by all or even most who are working with D. I recall Liran Zvibel repeatedly mentioned this as a big pain point at Weka.io. I will reach out to him. On what basis do you extrapolate from personal experience to most or all who are working with D? It's a problem being experienced with Phobos, but there are very valid solutions to that problem that don't involve an addition to the language - only some refactoring. I think that makes the argument in favor somewhat weak. How is Phobos special? What kind of refactoring do you envision would improve dependency management and build times? Thanks, Andrei
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 11:06:58 UTC, Temtaime wrote: Let's write a D, phobos. Then we drop all the features and use a BetterC mode. Then we write all modules against betterC mode. Sounds like a plan! --Ilya
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On Sunday, 18 December 2016 at 11:47:06 UTC, Jacob Carlborg wrote: I think any module with more than 2000 lines of code is too big. newCTFE has more than 2000 lines of code and is still growing, There is no way it could be split up.
Re: DIP10005: Dependency-Carrying Declarations is now available for community feedback
On 2016-12-18 01:34, Andrei Alexandrescu wrote: Yeah, std/datetime.d is a monster, from what I can tell owing to a rote and redundant way of handling unittesting. I didn't look at its dependencies, but I doubt they are special. I was quite vocal about breaking it up, but I got mellower with time since (a) someone measured its size without unittests and it was something like one order of magnitude smaller, and (b) there was really no more trouble using or maintaining it than with anything else in Phobos. Most other languages don't have inline unit tests, which saves a lot of lines of code. Not sure if this is the case. But if we have unit tests that are more on the functional/integration side perhaps those should be moved to a separate file structure. I should also add that each large project has a couple of outliers like that. I even recall a switch of a couple thousand lines once :o). Just because another project is worse doesn't mean we're in a good position. Rubocop, the major linter in the Ruby world, will complain if a class is more than 100 lines of code. I think that is on the extreme side but I think any module with more than 2000 lines of code is too big. -- /Jacob Carlborg
Re: Linux Kernel in D?
On Sunday, 18 December 2016 at 10:51:48 UTC, rikki cattermole wrote: Alternatively help out Wild with his PowerNex project[0]. Pure D port isn't all that exciting, pure D OS that actually tries to do things on its own, now that's something to write home about! [0] https://github.com/Vild/PowerNex If you read the README in the github master, you will notice that it isn't just porting the kernel, it is also porting glibc to D, as well as all the userland applications to D. Essentially making an entire D OS as you say, but using already established designs/technology as a base point. Linux is effective, powerful, widely used, and there are already thousands of programmers that program for Linux, so creating an entire OS in D based on Linux would encourage (hopefully) a lot of Linux programmers to also write for a D Linux. Since its largely familiar, the learning process would be relatively painless.
Re: feature request: nim's compilation option pragmas
On Sunday, 18 December 2016 at 04:46:34 UTC, Timothee Cour wrote:
Could we support something similar to nim's compilation option
pragmas [1]:
```
{.push checks: off.}
# compile this section without runtime checks as it is speed
critical
# ... some code ...
{.pop.} # restore old settings
```
in D could look like:
```
pragma(push, "-noboundscheck -O -release");
// compile this section with specified flags as it's critical
pragma(pop);
```
[1]
http://nim-lang.org/docs/manual.html#pragmas-compilation-option-pragmas
Afaik this Nim feature is inspired from Object Pascal:
http://www.freepascal.org/docs-html/3.0.0/prog/progsu63.html.
Re: A betterC modular standard library?
On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko wrote: Hi, Who is interested in betterC _modular_* standard library? I am planing to make libmir org a community for it. Thought and concerns? We already have better `cpuid` and better `random` packages. The betterC std.range and std.algorithm analogs would be released with new ndslice implementation. Mir's algorithm would be faster then Phobos and will generate less template bloat. Then lightweight threads, multithread GLAS, matrix inversion. Fastest I/O and http2 ... (they all are betterC modular libraries) Andrei and probably 90% of existing D users don't want Phobos to move this direction. In other hand I need a commercial attractive D infrastructure for large and heavy system projects. There is no commercial perspective for me to contribute to Phobos because: 1. Phobos version depends on compiler version. Delay with LDC release is too large. It should / can be one day. 2. We can not do hot fixes in Phobos without compiling and deploying patched libphobos. Mir's DUB package are much more flexible. 3. Dependencies should be clear. Modularity is a proper way for large std library. In phobos everything integrated with everything. DRuntime -> Phobos abstraction is weird for betterC because system modules can depends universal algorithms, but universal algorithm are more portable if they have not system dependencies. 4. Phobos does not provide and will not provide betterC guaranties. If something works for betterC mode now it may not work in the future release. [1] https://github.com/libmir/ * - separate github/DUB libraries with clear dependencies. Best regards, Ilya Let's write a D, phobos. Then we drop all the features and use a BetterC mode. Then we write all modules against betterC mode.
Re: Linux Kernel in D?
On 18/12/2016 11:13 PM, Whatsthisnow wrote: On Sunday, 18 December 2016 at 08:05:20 UTC, Jacques Müller wrote: A Minix port could be interesting as well. The kernel seems to be pretty small. Well if others want to work on porting those kernels they can, we could potentially look into expanding my repo to include all kinds of kernel projects, assuming there is enough interest in them, then we have a solid an entire project collection of potential D kernel solutions. That would be one heck of a boost for D I think. Alternatively help out Wild with his PowerNex project[0]. Pure D port isn't all that exciting, pure D OS that actually tries to do things on its own, now that's something to write home about! [0] https://github.com/Vild/PowerNex
Re: Making preconditions better specified and faster
On Sunday, 18 December 2016 at 09:32:53 UTC, Caspar Kielwein wrote: On Thursday, 15 December 2016 at 18:48:22 UTC, Andrei Alexandrescu wrote: https://issues.dlang.org/show_bug.cgi?id=16975 I'd love if preconditions where available at the caller. This would make it possible to use preconditions for input scrubbing, even in release builds with disabled asserts. That is exactly what they are not for.
Re: Linux Kernel in D?
On Sunday, 18 December 2016 at 08:05:20 UTC, Jacques Müller wrote: A Minix port could be interesting as well. The kernel seems to be pretty small. Well if others want to work on porting those kernels they can, we could potentially look into expanding my repo to include all kinds of kernel projects, assuming there is enough interest in them, then we have a solid an entire project collection of potential D kernel solutions. That would be one heck of a boost for D I think.
Re: Making preconditions better specified and faster
On Thursday, 15 December 2016 at 18:48:22 UTC, Andrei Alexandrescu wrote: https://issues.dlang.org/show_bug.cgi?id=16975 I like it. It's a step in the right direction of making contracts more powerful. I'd love if preconditions where available at the caller. This would make it possible to use preconditions for input scrubbing, even in release builds with disabled asserts.
A betterC modular standard library?
Hi, Who is interested in betterC _modular_* standard library? I am planing to make libmir org a community for it. Thought and concerns? We already have better `cpuid` and better `random` packages. The betterC std.range and std.algorithm analogs would be released with new ndslice implementation. Mir's algorithm would be faster then Phobos and will generate less template bloat. Then lightweight threads, multithread GLAS, matrix inversion. Fastest I/O and http2 ... (they all are betterC modular libraries) Andrei and probably 90% of existing D users don't want Phobos to move this direction. In other hand I need a commercial attractive D infrastructure for large and heavy system projects. There is no commercial perspective for me to contribute to Phobos because: 1. Phobos version depends on compiler version. Delay with LDC release is too large. It should / can be one day. 2. We can not do hot fixes in Phobos without compiling and deploying patched libphobos. Mir's DUB package are much more flexible. 3. Dependencies should be clear. Modularity is a proper way for large std library. In phobos everything integrated with everything. DRuntime -> Phobos abstraction is weird for betterC because system modules can depends universal algorithms, but universal algorithm are more portable if they have not system dependencies. 4. Phobos does not provide and will not provide betterC guaranties. If something works for betterC mode now it may not work in the future release. [1] https://github.com/libmir/ * - separate github/DUB libraries with clear dependencies. Best regards, Ilya
Re: Linux Kernel in D?
A Minix port could be interesting as well. The kernel seems to be pretty small.
