Re: DConf 2013 Day 3 Talk 4: LDC by David Nadlinger
On Monday, 17 June 2013 at 15:19:27 UTC, Andrej Mitrovic wrote: There seems to be some audio glitching every couple of seconds (at the beginning). I've noticed this in other videos as well. It's mostly minimal though, not much harm done. https://www.youtube.com/watch?v=m6jsXQm5IrM#t=106s :-)
Re: DConf 2013 Day 3 Talk 4: LDC by David Nadlinger
... slightly more serious response: really nice talk, David, and thanks for the mention of Dregs. :-)
Re: DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu
On Tuesday, 25 June 2013 at 15:44:02 UTC, Joakim wrote: Just finished watching Andrei's talk, it was up to his usual high standard. I found the bits about professionalism a bit weird though: can we really expect that from a volunteer effort? I'm pretty sure the A/V guys at the conference weren't volunteers, ie they were paid. Along the line that QAston started, if you want more professionalism, is there any interest in producing a commercial D compiler? If not, why not? I notice that Walter sells C and C++ compilers and source on digitalmars.com, but strangely not D. There are interesting business/source models nowadays where you can be mostly open source and still sell a commercial product. For example, Walter has often talked about optimizations in the compiler that he'd like to get to. There could be two compilers: one where the source is fully publicly available, another made available to paying users, which has additional optimizations done either by Walter or others who he supervises, but the source for those optimizations would not be available publicly, though perhaps made available only to the buyers under a non-OSS license. After enough time has passed for the optimization work to be paid for, the optimization patches would eventually be merged into the slower, non-paid version. Android uses a similar hybrid model, which has obviously been enormously successful. Another possibility is a bounty system, where users pledge money towards needed features or bug fixes. It'd basically be a more distributed version of the hybrid approach I've outlined. I wonder what the response would be to injecting some money and commercialism into the D ecosystem. Given how D's whole success stems from its community, I think an "open core" model (even with time-lapse) would be disastrous. It'd be like kicking everyone in the teeth after all the work they put in.
Re: DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu
On Tuesday, 25 June 2013 at 21:38:01 UTC, Joakim wrote: I don't know the views of the key contributors, but I wonder if they would have such a knee-jerk reaction against any paid/closed work. The current situation would seem much more of a kick in the teeth to me: spending time trying to be "professional," as Andrei asks, and producing a viable, stable product used by a million developers, corporate users included, but never receiving any compensation for this great tool you've poured effort into, that your users are presumably often making money with. Obviously I can't speak for the core developers, or even for the community as a group. But I can make the following observations. D's success as a language is _entirely_ down to volunteer effort -- as Walter highlighted in his keynote. Volunteer effort is responsible for the development of the compiler frontend, the runtime, and the standard library. Volunteers have put in the hard work of porting these to other compiler backends. Volunteers have made and reviewed language improvement proposals, and have been vigilant in reporting and resolving bugs. Volunteers also contribute to vibrant discussions on these very forums, providing support and advice to those in need of help. And many of these volunteers have been doing so over the course of years. Now, in trying to drive more funding and professional effort towards D development, do you _really_ think that the right thing to do is to turn around to all those people and say: "Hey guys, after all the work you put in to make D so great, now we're going to build on that, but you'll have to wait 6 months for the extra goodies unless you pay"? How do you think that will affect the motivation of all those volunteers -- the code contributors, the bug reporters, the forum participants? What could you say to the maintainers of GDC or LDC, after all they've done to enable people to use the language, that could justify denying their compilers up-to-date access to the latest features? How would it affect the atmosphere of discussion about language development -- compared to the current friendly, collegial approach? ... and -- how do you think it would affect uptake, if it was announced that access to the best features would come at a price? There are orders of magnitude of difference between uptake of free and non-free services no matter what the domain, and software is one where free (as in freedom and beer) is much more strongly desired than in many other fields. I understand that such a shift from being mostly OSS to having some closed components can be tricky, but that depends on the particular community. I don't think any OSS project has ever become popular without having some sort of commercial model attached to it. C++ would be nowhere without commercial compilers; linux would be unheard of without IBM and Red Hat figuring out a consulting/support model around it; and Android would not have put the linux kernel on hundreds of millions of computing devices without the hybrid model that Google employed, where they provide an open source core, paid for through increased ad revenue from Android devices, and the hardware vendors provide closed hardware drivers and UI skins on top of the OSS core. There's a big difference between introducing commercial models with a greater degree of paid professional work, and introducing closed components. Red Hat is a good example of that -- I can get, legally and for free, a fully functional copy of Red Hat Enterprise Linux without paying a penny. It's just missing the Red Hat name and logos and the support contract. In another email you mentioned Microsoft's revenues from Visual Studio but -- leaving aside for a moment all the moral and strategic concerns of closing things up -- Visual Studio enjoys that success because it's a virtually essential tool for professional development on Microsoft Windows, which still has an effective monopoly on modern desktop computing. Microsoft has the market presence to be able to dictate terms like that -- no one else does. Certainly no upcoming programming language could operate like that! This talk prominently mentioned scaling to a million users and being professional: going commercial is the only way to get there. It's more likely that closing off parts of the offering would limit that uptake, for reasons already given. On the other hand, with more and more organizations coming to use and rely on D, there are plenty of other ways professional development could be brought in. Just to take one example: companies with a mission-critical interest in D have a corresponding interest in their developers giving time to the language itself. How many such companies do you think there need to be before D has a stable of skilled professional developers being paid explicitly to maintain and develop the language? Your citation of the Linux kerne
Re: DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu
On Wednesday, 26 June 2013 at 12:39:05 UTC, Jacob Carlborg wrote: On 2013-06-26 12:16, Leandro Lucarella wrote: Yeah, right, probably Python and Ruby have only 5k users... There are companies backing those languages, at least Ruby, to some extent. They don't own them, though -- they commit resources to them because the language's ongoing development serves their business needs.
Re: DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu
On Wednesday, 26 June 2013 at 15:52:33 UTC, Joakim wrote: I suggest you read my original post more carefully. I have not suggested closing up the entire D toolchain, as you seem to imply. I have suggested working on optimization patches in a closed-source manner and providing two versions of the D compiler: one that is faster, closed, and paid, with these optimization patches, another that is slower, open, and free, without the optimization patches. Over time, the optimization patches are merged back to the free branch, so that the funding from the closed compiler makes even the free compiler faster, but only after some delay so that users who value performance will actually pay for the closed compiler. There can be a hard time limit, say nine months, so that you know any closed patches from nine months back will be opened and applied to the free compiler. I suspect that the money will be good enough so that any bugfixes or features added by the closed developers will be added to the free compiler right away, with no delay. Perhaps you'd like to explain to the maintainers of GDC and LDC why, after all they've done for D, you think it would be acceptable to turn to them and say: "Hey guys, we're going to make improvements and keep them from you for 9 months so we can make money" ... ? Or doesn't the cooperative relationship between the 3 main D compilers mean much to you? Thanks for the work that you and Don have done with Sociomantic. Why do you think more companies don't do this? My point is that if there were money coming in from a paid compiler, Walter could fund even more such work. Leaving aside the moral issues, you might consider that any work paid for by revenues would be offset by a drop in voluntary contributions, including corporate contributors. And sensible companies will avoid "open core" solutions. A few articles worth reading on these factors: http://webmink.com/essays/monetisation/ http://webmink.com/essays/open-core/ http://webmink.com/essays/donating-money/ I think this ignores the decades-long history we have with open source software by now. It is not merely "wanting to make the jump," most volunteers simply do not want to do painful tasks like writing documentation or cannot put as much time into development when no money is coming in. Simply saying "We have to try harder to be professional" seems naive to me. Odd that you talk about ignoring things, because the general trend we've seen in the decades-long history of free software is that the software business seems to getting more and more open with every year. These days there's a strong expectation of free licensing. If I understand your story right, the volunteers need to put a lot of effort into "bootstrapping" the project to be more professional, companies will see this and jump in, then they fund development from then on out? It's possible, but is there any example you have in mind? The languages that go this completely FOSS route tend not to have as much adoption as those with closed implementations, like C++. It's hardly fair to compare languages without also taking into account their relative age. C++ has its large market share substantially due to historical factors -- it was a major "first mover", and until the advent of D, it was arguably the _only_ language that had that combination of power/flexibility and performance. So far as compiler implementations are concerned, I'd say that it was the fact that there were many different implementations that helped C++. On the other hand, proprietary implementations may in some ways have damaged adoption, as before standardization you'd have competing, incompatible proprietary versions which limited the portability of code. And yet the linux kernel ships with many binary blobs, almost all the time. I don't know how they legally do it, considering the GPL, yet it is much more common to run a kernel with binary blobs than a purely FOSS version. The vast majority of linux installs are due to Android and every single one has significant binary blobs and closed-source modifications to the Android source, which is allowed since most of Android is under the more liberal Apache license, with only the linux kernel under the GPL. The binary blobs are nevertheless part of the vanilla kernel, not something "value added" that gets charged for. They're irrelevant to the development model of the kernel -- they are an irritation that's tolerated for practical reasons, rather than a design feature. Again, I don't know how they get away with all the binary drivers in the kernel, perhaps that is a grey area with the GPL. For example, even the most open source Android devices, the Nexus devices sold directly by Google and running stock Android, have many binary blobs: https://developers.google.com/android/nexus/drivers Other than Android, linux is really only popular on servers, where
Re: DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu
On Wednesday, 26 June 2013 at 19:26:37 UTC, Iain Buclaw wrote: I can't be bothered to read all points the both of you have mentioned thus far, but I do hope to add a voice of reason to calm you down. ;) Quick, nurse, the screens! ... or perhaps, "Someone throw a bucket of water over them"? :-P From a licensing perspective, the only part of the source that can be "closed off" is the DMD backend. Any optimisation fixes in the DMD backend does not affect GDC/LDC. To be honest, I can't see the "sales value" of optimization fixes in the DMD backend given that GDC and LDC already have such strong performance. The one strong motivation to use DMD over the other two compilers is (as you describe) access to the bleeding edge of features, but I'd have thought this will stop being an advantage in time as/when the frontend becomes a genuinely "plug-and-play" component. By the way, I hope you didn't feel I was trying to speak on behalf of GDC -- wasn't my intention. :-) Having used closed source languages in the past, I strongly believe that closed languages do not stimulate growth or adoption at all. And where adoption does occur, knowledge is kept within specialised groups. Last year I had the dubious privilege of having to work with MS Visual Basic for a temporary job. What was strikingly different from the various open source languages was that although there was an extensive quantity of documentation available from Microsoft, it was incredibly badly organized, much of it was out of date, and there was no meaningful community support that I could find. I got the job done, but I would surely have had a much easier experience with any of the open source languages out there. Suffice to say that the only reason I used VB in this case was because it was an obligatory part of the work -- I'd never use it by choice. - The development model of D on github has adopted a "pull, review and merge" system, where any changes to the language or compiler do not go in unless it goes through proper coding review and testing (thank's to the wonderful auto-tester). So your suggestion of an "open core" model has a slight fallacy here in that any changes to the closed off compiler would have to go through the same process to be accepted into the open one - and it might even be rejected. I had a similar thought but from a slightly different angle -- that allowing "open core" in the frontend would damage the effectiveness of the review process. How can you restrict certain features to proprietary versions without having also a two-tier hierarchy of reviewers? And would you be able to maintain the broader range of community review if some select, paid few had privileged review access?
Re: DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu
On Wednesday, 26 June 2013 at 19:01:42 UTC, Joakim wrote: Why are they guaranteed such patches? They have advantages because they use different compiler backends. If they think their backends are so great, let them implement their own optimizations and compete. I could respond at greater length, but I think that substantial flaws of your point of view are exposed in this single paragraph. GDC and LDC aren't competitors, they are valuable collaborators.
Re: DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu
On Wednesday, 26 June 2013 at 21:29:12 UTC, Iain Buclaw wrote: Don't call be Shirley... Serious? :-) By the way, I hope you didn't feel I was trying to speak on behalf of GDC -- wasn't my intention. :-) I did, and it hurt. :o) Oh no. 50 shades of #DD ? :-)
Re: DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu
On Thursday, 27 June 2013 at 08:21:12 UTC, Joakim wrote: I'm familiar with its arguments from a summary, not particularly interested in reading the whole thing. You know, I think I see what your problem is ... :-)
Re: DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu
On Saturday, 29 June 2013 at 08:37:48 UTC, Walter Bright wrote: The bottom line was the open source movement was not a very significant force in the 1980's when C++ gained traction. Open source really exploded around 2000, along with the internet. I wonder if open source perhaps needed the internet in order to be viable. That's a very good point. It's before my time really, but if I understand the history right, the main way to get hold of copies of stuff like GCC in the early days was to pay for a set of disks with it on -- and there was no infrastructure for easily sharing changes. So neither the free-as-in-beer or free-as-in-freedom advantages were as readily apparent or effective as they are today.
Re: DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu
On Sunday, 30 June 2013 at 19:45:06 UTC, Joakim wrote: OK, glad to hear that you wouldn't be against it. You'd be surprised how many who use permissive licenses still go nuts when you propose to do exactly what the license allows, ie close up parts of the source. Because people don't just care about the strict legal constraints, but also about the social compact around software. Often people choose permissive licenses because they want to ensure other free software authors can use their software without encountering the licensing incompatibilities that can result from the various forms of copyleft. Closing up their software is rightly seen as an abuse of their goodwill. In other cases there may be a broad community consensus that builds up around a piece of software, that this work should be shared and contributed to as a common good (e.g. X.org). Attempts to close it up violate those social norms and are rightly seen as an attack on that community and the valuable commons they have cultivated. Community anger against legal but antisocial behaviour is hardly limited to software, and is a fairly important mechanism for ensuring that people behave well towards one another. Since you have been so gracious to use such permissive licenses for almost all of D, I'm sure someone will try the closed/paid experiment someday and see if which of us is right. :) Good luck with that :-) By the way, you mentioned a project of your own where you employed the short-term open core model you describe. Want to tell us more about that? Regardless of differences of opinion, it's always good to hear about someone's particular experience with a project.
Re: DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu
On Sunday, 30 June 2013 at 03:29:06 UTC, Walter Bright wrote: On 6/29/2013 5:08 AM, Joseph Rushton Wakeling wrote: True, distribution was mainly by physical mail. There was some via BBS's and Usenet, but these were severely limited by bandwidth. I'd receive bug reports by fax, paper listings, and mailed floppies. This was also the heyday of the BBC Micro in UK schools, and I remember well the shelves full of books of sample programs in BBC Basic. We had lots of fun typing them up, working out how they worked, and then twisting them to our more evil designs.
Re: DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu
On Monday, 1 July 2013 at 21:20:39 UTC, Walter Bright wrote: On 7/1/2013 2:04 PM, Brad Roberts wrote: Actually, Boost was specifically chosen because it didn't require attribution when redistributing. If BSD hadn't had that clause we probably would be using it instead. That was indeed another important reason for it. But we were well aware of and approved of the idea that people could take it and make closed source versions. It was always clear (and logical) to me why the core libraries were permissively licensed, but the no-need-to-give-attribution-for-non-source-distribution feature was a subtlety I hadn't considered before.
Re: DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu
On Monday, 1 July 2013 at 17:45:59 UTC, Joakim wrote: Then they should choose a mixed license like the Mozilla Public License or CDDL, which keeps OSS files open while allowing linking with closed source files within the same application. If they instead chose a license that allows closing all source, one can only assume they're okay with it. In any case, I could care less if they're okay with it or not, I was just surprised that they chose the BSD license and then were mad when someone was thinking about closing it up. The trouble is, even very weak copyleft licenses like MPL and CDDL can result in licensing incompatibilities. Only by granting very permissive licensing terms can you guarantee that your software will be usable by the full range of free software alternatives. For what it's worth, I have also made the argument on many occasions that projects shouldn't pick permissive licenses unless they're happy to see their work turned into proprietary products. But if a developer releases software under a liberal license, saying "I'm doing this so that everyone can use it but please keep it free," I think they have a right to be pissed off when someone ignores their moral request. There's no doubt that even if they chose a permissive license like the MIT or BSD license, these communities work primarily with OSS code and tend to prefer that code be open. I can understand if they then tend to rebuff attempts to keep source from them, purely as a social phenomenon, however irrational it may be. That's why I asked Walter if he had a similar opinion, but he didn't care. Yes, the conscious choice of an extremely permissive license for druntime and Phobos is a different situation. It's completely right in this case to facilitate all forms of development and re-use, under all licensing scenarios. I still think it's ridiculous to put your code under an extremely permissive license and then get mad when people take you up on it, particularly since they never publicly broadcast that they want everything to be open. It is only after you talk to them that you realize that the BSD gang are often as much freetards as the GPL gang, just in their own special way. ;) It's a shame that you feel the need to resort to name-calling because someone has come to a considered moral or strategic position that's different from yours. It also doesn't really help your position -- you're better off just getting on with developing software using your strategy and showing how it serves free software in the long run. I wouldn't call closing source that they legally allowed to be closed antisocial. I'd call their contradictory, angry response to what their license permits antisocial. :) Personally speaking, I find there are a lot of things in life which I prefer to be legally permitted, but still nevertheless consider antisocial -- and I don't think there need be a contradiction there. http://www.phoronix.com/scan.php?page=article&item=sprewell_licensing Note that this article was written when Android had less than 10% of the almost billion users it has today, by using a similar hybrid model, and I was thinking up these ideas years before, long before I'd heard of Android. My project was a small one, so it couldn't be a resounding proof of my time-limited version of the hybrid model, but it worked for its purpose and I'm fairly certain it will be the dominant model someday. :) Thanks for the interesting read. I think you have a point inasmuch as this is a model that clearly works very well from a business perspective where apps are concerned -- and if you're going to have open core, I'd rather it be one where the closed parts are guaranteed to eventually be opened up. Of course, this is not the same as moral approval :-) What I'd say, though, is that what works for apps isn't going to be what works for languages or their core development tools. Most apps seem to be single- or small-team developments, not community projects, they are targeting niche requirements, and ultimately they're being delivered to a target audience that's used to paying for software. On the other hand with a language your overwhelming goal is to grow the user community, and (unless you're Microsoft or Apple, who can dictate terms to software developers) the best way by far to do that is to secure the language quality while keeping the development tools available free of charge. You'll get more mileage out of monetising other things -- e.g. bug-fixing services, support, consultancy -- than you will out of restricting access to tools that enable people to use the language effectively. You also have to consider the user perspective. If I was offered a new language whose tools were delivered with open core terms, I would almost certainly refuse -- I'd feel unable to trust that I wouldn't at some point find all future releases locked up, leaving me with the
Re: Is this D or is it Javascript?
On Saturday, 6 July 2013 at 00:27:45 UTC, Adam D. Ruppe wrote: I really should either finish it or just give in and use a third party service or something (the closest I've come is this newsgroup!), since at least blog blabbing would be *some* documentation for half my random stuff, but just blargh, words cannot describe my hatred for WordPress and friends. I used to think similarly, and spent years starting and never finishing websites using all sorts of different solutions. Finally I realized that, to run a blog, WordPress was actually by far the simplest solution. YMMV of course. I've never felt the need to write my own plugins so never had to deal with the painful side of WordPress.
Re: My first email to Walter, ever
On Sunday, 7 July 2013 at 22:03:17 UTC, Walter Bright wrote: On 7/7/2013 5:09 AM, Andrej Mitrovic wrote: That's a cool teaser, but how did the discussion continue? :) Generally along these lines: "And you, Scarecrow, have the effrontery to ask for a brain, you billowing bale of bovine fodder!" What happened when Toto pulled back the curtain? :-)
Re: A very basic blog about D
On Monday, 8 July 2013 at 17:39:46 UTC, Baz wrote: On Sunday, 7 July 2013 at 15:00:43 UTC, That's interesting...but I'm not a big fan of collecting hundreds of links...I think that someone should create something like http://www.delphifeeds.com/ but for D... blogs.dlang.org ... ? There'd need to be some way of filtering upstreams by topic. I blog about D, but not _just_ about D.
Re: A very basic blog about D
On Monday, 8 July 2013 at 18:54:30 UTC, Baz wrote:
You're wrong, there's a real need for promoting D worldwide.
Did I say otherwise? I am not sure you are reacting to what I
actually wrote.
Just for example, this mainstream (french) programming site has
(had?) a forum for D which is not updated or used at all:
http://www.developpez.net/forums/forumdisplay.php?s=fadd36f8505c59f0714e4e24a0c5a195&f=1180
That's a shame.
Fedora a few years ago has proposed the D language as a part of
their very "extremist open source ashole repository". The fact
is that D is totally missing from their dev packages ("sudo yum
i want some only opensource douche stuff even if I have to type
make make install every two minutes"). And If you setup dmd
manually they'll propose you to setup ldc, which is not
possible due to some broken package dependencies...
I don't understand your visceral hostility here. No one is
excluding D on licensing grounds -- there might be some distros
that would prefer not to include DMD, but they'd be happy to
include GDC and/or LDC.
If D compilers are missing from a distro, or have broken
dependencies, it's because no one is stepping up to take
responsibility for packaging.
Blogs are usefull, D can be used in many editors and compiled
in two portables IDE (Xamarin and Geany) and in another
mainstream win-only-IDE(VS)...
Yes, blogs are useful. I'm still not sure who your argument is
with, though.
Complex networks in D
Following the discussion on digitalmars.D, I've put together a
little (... er, long ...) blog post discussing the basics of my D
graph library:
http://braingam.es/2013/07/complex-networks-in-d/
The main slant of this post is the ease of writing this stuff in
D. Later posts will follow up on performance issues and fill in
some more detail about the workings of the library.
{Enj,Destr}oy :-)
Re: Complex networks in D
On Tuesday, 16 July 2013 at 08:27:07 UTC, Paulo Pinto wrote: On Tuesday, 16 July 2013 at 07:17:59 UTC, Walter Bright wrote: https://news.ycombinator.com/item?id=6050404 http://www.reddit.com/r/programming/comments/1iegj9/complex_networks_in_d/ Thanks! :-)
Re: GHC 2013 in Paris
On Tuesday, 16 July 2013 at 11:02:10 UTC, Iain Buclaw wrote: GHC 2013 Been Haskelling too much recently? :-)
Re: Complex networks in D
On Tuesday, 16 July 2013 at 14:18:02 UTC, bearophile wrote:
size_t vertexCount() @property const pure nothrow
{
assert(_sumHead.length == _sumTail.length);
return _sumHead.length - 1;
}
Is that better written in a struct/class invariant?
Nice thought -- probably; it's a condition that must always hold.
size_t degreeIn(immutable size_t v) const pure nothrow
{
assert(v + 1 < _sumTail.length);
return _sumTail[v + 1] - _sumTail[v];
}
Here you are looking for the method pre-condition.
Ahh, you mean inside in { ... } brackets? I did consider writing
it like that. It wasn't clear to me what the benefits were,
though, especially as I did consider making this an enforce()
rather than an assert().
And "in size_t v" is enough compared to "immutable size_t v".
Does "in" allow for subsequent mutation _within_ the function?
For such kind of code I suggest to use UFCS chains.
Can you explain in a little more detail? It's not an aspect of
programming I'm familiar with.
Also be careful with the performance of such range-based code,
writing benchmarks. Unfortunately often DMD doesn't compile it
efficiently.
Yes, this is a concern of mine too. In benchmarks I've carried
out, the calls to e.g. neighbours() take up a substantial chunk
of the overall runtime -- but that said, the number of calls to
them is very, very large. It works out as on the order of
between 1e-9 and 1e-8 seconds per call.
These kinds of range-based solutions seem to be a part of D where
LDC typically produces the best performance. But I would not use
DMD for serious number crunching of any kind -- as it stands it
can't match either of the other two compilers.
Anyway, thanks very much for the useful feedback :-)
Re: Complex networks in D
On Tuesday, 16 July 2013 at 18:22:31 UTC, Walter Bright wrote: People are much more likely to read your article from links in reddit and hackernews if you put in as a comment some description of it. Don't wait for others to do it for you! They may mischaracterize it, or worse, the opportunity will slip by. Done. Thanks for the advice!
Re: Complex networks in D
On Tuesday, 16 July 2013 at 15:57:06 UTC, bearophile wrote: For such kind of code I suggest to use UFCS chains. Can you explain in a little more detail? It's not an aspect of programming I'm familiar with. auto r1 = iota(_sumHead[v], _sumHead[v + 1]).map!(a => _tail[_indexHead[a]]); auto r2 = iota(_sumTail[v], _sumTail[v + 1]).map!(a => _head[_indexTail[a]]); return chain(r1, r2); Ahh, OK. To be sure I understand what you're getting at, is it just that it's more elegant to write it this way (I agree:-), or is there a performance benefit in the iota().map!() form (or to separately generating the ranges and then chaining them)?
Re: monarch dodra granted write access to phobos, druntime, and tools
On Monday, 22 July 2013 at 18:08:36 UTC, Andrei Alexandrescu wrote: Please join me in congratulating monarch dodra for his admission among our github committers. We're starting with phobos, druntime, and tools access, and if all goes well, we'll extend write rights to dmd also. Congratulations! Well deserved :-)
Re: monarch dodra granted write access to phobos, druntime, and tools
On Tuesday, 23 July 2013 at 19:24:10 UTC, Andrei Alexandrescu wrote: I'm very surprised by your outlook. My perception is that the long queue of pending pull requests not being reviewed is the single most important bottleneck at this point in history in the path of D. By my estimates I think we'd improve the speed of D's development by at least one third if we solve this one issue. There's no other issue offering so much impact. I agree it's the major bottleneck but disagree slightly about the details. My recent experience has been that my Phobos pull requests get _reviewed_ quite quickly but then it may take quite some time to actually get merged. Confusion can be added because the reviewers don't always indicate explicit approval of the code, so the submitter can be sitting in limbo not knowing if the lack of merge is down to the code still being inadequate or just the reviewer not getting round to merging it yet. The latter kind of delay tends to result from the situation where the reviewer is waiting for the test suite to pass. Because there's no option to auto-merge on pass, and no alert to reviewers that a pull request has passed testing, it's easy to miss windows of opportunity to merge. This only has to happen a few times for the pull request to get stuck at the bottom of the test queue and for the delay in merging just to stretch. So, I'd propose that if possible the review process include a way for reviewers to explicitly indicate, "This pull request is provisionally approved subject to testing." Approved pull requests would go on a separate priority test queue with "first in, first out" policy. If the test suite passes, they're auto-merged, if it fails they are removed from the queue and must be re-approved. Ideally it should be possible to distinguish actual test failures from something going wrong with the test procedure itself (e.g. a test process not spawning correctly) and in the latter case keeping the pull request in the approved queue. Does this sound workable/useful?
Re: echo: "-n", the next installment
On Saturday, 27 July 2013 at 12:19:44 UTC, John Colvin wrote: I'm pretty sure it wouldn't work ideally for echo as the behaviour depends on the order of the arguments. It also has some odd little niggles -- e.g. it's not nice that with a short option you can have --t 5 and --t=5 but not -t 5 or -t=5 (according to docs you can only use -t5 ).
Re: New malloc() for win32 that should produce faster DMD's and faster D code that uses malloc()
On Sunday, 4 August 2013 at 06:07:54 UTC, dennis luehring wrote: ever tested nedmalloc (http://www.nedprod.com/programs/portable/nedmalloc/) or other malloc allocators? "Windows 7, Linux 3.x, FreeBSD 8, Mac OS X 10.6 all contain state-of-the-art allocators and no third party allocator is likely to significantly improve on them in real world results." So there may be minimal returns from incorporating nedmalloc on modern OS's ... ?
Re: D reaches 1000 questions on stackoverflow
On Thursday, 15 August 2013 at 00:23:16 UTC, Andrei Alexandrescu wrote: You can define a filter that emails you whenever there are new questions on the "D" tag. Why not set up D.learn (or a new mailing list) to track that filter? That should help prompt the community here to engage with any new questions.
Re: GHC 2013 in Paris
On Tuesday, 20 August 2013 at 07:31:29 UTC, Iain Buclaw wrote: Will be arriving in Paris tomorrow. Is it only you two who will be around? Sad to say I can't make it. :-( Will the talk be videoed?
Re: GHC 2013 in Paris
On Tuesday, 20 August 2013 at 08:35:57 UTC, Iain Buclaw wrote: I don't know, will certainly ask... In any case, have fun (and look out for the Space Invaders...:-)
Updates to D graph library
Hello all, Today I pushed a number of major (and breaking) changes to the master repository of the D graph library. I've provided a brief summary on my blog, which also describes how to revise any programs to work with the new code: http://braingam.es/2013/09/d-graph-library-updates/ I hope that these changes don't unduly inconvenience anyone currently using the library. My own take was that it's worth it in terms of moving to a generic design and offering some new higher-performance graph data structures, which I'll be describing in more detail some time soon. Feedback on the new code is welcome. I would also like to offer thanks to all the nice people on D.learn who contributed useful ideas that helped with this update :-) Thanks & best wishes, -- Joe
Re: [OT] My C++ talk at GoingNative 2013
On Tuesday, 10 September 2013 at 13:08:29 UTC, Iain Buclaw wrote: All class methods are virtual by default in D, unless declared 'final'. There was an intense discussion a while back which ended in (I think) a decision by Walter to switch to final-by-default, but there has so far been no practical follow-up.
Re: Updates to D graph library
On Friday, 13 September 2013 at 07:49:49 UTC, ilya-stromberg wrote: Do you have any plans to change license from GPLv3 to something more liberal like Boost, MIT or BSD? Without this it's impossible to use your library for commercial purposes. The licence is GPLv3+ because the code is closely influenced by igraph, which is GPLv3-licensed. It's not like there's copy-pasting, but it's not clean-room either, so offering a permissive licence might put users in an invidious situation if the igraph authors chose to make an issue of it. Unlikely, but better safe than sorry. If anyone wants to use it in a commercial application the best thing probably to let me know and I can discuss with the igraph authors. I will probably do so anyway once the library is more feature-complete, less out of concern for commercial apps than in order not to be incompatible with other free licenses.
Re: Updates to D graph library
On Friday, 13 September 2013 at 08:45:45 UTC, ilya-stromberg wrote: I see. You can use Boost Graph Library (BGL) as a initial point. It's under Boost license that allows commercial usage. http://www.boost.org/doc/libs/1_54_0/libs/graph/doc/index.html I'm aware of the BGL, but I didn't find it a very nice design. I was also influenced by the fact that among colleagues who work with the various graph libraries, none of them seem to favour it -- whereas igraph seems both popular and high-performing (I believe it can scale to larger data size than any other solution out there). But of course I may revisit that in future. Also, it would be nice to have graph library in Phobos in a future, and in that case module must be under Boost license. Understood, although I never particularly saw Dgraph as being a Phobos candidate, it feels a bit too specialized. My impression is that graph libraries tend in practice to sprawl out into massive constructions, rather too extended for something like a standard library. But if the core functionality is something people are interested in for Phobos, then again, I'm happy to open up the licensing discussion. I'm sure that the igraph authors would be supportive.
Implementing and optimizing a simple graph metric
Hello all,
I thought I'd do a writeup of the process of implementing and
optimizing one of the graph metrics in Dgraph, starting from a
fairly straight copy of pseudo-code in a research paper all
through the various incremental tweaks that improve performance.
http://braingam.es/2013/09/betweenness-centrality-in-dgraph/
I guess the optimizations made in this process are trivial for
anyone experienced in D, but I hope the detailed description of
what changes produce useful speedups might be useful for people
new to the language.
{Enj,Destr}oy :-)
Best wishes,
-- Joe
Re: Implementing and optimizing a simple graph metric
On Wednesday, 18 September 2013 at 13:39:29 UTC, bearophile wrote: Just for a test, try to allocate all those arrays in a different way: - First try a std.array.Array using the LDC2 compiler; - Another thing to try is to allocate them on the stack using core.stdc.stdlib.alloca: auto p = cast(T*)alloca(T.sizeof * g.vertexCount); if (!p) throw new MemoryError... auto centrality = p[0 .. g.vertexCount]; This probably can't be used for very large graphs, so you have to add even more logic to allocate the arrays on the C heap if they are too much large. This could be useful if you call betweenness() many many times. - Try to optionally accept the buffers from outside. Nice suggestions, I'll try those out! :-) I think I did give std.array.Array a trial when trying to speed up its performance, and I don't remember it making any difference (if anything it may have slowed things down). But I'll give it a second look and report back. I haven't yet tried alloca or other manual memory management -- I felt a bit resistant to this as I'd prefer to keep the code simple and readable -- but I'll give that a go too just to see how it goes. It's interesting that you pick up on my use of to!T(0) etc. I had some concern about whether that would affect speed at all. At the time of originally writing the code, my impression was that not having it actually made things worse, and that the compiler was smart enough to carry out the conversion at compile-time. However, my impression now is that having or not having it, or any other method (e.g. enum T0, T1, etc.) makes absolutely no difference, and that I might as well be simple and write 0 for integral-types, 0.0 for floating-point.
Re: Implementing and optimizing a simple graph metric
On Wednesday, 18 September 2013 at 15:22:51 UTC, bearophile wrote: Joseph Rushton Wakeling: I haven't yet tried alloca or other manual memory management -- I felt a bit resistant to this as I'd prefer to keep the code simple and readable -- but I'll give that a go too just to see how it goes. I'd like some stack-allocated variable-length array in D+Phobos, as in C++14. It's also a basis to build several other stack-allocated data structures. Yes, that'd be useful, although in the case of this code I think that stack vs. heap probably isn't that important. If the data is small enough for stack allocation, the calculation will be quick anyway. However, my impression now is that having or not having it, or any other method (e.g. enum T0, T1, etc.) makes absolutely no difference, and that I might as well be simple and write 0 for integral-types, 0.0 for floating-point. Right. And this could be right even if you want T=Rational. Actually, on re-testing this just now, I'm returning to my original view. I find that if you put in raw floating-point numbers like 0.0, 1.0 etc. the resulting code can be slower in the case where T = float. Putting to!T or using enums as you suggest appears to make no difference. This is a guess rather than confirmed by looking at assembly, but I'm presuming that to!T(0) and other conversions of compile-time constants are evaluated at compile time, so you don't in practice carry the cost you'd normally get from using to().
Re: Implementing and optimizing a simple graph metric
On Wednesday, 18 September 2013 at 15:17:25 UTC, Joseph Rushton Wakeling wrote: I think I did give std.array.Array a trial when trying to speed up its performance, and I don't remember it making any difference (if anything it may have slowed things down). But I'll give it a second look and report back. I tried rewriting the variable declarations: Array!T centrality; centrality.length = g.vertexCount; centrality[] = to!T(0); Array!size_t stack; stack.length = g.vertexCount; Array!T sigma; sigma.length = g.vertexCount; Array!T delta; delta.length = g.vertexCount; Array!long d; d.length = g.vertexCount; auto q = VertexQueue(g.vertexCount); Appender!(size_t[])[] p = new Appender!(size_t[])[g.vertexCount]; It results in a moderate slowdown of the code, at a guess because it increases the total number of mallocs/deallocs. I note that there seems to be no way to initialize std.container.Array as having a certain length ... ? I tried instead using std.array.uninitializedArray to initialize the arrays in the betweenness() function -- it was a bit difficult to judge here: with a relatively small number of calls to betweenness() it seems to have resulted in a speedup, but with very many calls, it seems to have been slower.
Re: Implementing and optimizing a simple graph metric
On Wednesday, 18 September 2013 at 13:39:29 UTC, bearophile wrote:
- Try to optionally accept the buffers from outside.
Does this look good to you?
/
auto ref betweenness(T = double, Graph)(ref Graph g, bool[]
ignore = null)
if (isFloatingPoint!T && isGraph!Graph)
{
T[] centrality = new T[g.vertexCount];
return betweenness!(T, Graph)(g, centrality, ignore);
}
auto ref betweenness(T = double, Graph)(ref Graph g, ref T[]
centrality, bool[] ignore = null)
{
centrality.length = g.vertexCount;
centrality[] = to!T(0);
// ... the rest as before
}
/
Re: Implementing and optimizing a simple graph metric
On Wednesday, 18 September 2013 at 17:13:28 UTC, bearophile wrote: How many times or how often do you need to call betweenness()? If it's called only few times or once in a while then using the GC is good enough. But if you have to call it many millions of times, all those GC array allocations could cost some time, even if they are small arrays. In this case allocating them on the stack (or not allocating them, using externally allocated buffers) helps. I think millions of times is excessive. In my test code I have an example with a 50-node network calculating betweenness centrality 1 times, which takes under 1.5 seconds. So obviously if you scale up to millions, small improvements could make a difference, but it's not going to be on such a scale that it's worth prioritizing, at least for now. I think even 1 calculations is excessive for any practical case I can think of. You're right to call for the opportunity to pass a buffer, though, and I've enabled that. In the current test code it doesn't seem to improve anything, but that may be simply a matter of scale.
Re: Implementing and optimizing a simple graph metric
On Wednesday, 18 September 2013 at 13:39:29 UTC, bearophile wrote: auto centrality = minimallyInitializedArray!(typeof(return))(g.vertexCount); centrality[] = T0; auto stack = new size_t[g.vertexCount]; auto sigma = minimallyInitializedArray!T(g.vertexCount); sigma[] = T0; auto delta = minimallyInitializedArray!T(g.vertexCount); delta[] = T0; auto d = minimallyInitializedArray!long(g.vertexCount); d[] = -1; auto q = VertexQueue(g.vertexCount); auto p = new Appender!(size_t[])[g.vertexCount]; As an experiment I tried something along these lines -- using uninitializedArray for most of the arrays here and minimallyInitializedArray for p. It seems to result in a small speed improvement, although I'm not certain about that -- certainly not a speed loss. On the other hand, if inside the VertexQueue implementation, I replace the "new" declaration with an uninitializedArray call, the code gets slower by a noticeable amount. Very odd -- any ideas why that might be?
Re: Implementing and optimizing a simple graph metric
On Tuesday, 24 September 2013 at 22:14:30 UTC, bearophile wrote: minimallyInitializedArray is not stupid, if the specified type has no indirections, it's equivalent to using uninitializedArray, but it's safer if you later change the type. So in general it's not a good idea to use uninitializedArray, unless you have special needs. The two functions are not equivalent, one of them is for normal performance tuning, and the other is for special usages. I have not found this -- using minimallyInitializedArray for the arrays of built-in types is slower than if I use uninitializedArray. These arrays have their values initialized immediately -- this is the actual code from inside the betweenness centrality function: size_t[] stack = uninitializedArray!(size_t[])(g.vertexCount); T[] sigma = uninitializedArray!(T[])(g.vertexCount); T[] delta = uninitializedArray!(T[])(g.vertexCount); long[] d = uninitializedArray!(long[])(g.vertexCount); auto q = VertexQueue(g.vertexCount); Appender!(size_t[])[] p = minimallyInitializedArray!(Appender!(size_t[])[])(g.vertexCount); sigma[] = to!T(0); delta[] = to!T(0); d[] = -1L; ... so I don't see the safety problem here. uninitializedArray is used only for arrays of built-in types, minimallyInitializedArray is used otherwise. On the other hand, if inside the VertexQueue implementation, I replace the "new" declaration with an uninitializedArray call, the code gets slower by a noticeable amount. Very odd -- any ideas why that might be? See above, use uninitializedArray only in special situations and when you know what you are doing. Here you do not know what you are doing, so use minimallyInitializedArray. uninitializedArray or minimallyInitializedArray, using either inside the VertexQueue creates a significant slowdown. uninitializedArray creates random pointers, and aliases that could increase the work done by the GC and cause (temporary if you initialized all your data) memory leaks. Try to totally disable the GC and time the two versions of the code, with and without uninitializedArray. If the GC is the cause of speed differences and you disable it, you will see no performance difference any more. You seem to be suggesting that using uninitializedArray could cause general slowdown, but in general it results in a speedup compared to minimallyInitializedArray. In the one case where it causes a slowdown, minimallyInitializedArray does too, and by a similar amount.
Re: Implementing and optimizing a simple graph metric
On Thursday, 26 September 2013 at 20:56:39 UTC, bearophile wrote:
Joseph Rushton Wakeling:
I have not found this -- using minimallyInitializedArray for
the arrays of built-in types is slower than if I use
uninitializedArray.
Then minimallyInitializedArray should be improved :-)
It's odd, because the two actually use the same underlying
function, with minimallyInitializedArray potentially triggering
this section:
else static if(minimallyInitialized && hasIndirections!E)
{
ret[] = E.init;
}
... but if E is (say) an int, or a size_t, or any other built-in
type, this should not be triggered -- right?
So it's bizarre that there is a performance difference here.
Incidentally, this was why I felt safe using uninitializedArray
for arrays of e.g. long, size_t, etc., because the only
difference at the end of the function could be whether the values
contained in the array have been set or not. As I do ensure that
any array entry used also has its value set, it should be OK.
Re: Implementing and optimizing a simple graph metric
On Thursday, 26 September 2013 at 21:29:42 UTC, bearophile wrote: You also have arrays of T. Someday T could be something with indirections :-) So minimallyInitializedArray is safer regarding future changes in your code. T is qualified via isFloatingPoint :-)
Re: Implementing and optimizing a simple graph metric
On Thursday, 26 September 2013 at 22:03:12 UTC, bearophile wrote: Joseph Rushton Wakeling: T is qualified via isFloatingPoint :-) I know, but that qualification could change in future evolutions of your code. Strong type safety means that if you change a type in your code, with a localized change (like removing isFloatingPoint at the top of your function) the whole code that depends on that type (like the body of this function of yours) will keep working as safely as before :-) OK, I accept your argument :-) As things stand I'm inclined to leave the code using "new" for now. I'll see if I can work out anything about why minimallyInitializedArray might be problematic -- at a guess, perhaps it accidentally gets in the way of some potential optimizations?
Re: Facebook is using D in production starting today
On Friday, 11 October 2013 at 00:36:12 UTC, Andrei Alexandrescu wrote: Today I committed the first 5112 lines of D code to Facebook's repository. The project is in heavy daily use at Facebook. Compared to the original version (written in C++) we've measured massive wins in all of source code size, build speed, and running speed. In all likelihood we'll follow up with a blog post describing the process. Congratulations! This is wonderful news.
Re: Facebook is using D in production starting today
On Friday, 11 October 2013 at 05:11:49 UTC, Walter Bright wrote: It's the first battle signaling the end of Middle Earth, and the rise of the Age of D. The old guard will be sailing to the Grey Havens soon. Hmmm, dodgy metaphor. The departure of the ringbearers heralds the new age, but it's the one in which all the beautiful magical things of Middle Earth will fade away and be lost ... :-P (Can you tell who re-read LOTR recently?:-)
Re: Facebook is using D in production starting today
On Saturday, 12 October 2013 at 12:08:03 UTC, Todor wrote: On Friday, 11 October 2013 at 05:11:49 UTC, Walter Bright wrote: On 10/10/2013 10:05 PM, Nick Sabalausky wrote: Awesome! Great bragging rights for D :) It's the first battle signaling the end of Middle Earth, and the rise of the Age of D. The old guard will be sailing to the Grey Havens soon. They're taking the Hobbits to Isengard! Actually, I think this development is akin to the March of the Ents. They spend a long time thinking and are slow to rouse ... but when they are roused ... :-P https://www.youtube.com/watch?v=h5YwMpSN6CU
Re: LDC 0.12.0 has been released
On Tuesday, 22 October 2013 at 22:42:14 UTC, David Nadlinger wrote: LDC 0.12.0, the LLVM-based D compiler, is available for download! It is built on the 2.063.2 frontend and standard library and supports LLVM 3.1-3.3 (OS X: 3.2 only). Congratulations David and team :-)
Re: "Programming in D" book is about 95% translated
On Saturday, 2 November 2013 at 00:03:51 UTC, Ali Çehreli wrote: I have continued with the translation of the book. There are 36 of the 727 pages still to be translated. (However, I still need to write the UDA chapter.) In addition to many corrections and additions throughout the book, there are the following chapters translated: * Tuples * More Templates * More Functions * Mixins * More Ranges That's fantastic, Ali, thanks and many congratulations! :-) One small linguistic note -- in the "More Ranges" chapter, it may be better to call your example range "Opposite" rather than "Inverse". In a strict mathematical sense you're correct -- swapping sign gives you the additive inverse -- but in a more colloquial sense the term "inverse" is usually in English assumed to be the multiplicative inverse. This is to an extent a matter of taste, of course, so feel free to ignore me.
Re: "Programming in D" book is about 95% translated
On Saturday, 2 November 2013 at 20:36:39 UTC, Andrei Alexandrescu wrote: "Converse"? (Haven't read the section discussed.) Could also work. The range in question wraps an input range r and sets front to return -r.front.
Re: "Programming in D" book is about 95% translated
On Saturday, 2 November 2013 at 22:45:13 UTC, Ali Çehreli wrote: I spent considerable amount of time on those names. Like you, I am not happy with Inverse. :) I wanted to say struct Negate and function negate(). But ! is the negation operator. I like opposite better but the Wikipedia article calls it "additive inverse" so inverse is still acceptable, I guess? :) http://en.wikipedia.org/wiki/Operators_in_C_and_C++ Well, if you don't want to change it now that the chapter has been released, don't worry -- it's not wrong per se, just somewhat unintuitive to my native English ear. I think that Negative and negative() (not Negation) would be fine. Bear in mind that ! is the _logical_ negation operator; without the qualifier, the word doesn't have the same meaning. "The negative of x" is just -x. The trouble with inverse (for me) is that its colloquial meaning is the multiplicative inverse, while in a strict mathematical sense it is too general -- you can have an inverse of _any_ function, so the term always needs to be qualified as the "inverse of ..." (whether of addition, multiplication, some other choice of function). But this may be my fussing too much. ;-)
Re: "Programming in D" book is about 95% translated
On Sunday, 3 November 2013 at 22:42:37 UTC, Tove wrote: I'm not a native English speaker, but FWIW I would have chosen: http://en.wiktionary.org/wiki/numeric_complement I knew there was another term out there somewhere :-)
Re: DConf 2014 Call for Submissions is now open
On Wednesday, 27 November 2013 at 05:44:36 UTC, Jonathan M Davis wrote: And now I have to wrack my brain for ideas. :) I could probably answer questions about D all day, but coming up with something useful to talk about on my own never seems to be as easy as it should be... Well, you have 'til the end of January ... :-) Why not talk about exactly that -- about providing effective on-list guidance, the kind of things that are simple or tricky to deal with, the kind of supporting infrastructure that might be helpful for minimizing the number of repeat questions I'm sure you can extend this list.
Re: DConf 2014 Call for Submissions is now open
On Wednesday, 27 November 2013 at 14:37:11 UTC, Dicebot wrote: I had some until I have started to think about "Credentials: What qualifies you to talk on the topic of choice?". The fact that you are curious about the topic in question, have given it some thought and come up with something interesting to say? :-) It's not all about degrees and job titles.
Re: Range-Based Graph Search in D (blog post)
On Friday, 10 January 2014 at 11:07:13 UTC, John Colvin wrote: Nice :) I presume you are aware of https://github.com/WebDrake/Dgraph Good inspiration for me to get back to work on that :-) @Peter -- this is really exciting to see and I will be looking into your work with great interest. Let me know if there is any overlap in our projects that we can make use of. At first glance (I'm in the midst of house-moves right now so can't give your code the attention it deserves immediately:-) it is interesting to see how our algorithms and problems of interest have resulted in different design concerns. We should definitely try and set up some mutual coding challenges to try and test scalability, performance etc. :-)
Re: Facebook open sources flint, a C++ linter written in D
On Wednesday, 26 February 2014 at 00:57:55 UTC, deadalnix wrote: Usually, I understand XXX evangelist as "My job is to use twitter". Oh, so _that's_ why the text of the Bible comes in individual numbered verses of less than 140 characters each!
1st draft of complete class-based std.random successor
Hello all,
As some of you may already know, monarch_dodra and I have spent
quite a lot of time over the last year discussing the state of
std.random. To cut a long story short, there are significant
problems that arise because the current RNGs are value types
rather than reference types. We had quite a lot of back and
forth on different design ideas, with a lot of helpful input from
others in the community, but at the end of the day there are
really only two broad approaches: create structs that implement
reference semantics internally, or use classes. So, as an
exercise, I decided to create a class-based std.random.
The preliminary (but comprehensive) results of this are now
available here:
https://github.com/WebDrake/std.random2
Besides re-implementing random number generators as classes
rather than structs, the new code splits std.random2 into a
package of several different modules:
* std.random2.generator, pseudo-random number generators;
* std.random2.device, non-deterministic random sources;
* std.random2.distribution, random distributions such as
uniform,
normal, etc.;
* std.random2.adaptor, random "adaptors" such as randomShuffle,
randomSample, etc.
* std.random2.traits, RNG-specific traits such as isUniformRNG
and isSeedable.
A package.d file groups them together so one can still import all
together via "import std.random2". I've also taken the liberty
of following the new guideline to place import statements as
locally as possible; it was striking how easy and clean this made
things, and it should be easy to port that particular change back
to std.random.
The new package implements all of the functions, templates and
range objects from std.random except for the old
std.random.uniformDistribution, whose name I have cannibalized
for better purposes. Some have been updated: the
MersenneTwisterEngine has been tweaked to match the corresponding
code from Boost.Random, and this in turn has allowed the
definition of a 64-bit Mersenne Twister (Mt19937_64) and an
alternative 32-bit one (Mt11213b).
There are also a number of entirely new entries.
std.random2.distribution contains not just existing functions
such as dice and uniform, but also range-based random
distribution classes UniformDistribution, NormalDistribution and
DiscreteDistribution; the last of these is effectively a
range-based version of dice, and is based on Chris Cain's
excellent work here:
https://github.com/D-Programming-Language/phobos/pull/1702
The principal weak point in terms of functionality is
std.random2.device, where the implemented random devices (based
on Posix' /std/random and /std/urandom) are really very primitive
and just there to illustrate the principle. However, since their
API is pretty simple (they're just input ranges with min and max
defined) there should be plenty of opportunity to improve and
extend the internals in future. Advice and patches are welcome
for everything, but particularly here :-)
What's become quite apparent in the course of writing this
package is how much more natural it is for ranges implementing
randomness to be class objects. The basic fact that another
range can store a copy of an RNG internally without creating a
copy-by-value is merely the start: for example, in the case of
the class implementation of RandomSample, we no longer need to
have complications like,
@property auto ref front()
{
assert(!empty);
// The first sample point must be determined here to avoid
// having it always correspond to the first element of the
// input. The rest of the sample points are determined
each
// time we call popFront().
if (_skip == Skip.None)
{
initializeFront();
}
return _input.front;
}
that were necessary to avoid bugs like
https://d.puremagic.com/issues/show_bug.cgi?id=7936; because the
class-based implementation copies by reference, we can just
initialize everything in the constructor. Similarly, issues like
https://d.puremagic.com/issues/show_bug.cgi?id=7067 and
https://d.puremagic.com/issues/show_bug.cgi?id=8247 just vanish.
Obvious caveats about the approach include the fact that classes
need to be new'd, and questions over whether allocation on the
heap might create speed issues. The benchmarks I've run (code
available in the repo) seem to suggest that at least the latter
is not a worry, but these are obviously things that need to be
considered. My own feeling is that ultimately it is a
responsibility of the language to offer nice ways to allocate
classes without necessarily relying on new or the GC.
A few remarks on design and other factors:
* The new range objects have been implemented as final classes
for
speed purposes. However, I tried another approach where the
RNG
class templates were abstract classes, and the individual
parameterizations were
Re: 1st draft of complete class-based std.random successor
On Wednesday, 19 March 2014 at 23:58:36 UTC, Rikki Cattermole wrote: Out of interest but, shouldn't in the device module have a static assert(0, "Not implemented yet") type of deal with the version(Posix) block? Not really. There's still usable functionality in there for all architectures (although I'm not sure how practically useful).
Re: 1st draft of complete class-based std.random successor
On Thursday, 20 March 2014 at 00:09:51 UTC, bearophile wrote: Do you have a simple but very fast function that generates uniforms in [0.0, 1.0]? :-) No, but it's planned. Jerro wrote quite a nice one in the course of his work on the Ziggurat algorithm, and I'm sure he'd be happy for me to adapt it accordingly.
Re: 1st draft of complete class-based std.random successor
On Thursday, 20 March 2014 at 00:05:20 UTC, Joseph Rushton
Wakeling wrote:
Not really. There's still usable functionality in there for
all architectures (although I'm not sure how practically
useful).
Just to expand on that remark: my impression is that individual
random devices are inevitably going to be architecture-dependent.
/dev/random and /dev/urandom are Posix devices; Windows AFAIK
has its own alternative. So the broad idea is that you'd have as
much generic functionality as possible available to all
architectures (mostly related to what sources you read from; a
file, a socket, something else?), and then individual
architecture-dependent aliases would map this to particular
random sources available to them.
Then, finally, you'd have some default alias RandomDevice that
would point to an appropriate architectural default; so e.g.
version (Posix)
{
alias RandomDevice = DevURandom!uint;
}
else version (Windows)
{
alias RandomDevice = ...
}
// etc.
... so, unless you were quite specific about your requirements,
90% of the time you could just use RandomDevice and expect it to
Just Work whatever your platform.
But as random devices are not my strongest area of expertise,
I'll happily take advice here.
Re: 1st draft of complete class-based std.random successor
On Thursday, 20 March 2014 at 00:09:51 UTC, bearophile wrote: Please don't use stuttering names like "std.random2.randomShuffle". "std.random2.shuffle" is enough. I don't object to rewriting the names if there's a valid case for it, but it does seem to me to be important to try and match as much as possible the names that are already out there in std.random. The idea is to minimize the amount of rewriting anyone will have to do to adapt their code, and as far as I can tell where the contents of std.random2.adaptor are concerned (randomShuffle, randomCover, randomSample) it should require no rewriting at all. Besides, while std.random2.adaptor.randomShuffle may be the fully-qualified name, in practice, no one will write all that out, so the redundancy is less bad; and in any case, as any magician will tell you, a shuffle is not necessarily random ;-) I don't think the language is yet there. So I think currently this is not a good idea. If the aim were to overwrite std.random, I would agree with you, but there is no need to do that. It's named std.random2 for a reason :-) However, I do think that merging it into Phobos (assuming all other factors are OK) may have to be conditional on improvements in the available allocation strategies.
Re: 1st draft of complete class-based std.random successor
On Thursday, 20 March 2014 at 00:39:43 UTC, bearophile wrote: Note: I meant a simple but very fast function that generates just one value in [0.0, 1.0] (not a range). There will be both. :-) Off the top of my head I'm not sure whether the interval will be [0.0, 1.0], [0.0, 1.0) or whether it might be possible to make it work with arbitrary boundaries. If I recall right, most uniform01 implementations are [0.0, 1.0) ... ? It's the best chance to improve naming, so do not throw it away for nothing: https://d.puremagic.com/issues/show_bug.cgi?id=9106 If you want you can keep a deprecated randomShuffle alias name for some time in std.random2. Yes, in that case, I'd be happy to make the change (and maintain the old names via aliases). Thanks for pointing me to the bug report; I'd forgotten that this was an open issue :-) We will probably have the nice Andrei's allocators in Phobos, but not in a short time. So I suggest to not rely on them for std.random2. No, I don't intend to rely on them arriving soon. But while of course a random3 is always possible too, I'd rather not be faced with the situation of needing breaking changes to handle support for alternative allocation strategies. So if necessary, I'd rather maintain std.random2 outside of Phobos for a while and get things right when it finally lands, than push it in too early and need to make breaking changes.
Re: 1st draft of complete class-based std.random successor
On Thursday, 20 March 2014 at 01:07:54 UTC, bearophile wrote: In Bugzilla probably there are many bug reports/enhancement requests about std.random, so I suggest you to read them. Some of them can be useful, while other are probably already addressed in the current (or planned) std.random2. Yes, indeed. Quite a few of them _are_ addressed, I think, but now that I've got the essentials of the design laid out, I should be systematic and go through them. Another random one that was just commented by Infiltrator: https://d.puremagic.com/issues/show_bug.cgi?id=5901 Well, you already have the NormalDistribution in std.random2.distribution ;-) I clearly can also implement function-only Box-Muller variant that spends 2 random variates to generate a single normally-distributed value, as this doesn't have the problem of needing to store state or allocate memory, so I will add that at some stage. I'm reluctant to add a specific fastRandom because I think here the better option is a really nice range-based algorithm that can generate high quality variates at speed (e.g. the Ziggurat algorithm is a good candidate here). There's a quite good review of different algorithms here: http://www.cse.cuhk.edu.hk/~phwl/mt/public/archives/papers/grng_acmcs07.pdf But of course I'm open to arguments here :-)
Re: 1st draft of complete class-based std.random successor
On Thursday, 20 March 2014 at 01:32:41 UTC, Chris Williams wrote: Any chance that you could describe them? I was about to resume porting the dcrypt library into Phobos, and had intended to flip the classes into structs, to match what the rest of the library was doing. I think there's a good case for a std.random2.crypto module that contains RNGs that are specifically suitable for cryptography. That said I think the bar here has to be set VERY high, which is why I didn't even begin working on it yet. It has been argued by some that where crypto in Phobos is concerned, we shouldn't take community contributions but we should hire security experts to write the functionality for us. Anyway, let's keep in touch about this and discuss how we could support one another's efforts. About the issues with value-type RNGs (as monarch_dodra says, it's not structs vs. classes per se, as you can implement reference types via structs; it's just more finnicky to do so), probably the best starting point is to read through the various bugs that have been reported as a result of this: https://d.puremagic.com/issues/show_bug.cgi?id=7067 https://d.puremagic.com/issues/show_bug.cgi?id=7936 https://d.puremagic.com/issues/show_bug.cgi?id=8247 https://d.puremagic.com/issues/show_bug.cgi?id=10322 Although some of these are marked as fixed, the fixes are pretty unpleasant and are workarounds rather than solutions of the underlying problem. It may look like only a few issues, but the implications are nasty. We had extensive discussions about this over the last year: http://forum.dlang.org/thread/mailman.259.1357667544.22503.digitalmar...@puremagic.com http://forum.dlang.org/thread/mailman.1017.1370879340.13711.digitalmar...@puremagic.com http://forum.dlang.org/thread/mailman.1157.1371497540.13711.digitalmar...@puremagic.com http://forum.dlang.org/thread/mailman.1209.1371565034.13711.digitalmar...@puremagic.com http://forum.dlang.org/thread/mailman.443.1377369357.1719.digitalmar...@puremagic.com http://forum.dlang.org/thread/5218fd04.8040...@webdrake.net The bottom line is that implementing your RNGs as classes automatically gets you out of the worst of these traps by giving you reference semantics from the get-go. Whether there are other problems that arise from this that make you prefer another design is a question you'll have to answer for yourself -- someone may yet come up with an objection that shows my current design is a Very Bad Idea ;-) Anyway, the example with rndGen.take(10).writeln that monarch_dodra gave is probably the best argument one can make. Imagine a cryptographic application where you're generating (supposedly) two different sets of random data, and because of an unintended value-type copy like this they turn out to be identical. Insecure much? :-)
Re: 1st draft of complete class-based std.random successor
On Thursday, 20 March 2014 at 08:30:09 UTC, ponce wrote: Related: please consider using parts of SimpleRNG the excellent work of John D. Cook which provides many random distributions in a compact and documented way. https://github.com/p0nce/gfm/blob/master/math/gfm/math/simplerng.d (here a port) Good call, I'll take a close look at that. Can you provide me with a link to the original project too? (Yes, I can just Google it, I'm being lazy:-)
Re: 1st draft of complete class-based std.random successor
On Thursday, 20 March 2014 at 08:51:08 UTC, monarch_dodra wrote:
Agreed.
There is consensus it seems. I will make the fix ;-)
I think there is 0 doubt that reference semantics is the way to
go. An advantage of using class is that it is still *possible*
to place them on the stack with Scoped, or with some future
language mechanic. On the other hand, if we implement as
reference structs, then that's that.
I suppose the one concern I have is whether these reference-type
RNGs might generate unpleasant unintended effects with other
range objects in Phobos. One thing that I really must do now
that the basic design is in place is to systematically go through
all the different ways in which these ranges could interact with
deterministic ranges, and whether there are any issues to address.
Furthermore, even in terms of performance, I think a heap
allocated PRNG will still flat-out beat the value based one, if
only because of the size of the damn thing.
I don't know if you or anyone else has run the simple benchmark
programs I created, but my impression was that for the RNGs and
other functions here there is no significant speed difference
between the std.random2 class implementations and their
std.random struct predecessors. Where there _is_ a difference it
seems more likely to be down to algorithm rather than
class/struct or heap/stack.
For example, my new Mersenne Twister is slightly slower, but
probably because it's carrying extra parameters compared to that
of std.random. On the other hand, generating random numbers by
foreach'ing over uniform() calls does not seem to have any speed
difference with popFrontN()'ing over a Uniform Distribution.
That said, being able to allocate them on the malloc heap, and
not the GC heap, would be (IMO) also a valid design.
A simple and dumb design might be to still implement them with
value semantic but:
1. Disable postblit.
2. Make .save() return a "Random*"
This would mean
1. No dangers of accidental copy.
2. Range* is a ForwardRange.
3. Trivially allows GC/malloc/stack allocation.
With good aliases ("alias Random = RadomImpl*;"), and a "make!"
template we could make the "default useage" transparent to this
mechanism yet make it easy to get our hands under the hood.
One strict objection here: .save returning a Random* would mean
that this kind of unittest will fail, no?
auto rng1 = someRandomGenType;
auto rng2 = rng1.save;
rng1.popFrontN(10);
rng2.popFrontN(10);
assert(rng1.front == rng2.front);
More generally, I think that, while I don't object to doing
complicated stuff behind the scenes to get things simple and easy
for the user, the problem I have with the above is that it really
seems to require so much effort to create something which comes
naturally with the current std.random2 design.
I didn't check the code yet, but a "middle ground" could be to
make all constructors private, and disable T.init. Then, we
force construction through a make! template.
This might not be what's most convenient, but it would allow us
to potentially change the design at a later stage, without
breaking user code.
The idea of making constructors private and forcing the user to
use the convenience functions is a very interesting one. As long
as they provide an adequate interface to completely control all
implementation parameters, it could provide a way to have
significant leeway in controlling exactly how RNG instances are
initialized.
On the other hand it really feels obnoxious to cut users off from
being able to use objects directly :-(
Do you have a simple but very fast function that generates
uniforms in [0.0, 1.0]? :-)
AFAIK, the allocation issue is only for ranges? "uniform" is
just a function, I don't think it affected by the issue. Even
if you are operating on a "passed range", either ranges are
reference semantics, and you take by value, or they are value
semantic, and you take by ref. Either way, you have to pay for
the indirection.
I think the issue here is just that it's possible to implement a
really fast high-quality algorithm for uniformly-distributed
floating point numbers in [0, 1). That has all sorts of uses not
just for Phobos users but also internally in e.g. random
distributions (for example, it'll give a significant speed boost
to NormalDistribution).
Re: 1st draft of complete class-based std.random successor
On Thursday, 20 March 2014 at 21:42:13 UTC, Chris Williams wrote: To be certain that the implementation doesn't have any security holes? Yes. Of course, in the current climate one might fear that they'd be the ones introducing them ... :-)
Re: 1st draft of complete class-based std.random successor
On Thursday, 20 March 2014 at 18:43:49 UTC, Chris Williams wrote:
That's only completely true if structs are referred to by
pointer. ref parameters/returns aren't quite sufficient to keep
a struct acting as a reference for all purposes.
As far as I can tell, you're thinking of _passing_ struct
parameters, and here, indeed, passing by ref is sufficient.
The problem comes when you want to _store_ them. It's not safe
to just store a pointer, because the (value type) struct that's
being pointed to might go out of scope and be deleted.
However, you can make structs behave like reference types behave
like reference types, simply by making them contain (safe)
references to the actual data they contain. E.g. (stupidly
simple example):
struct Foo
{
private:
int *_a;
public:
this(int val)
{
_a = new int;
*_a = val;
}
ref int a() @property
{
return *_a;
}
}
unittest
{
auto foo1 = Foo(23);
auto foo2 = foo1;
foo2.a = 4;
writeln(foo1.a);
}
Most of the discussion over RNGs in the last year is about
whether we need to take a (more sophisticated) variant of this
kind of approach to implement reference-type semantics for RNGs,
or whether we should do something different. std.random2 is ...
something different ;-)
Re: 1st draft of complete class-based std.random successor
On Thursday, 20 March 2014 at 00:39:43 UTC, bearophile wrote: It's the best chance to improve naming, so do not throw it away for nothing: https://d.puremagic.com/issues/show_bug.cgi?id=9106 I think the following patch should fix that for you: https://github.com/WebDrake/std.random2/commit/fb5429de77b3c1f7fe3968fd0bd92209c9021f31 I've also made shuffle composable as per your request. Looks good? :-)
Re: 1st draft of complete class-based std.random successor
Latest patches rename randomSample => sample, again offering a documented alias to assist migration. It would be nice to complete the set and eliminate randomCover, but in this case "cover" seems too vague a name to use. Any suggestions for alternatives? I wasn't able to readily find an equivalent in other random number libraries. "permutation" seems good to me (or "permute"?), but perhaps others have suggestions or can point to a typical naming practice from other languages?
Re: 1st draft of complete class-based std.random successor
On Saturday, 22 March 2014 at 20:09:00 UTC, bearophile wrote: Perhaps it's better to not document this alias. For now it will be documented, for clarity if nothing else. Whether that documentation makes it into a Phobos submission, I think should depend on formal review. I'd like a permutations() range in std.range or std.combinatorics. permutation() sounds a bit too much close, despite it is inside another module. How does your desired concept relate to the existing std.algorithm.nextPermutation ... ?
Re: 1st draft of complete class-based std.random successor
Latest patches just pushed to repo make the randomSample => sample change and introduce a fast uniform01 and uniform01Distribution :-)
Re: 1st draft of complete class-based std.random successor
On Saturday, 22 March 2014 at 23:56:35 UTC, bearophile wrote: They seem good. Excellent! There may need to be some attention to the internals of uniform01. Its correctness depends on whether one can always trust a float-based RNG to return values in [min, max) or whether [min, max] is also going to be supplied by some. More ideas: "Three suggestions for std.random": https://d.puremagic.com/issues/show_bug.cgi?id=4851 I think all std.random functions now support a default RNG. There were some bugs related to that (e.g. the "can't use Xorshift" one) that I fixed last year. The problem you identify with, int r = randomCover(data, rndGen).front; always returning the same value, is down to the fact that rndGen is being copied inside the RandomCover struct by value, so of course the original rndGen is never updated and each of these calls will produce the same result. The new std.random2 fixes that, because the RNGs are reference types. However, I'd have thought that int r = data.sample(1, rndGen).front; would have been a more efficient way to implement "choice", as it can operate on any input range, as long as it has the .length property; and it ought to be _much_ faster than even a single call to randomCover. One could always use this as a default option, with a specialization where data is a RandomAccessRange to use the more efficient int r = data[uniform!"[)"(0, data.length)]; "Strongly pure random generator": https://d.puremagic.com/issues/show_bug.cgi?id=5249 .front and .popFront at least are pure for _all_ the RNGs currently implemented in std.random2.generator. See e.g.: https://github.com/WebDrake/std.random2/blob/master/std/random2/generator.d#L266-L272 https://github.com/WebDrake/std.random2/blob/master/std/random2/generator.d#L506-L517 https://github.com/WebDrake/std.random2/blob/master/std/random2/generator.d#L821-L834 Of course this is not strongly pure in line with your request, but it should enable use of these RNGs in many other scenarios where purity is important. I hope a gaussian (normal distribution) generator is planned or present. https://github.com/WebDrake/std.random2/blob/master/std/random2/distribution.d#L326 This is a range implementation; there will also be a function implementation, which will probably follow the inefficient Box-Muller variant that uses 2 uniform random variates to generate a single normal variate (as per the example you posted in your feature request).
Re: 1st draft of complete class-based std.random successor
On Sunday, 23 March 2014 at 10:15:32 UTC, bearophile wrote: Is the issue is already fixed in std.random you can close it :-) Well, your request for a "choice" method is still open ... :-) The best thing is to add an efficient choice() function, so no efficiency mistake happens :-) Sure, I'm simply raising a couple of simple internal implementations that could be used for an effective first draft of that function. So you are saying that those RNGs are already weakly pure and they can't become strongly pure because they take the engine as mutable class reference. And even if you design a very small random engine that can be created every time you call a random generator, the API of all the random functions is not compatible with it. So it's not a simple problem... I think I need to make some detailed research into how Haskell and other functional languages handle randomness before commenting here. What it does seem to me at this stage is that while a weakly pure range-based RNG is readily possible (as implemented in std.random2.generator now), I'm not sure that the range-based approach typical of Phobos plays nicely with strong purity where random number generation is concerned. A possibility is to also add a less precise (more approximate) but faster function implementation. Again, this is something I'll look into. I need to re-read the paper on gaussian-distribution algorithms that I linked to earlier in this thread, but my recollection is that the speed/precision tradeoff is something of a false dichotomy given the algorithms out there now; so a good range-based solution (which stores internal state) will probably be able to provide high-quality normal variates faster than a low-quality, purely function-based implementation. Are the ddocs produced by std.random2 online somewhere? Not yet. I can make that happen :-)
Re: 1st draft of complete class-based std.random successor
On Thursday, 20 March 2014 at 00:09:51 UTC, bearophile wrote: Do you have a simple but very fast function that generates uniforms in [0.0, 1.0]? :-) On that note: https://github.com/D-Programming-Language/phobos/pull/2050 Hope you don't mind me jumping ahead of your existing PR on this -- it's been inactive so I didn't know if you were planning on following up. I'd be very happy to see you take what's good from the above into your own patchset, we need to land a contribution from you in Phobos :-)
Re: 1st draft of complete class-based std.random successor
On Tuesday, 25 March 2014 at 00:08:27 UTC, bearophile wrote: I don't mind, I am happy :-) Thank you for adding a sorely needed function. You are very kind, and far too modest. :-)
Re: 1st draft of complete class-based std.random successor
On Tuesday, 25 March 2014 at 00:08:27 UTC, bearophile wrote: I don't mind, I am happy :-) Thank you for adding a sorely needed function. It's been merged :-)
Re: New newsgroup: digitalmars.D.ldc
On Sunday, 16 December 2012 at 10:27:05 UTC, David Nadlinger wrote: For everyone wondering why it does not show up in the web interface (forum.dlang.org) yet What about mailing-list access? I see nothing on lists.puremagic.com.
Re: DConf 2013 Day 1 Talk 5: Using D Alongside a Game Engine by Manu Evans
Really exciting to see how D is being used in this context, and the C++ binding examples are inspiring. Just one niggle -- the word "postmortem" makes it sound like a study of how something died, whereas this seems to be work that is very much alive and kicking!
Re: DConf 2013 Day 1 Talk 5: Using D Alongside a Game Engine by Manu Evans
On Friday, 17 May 2013 at 22:52:19 UTC, Adam D. Ruppe wrote: The last question in the video was if it would work in Linux too, and idk about the rest of their setup, but the C++ binding is something I've done before, and the code is virtually identical to what Manu did. That's good to know, as Linux is my platform too and (though I've avoided it 'til now) I am always concerned that at some point I'll have to bind with C++.
Re: DConf 2013 Day 1 Talk 7: Panel with Walter Bright and Andrei Alexandrescu
On Wednesday, 22 May 2013 at 13:44:10 UTC, Dicebot wrote: Eh, official definition of "breaking change" keeps breaking my heart. But I guess this is a mindset set in stone now and changing it is close to impossible. Can you elaborate a little bit? I felt personally that what the discussion didn't really address was that these days the "breaking" problems are primarily not with the language, but with the standard library, and this is not much less problematic. Yes, it was mentioned, but not really considered in detail, and that's where a significant amount of attention needs to be. You can extend this to spec -- whereas D as a language is substantially well defined, the standard library doesn't seem to be, and there are clearly areas where this needs to be addressed. Just as one example, which strongly affects my own use of and engagement with D: the de facto behaviour of random number generators, which are implemented as value types despite there being strong reasons why they should instead be reference types. I understand that resolving the last major language spec issues probably needs to be first priority, but I don't think this is the major _usability_ issue any more.
Re: DConf 2013 Day 2 Talk 1: GDC by Iain Buclaw
Wonderful talk, Iain. :-) One question about the copyright assignment issue. How does this operate in practice? Is it going to be that the D frontend will simply go forward as copyright (c) FSF (which isn't a problem DMD-wise as their assignment agreement immediately licenses the code back under completely unrestricted terms), or will it operate on a per-release basis (where basically every time a new D frontend release is pushed to GCC, the copyright is also transferred)?
Re: Five Projects Selected for SAOC 2019
On Monday, 26 August 2019 at 03:55:43 UTC, Andrej Mitrovic wrote: On Sunday, 25 August 2019 at 13:38:24 UTC, Mike Parker wrote: ... Solve Dependency Hell: This is considered as a crucial first step in making Phobos available via the DUB registry I'm guessing this means we might even be able to use multiple versions of Phobos one day. However before we do that, we will really need to fix the use of globals in Phobos. Honestly, I'm a bit leery about this use-multiple-versions approach. It feels like it carries some serious risks of code bloat, which may be accentuated by the fact that DUB has some serious dependency resolution bugs even with single version dependencies... :-\
Re: D at 20: Hits and Misses, and what I learned along the way Oct 19
On Tuesday, 24 September 2019 at 23:27:44 UTC, Walter Bright wrote: On 9/23/2019 3:01 PM, H. S. Teoh wrote: On Mon, Sep 23, 2019 at 02:55:00PM -0700, Walter Bright via There should be redundant, decoupled camera/operator crew to solve this problem. ;-) I know. The same thing happened at DConf 2018, where the first morning's sessions were all lost. Going by the lessons of DConf 2015, I reckon we should always have at least one person using their laptop to stream to YouTube ;-)
Re: Blog Post: Beating std::visit Without Really Trying
Speaking of performance, I was intrigued by the Reddit response noting that Rust can go one better by eliminating the error path at compile time: https://www.reddit.com/r/programming/comments/ddi5wb/comment/f2iow4u The commenter suggests that's because Rust bakes the functionality into the compiler instead of making it a library type. @Walter, @Andrei, that might be an interesting data point for the question of what should have compiler vs. library support?
Re: Blog Post: Beating std::visit Without Really Trying
On Sunday, 6 October 2019 at 03:47:25 UTC, Seb wrote: My earlier post tried to point out that SumType is an excellent candidate for v2. Sorry, Seb, but I don't get this. There's no reason to wait for a v2 to introduce a new SumType symbol that outperforms the old Variant (assuming it's not possible to just rewrite the latter implementation under the hood without changing behaviour). On the contrary, the best way to do a v2 is likely to be that all or most of the stuff we want in it is already there and has been battle tested in the wild.
Re: Blog Post: Beating std::visit Without Really Trying
On Sunday, 6 October 2019 at 08:27:35 UTC, Seb wrote: Well, my guess it will be similar [...] If you're not the one making those decisions it may be better not to prejudge them. A significant performance improvement is a different beast to moderate API/usability improvements. A standard library is _not_ supposed to be a place where actual battle-testing happens. It's where things move when they have been tested and no longer change. You misunderstand what I mean by "battle-testing". Clearly designs should go through a high level of testing and usage before they go anywhere near the standard library. But the very fact of being placed in the standard library exposes them to orders of magnitude more usage, and hence gives a much stronger guarantee of establishing their correctness (or identifying their flaws). It's much better to get newer and apparently better designs exposed to this level of scrutiny _before_ making them the only option in a new major release. That way you are much less likely to get hit by a showstopper edge case that no one anticipated.
Re: Blog Post: Beating std::visit Without Really Trying
On Sunday, 6 October 2019 at 14:08:07 UTC, Adam D. Ruppe wrote: D can eliminate error paths at compile time too, e.g. static assert - which can be used to create all kinds of new useful errors. So I am guessing this is just a case of the code needing a lil tweak or the compiler being conservative and putting the code in even though it is never supposed to happen (like final switch keeps an error path because you can do like cast(some_enum) value_not_in_enum and then better to have the assertion failure than UB.) Good to hear. I confess I was a bit mystified about why it should be an issue for D or why compiler vs. library implementation should make a difference to the ability to eliminate the error path (I infer from your remarks that it shouldn't, in principle). I'm not fluent in assembly so, leaving the error path aside, I wasn't sure how to interpret the "main path" assembly from the D and Rust code, and whether they were practically equivalent (clearly the assembly posted looked different). Can anyone offer any interpretation there?
Re: Blog Post: Beating std::visit Without Really Trying
On Monday, 7 October 2019 at 01:38:04 UTC, Paul Backus wrote: Just to clarify: SumType isn't, and was never intended to be, a drop-in replacement for Algebraic. Their interfaces are similar enough that porting code from Algebraic to SumType shouldn't be too difficult, but even within the common subset, there are incompatibilities. For example, here's a post from an old sumtype announcement thread where I discuss the differences between SumType's `match` and Algebraic's `visit`: https://forum.dlang.org/post/xipgrrfljdnyhrhnm...@forum.dlang.org I don't follow. The visit and match template signatures are identical, and from what you describe the SumType match implementation should support the same handler choices as visit, while allowing extra/more flexible choices. Where's the breaking change? Are there selections of handlers that work for visit and don't work with match?
Re: dud: A dub replacement
On Monday, 11 November 2019 at 13:44:28 UTC, Robert Schadek wrote: So dub has some problems, and personally I find its code base very hard to get into. At Symmetry we are a very heavy user of dub, resulting in many wasted hours. So I started to write dud [1]. I kept some boring/nice parts from dub, but most code so far is a complete rewrite. The goal of dud is mostly do what dub does, but more understandable. Cool :-) Since I have also been experiencing a fair bit of production-use DUB pain in the last year, I really appreciate your taking action on this. A few things that would be good to understand up front: * what are the particular issues with DUB that you want to address? - making the codebase cleaner and more functional is obviously nice but is at most a means to an end -- what's the real end you have in mind? - I would imagine getting dependency resolution really right would be top of the list -- it would be good to aim to fix issues like https://github.com/dlang/dub/issues/1732 - I would personally really appreciate it if you would make it a design goal to better separate concerns in terms of what DFLAGS are used and why (for example, the fact that right now `dub test --build=release` will not actually run unittests, as `--build=release` drops the `-unittest` flag) * are there any particular known issues with DUB that this definitely will NOT address? * are there any key _breaking_ changes you have in mind? * where does this stand w.r.t. some of the proposals to break DUB apart into more cleanly separated components, e.g. determining compatible dependencies, downloading dependencies, building or running tests ... ? Some concrete feedback on the project as it stands: * the tickboxes of compatible commands are nice, but it would be good to have a more contextualized roadmap, in particular outlining the design concerns for core data structures and functionality - this should probably be in issues rather than the README, but it needs to be somewhere, otherwise it's hard for anyone outside to identify things they can do - it might be nice to use a GitHub project to manage things so that us outside folks can identify better what's being worked on and what's blocked by what * I don't mind breaking changes in either the config format or the command line API if it gets us to a nicer tool, so please embrace the opportunity where appropriate - I can imagine this might be the reason why the aim is to support a "tasteful subset" of DUB's features: it means that others can be re-implemented in an incompatible but better way - auto-conversion mechanisms may be preferable to outright support for old formats and commands * I really recommend trying to start writing clean, clear commit messages from the very start -- think of this as another form of code documentation that communicates to all interested readers the intent and considerations behind any particular change to the codebase. This makes it much easier for outsiders to get a clear understanding of the project and get into the habit of contribution - right now, pretty much all the commit messages read like spontaneous notes where even YOU won't remember the whys or wherefores in a few months' time - long term it saves a LOT of time when trying to look back and understand "Why did we do things that way?" -- particularly useful when trying to fix some subtle design bug * for the same reasons, really try to provide good documentation and comments for all code from the start -- this really makes it much easier for anyone interested to grasp the major design concerns and get contributing * these concerns are going to be strongest for the key data structures and algorithms -- please make sure these are REALLY documented well, from the very start Hope all of that's helpful, and best wishes for taking this forward -- I will try to help as I can, but time right now is very constrained ... ;-) Thanks & best wishes, -- Joe
Re: dud: A dub replacement
On Monday, 18 November 2019 at 13:19:12 UTC, Paolo Invernizzi wrote: Closing this kind of discussions and letting anyone to choose "tabs or spaces" is a constructive solution, I think. It is quite extraordinary how readily folks fall to arguing over what the config format should be, rather than what the app should actually be able to do. :-\
Re: dud: A dub replacement
On Monday, 18 November 2019 at 19:44:46 UTC, Russel Winder wrote: On Mon, 2019-11-18 at 15:35 +, Joseph Rushton Wakeling via Digitalmars-d- announce wrote: […] It is quite extraordinary how readily folks fall to arguing over what the config format should be, rather than what the app should actually be able to do. :-\ Perhaps because writing the configuration files is a critical part of the usability of the tool. Fair. My remark was maybe a little too intemperate :-) I'm simply concerned that if we don't put enough scrutiny on the app features and behaviour, we run the risk of simply reproducing some of the problematic design decisions of the existing tool. As an example -- try running `dub test --build=release`. Intuition suggests that this is testing a release build. But actually it strips out your unittests, because `--build=release` determines not only the optimization flags, but the complete set of DFLAGS used -- and `-unittest` isn't among them. I ran `dub test && dub test --build=release` as a matter of habit for some time before discovering the latter wasn't actually testing anything. And while I now know that I _can_ define a custom build to get what I want, that's clumsy and hard(er) to discover, and annoying to have to repeat for every project I create. That's the kind of usability and design concern we really ought to be revisiting in any rewrite.
Re: dud: A dub replacement
On Monday, 18 November 2019 at 20:48:53 UTC, bachmeier wrote: IMO this is one of the most important parts of the first five minutes with the language. Someone has installed the compiler, and now they want to test it out. If they have a bad experience with Dub, they will not continue with the language. A package manager, including the choice of format, is something you have to get right. Rust understands this. Fair point. But that isn't something that has to be decided at the _start_ of a rewrite: better to focus on wanted behaviour, and then derive the ideal config format from that.
Re: dud: A dub replacement
On Monday, 18 November 2019 at 16:31:09 UTC, Nick Sabalausky (Abscissa) wrote: As has been discussed elsewhere a few months ago, dependency resolution should be outsourced to an established SAT I have no objections to that in principle, but I am not sure that will resolve the issue I posted: IIRC DUB's key problem is not so much how it resolves dependencies _per se_ but the fact that it includes dependency constraints that actually don't apply to a given build (e.g. constraints that derive from the testing requirements of dependencies).
Re: dud: A dub replacement
On Wednesday, 20 November 2019 at 11:40:19 UTC, Robert Schadek wrote: Here is disagree, to a degree I consider comments a code smell. If I have to write them, I failed to convey the information needed to understand the code in the code. That depends on what you're using documentation and comments for. It's obviously great to try to write code that is as clean and clear and comprehensible as possible. But in general even the cleanest and clearest code rarely communicates the WHY behind design decisions. "This design was chosen because ..." "These are the assumptions made ..." etc. That WHY is what you want to clearly document, because that's what saves the most time for anyone looking to understand and modify the codebase.
Re: dud: A dub replacement
On Thursday, 21 November 2019 at 14:50:59 UTC, Steven Schveighoffer wrote: Assuming the code you wrote does what you wanted it to do... Often times, comments convey what you're thinking, and it's much easier to understand a description than mentally compiling and running the code to figure out what you were thinking. Exactly. And of course this isn't just for the benefit of others -- future you often needs the exact same help to understand what past you was thinking ...
Re: Proposal for porting D runtime to WebAssembly
On Saturday, 23 November 2019 at 09:51:13 UTC, Sebastiaan Koppe wrote: This is my proposal for porting D runtime to WebAssembly. I would like to ask you to review it. You can find it here: https://gist.github.com/skoppe/7617ceba6afd67b2e20c6be4f922725d Thanks for putting this together, it looks very carefully thought out. On this particular part: Exceptions can be thrown but not catched. A thrown exception will terminate the program. Exceptions are still in the proposal phase. When the proposal is accepted exceptions can be fully supported. This would suggest that there may be some benefit in D providing improved support for return-type-based error propagation (as with `Result` from Rust), no ... ?
