Re: A brief survey of build tools, focused on D
On Mon, 2018-12-10 at 13:01 -0800, H. S. Teoh via Digitalmars-d-announce wrote: > […] > Wow. Thanks for the writeup that convinces me that I don't need to > waste time looking at Meson/Ninja. […] The article is a personal opinion and that is fine. For me it is wrong. No mention of SCons, nor that Gradle build C++ as well as for the JVM languages. Some of the points about Meson are right, some wrong, but it is a personal opinion and that is fine. I shall continue to use Meson and Ninja because they are way, way better than Autotools (not mentioned but still used a lot) and better than SCons for many use cases. But this is also a personal opinion. -- Russel. === Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: A brief survey of build tools, focused on D
On Mon, Dec 10, 2018 at 10:30 AM Neia Neutuladh via Digitalmars-d-announce wrote: > > I wrote a post about language-agnostic (or, more accurately, cross- > language) build tools, primarily using D as an example and Dub as a > benchmark. > > Spoiler: dub wins in speed, simplicity, dependency management, and > actually working without modifying the tool's source code. > > https://blog.ikeran.org/?p=339 Why isn't premake in the list? It's the only buildtool that works reasonably well with IDE's, and it's had D well supported for almost 6-7 years. It also doesn't depend on a horrible runtime language distro.
Re: A brief survey of build tools, focused on D
On Mon, 10 Dec 2018 13:01:08 -0800, H. S. Teoh wrote: > It also requires network access. On *every* invocation, unless > explicitly turned off. And even then, it performs time-consuming > dependency resolutions on every invocation, which doubles or triples > incremental build times. Again, unacceptable. I feel like those should be configuration options at the very worst. And dub probably shouldn't even bother verifying your dependencies if you haven't changed dub.json. > Then it requires a specific source layout, with incomplete / > non-existent configuration options for alternatives. Which makes it > unusable for existing code bases. Unacceptable. A lot of people do find it acceptable to have a build tool that makes assumptions about your source code layout, but that's certainly not always possible or desirable. > Worst of all, it does not support custom build actions, which is a > requirement for many of my projects. Yeah, there's a lot of neat metaprogramming stuff in D (like pegged) where it's awesome with small projects that it's part of compilation, but when I'm dealing with a nontrivial instance of it, I want to split it into a separate build step. Dub doesn't help me accomplish that. > After so many decades of "advancement", we're still stuck in the > gratuitously incompatible walled gardens, like the gratuitous browser > incompatibilities of the pre-W3C days of the Web. And on modern CPUs > with GHz clock speeds, RAM measured in GBs, and gigabit download speeds, > building Hello World with a system like dub (or Gradle, for that matter) > is still just as slow (if not slower!) as running make back in the 90's > on a 4 *kHz* processor. It's ridiculous. Solving an NP-complete problem every time you build is not a great start. > Why can't modern source code come equipped with dependency information > in a *standard format* that can be understood by *any* build system? Kythe is an attempt to make the relevant information available in a language-agnostic way. Might be a reasonable basis for a standardized build system. No clue how well it works or what it actually supports. https://kythe.io/ > Build systems shouldn't need to reinvent their own gratuitously > incompatible DSL just to express what's fundamentally the same old > decades-worn directed graph. And programmers shouldn't need to repeat > themselves by manually enumerating individual graph edges (like Meson > apparently does). Meson doesn't have you enumerate individual graph edges at that level. It just doesn't build your project correctly. Change a struct size in one file, and you get a host of weird errors when another file uses it. Maven and Gradle also don't really have a DAG like that. If any file changed, your whole project needs to be rebuilt, and all your dependencies are immutable. Bazel has a DAG across build rules, not across individual files. > - Efficient: the amount of work done by the build should be proportional > to the size of changes made to the source code since the last build, > NOT proportional to the size of the entire source tree (SCons fails in > this regard). Would be great if the tool could pay attention to whether incremental builds saved time on average and just do a full build if it's better. > - Language-agnostic: the build system should be essentially a dependency > graph resolver. It should be able to compile (possibly via plugins) > source code of any language using any given compiler, provided such a > combination is at all possible. In fact, at its core, it shouldn't > even have the concept of "compilation" at all; it should be able to > generate, e.g., .png files from POVRay scene description files, run > image post-processing tools on them, then package them into a tarball > and upload it to a remote webserver -- all driven by the same > underlying DAG. You could support rsync just fine, but if it's just an HTTP upload, there's no standard way to tell if the server's got the file already.
Re: A brief survey of build tools, focused on D
On Tue, 11 Dec 2018 02:54:15 +, Mike Franklin wrote: > Why not just write your build/tooling scripts in D? That's what I > prefer to do, and there's been a recent effort to do just that for the > DMD compiler as well: > https://github.com/dlang/dmd/blob/master/src/build.d It still resembles > the makefiles it was modeled from, but in time, I think it will clean up > nicely. That's fine for executables that don't depend on external libraries. It's not good for libraries that I want other people to use; dub's the easiest way to publish a thing. It also means I need to replicate that dependency graph logic in every single project, which is worse than replicating it once per language. We really should have a standard build tool supporting per-language plugins, like H. S. Teoh is recommending.
Re: A brief survey of build tools, focused on D
On Monday, 10 December 2018 at 18:27:48 UTC, Neia Neutuladh wrote: I wrote a post about language-agnostic (or, more accurately, cross- language) build tools, primarily using D as an example and Dub as a benchmark. Spoiler: dub wins in speed, simplicity, dependency management, and actually working without modifying the tool's source code. https://blog.ikeran.org/?p=339 Why not just write your build/tooling scripts in D? That's what I prefer to do, and there's been a recent effort to do just that for the DMD compiler as well: https://github.com/dlang/dmd/blob/master/src/build.d It still resembles the makefiles it was modeled from, but in time, I think it will clean up nicely. Mike
Re: A brief survey of build tools, focused on D
On Mon, 10 Dec 2018 21:53:40 +, GoaLitiuM wrote: > The results for touching second file seems like an anomaly to me, The generated ninja file had one rule per source file. If your modules tend to import each other a lot, or if they transitively import the code that's doing expensive stuff, then one rule per source file is bad. If your modules have few transitive dependencies and they're each fast to compile, one rule per source file is good. My project used Pegged, and a lot of stuff referenced the grammar. That meant incremental builds went long and it would have been better to build the whole project at once. Separating the grammar into a different build would reduce compile times significantly, and that might make incremental builds fast. >From discussions on IRC about reducing compile times, though, using Phobos is a good way to get slow compilation, and I use Phobos. That alone means incremental builds are likely to go long. > You also have to make sure the dependencies are built with the same > compiler, which could explain the headache #3 in your article. I've been using dmd as my primary compiler for ages, cleared out all the cached dub builds I could find, ran `dub build -v` to ensure that it was invoking dmd, and explicitly told Meson to use dmd. Meson was still convinced that I'd built pegged with some other compiler. > The comparison and some of the other headaches with meson does not seem > to be fair as you are comparing dub, which is both a build system and a > package manager, to meson which is only a build system, you have to make > sure all the dependencies are installed to your system beforehand. That *would* be a reasonable objection, but Meson explicitly advertises that you can use dub dependencies. The two flaws are the extra work required and the fact that it's broken. If it had not even pretended to support dub dependencies, I could have avoided several of the problems and just used git submodules from the start. Just like with Bazel.
Re: A brief survey of build tools, focused on D
On Monday, 10 December 2018 at 21:01:08 UTC, H. S. Teoh wrote: [SNIP] Great rant! Do you think dub's current architecture is a lost cause or are there some leverage points where it can greatly improve? Also, do you have any recommendations? Currently I'm using dub because it's the standard, but I wish it were a bit faster and had more flexibility.
Re: A brief survey of build tools, focused on D
I switched away from dub to meson for my small game engine project, and the biggest benefit of this switch was the improved build times while doing small iterations to some files: dub build --arch=x86_64 --build=debug --compiler=dmd - full rebuild: 3960ms - touch file1 and build: 2780ms - touch file2 and build: 2810ms ninja -c build - full rebuild: 10280ms (includes some dependencies like ErupteD and SDL2 bindings) - touch file1 and build: 1410ms - touch file2 and build: 250ms The results for touching second file seems like an anomaly to me, but in practice the incremental build times are around the same with touching the first file, so that is already 2x improvement with incremental build times. If I touch multiple files, ninja can invoke multiple build commands at the same time so the work gets distributed along all the processor cores so the build time does not change a lot from editing one file (which does not seem to reflect in your build timing results for some reason?). But as you mentioned in the article, there are some caveats with this, mainly the lack of dependency graph which may cause some weird bugs in the program if you drastically change one module or work with templates. You also have to make sure the dependencies are built with the same compiler, which could explain the headache #3 in your article. The comparison and some of the other headaches with meson does not seem to be fair as you are comparing dub, which is both a build system and a package manager, to meson which is only a build system, you have to make sure all the dependencies are installed to your system beforehand. While I agree with the headache #1, other headaches are simply due to you not using meson the way it was intended to.
Re: A brief survey of build tools, focused on D
On Monday, 10 December 2018 at 21:01:08 UTC, H. S. Teoh wrote: And almost no build system handles reliable builds correctly when the build description is changed -- Button does, but it's in the extreme minority, and is still a pretty young project that's not widely known). Tup [1] does, and it's pretty reliable on that: I think Botton was inspired by it. Anyway, you are totally right: +1 on all your points! [1] http://gittup.org/tup/ -- Paolo
Re: A brief survey of build tools, focused on D
On Mon, Dec 10, 2018 at 06:27:48PM +, Neia Neutuladh via Digitalmars-d-announce wrote: > I wrote a post about language-agnostic (or, more accurately, cross- > language) build tools, primarily using D as an example and Dub as a > benchmark. > > Spoiler: dub wins in speed, simplicity, dependency management, and > actually working without modifying the tool's source code. > > https://blog.ikeran.org/?p=339 Wow. Thanks for the writeup that convinces me that I don't need to waste time looking at Meson/Ninja. I find the current landscape of build systems pretty dismal. Dub may be simple to use, but speed, seriously?! If *that's* the generally accepted standard of build speed out there these days, then hope is slim. Convenience and simplicity, sure. But speed? I'm sorry to say, I tried dub for 2 days and gave up in frustration because it was making my builds *several times longer* than a custom SCons script. I find that completely unacceptable. It also requires network access. On *every* invocation, unless explicitly turned off. And even then, it performs time-consuming dependency resolutions on every invocation, which doubles or triples incremental build times. Again, unacceptable. Then it requires a specific source layout, with incomplete / non-existent configuration options for alternatives. Which makes it unusable for existing code bases. Unacceptable. Worst of all, it does not support custom build actions, which is a requirement for many of my projects. It does not support polyglot projects. It either does not support explicit control over exact build commands, or any such support is so poorly documented it might as well not exist. This is not only unacceptable, it is a show-stopper. This leaves only package management as the only thing about dub that I could even remotely recommend (and even that is too unconfigurable for my tastes -- basically, it's a matter of "take my way or the highway" -- but I'll give it credit for at least being *usable*, if not very pleasant). But given its limitations, it means many of my projects *cannot* ever be dub projects, because they require multiple language support and/or code generation rules that are not expressible as a dub build. Which means the package management feature is mostly useless as far as my projects are concerned -- if I ever have a dependency that requires code generation and/or multiple languages, dub is out of the question. So I'm back to square one as far as dependency management and build system are concerned. This dismal state of affairs means that if my code ever depends on a dub package (I do have a vibe.d project that does), I have to use dub as a secondary tool -- and even here dub is so inflexible that I could not make coax it work nicely with the rest of my build system. In my vibe.d project I had to resort to creating a dummy empty project in a subdirectory, whose sole purpose is to declare dependency on vibe.d so that I can run dub to download and build vibe.d (and generate a dummy executable that does nothing). Then I have to manually link in the vibe.d build products in my real build system as a separate step. // Taking a step back, this state of affairs is completely ridiculous. The various build systems out there are gratuitously incompatible with each other, and having dependencies that cross build system boundaries is completely unthinkable, even though at its core, it's exactly the same miserable old directed acyclic graph, solved by the same old standard graph algorithms. Why shouldn't we be able to integrate subgraphs of different origins into a single, unified dependency graph, with standard solutions by standard graph algorithms? Why should build systems be effectively walled gardens, with artificial barriers that prevent you from importing a Gradle dependency into a dub project, and importing *that* into an SCons project, for example? After so many decades of "advancement", we're still stuck in the gratuitously incompatible walled gardens, like the gratuitous browser incompatibilities of the pre-W3C days of the Web. And on modern CPUs with GHz clock speeds, RAM measured in GBs, and gigabit download speeds, building Hello World with a system like dub (or Gradle, for that matter) is still just as slow (if not slower!) as running make back in the 90's on a 4 *kHz* processor. It's ridiculous. Why can't modern source code come equipped with dependency information in a *standard format* that can be understood by *any* build system? Build systems shouldn't need to reinvent their own gratuitously incompatible DSL just to express what's fundamentally the same old decades-worn directed graph. And programmers shouldn't need to repeat themselves by manually enumerating individual graph edges (like Meson apparently does). It should be the compilers that generate this information -- RELIABLY -- in a standard format that can be processed by any tool that understands the common format. You should be able to dow
Re: A brief survey of build tools, focused on D
On Monday, 10 December 2018 at 18:27:48 UTC, Neia Neutuladh wrote: I wrote a post about language-agnostic (or, more accurately, cross- language) build tools, primarily using D as an example and Dub as a benchmark. Spoiler: dub wins in speed, simplicity, dependency management, and actually working without modifying the tool's source code. https://blog.ikeran.org/?p=339 Pretty nice post, thanks for sharing this. Maybe it worths to note that dub is also able to retrieve packages from maven repositories. Kind regards Andre
A brief survey of build tools, focused on D
I wrote a post about language-agnostic (or, more accurately, cross- language) build tools, primarily using D as an example and Dub as a benchmark. Spoiler: dub wins in speed, simplicity, dependency management, and actually working without modifying the tool's source code. https://blog.ikeran.org/?p=339
Re: Google Summer of Code 2019
On Sunday, 25 November 2018 at 13:58:25 UTC, Mike Parker wrote: The time has come to start thinking about GSoC 2019. The application deadline for mentoring organizations is on February 6. I'd like to get a solid list of project ideas for potential student applications. I've set up a new page at the Wiki to collect ideas and seeded it with two from the GSoC 2018 page: https://wiki.dlang.org/GSOC_2019_Ideas I invite everyone to add ideas to the list. Please be as descriptive as you can in your summaries, and be explicit about the goals the project should achieve. We want projects that are both necessary and challenging. [snip] The data frames project might mention libmir. It would be nice if anything done on that front builds on ndslices.
Re: OFFTOPIC Re: I've just released Vasaro
On Monday, 10 December 2018 at 10:47:42 UTC, Jacob Carlborg wrote: If you click on the Apple menu in the top left corner and choose "About This Mac", it will say which model and which year in the window that appears. It will also specify which version of the OS it's running. Ah, there it is: 10.9.5, 1.6 GhZ Core i5, 2 GB. (c) 2016. Actually not that old. If you don't want to keep it you could always donate it as a testing machine, if the Dlang foundation will accept it. Well, I still want to add the support to my library anyway. At least the minimal stuff - create window, create opengl context, dispatch events, so it can serve as a base for other users to PR the other functions if they use it. I am also very slowly working on some objc helper functions (though I wish the headers were at least in druntime like win32 is now) That's why I got this and it is still on my list, it is just somewhat far down on my list.
Re: Google Summer of Code 2019
On Monday, 10 December 2018 at 11:22:04 UTC, Francesco Mecca wrote: I can see from the previous GSOC entries in the wiki that there are many projects that are still interesting IMHO. Even my entry is just a rehash of the interest around calypso given that we now have dpp. Why aren't they included in the current GSOC page? I don't want to blindly copy project ideas from the old pages to the new one. I don't know what information is still relevant, any new forums discussions or other links that can provide more background, etc. I encourage anyone who added ideas to the older pages to update them as needed for the new page. If we don't have a good number of ideas before I submit our application, I'll do what I need to do to flesh out the list. Also, shouldn't students propose after the Dlang foundation gets accepted? Students should submit their applications to Google at that time, yes. But the timeline for how we handle our own process is entirely up to us. Before I submit our organization application, I want to have a good idea of how many students are interested, how many mentors are interested, and have as many of them paired up as we can get. That will help us move things along more smoothly.
MORE OFFTOPIC Re: I've just released Vasaro
On Monday, 10 December 2018 at 10:47:42 UTC, Jacob Carlborg wrote:
On 2018-12-08 18:01, Adam D. Ruppe wrote:
The one I have is a macbook air with a broken, but usable
screen (I got it for free yay). I don't know how old it is, I
*think* it is a 2013 model.
If you click on the Apple menu in the top left corner and
choose "About This Mac", it will say which model and which year
in the window that appears. It will also specify which version
of the OS it's running.
I know it won't take the new OS update from Apple, but it was
able to run dmd on it last time I tried (which was like 9
months ago lol).
DMD will run on Mavericks (10.9) or later.
I have a 10.6 that I will be re-adding port of druntime/phobos to
(2.076 and later), and I think gcc also has a 10.4 in their
server farm.
Anything that I find is missing that belongs in common parts I'll
raise a pull to re-add, as it costs us nothing to support it.
Is there any consideration apart from section/tls support? I'm
just going to fork the current rt.sections stuff (I.e:
gcc.sections.{elf,macho,pef,x off}) as apart from linux/elf, the
rest is not of any use or is specific to dmd object format. As
for tls, well there is still no native support in gcc if I
understand correctly.
Re: Google Summer of Code 2019
On Sunday, 25 November 2018 at 13:58:25 UTC, Mike Parker wrote: The time has come to start thinking about GSoC 2019. The application deadline for mentoring organizations is on February 6. I'd like to get a solid list of project ideas for potential student applications. I've set up a new page at the Wiki to collect ideas and seeded it with two from the GSoC 2018 page: https://wiki.dlang.org/GSOC_2019_Ideas I invite everyone to add ideas to the list. Please be as descriptive as you can in your summaries, and be explicit about the goals the project should achieve. We want projects that are both necessary and challenging. Anyone who is interested in participating as a student or a mentor, please contact me (aldac...@gmail.com). Be sure to visit the GSOC FAQ for links to details about what both roles entail: https://developers.google.com/open-source/gsoc/faq I'll be putting out more information in the coming weeks, here and on the blog. Hi Mike, I can see from the previous GSOC entries in the wiki that there are many projects that are still interesting IMHO. Even my entry is just a rehash of the interest around calypso given that we now have dpp. Why aren't they included in the current GSOC page? Also, shouldn't students propose after the Dlang foundation gets accepted?
Re: Updates in D Land
On Monday, 10 December 2018 at 10:41:25 UTC, Mike Parker wrote: I've just published a post with a few updates and reminders on some major current and upcoming events in the world of D: fundraising, SAoC, DIPs, GSoC, and DConf 2019. Blog: https://dlang.org/blog/2018/12/10/updates-in-d-land/ Reddit: https://www.reddit.com/r/d_language/comments/a4ul8h/updates_in_d_land/ DIP1013, DIP1016 and copy is implement will made D very power in betterC mode.
Re: OFFTOPIC Re: I've just released Vasaro
On 2018-12-08 18:01, Adam D. Ruppe wrote: The one I have is a macbook air with a broken, but usable screen (I got it for free yay). I don't know how old it is, I *think* it is a 2013 model. If you click on the Apple menu in the top left corner and choose "About This Mac", it will say which model and which year in the window that appears. It will also specify which version of the OS it's running. I know it won't take the new OS update from Apple, but it was able to run dmd on it last time I tried (which was like 9 months ago lol). DMD will run on Mavericks (10.9) or later. If you don't want to keep it you could always donate it as a testing machine, if the Dlang foundation will accept it. -- /Jacob Carlborg
Updates in D Land
I've just published a post with a few updates and reminders on some major current and upcoming events in the world of D: fundraising, SAoC, DIPs, GSoC, and DConf 2019. Blog: https://dlang.org/blog/2018/12/10/updates-in-d-land/ Reddit: https://www.reddit.com/r/d_language/comments/a4ul8h/updates_in_d_land/
S2 Geometric Library for D
I would like to announce the release of the S2 Geometric Library in the D Programming Language. Who: This is of interest to engineers who need to process very large amounts of geographic coordinate data very quickly. E.g. ride-shares, bikes, maps, etc. What: The S2 Geometric Library was originally developed by Eric Veach from Google to serve as a highly performance geo-spatial library with very high accuracy. It achieves this by keeping coordinates as 3D points which are projected onto the Earth's surface. http://s2geometry.io/ Where: DUB Package - https://code.dlang.org/packages/s2geometry-d When: Right now, the first version with most features is ready. The remaining features will be added in the following couple of months. Why: I have used the Java port of this library (https://github.com/google/s2-geometry-library-java) in systems handling thousands of inputs per second with great results. The Java version, however, is 7 years old, and because performance was my interest, I decided to port this library from it's modern C++ version to D. How: Lots and lots of work over a long time: https://github.com/vnayar/s2geometry-d/graphs/code-frequency
