Re: From the D Blog: Driving with D
On Friday, 4 June 2021 at 21:47:21 UTC, Iain Buclaw wrote:
On Friday, 4 June 2021 at 21:28:00 UTC, Max Samukha wrote:
This doesn't [work]:
ubyte[] slice;
foreach (ubyte i; slice) {
}
Invalid bitcast
%17 = bitcast i16 %15 to i32
I guess the cause is the same - slice.length.sizeof == 4,
while slice.sizeof == 4, slice.ptr.sizeof == 2, and
size_t.sizeof == 2.
You should have better luck using gdc on avr.
https://explore.dgnu.org/z/bos5ee
Hi Max,
GDC and LDC are both happy to support many platforms (it's
fun), so keep filing bug reports in our bug trackers! (LDC's bug
tracker is on Github, please also mention the triple you are
using)
cheers,
Johan
Re: LDC 1.25.0
On Monday, 22 February 2021 at 07:20:16 UTC, kinke wrote: On Sunday, 21 February 2021 at 21:58:48 UTC, Dennis wrote: On Sunday, 21 February 2021 at 18:26:38 UTC, kinke wrote: - Profile/trace LDC invocations via --ftime-trace. Is there more info on this? I tried it on a simple file, it generates a large json file. Can this be inspected to reduce compile times? The changelog and pull request do not reveal much info on how to use it for the uninformed. See Johan's comment here: https://forum.dlang.org/post/usyrbgwzzwxnzzhqe...@forum.dlang.org Thanks kinke. This article explains the basics: https://www.snsystems.com/technology/tech-blog/clang-time-trace-feature I'm aware there is very little documentation about this... I lack the time to write up more details and promote it better. Now that it's released, I hope promotion can be picked up by LDC's users. Here is an example of how it can help: https://github.com/dlang/phobos/pull/7676#issuecomment-715373535 There is definitely room for improvement, but it requires user feedback and consideration of their different use cases. -Johan
Re: LDC 1.24.0
On Saturday, 24 October 2020 at 15:11:08 UTC, kinke wrote: Glad to announce LDC 1.24 - some highlights: - Based on D 2.094.1+. - Support for LLVM 11. The prebuilt packages use v11.0.0, and the x86 packages newly include the LLVM backend for AMD GPUs. - Experimental support for targeting macOS on 64-bit ARM. The macOS package includes prebuilt druntime/Phobos libraries for cross-compilation via `-mtriple=arm64-apple-macos`. Full release log and downloads: https://github.com/ldc-developers/ldc/releases/tag/v1.24.0 Thanks to all contributors & sponsors! Thanks Martin! FYI: this is the first release that includes a working ThreadSanitizer. Curious to hear people's experiences with it. Cmdline flag: -fsanitize=thread https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual Examples: https://github.com/ldc-developers/ldc/blob/master/tests/sanitizers/tsan_tiny_race.d https://github.com/ldc-developers/ldc/blob/master/tests/sanitizers/tsan_tiny_race_TLS.d cheers, Johan
Re: LDC 1.24.0-beta1
On Tuesday, 20 October 2020 at 20:21:56 UTC, aberba wrote: On Tuesday, 20 October 2020 at 17:36:11 UTC, kinke wrote: On Tuesday, 20 October 2020 at 16:08:47 UTC, aberba wrote: It's an option but doesn't fill the need for an installer. Not sure why its hasn't been done. See https://github.com/ldc-developers/ldc/issues/1754. From the discussions, it seems you still don't see the value of an installer...backing it with the idea that LDC is for "developers". I'm a developer myself and I use installers all the time when on Windows...there are very few people I personally know who would go for an archive file to set it up themselves. So not everyone is like you. The reason why I personally go for DMD over LDC is convenience (especially when introducing D to newbies)...even though LDC is more optimized for performance. Unless what you guys are doing is an artificial barrier to get others to not use it. Guys, all points have been made, there is no wrong and right here, let's stop arguing over this. What is needed is someone who thinks it is useful to have an exe installer and wants to do the work. It cannot be done by someone who thinks it is not useful, because there are decisions to be made (like which folder to install it in, whether to overwrite old or not), that can only be made by someone who actually cares about it. There is no point in trying to convince kinke or me. I'm sure noone will be against uploading the installer exe onto github release page once it's been made and checked. -Johan
Re: Proposal for porting D runtime to WebAssembly
On Saturday, 23 November 2019 at 09:51:13 UTC, Sebastiaan Koppe wrote: This is my proposal for porting D runtime to WebAssembly. I would like to ask you to review it. You can find it here: https://gist.github.com/skoppe/7617ceba6afd67b2e20c6be4f922725d Perhaps you can explicitly clarify that "port" in this context means that you will add the required version(WebAssembly) blocks in the official druntime, rather than in a fork of druntime. (WebAssembly predefined version now explicitly mentions that it is for 32bit. Do you want to broaden this to 64bit aswell, or add a new version identifier?) I read that Clang uses a triple with explicit mention of WASI: --target wasm32-wasi Are you planning for the same with LDC? Will you need a new predefined version identifier for WASI-libc? Perhaps group all required compiler features in a section (and move the `real` story there). Can you elaborate on how you envision CI testing? Do you want to add that to LDC testing? (this may also mean that you first add a new change to LDC's druntime, confirming functionality with LDC CI, and then upstreaming the change) I'm assuming you already started some work in this area? Where can we track it? Great initiative! Johan
Re: D code running on the Nintendo 3DS
On Sunday, 20 October 2019 at 06:06:48 UTC, TheGag96 wrote: Hi, all. I wanted to get into the world of 3DS homebrew, but I really didn't feel like coding in C or C++. So, through an effort of sheer will, I somehow got a hello world example written in D up and running, along with bindings for most of libctru and citro3d. https://github.com/TheGag96/3ds-hello-dlang Included are instructions on how to set this up (although it's pretty hacky). I imagine one could easily start writing Switch homebrew in D by following most of these steps as well. Once GCC 10 comes out with an updated GDC, it might become a pretty attractive alternative to C/++ for such projects if some work is put into bindings and stuff. Hope someone finds this interesting! Definitely :-) Did you try with LDC? Just a wild guess but perhaps this triple will work: `-mtriple=armv6k-unknown-eabi`. -Johan
Re: Blog Post: Beating std::visit Without Really Trying
On Saturday, 5 October 2019 at 02:59:58 UTC, Paul Backus wrote: I was curious how C++17's std::variant compared to the options we have in D, like Algebraic and SumType, so I did a simple comparison of the generated assembly for each of them. You can read about it at the link below. And as you can probably guess from the title, D comes out ahead, in the end. https://pbackus.github.io/blog/beating-stdvisit-without-really-trying.html This is my first attempt at sharing something like this, so any comment or feedback is very much appreciated! It'd be interesting if you could investigate the case of non-trivial lambdas, where inlining the lambdas is not profitable. Perhaps easiest if you call opaque (declaration-only) functions in the visit list. Conclusion, performance-wise: - SumType is great - std::visit is OK'ish - D Algebraic is terrible Do you agree? cheers, Johan
Re: utiliD: A library with absolutely no dependencies for bare-metal programming and bootstrapping other D libraries
On Friday, 10 May 2019 at 23:58:37 UTC, Mike Franklin wrote: I don't know how a proper assembly implementation would not be performant. Perhaps you could elaborate. Inline assembly prevents a lot of optimizations that give large performance gains such as constant propagation. Say you implement a memcpy with a different signature than C's mempcy (because of slices instead of pointers), then the optimizer does not know what the semantics of that function are and will need the function to be transparent (not assembly) to do such optimizations. But I'm sure you know all that, so that's not your question. :) In the case of reimplementing memcpy/mem* in a function with the same signature as libc, that is not supposed to be inlined (like the current libc functions), then I also think the use of inline asm will not give a perf penalty. Be careful to recreate the exact same semantics as those libc functions because the optimizer is going to _assume_ it knows _exactly_ what those functions are doing. cheers, Johan
Re: DConf 2019 Day 3 Slides
On Friday, 10 May 2019 at 08:03:52 UTC, Mike Parker wrote: The slides for Andrei's keynote can be found here: https://drive.google.com/open?id=1nrya9553FSMyBfLUmioYovVbqOoYycMe Thank you Andrei for presenting those last 3 slides. -Johan
Re: utiliD: A library with absolutely no dependencies for bare-metal programming and bootstrapping other D libraries
On Friday, 10 May 2019 at 17:16:24 UTC, Mike Franklin wrote: On Friday, 10 May 2019 at 05:20:59 UTC, Eugene Wissner wrote: - Memcmp, memcpy, memmove and memset are named equal, copy, copyBackward and fill respectively. I just wanted to create native implementations that are bit safer than their C counterparts. So they do the same job, but accept void[] instead of pointers. There are also templated functions with the same names, that work with ranges. I‘m not very comfortable with GCC‘s inline asm and it doesn‘t support naked asm as DMD does, so I put asm in the .S files and compile them separately. But I‘m fine with inline asm too. Why would you use inline assembly ? (generalizing but: extremely bad portability, bad performance, bad readability) Recent discussion on LLVM mailinglist about the problem of the optimizer recognizing memcmp implementation and substituting it with a call to memcmp: https://lists.llvm.org/pipermail/llvm-dev/2019-April/131973.html cheers, Johan
Re: Beta 2.084.1
On Tuesday, 5 February 2019 at 04:18:55 UTC, Martin Nowak wrote: Glad to announce the first beta for the 2.084.1 point release, Any chance of getting this in? https://github.com/dlang/phobos/pull/6814 Phobos std.file has become unusable after 2.082 because of [1] leading to memory corruption. It's pretty bad: dmd 2.082 and 2.083, and LDC 1.12 and 1.13 cannot be used for serious work (unless you prevent people from using std.file by deleting it?). Cheers, Johan [1] the trigger of the bug: https://github.com/dlang/phobos/pull/6584
LDC "nightly" or latest CI build
Hi all, The dlang.org install.sh script is now able to install the latest successful CI build of LDC master, called "ldc-latest-ci" ! That should make it easier for you to test the newest of the newest of LDC master. On Travis, you can use "d: ldc-latest-ci" to test your project with it. On d.godbolt.org, you can find it as ldc latest CI (dmd nightly is there too). Note: the CI build includes assertions for extra testing, which means that the compiler will be much slower than normal releases. Cheers, Johan
Re: Fuzzed - a program to find DMDFE parser crash
On Saturday, 15 December 2018 at 11:29:45 UTC, Basile B. wrote: Fuzzed [1] is a simple fuzzer for the D programming language. Are you familiar with libFuzzer and LDC's integration? https://johanengelen.github.io/ldc/2018/01/14/Fuzzing-with-LDC.html You can feed libFuzzer with a dictionary of keywords to speed up the initial fuzzing phase, where the keywords are the tokens strings that you use. Besides finding crashes, it's also good to enable ASan to find memory-related bugs that by luck didn't crash the program. The time to write this announce, already 5 "crashers" found. Great :) The other day I was reminded of OSS Fuzz and that it'd be nice if we would setup fuzzing for the frontend and phobos there... -Johan
Re: LDC 1.13.0-beta1
On Friday, 2 November 2018 at 21:04:13 UTC, kinke wrote: Glad to announce the first beta for LDC 1.13: Note how fast Martin produced this beta release after the DMD 2.083 release. Thanks to all contributors! The main contributor by far is you Martin, thank _you_! -Johan
Re: LDC 1.11.0 beta
On Saturday, 7 July 2018 at 18:17:49 UTC, Seb wrote: Would be great to include https://github.com/dlang/dmd/pull/8456 as it's a serious regression and the reason for the early 2.081.1 release. Because the quality of new DMD releases is often subpar, the LDC release plan is to only release after a good point release of DMD (usually *.*.1, but if this is a hasty .1 release, we probably better wait until 2.081.2) -Johan
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 15:18:40 UTC, Jacob Carlborg wrote: On Thursday, 22 March 2018 at 11:00:31 UTC, Atila Neves wrote: Direct link: https://atilanevesoncode.wordpress.com/2018/03/22/keep-d-unittests-separated-from-production-code/ I completely agree. Although my reason is mostly because there will be too much code in a single file if the regular code and unit tests are mixed in the same file. Fully agree with this "too much code in a single file" point. I am confident that part of the reason of Phobos unittesting being very incomplete, is that adding unittests further clutters the codebase. Moving all unittests to the bottom of the file (pulling them out of classes too) would resolve this issue in part. - Johan
Re: Updated D packages in GNU Guix for x86_64-linux, i686-linux, armhf-linux
On Sunday, 11 March 2018 at 08:43:25 UTC, Pjotr Prins wrote: The GNU package manager updated LDC to 1.7.0 which now provides x86_64-linux, see https://www.gnu.org/software/guix/packages/L/. Runtime dependencies and build are listed at https://hydra.gnu.org/build/2525193#tabs-runtime-deps, including 0.17.4 (bootstrap) and LLVM 3.8.1. LDC 0.17.4 provides x86_64-linux, i686-linux, armhf-linux. We aim to add D compilers for all supported architectures. Kai has been working recently [1] on updating LDC 0.17.x (ltsmaster) to work with LLVM 6, PowerPC, AArch64, ... You may want to take advantage of that activity to get better bootstrap compiler support for your platforms. For example by filing bug reports, or better: helping out with testing and fixing things ;-) -Johan [1] See e.g. https://github.com/ldc-developers/dmd-testsuite/pull/34
llvm-mca: a static performance analysis tool
I find this such great LLVM news that I'd share it here: llvm-mca: a static performance analysis tool http://lists.llvm.org/pipermail/llvm-dev/2018-March/121490.html cheers, Johan
Re: LDC 1.7.0
On Tuesday, 30 January 2018 at 09:38:26 UTC, aberba wrote: On Monday, 29 January 2018 at 07:40:10 UTC, Dominikus Dittes Scherkl wrote: On Saturday, 27 January 2018 at 21:42:49 UTC, aberba wrote: [...] Ubuntu 16.04 This is a long-term support distribution. Don't expect those to have actual tip versions of any SW package! They rely on stabe versions that don't have the latest features but only those very well tested. The semver 1.7 is not an unstable package. Its that their reason for no updates? LDC 1.7.0 includes major changes to the frontend and is not well-tested. If you want a better-tested recent LDC, I recommend LDC 1.6.0, which is used in production at Weka. - Johan
Re: TSV Utilities release with LTO and PGO enabled
On Wednesday, 17 January 2018 at 04:37:04 UTC, Jon Degenhardt wrote: Clearly personal judgment played a role. However, the tools are reasonably task focused, and I did take basic steps to ensure the benchmark data and tests were separate from the training data/tests. For these reasons, my confidence is good that the results are reasonable and well founded. Great, thanks for the details, I agree. Hope it's useful for others to see these details. (btw, did you also check the performance gains when using the profile of the benchmark itself, to learn about the upper-bound of PGO for your program?) I'll merge the IR PGO addition into LDC master soon. Don't know what difference it'll make. -Johan
Re: TSV Utilities release with LTO and PGO enabled
On Tuesday, 16 January 2018 at 02:45:39 UTC, Jon Degenhardt wrote: On Tuesday, 16 January 2018 at 00:19:24 UTC, Martin Nowak wrote: On Sunday, 14 January 2018 at 23:18:42 UTC, Jon Degenhardt wrote: Combined, LTO and PGO resulted in performance improvements greater than 25% on three of my standard six benchmarks, and five of the six improved at least 8%. Yay, I'm usually seeing double digit improvements for PGO alone, and single digit improvements for LTO. Meaning PGO has more effect even though LTO seems to be the more hyped one. Have you bothered benchmarking them separately? Last spring I made a few quick tests of both separately. That was just against the app code, without druntime/phobos. Saw some benefit from LTO, mainly one of the tools, and not much from PGO. Because PGO optimizes for the given profile, it would help a lot if you clarified how you do your PGO benchmarking. What kind of test load profile you used for optimization and what test load you use for the time measurement. Regardless, it's fun to hear your test results :-) Johan
Re: Article: Fuzzing D code with LDC
On Monday, 15 January 2018 at 09:49:24 UTC, Temtaime wrote: // `Throwable`s thrown are not bugs (in contrast to `Errors`). They _can_ be bugs. class Error : Throwable class Exception : Throwable Good catch ( ;-) ) thanks. -Johan
Re: Article: Fuzzing D code with LDC
On Saturday, 13 January 2018 at 23:59:52 UTC, Johan Engelen wrote: It's been a work-in-progress for half a year, but finished now: http://johanengelen.github.io/ldc/2018/01/14/Fuzzing-with-LDC.html "A not-so-well-written article about the fuzzing capability recently added to LDC Argh, and just now I find out about: - https://www.youtube.com/watch?v=k-Cv8Q3zWNQ - https://github.com/google/oss-fuzz Anybody interested in looking at OSS Fuzz? "OSS-Fuzz - Continuous Fuzzing for Open Source Software Status: Beta. We are now accepting applications from widely-used open source projects." Cheers, Johan
Re: Article: Fuzzing D code with LDC
On Sunday, 14 January 2018 at 17:33:45 UTC, Joakim wrote: Nice post, enjoyed reading it. I need to look into trying out the shift sanitizer sometime. Definitely would be *great* to have `-fsanitize=undefined` ! - Johan
Article: Fuzzing D code with LDC
It's been a work-in-progress for half a year, but finished now: http://johanengelen.github.io/ldc/2018/01/14/Fuzzing-with-LDC.html "A not-so-well-written article about the fuzzing capability recently added to LDC, using LLVM’s libFuzzer. Compiling code with -fsanitize=fuzzer adds control-flow instrumentation used to guide the fuzzing and links-in the libFuzzer library that drives the fuzz testing (same as Clang). -fsanitize=fuzzer is available from LDC 1.4.0, not on Windows. LDC 1.6.0 was used for the examples in this article." Hope the article gives you enough information to start fuzz testing your own projects. cheers, Johan (per Ali's request, the article has a nice fuzzy font for some of you :P)
D code on CodeOcean
Hi all, I made two example "capsules" on CodeOcean.com that demonstrate how to use D on that platform. CodeOcean is a platform for publishing reproducible code accompanying scientific publications for example. Hello World: https://codeocean.com/2018/01/04/d-lpar-dlang-rpar-hello-world VectorFlow (using dub): https://codeocean.com/2018/01/04/vectorflow-mnist-example-lpar-dlang-dub-rpar So if any of you is using D code in publications (Andrei?), consider publishing your code there :) Cheers, Johan
Re: Article: Finding memory bugs in D code with AddressSanitizer
On Wednesday, 3 January 2018 at 23:16:45 UTC, Martin Nowak wrote: On 12/25/2017 06:03 PM, Johan Engelen wrote: I've been writing this article since August, and finally found some time to finish it: http://johanengelen.github.io/ldc/2017/12/25/LDC-and-AddressSanitizer.html Just built dmd with AddressSanitizer and ran dmd's, druntime's, and phobos' test-suite. Nice. Plans to make it part of CI ? -Johan
Re: Article: Finding memory bugs in D code with AddressSanitizer
On Monday, 25 December 2017 at 17:03:37 UTC, Johan Engelen wrote: I've been writing this article since August, and finally found some time to finish it: http://johanengelen.github.io/ldc/2017/12/25/LDC-and-AddressSanitizer.html Is it a good fit with /r/programming ? -Johan
Re: Article: Finding memory bugs in D code with AddressSanitizer
On Tuesday, 26 December 2017 at 22:11:18 UTC, Jon Degenhardt
wrote:
On Monday, 25 December 2017 at 17:03:37 UTC, Johan Engelen
wrote:
I've been writing this article since August, and finally found
some time to finish it:
http://johanengelen.github.io/ldc/2017/12/25/LDC-and-AddressSanitizer.html
Nice article. Main question / comment is about the need for
blacklisting D standard libraries (druntime/phobos). If someone
wants to try ASan out on their own code, can they start by
ignoring the D standard libraries? And, for programs that use
druntime/phobos, will this be effective? If I understand the
post, the answer is "yes", but I think it could be more
explicit.
Indeed, yes. I've used ASan successfully on the ddmd lexer.
"successfully" = I found and fixed an actual bug with it.
Without ASan-enabled standard libs, ASan testing will cover your
code and (most) std lib _templated_ code.
A blacklist may be needed for templated std lib code that doesn't
work with ASan (yet), either because of a bug in the std lib (not
very likely I think) or something else. We need much more testing
of LDC+ASan.
Second comment is related - If the reader was to try
instrumenting druntime/phobos along with their own code, how
much effort should be expected to correctly blacklist
druntime/phobos code? Would many programs have smooth sailing
if they took the blacklist published in the post? Or is this
early stage enough that some real effort should be expected?
Very early stage. I myself have not worked on ASan-enabled
druntime/phobos for more than 30 minutes. Already found some
trouble with cpuid functions (inline asm): `fun:_D4core5cpuid*`
must be added to the blacklist.
I think the first goal should be to make a blacklist such that
all tests pass, adding blacklist items in a "# not reviewed"
section. Then afterwards, we can reduce the blacklist bit-by-bit
by figuring out exactly why ASan triggers: either a bug, expected
behavior, or an ASan bug.
A counterpart to the blacklist file is an
`@no_sanitize("address")` magic UDA; to disable ASan and document
it inside the code. This should be done in such a way that it is
upstreamable. (e.g. version(LDC) static import ldc.attributes,
alias no_sanitize = ...)
Also, if the blacklist file in the post represents a meaningful
starting point,
it does
perhaps it makes sense to check it in and distribute it. This
would provide a place for contributors to start making
improvements.
Definitely makes sense. I think this should be inside the runtime
libraries' repos, right? (So one blacklist for druntime, and
another for Phobos).
(I'm even thinking about adding `-fsanitize-blacklist=<...>` to
the shipped blacklist in `ldc.conf`.)
I'll figure out how to incorporate your comments into the
article, thanks.
cheers,
Johan
Re: Article: Finding memory bugs in D code with AddressSanitizer
On Monday, 25 December 2017 at 20:31:18 UTC, Walter Bright wrote: Thanks for the great article! Some suggestions: Thanks for your comments, I've incorporated them (to my liking). 1. The gray-on-white text is not very legible. Looks great here, I like it, sorry. (made it completely black now, can't see the difference here though) Snips: This comes across as unduly negative. This is a contrived example, and implies that normal D code is written like C++ code. It would be nice to add a paragraph mentioning things about D that make it a more memory safe language. This code should be rejected by the compiler if using -dip1000. It is not, so I filed a bug report I'd rather use examples that didn't rely on compiler/library bugs. You're right, the examples are (of course) contrived. However, I didn't want to write a marketing article, and I also want to show examples found in the wild. I think one of the use cases of ASan is exactly that it can help discover bugs whereever they are, even in the compiler / standard library. I've added bits and pieces to indicate some facilities of D to mitigate these kinds of bugs, but the reality is that a lot of D code is not idiomatic and does not use the safety features (for diverse reasons). The article is not meant as a marketing piece (only for ASan), but also shouldn't be overly critical of D. Hope that the balance is a bit better now with the modifications. -Johan
Article: Finding memory bugs in D code with AddressSanitizer
I've been writing this article since August, and finally found some time to finish it: http://johanengelen.github.io/ldc/2017/12/25/LDC-and-AddressSanitizer.html "LDC comes with improved support for Address Sanitizer since the 1.4.0 release. Address Sanitizer (ASan) is a runtime memory write/read checker that helps discover and locate memory access bugs. ASan is part of the official LDC release binaries; to use it you must build with -fsanitize=address. In this article, I’ll explain how to use ASan, what kind of bugs it can find, and what bugs it will be able to find in the (hopefully near) future." Thanks for your proof-reading. cheers, Johan
Re: Silicon Valley D Meetup - December 14, 2017 - "Experimenting with Link Time Optimization" by Jon Degenhardt
On Saturday, 16 December 2017 at 19:40:14 UTC, Jon Degenhardt wrote: On Saturday, 16 December 2017 at 11:52:37 UTC, Johan Engelen wrote: Can't wait to see the results of LTO on Weka.io's (LARGE) applications. Work in progress...! Agreed. It'd be great to see the experience of a few more apps. Don't have performance numbers yet. But the executable size of the release build with ThinLTO drops to a third of the non-LTO executable size...!!! 809 MB to 250 MB. (That's without an LTO build of druntime/Phobos.) -Johan
Re: D's Newfangled Name Mangling
On Wednesday, 20 December 2017 at 13:57:20 UTC, Mike Parker wrote: Many thanks to Rainer for his insightful new article for the D Blog outlining the new name mangling algorithm. Nice! "D and C++ avoid this problem by adding more information to the symbol name, i.e. they encode into a symbol name the scope in which the symbol is defined, the function argument types, and the return type. " I would change it to "... and the return type (D only)". C++ does not include the return type in the mangle for normal functions (it does for templates). An important difference, concerning the earlier remark about programs crashing when "fail to update and recompile all source files that use the new declarartion" -Johan
Re: Silicon Valley D Meetup - December 14, 2017 - "Experimenting with Link Time Optimization" by Jon Degenhardt
On 11/21/2017 11:58 AM, Ali Çehreli wrote: Meetup page: https://www.meetup.com/D-Lang-Silicon-Valley/events/245288287/ LDC[1], the LLVM-based D compiler, has been adding Link Time Optimization capabilities over the last several releases. [...] This talk will look at the results of applying LTO to one set of applications, eBay's TSV utilities[2]. [...] Jon Degenhardt is a member of eBay's Search Science team. [...] D quickly became his favorite programming language, one he uses whenever he can. On Friday, 15 December 2017 at 03:08:35 UTC, Ali Çehreli wrote: This should be live now: http://youtu.be/e05QvoKy_8k Great! I've added some comments there, pasted here: Jon, thanks for the extensive talk and testing on LTO! And thanks for recording / broadcasting :-) (times are approximate) 7:45 Full vs Thin LTO further clarification: Full LTO is single threaded optimization and codegen (comparable with putting all source in one module). Thin LTO loads each module separately and imports functions it needs from other modules, then after the optimization and codegen happen in parallel for each module (and normal linking happens afterwards). LTO's capabilities stem from having access to functions' source code of other modules, and knowing which functions are internal to the program (so that they can be removed, non-ABI-conformant calling convention, etc., also discussed around 41:30); the importing+optim that happens at the start of Thin LTO gives you that, with the added advantage of parallel optim+codegen afterwards. 14:00 If the question was: do you need all libraries to be in IR: no. LTO works with mixed IR-object files and normal object files and libraries. Even if linking with non-IR libraries, it helps to know that no other object file references a symbol (so you can internalize it and generate better code). But indeed, for _much_ better optimization potential: the more source you have compiled with LTO enabled the better. 15:30 Whole source optimization at D-level has indeed higher potential; at the moment I don't think we do many optimizations that are only possible at D-level (and so they are done at IR level; or not at all... I'm working e.g. on devirtualization). Extra remark: the first step towards that is much deeper and well-defined spec of D semantics, in abstract machine terms. 15:45 Testing == contributing! And you're testing has greatly improved LDC's LTO, thanks! 15:50 The ldc-build-runtime tool was made by Martin Kinkelin, and as you mention it is the enabler for most of your work. 16:15 LDC LTO Windows == integrating LLD into LDC (or using lld-link.exe), https://github.com/ldc-developers/ldc/issues/2028 ~30:00 IIRC, the performance regression is due to cross-module inlining/optim (as you mention), which we get for free with LTO :-) (that is not to say that we wouldn't like to do cross-module inlining without LTO) 33:20 Compilation time. LTO skips machine codegen during the normal compilation, as machine codegen is done in the LTO linking step. So the slowdown with Thin LTO may not be too much (Thin LTO being a parallel build). An extreme case where LTO may actually result in faster codegen: if you have 1 million template function instantiations in CTFE, but they are not called during runtime, LTO may easily discard them before they reach the optimization and machine codegen stage. In such a case, LTO may very well be faster (optimized machine codegen is time consuming); however, the IR does have to be created and written to disk, and then read from disk, that takes time too... Overall, Thin LTO is slower than a normal `-O3` build, but only by a small ratio, but it also does more work (the added optimization). The compile speed difference between Full LTO and Thin LTO is very large (Full LTO is several times slower). 39:40 Indeed, D doesn't require codegen of templates if we can prove that it is already codegenned in the library itself: i.e. you _have_ to _link_ with a template-only library. In C++, codegen of templates is mandatory (afaik), and thus you do not have to link with a template-only library (e.g. headers files only). In D, this culling of template codegen is done to increase compile speed; in that sense not a fair comparison with C++. For cross-module inlining / inlining of templated functions: in C++ all template code is available in each codegenned module, so LTO is not needed to improve things; in D, using LTO makes template code available that otherwise wouldn't ---> larger (potentially much larger) relative gains with LTO for D. (this is somewhat particular to LDC currently; GDC does better cross-module inlining; try LDC's `-enable-cross-module-inlining`) 56:40 Fully share your thinking that cross-module inlining is the main source of performance gains Can't wait to see the results of LTO on Weka.io's (LARGE) applications. Work in progress...! Could you add the reference
Re: Interfacing D with C: Getting Started
On Tuesday, 5 December 2017 at 16:06:19 UTC, Mike Parker wrote: This is the first post in a new tutorial series I'm doing on the blog. Hi Mike, Nice. It's always much easier to comment than to write an article from scratch. So take all this with a grain of salt: Overall I find that it'd be much nicer if you focus on C-D interaction only. Currently you've added a lot of things that people really would already know before reading the text: I think preknowledge should be how to create programs in C and in D (including compilers, etc.). Especially on a second read, it's a lot of scrolling back and forth to find the good nuggets of info. - I would make it clearer that you are discussing calling C code from a D application. If you are going to discuss the other way around too, always make it super clear in which language the "main" part of the program is. Perhaps explain the differences in how to setup a "hello world" (e.g. druntime initialization). - I would remove all related to DMC, it's needless extra complication. - Add a sentence that all should work identically for GDC and LDC, except for the `-m32mscoff` (I think `-m32` or `-m64` is superfluous for LDC). - The text about "both files have the same name" is also needless, I think. You are writing a text about how to call C-code from D-code, using the commandline no less! ;-) Instead, I would add the output filename explicitly on the cmdlines. It's short, and more clearly shows which files are created when. - Commandline invocation and installation of compilers is mixed now; pull it apart for clarity. Fit the installation of compilers in one or two sentences (D and C). - typo: "offical" --> "official" - The code fragments don't need to be packed tight. Perhaps add comments inside them, to beef them up a bit and make them stick out more. (you could add comments about "forward declaration", "definition", ...) - Separate the hello world example into its own section: "see how easy it is?" :-) Ideas: - Idea for future texts: how to set this up using dub, and other build mechanisms like cmake, makefile, IDEs, ... - Perhaps an example of why one might want this, external libraries is what I am thinking. Forward reference to future blog article? (dstep?) Cheers, Johan
Re: LDC 1.5.0-beta1
On Monday, 23 October 2017 at 21:26:46 UTC, Guillaume Piolat wrote: Some measurements for some audio processing. Output are identical across compilers (30 measures). Guillaume, can you try with building your own LTO version of the standard library and do new measurements? Something like this should work: ``` ldc-build-runtime --reset --dFlags="-flto=full" BUILD_SHARED_LIBS=OFF ldc2 -your-normal-compile-flags -flto=full -L-L./ldc-build-runtime.tmp/lib ``` Use `--dFlags="-flto=full;-ar="` for now if you are using macOS (will not be needed with the next beta). (you can try with `-flto=thin` too for much faster link times) Cheers, Johan
Re: DMD library available as DUB package
On Sunday, 30 July 2017 at 23:41:40 UTC, Johan Engelen wrote: On Tuesday, 18 July 2017 at 12:07:27 UTC, Jacob Carlborg wrote: During the dconf hackathon I set out to create a DUB package for DMD to be used as a library. This has finally been merged [1] and is available here [2]. It contains the lexer and the parser. This is great news of course! But I have some bad news ;-) Now that the Lexer nicely separated, it is very easy for me to testdrive libFuzzer+AddressSanitizer on the lexer and... Expect many bug reports in the next days. OK, this wasn't entirely fair. 1. I didn't read the API: the buffer needs to be null-terminated. 2. With a fix [1] to prevent reading beyond the input buffer, I have yet to find a new bug. The fuzzer is running now... I wonder how long it takes to find the next bug, if any. -Johan [1] https://github.com/dlang/dmd/pull/7050
Re: DMD library available as DUB package
On Tuesday, 18 July 2017 at 12:07:27 UTC, Jacob Carlborg wrote:
During the dconf hackathon I set out to create a DUB package
for DMD to be used as a library. This has finally been merged
[1] and is available here [2]. It contains the lexer and the
parser.
This is great news of course!
But I have some bad news ;-)
Now that the Lexer nicely separated, it is very easy for me to
testdrive libFuzzer+AddressSanitizer on the lexer and... Expect
many bug reports in the next days. I am testing this code:
```
void fuzzDMDLexer(const(char*) data, size_t length)
{
scope lexer = new Lexer("test", data, 0, length, false,
false);
lexer.nextToken;
do {
auto drop = lexer.token.value;
}
while (lexer.nextToken != TOKeof);
}
```
A short list of heap-overflow memory access bugs (params data and
length are consistent):
1. length == 0
2. data == "\n" (line feed, 0xa)
3. data == "only_ascii*" (nothing following the "*" is the
problem)
4. data == "%%"
5. data == "*ô"
6. data == "\nÜÜÜ"
7. data ==
"\x0a''"
8. data == ")\xf7"
`void scan(Token* t)` is to blame for most of the bugs I found so
far. See e.g. line 980 that causes bug 3:
https://github.com/dlang/dmd/blob/154aa1bfd36333a8777d571e39690511e670bfcf/src/ddmd/lexer.d#L979-L980
Example of stacktrace (bug 8):
```
==11222==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x60200952 at pc 0x0001028915b5 bp 0x7fff5d3941f0 sp
0x7fff5d3941e8
READ of size 1 at 0x60200952 thread T0
#0 0x1028915b4 in _D4ddmd5lexer5Lexer9decodeUTFMFZk
lexer.d:2314
#1 0x102887cae in
_D4ddmd5lexer5Lexer4scanMFPS4ddmd6tokens5TokenZv lexer.d:1019
#2 0x102875089 in
_D4ddmd5lexer5Lexer9nextTokenMFZE4ddmd6tokens3TOK lexer.d:222
#3 0x1028c5d20 in _D9fuzzlexer12fuzzDMDLexerFxPhmZv
fuzzlexer.d:31
```
I am very excited to see the fuzzer+asan working so nicely!
:-)
Johan
Re: DCompute: GPGPU with Native D for OpenCL and CUDA
On Monday, 17 July 2017 at 13:50:22 UTC, Mike Parker wrote: Nicholas Wilson has put together a blog post on his progress with DCompute Great, Nick!
Re: Cap'n Proto for D v0.1.2
On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote: This is the initial public release of my optimized port of the Java implementation of Cap'n Proto. http://code.dlang.org/packages/capnproto-dlang https://github.com/ThomasBrixLarsen/capnproto-dlang Hi Thomas, Great that you are working on this. I strongly advise you to advertise your work on this page: https://capnproto.org/otherlang.html Cheers, Johan
Re: Snap packages for DMD and DUB
On Monday, 8 May 2017 at 20:05:01 UTC, Joseph Rushton Wakeling wrote: The `dmd` package also contains `rdmd`, `dustmite` and `ddemangle`. Note that by default the snap package exposes these namespaced on the name of the package (i.e. `dmd.rdmd`, etc.). To avoid this, use the `snap alias` command: sudo snap alias dmd rdmd sudo snap alias dmd dustmite sudo snap alias dmd ddemangle Have you thought about creating a `dtools` package with rdmd, dustmite, and ddemangle in it? They are useful for LDC and GDC too, perhaps people would like to be able to install them without needing to install DMD. (Perhaps the DMD/LDC/GDC package can automatically install such a "dtools" package?) cheers, Johan
Re: D for Android beta
On Thursday, 1 June 2017 at 19:31:28 UTC, Joakim wrote: [awesome text] This is great stuff Joakim! It's very nice to see your detailed release notes, with links to the patches. Hope we can get much of that into LDC master soon. Cheers, Johan
Re: Trip notes from Israel
On Monday, 22 May 2017 at 15:05:24 UTC, Andrei Alexandrescu wrote: http://dlang.org/blog/2017/05/22/introspection-introspection-everywhere/ -- Andrei A fun read! "(Late at night, I double checked. Mozilla’s CheckedInt is just as bad as I remembered. They do a division to test for multiplication overflow. Come on, put a line of assembler in there! Portability is worth a price, just not any price.)" Shocked: do you use assembly in Checked and cripple the optimizer?!?! Luckily, no. But LDC and GDC do create the `seto` instruction I think you were hinting at: https://godbolt.org/g/0jUhgs (LDC doesn't do as good as it could, https://github.com/ldc-developers/ldc/issues/2131) cheers, Johan
Re: A New Import Idiom`
On Monday, 13 February 2017 at 20:09:45 UTC, Daniel Nielsen wrote: I still suspect there will be a difference in real-world applications even with LDC. Why would you expect there to be a difference? Module ctors/dtors are still pulled in, regardless of the import being selective or not. What am I missing? Thanks, Johan
Re: A New Import Idiom`
On Monday, 13 February 2017 at 14:28:20 UTC, Mike Parker wrote: Daniel Nielsen put together a post ... [1] https://dlang.org/blog/2017/02/13/a-new-import-idiom/ I also liked the post, nicely written. I didn't check, but if this idiom is not already tested in dmd-testsuite, please add it! -Johan
Re: A New Import Idiom`
On Monday, 13 February 2017 at 15:04:53 UTC, Daniel N wrote:
On Monday, 13 February 2017 at 15:00:24 UTC, jmh530 wrote:
With the Jack Stouffer comparison, wouldn't it be fairer to do:
import std.datetime : Systime;
import std.traits : isIntegral;
Probably, please help measuring your idea and post it here. I
intentionally chose a test made by someone else to avoid
biasing from my part. It was the only test I found at time of
writing.
In case you don't actually use the function, things are faster
and you get a smaller object file because the template is never
instantiated. So if the function is never used in the module (it
is just there for other modules to use), then things are indeed
faster and smaller. But you also don't get compile checking on
it; putting only
```
void func(T)(from!"std.datetime".SysTime a, T value)
if (from!"std.traits".isIntegral!T)
{
import std.stdio : writeln;
writeln(a, value);
}
```
in a file compiles fast without complaints about an undefined
`from`.
But for a full program, compile time is the same (it's really
short so you can't really conclude much from it), and binary size
as well (LDC 1.1.0, -O3). For my test cases below, the binaries
are identical, except for around 30 bytes...
My test cases:
```
template from(string moduleName)
{
mixin("import from = " ~ moduleName ~ ";");
}
void func(T)(from!"std.datetime".SysTime a, T value)
if (from!"std.traits".isIntegral!T)
{
import std.stdio : writeln;
writeln(a, value);
}
void main()
{
func!int(from!"std.datetime".SysTime(), 1);
}
```
```
import std.datetime;
import std.traits;
void func(T)(SysTime a, T value) if (isIntegral!T)
{
import std.stdio : writeln;
writeln(a, value);
}
void main()
{
import std.datetime : SysTime;
func!int(SysTime(), 1);
}
```
-Johan
Re: LDC talk @ FOSDEM'17
On Saturday, 21 January 2017 at 12:37:26 UTC, Kai Nacke wrote: Hi everybody! Last year turned out to be difficult for D development for me. I try to come back this year. Great! :-) Keep me in the loop when preparing your slides! ;-) cheers, Johan
Re: New GDC binaries: 2.068.2, shared library support, multilib support & a new gdmd tool
On Sunday, 25 December 2016 at 19:41:38 UTC, Johannes Pfau wrote: Happy holidays everybody, I'm happy to finally announce the release of new GDC binaries at https://gdcproject.org/downloads . Gratz! Hope you can add it to d.godbolt.org soon. ;) -Johan
Re: Many documentation examples can now be run online
On Tuesday, 20 December 2016 at 07:04:38 UTC, Seb wrote: https://github.com/dlang/dlang.org/pull/1527 Nice. It's pretty awesome! When clicking the "edit" button, a new box appears with the example's code in it but editable. Then when clicking "run", the editted code disappears (it's saved for future use, pfew!). How about this: when clicking _either_ "edit" or "run", the non-editable code box disappears, and the edittable code box appears and stays there forever. (also, the height of the editable-code-box is too small; should fit the whole example's code?) cheers, Johan
Re: Mir Blog: Writing efficient numerical code in D
On Monday, 12 December 2016 at 21:58:23 UTC, Relja Ljubobratovic wrote: Hey guys, We have just published another post on "Writing efficient numerical code in D", to Mir's Blog[1]. Nice :) Some comments: - You can make the numbers in the table more readable by adding commas? For example, 1624687 --> 1,624,687. Probably even better is to only show the most significant digits. 1624687 us --> 1.625 s for example. - Add a sentence about the benchmarking set up. " The numbers shown are the best times of 10 trials, with LDC commandline options `ldc2 -O3 -release`" Something like that. Or add it to the configuration table. - For the "convolution" example, could you elaborate on why the Mir-based implementation is so much faster than the C-like implementation? From the big table, I see that the speedup for DCV's convolution algorithms is not as large as 10x. - The "threshold" algorithm has a 19x speed up?! (I would remove "threshold" from the average improvement calculation. The numbers are already awesome, no need to oversell it!) - Overall, does this mean that DCV became more than twice as fast? How does it stack up to other vision libs after this massive speed boost? cheers, Johan
Re: Link Time Optimization in LDC
On Thursday, 10 November 2016 at 16:23:03 UTC, Dicebot wrote: Does that mean that cross-module/cross-package inlining with separate compilation now works? When all is compiled with `-flto=...`, yes. -Johan
Re: Link Time Optimization in LDC
On Thursday, 10 November 2016 at 16:20:33 UTC, Johan Engelen wrote: Hi all, Yesterday I merged LTO capability into LDC. Here a short article about it: https://johanengelen.github.io/ldc/2016/11/10/Link-Time-Optimization-LDC.html And the reddit thread: https://www.reddit.com/r/programming/comments/5c8p58/link_time_optimization_cd_crosslanguage/
Link Time Optimization in LDC
Hi all, Yesterday I merged LTO capability into LDC. Here a short article about it: https://johanengelen.github.io/ldc/2016/11/10/Link-Time-Optimization-LDC.html For the folks building LDC themselves: let me know your issues :) cheers, Johan
Re: d.godbolt.org compiler explorer now has LDC too
On Thursday, 20 October 2016 at 08:23:33 UTC, Johan Engelen wrote: https://twitter.com/mattgodbolt/status/788890061949509632 The compiler explorer at https://d.godbolt.org now also features LDC! And now it has LDC 1.1.0-beta3 aswell. See the magic of `-enable-cross-module-inlining`: https://godbolt.org/g/yk7OAo
Re: Release D 2.072.0
On Friday, 4 November 2016 at 00:42:48 UTC, Johan Engelen wrote: On Thursday, 3 November 2016 at 15:51:22 UTC, Steven Schveighoffer wrote: Any cycles that are "newly discovered" were already there. What was happening is that the runtime did not detect the cycle, and was arbitrarily choosing an order for initializing these modules. Either a) the arbitrary order happened to be right, or b) the order didn't matter. Most of the time it's b, because most static ctors don't depend on externally initialized items. My question is: any cycle is invalid? nevermind, I found this: https://dlang.org/spec/module.html#order_of_static_ctor
Re: Release D 2.072.0
On Thursday, 3 November 2016 at 15:51:22 UTC, Steven Schveighoffer wrote: Any cycles that are "newly discovered" were already there. What was happening is that the runtime did not detect the cycle, and was arbitrarily choosing an order for initializing these modules. Either a) the arbitrary order happened to be right, or b) the order didn't matter. Most of the time it's b, because most static ctors don't depend on externally initialized items. My question is: any cycle is invalid? -Johan
Re: Release D 2.072.0
On Wednesday, 2 November 2016 at 15:13:43 UTC, anonymous wrote: On Wednesday, 2 November 2016 at 15:08:26 UTC, anonymous wrote: I confirm, dmd 2.072 can't build dmd 2.071.2, same error, but boot since master straping works there's probably something that's been fixed in one or two of these ddmd modules, likely a static ctor... Maybe after this: https://github.com/dlang/dmd/commit/1d0ab8b9c136e46bf449c506ca25d2c8a784f7b9#diff-b4674e7b5d3a44178526afdefc9aa368 ddmd.attribs was removed https://github.com/dlang/dmd/commit/1d0ab8b9c136e46bf449c506ca25d2c8a784f7b9#diff-b4674e7b5d3a44178526afdefc9aa368L15 and it was also part of the cycle. Thanks for the detective work. I wonder where the bug is: in 2.071 or in 2.072 :)
Re: Release D 2.072.0
On Tuesday, 1 November 2016 at 16:40:42 UTC, Andrei Alexandrescu wrote: On 11/01/2016 11:41 AM, Johan Engelen wrote: On Monday, 31 October 2016 at 01:27:08 UTC, Martin Nowak wrote: Glad to announce D 2.072.0. DMD 2.072.0 miscompiles/uncovers a bug in LDC, so I switched back to DMD 2.071.2 for CI testing. :( Is there somebody working on that bug? Thanks. -- Andrei LDC built with DMD 2.072.0 gives the following error when run: object.Error@src/rt/minfo.d(356): Cyclic dependency between module ddmd.traits and ddmd.cond ddmd.traits* -> ddmd.attrib -> ddmd.cond* -> ddmd.expression -> ddmd.traits* Pretty much all of LDC's D code is DDMD front-end code, master is at front-end version 2.071.2. I hope someone can try to build DMD 2.071.2 using 2.072.0 and see if a similar issue is found.
Re: Release D 2.072.0
On Monday, 31 October 2016 at 01:27:08 UTC, Martin Nowak wrote: Glad to announce D 2.072.0. DMD 2.072.0 miscompiles/uncovers a bug in LDC, so I switched back to DMD 2.071.2 for CI testing. :( -Johan
Re: [Slides] Generic Low Level Programming with D - The Better C for your Business
On Friday, 28 October 2016 at 06:46:27 UTC, Ilya Yaroshenko wrote: https://docs.google.com/presentation/d/1w1cQ8vDluglRIt8Qdnm-sY7kqxoKZxbPEWW6tR3lPpo/edit?usp=sharing Nice :) Something I noticed on slide 6-8. The call __adEq2 is pretty lame, we have our fix ready: https://github.com/ldc-developers/ldc/pull/1719 (you really should number your slides!) -Johan
Re: Comparing compilation time of random code in C++, D, Go, Pascal and Rust
On Thursday, 27 October 2016 at 06:43:15 UTC, Sebastien Alaiwan
wrote:
From the article:
Surprise: C++ without optimizations is the fastest! A few
other surprises: Rust also seems quite competitive here. D
starts out comparatively slow."
These benchmarks seem to support the idea that it's not the
parsing which is slow, but the code generation phase. If code
generation/optimization is the bottleneck, a "ccache-for-D"
("dcache"?) tool might be very beneficial.
See
https://johanengelen.github.io/ldc/2016/09/17/LDC-object-file-caching.html
I also have a working dcache implementation in LDC but it still
needs some polishing.
-Johan
Re: d.godbolt.org compiler explorer now has LDC too
On Thursday, 20 October 2016 at 08:23:33 UTC, Johan Engelen wrote: The compiler explorer at https://d.godbolt.org now also features LDC! By the way, adding DMD should be doable. It looks like the code already supports objdump'ing a generated binary if assembly output is not available. Matt is super friendly, so for anyone interested: go submit PRs for DMD support! cheers, Johan
d.godbolt.org compiler explorer now has LDC too
https://twitter.com/mattgodbolt/status/788890061949509632 The compiler explorer at https://d.godbolt.org now also features LDC! Now you can view the assembly output of GDC and LDC side-by-side, for example: https://godbolt.org/g/y24RGI cheers, Johan
Re: Auto-gen list of D compiler versions: Improvements
On Tuesday, 11 October 2016 at 17:21:42 UTC, Nick Sabalausky wrote: The automatically-updated list of D compiler versions available on Travis-CI (and which front-end/back-end version they each use) has had a few small improvements lately: http://semitwist.com/travis-d-compilers Perhaps another nice addition is to list the "aliases" and what they currently point to: dmd --> dmd v2.071.2 dmd-beta --> ... ldc --> ... ldc-beta --> ... gdc --> ... -Johan
Re: Article: Increasing the performance of D math code
On Tuesday, 11 October 2016 at 17:29:47 UTC, Ali Çehreli wrote: On 10/11/2016 07:01 AM, Johan Engelen wrote: > I wrote a piece on LDC's fastmath stuff that Mir uses for > high-performance D math code: > > https://johanengelen.github.io/ldc/2016/10/11/Math-performance-LDC.html > > cheers, > Johan Kind of off topic and hopefully constructive critism: You have sentence that starts with "I’m afraid it won’t be a nice read..." on Reddit. No matter how correct, there is no need to use negative marketing for your work. :) How about something more positive like "I'm sure you'll enjoy this more if you're familiar with...". ;) Something to keep in mind next time!
Re: LDC 1.1.0-beta3 has been released!
On Tuesday, 11 October 2016 at 07:29:00 UTC, Sönke Ludwig wrote: Just noticed that the release binaries are missing from https://github.com/ldc-developers/ldc/releases/tag/v1.1.0-beta3 This means that the beta currently cannot be tested with TravisCI (wanted to test a DUB related regression fix). Whoops, that's my bad :( (I editted a little and clicked the "save draft" button which turned it into a draft again, I think) Thanks for letting us know! Fixed it. I also updated LATEST_BETA, so `ldc-beta` on Travis should now automatically use 1.1.0-beta3. -Johan
Re: LDC: Speed up incremental builds with object file caching
On Sunday, 18 September 2016 at 12:15:12 UTC, Daniel Kozak wrote: But on my android phone it is OK. So maybe I should bought some 2k or 4k monitor ;) Definitely one of the best upgrades I ever did (4k monitor). Warning: there's no way back.
Re: LDC: Speed up incremental builds with object file caching
On Sunday, 18 September 2016 at 08:14:47 UTC, Martin Nowak wrote: Interesting approach to speed up compilation without running into dmd's problems of template instance emission when compiling multiple modules to multiple objects. I think LDC has the same problems (template instances are emitted in the first module not the one that needed the instantiation?). Knowing this, you may be able to set up a case where things break, but I think it would have to involve recompilation with a different set of sources than the first compile. E.g. `ldc2 -c a.d b.d c.d` first, then `ldc2 -c b.d c.d`, and then trying to link. Which probably doesn't work well without caching either... We should deprecate non-singleobj compiles. Many people use noatime or relatime when mounting their filesystems, so access time isn't the best eviction strategy unless you touch the files. I forgot to mention this in the article, thanks! I read that a file's last access timestamp may also not be updated on Windows (default Windows 7 setting), so I already had LDC `touch` the cached file upon a cache hit. Added it as a footnote in the article.
Re: LDC: Speed up incremental builds with object file caching
On Saturday, 17 September 2016 at 19:17:44 UTC, Daniel Kozak wrote: Nice article, however font on this site is too small to read, I have to zoom in :( Do more people have this problem? For me, the font size is similar to, say, Github.
LDC: Speed up incremental builds with object file caching
I just finished another post about LDC: https://johanengelen.github.io/ldc/2016/09/17/LDC-object-file-caching.html Thanks in advance for letting me know about any bugs you find, in the text or in the code :) -Johan
Re: Beta D 2.071.2-b5
On Wednesday, 14 September 2016 at 12:16:51 UTC, Martin Nowak wrote: Fifth and hopefully last beta for the 2.071.2 release. This comes with two more fixes for Issue 16031 and 16460. LDC master is up-to-date with 2.071.2-b5!
Re: Beta D 2.071.2-b4
On Tuesday, 13 September 2016 at 14:00:26 UTC, Martin Nowak wrote: On Monday, 12 September 2016 at 07:47:19 UTC, Martin Nowak wrote: This comes with a different fix for Issue 15907 than 2.071.2-b3. There will be another beta tomorrow or so to include at least one more fix (for Issue 16460) and we'll soon release 2.071.2. I have merged 2.071.2-b4+the fix for issue 16460 into LDC master. As far as deprecations go, it's looking good on Weka's codebase. cheers, Johan
Re: Beta D 2.071.2-b4
On Monday, 12 September 2016 at 07:47:19 UTC, Martin Nowak wrote: Fourth beta for the 2.071.2 release. This comes with a different fix for Issue 15907 than 2.071.2-b3. http://dlang.org/changelog/2.071.2.html I think the changelog has not been updated yet?
Re: Beta D 2.071.2-b3
On Saturday, 3 September 2016 at 18:06:37 UTC, Dicebot wrote: On Saturday, 3 September 2016 at 14:40:37 UTC, Martin Nowak wrote: On Wednesday, 31 August 2016 at 09:56:17 UTC, Johan Engelen wrote: (I can only think of complicated stuff that requires pretty much whole-program analysis to prove validity, and in that case adding `private` doesn't help any more) Yes, it does help. As private prevents usage outside of a module it allows to do some optimizations that required whole program analysis otherwise, e.g. variables and functions can get internal linkage, thus reducing loads/stores and indirect calls. This is a very naive viewpoint. There are enough ways to leak access to private entity (alias, template argument, taking address, some compile-time introspection) to make such optimizations impossible without extensive program analysis. Exactly.
Re: Beta D 2.071.2-b3
On Thursday, 1 September 2016 at 20:46:50 UTC, Steven Schveighoffer wrote: On 8/31/16 5:56 AM, Johan Engelen wrote: On Tuesday, 30 August 2016 at 23:54:45 UTC, Martin Nowak wrote: Allowing access to private members has a lot of implications, e.g. breaks lots of optimizations b/c you can't know who accesses sth. "lots of optimizations" Can you mention a few? I'm not sure optimizations are had by private variables. Some optimizations can be had because the compiler can prove a variable could not have been changed. But I don't know that private suits that need, it's more due to const or immutable. Yep. I was trying to be polite :-)
Re: Beta D 2.071.2-b3
On Tuesday, 30 August 2016 at 23:54:45 UTC, Martin Nowak wrote: Allowing access to private members has a lot of implications, e.g. breaks lots of optimizations b/c you can't know who accesses sth. "lots of optimizations" Can you mention a few? (I can only think of complicated stuff that requires pretty much whole-program analysis to prove validity, and in that case adding `private` doesn't help any more) -Johan
Re: [GSoC] Mir.random.flex - Generic non-uniform random sampling
On Tuesday, 23 August 2016 at 13:01:29 UTC, Seb wrote: On Tuesday, 23 August 2016 at 12:12:30 UTC, Seb wrote: [...] I fixed that and also added how many samples were generated per run (10M), thanks! Btw I quickly added the run-time for C++ (for G++ and clang++ with -O3) I think you should add compiler/runtimelib versions for all measurements, and all relevant flags. I'm happy to see you are using LDC for benchmarking; don't forget to mention the LLVM version for LDC's version. You already wrote `-O3` for the C++ measurements, did you not use -mcpu=native or some other performance changing flags? Cheers, Johan
Re: Beta D 2.071.2-b2
On Tuesday, 9 August 2016 at 15:37:27 UTC, Martin Nowak wrote: Please report any bugs at https://issues.dlang.org I'd appreciate it if someone could have a look at this regression: https://issues.dlang.org/show_bug.cgi?id=16369
Re: Beta D 2.071.2-b2
On Tuesday, 9 August 2016 at 15:37:27 UTC, Martin Nowak wrote: Second beta for the 2.071.2 release. Nice. Merged into LDC master. -Johan
Re: Battle-plan for CTFE
On Sunday, 7 August 2016 at 16:52:28 UTC, Stefan Koch wrote: On Saturday, 6 August 2016 at 23:04:48 UTC, Stefan Koch wrote: No, not now, but very soon. I want to have _basic_ utf8 support before I am comfortable with enabling string operations. Tomorrow this is going to work. Hi Stefan, How much effort will it be to integrate your work into LDC? I'd like to test it on some Weka code. Definitely let us know once you can CTFE something like std.traits.ParameterStorageClassTuple. cheers, Johan
Re: Running a D game in the browser
On Friday, 5 August 2016 at 05:39:57 UTC, Sebastien Alaiwan wrote: On Thursday, 4 August 2016 at 19:17:34 UTC, Sebastien Alaiwan wrote: at the moment, I have a patch to making the build work (only for the binary "ldc2", not other tools of the package). I created a dedicated github branch "fastcomp-ldc". The patch: https://github.com/Ace17/dscripten/blob/fastcomp-ldc/ldc2.patch That patch doesn't look too bad. Could you introduce a CMake option for building with Emscripten-fastcomp? And a #define "LDC_LLVM_EMSCRIPTEN" or something like that, so that you can change `#if LDC_LLVM_VER >= 309 && 0` to `#if LDC_LLVM_VER >= 309 && !LDC_LLVM_EMSCRIPTEN`. Should be mergable into LDC master then! cheers, Johan
Re: ndBenchmarks #1: ndslice.algorithm vs std.numeric vs std.algorithm
On Wednesday, 3 August 2016 at 20:53:59 UTC, Ilya Yaroshenko wrote: Hi all, There are two first [1] benchmarks for upcoming ndslice.algorithm [2]. Recent LDC alpha based on LLVM 3.8 and recent Mir v0.16.0-alpha3 are required. @fasmath syntax may be changed a little bit and will be simplified anyway. Dot Product: ndReduce vectorized = 3 ms, 314 μs ndReduce = 14 ms, 767 μs **That's** the difference with or without fastmath?? (awesome work of course!) -Johan
Re: Beta D 2.071.2-b1
On Monday, 1 August 2016 at 11:02:41 UTC, Martin Nowak wrote: First beta for the 2.071.2 point release. Green on all testers: merged into LDC master! cheers, Johan
Article: "Profile-Guided Optimization with LDC"
Because you can all download binaries for LDC 1.1.0-alpha1 now, it is time to release the article on how to use profile-guided optimization with the new LDC release. Please test and report back to us! :) https://johanengelen.github.io/ldc/2016/07/15/Profile-Guided-Optimization-with-LDC.html " Three months ago, I measured a 7% performance gain of the compiler front-end D code using Profile-Guided Optimization (PGO) with LDC. Now, part of my PGO work was merged into LDC master on Jun 20, 2016! This means it will be available in the next LDC release (version 1.1.0, needs LLVM 3.7 or newer). You can play with it with the LDC 1.1.0-alpha1 release. Here I’ll discuss how to use it, saving the implementation details for another article. Using PGO with LDC is similar to using PGO with Clang, and much of this article also applies to Clang. Profile-Guided Optimization (PGO) “Profile-guided optimizations” are optimizations that the compiler can do when it is given information about the typical execution flow of your program, such that the typical execution flow of your program runs faster (and rare execution flows run slower). The typical execution flow information is the “profile” of your program: how many times was this function called, how often was a control-flow branch taken, how often was function A called by function B, what are likely values of variable X, how much memory is usually allocated for something, etc. ... "
Re: First dmd nightly shipping with dub
When packaging dub with a compiler, it'd be nice if dub per default selected the compiler it came packaged with, I think. (it currently doesn't) Perhaps people can chime in here with ideas? https://github.com/dlang/dub/issues/895 https://github.com/dlang/dub/pull/875 Thanks, Johan
Re: First dmd nightly shipping with dub
On Friday, 8 July 2016 at 20:43:58 UTC, Martin Nowak wrote: On 07/08/2016 11:22 AM, Dicebot wrote: On Friday, 8 July 2016 at 09:13:08 UTC, Martin Nowak wrote: What would be the use-case for those? Using newer dub versions with an older compiler? Or simply using dub with ldc/gdb without also installing dmd. Kai is working on shipping LDC releases with Dub too! -Johan
Re: Release D 2.071.1
On Monday, 27 June 2016 at 22:11:53 UTC, Martin Nowak wrote: Glad to announce D 2.071.1. Thanks. LDC master is now also at 2.071.1. -Johan
Re: [Semi OT] About code review
On Monday, 27 June 2016 at 00:01:34 UTC, deadalnix wrote: Several people during DConf asked abut tips and tricks on code review. So I wrote an article about it: http://www.deadalnix.me/2016/06/27/on-code-review/ It's a nice read. One comment: perhaps the balance has tipped a bit much to "making a good PR", rather than "doing a good review". I feel the merit of a review is to improve the contribution, rather than to decide whether it is mergable or not. Although it is in the article, I think it could be given a little more attention: the review itself should contribute to the project, i.e. the reviewer should (try hard to) propose alternatives if something should/could be improved. Criticism is very easy, _constructive_ criticism isn't; I think the latter is needed to gain a contributor, and the first does the opposite. -Johan
Re: Recursive SymbolNames solved.
On Thursday, 9 June 2016 at 04:58:45 UTC, Stefan Koch wrote: On Wednesday, 8 June 2016 at 13:28:19 UTC, Stefan Koch wrote: Hi, I solved the issue. PR is coming shortly. Solution is as follows: Keep a list of already visited symbols in the mangler. And information where we store the mangle. meaning the position in the mangle string. If a symbol is encountered for the second time store a backrefernce rather then the symbol name again. You mean this? https://github.com/weka-io/ldc/commit/ffd6b55a4b83b3007d9690cbd9a25d07b243a00e It did not result in a large reduction of symbol sizes. http://forum.dlang.org/post/szodxrizfmufqdkpd...@forum.dlang.org
Re: LDC+Dub+Vibe.d work on SmartOS 64bit now
On Wednesday, 8 June 2016 at 10:06:00 UTC, Oleg Nykytenko wrote: On Wednesday, 8 June 2016 at 08:43:59 UTC, Joakim wrote: Have you run the standard library's tests? There was a thread last summer about this: http://forum.dlang.org/thread/zhcduibirwprgbzqk...@forum.dlang.org We haven't run library's tests. What right way to run this tests? But: We ported our sufficiently big server application from Linux to SmartOS x64. All application tests is OK. Application working now on testing server in production like regime under full load. And it's working fine. We're planning to use it in production in future. Are you interested in running a buildbot for LDC on your platform to detect issues early? See: http://forum.dlang.org/thread/hikcbbqqnpvvyirxg...@forum.dlang.org Thanks, Johan
Re: LDC 1.0.0 has been released!
On Tuesday, 7 June 2016 at 16:54:55 UTC, Daniel N wrote: On Monday, 6 June 2016 at 10:45:39 UTC, Guillaume Piolat wrote: very nice! how about a howto for windows/VS users? 1) Download LDC unpack anywhere 2) Download VisualD install https://github.com/dlang/visuald/releases 3) Setup VisualD to find LDC Tools->Options->Projects and Solutions->Visual D Settings->LDC Directories ... Profit! Excellent :-) https://github.com/dlang/visuald/pull/61 (I am just "copy-pasting" there! Please create your own PR with better info! )
Re: D syntax highlighting support for Jekyll websites in v1.11.0
On Monday, 6 June 2016 at 20:03:11 UTC, Borislav Kosharov wrote: Hey guys I implemented D syntax highlighting for rouge that is the default syntax highlighter for Jekyll. If you have a github pages jekyll site you will be able to post D code and get highlighting when pages moves to version 1.11.0 . Awesome! Do you know when Github Pages will upgrade, roughly? cheers, Johan
Re: Beta D 2.071.1-b2
On Monday, 30 May 2016 at 09:24:21 UTC, Johan Engelen wrote: On Sunday, 29 May 2016 at 21:53:23 UTC, Martin Nowak wrote: Second beta for the 2.071.1 release. http://dlang.org/download.html#dmd_beta http://dlang.org/changelog/2.071.1.html Please report any bugs at https://issues.dlang.org I've merged it into LDC, CI testers are running! Thumbs up! LDC master is now at 2.071.1-b2, and it can compile Weka's codebase fine (one remaining issue, but most likely not frontend related). cheers, Johan
Re: Beta D 2.071.1-b2
On Sunday, 29 May 2016 at 21:53:23 UTC, Martin Nowak wrote: Second beta for the 2.071.1 release. http://dlang.org/download.html#dmd_beta http://dlang.org/changelog/2.071.1.html Please report any bugs at https://issues.dlang.org I've merged it into LDC, CI testers are running!
Re: My ACCU 2016 keynote video available online
On Thursday, 19 May 2016 at 12:54:48 UTC, Jens Müller wrote: For example what's equivalent to gdc's -ffast-math in ldc. This is: https://github.com/ldc-developers/ldc/pull/1472 Working on performance improvements is a lot of fun. Please feed us with code that doesn't run as fast as it should! :) Johan
Re: amoeba, a chess engine written in D
On Saturday, 21 May 2016 at 16:25:27 UTC, Richard Delorme wrote: With the above versions & my poor knowledge of the compilers, pgo & BUILD=popcount only worked with gdc. I will try to upgrade to more recent versions to see if things changed. With LDC, you can pass "-mattr=+popcnt" to use the popcount instruction. Or "-mattr=+sse42" to enable SSE4.2 (like you seem to do in the Makefile).
Re: amoeba, a chess engine written in D
On Friday, 20 May 2016 at 23:16:01 UTC, Richard Delorme wrote: I am pleased to announce the release of a chess engine written in D: https://github.com/abulmo/amoeba In the makefile, I see you tried LDC's PGO, awesome! I hope to get (part of) it into LDC master very soon. All feedback is very welcome. (small typo: better use ldc-profdata instead of llvm-profdata) cheers, Johan
Re: amoeba, a chess engine written in D
On Friday, 20 May 2016 at 23:16:01 UTC, Richard Delorme wrote: The source can be compiled with dmd, ldc or gdc, but the best performance are obtained with the latter (almost twice faster). Can you give cmdline details and compiler version data for this? (how much faster is gdc relative to ldc?) Thanks, Johan
[OT] Re: pl0stuff an optimizing pl0 > c transcompiler
On Friday, 20 May 2016 at 18:04:55 UTC, Stefan Koch wrote: Update I have implemented D codegen. The CodeGenerator as well as the optimizer work at CTFE. Therefore you can transcompile code at compileTime at call PL/0 functions as there were naively implemented in D. This is pretty cool :D Now I understand why you want to improve CTFE. You actually want better C++ interop. And to do that, you are going to CTFE compile the C++ code to D code, which is then mixed in, and all is good. Excellent idea! ;-) cheers, Johan
Re: It's alive! D building D building D, all on Android
Fantastic news! I hope we can find a good way to integrate automated testing of github branches/PRs for Android. cheers, Johan
