Re: What's going on with std.experimental.lexer?
On Wed, Jun 4, 2014 at 4:12 PM, Brian Schott via Digitalmars-d digitalmars-d@puremagic.com wrote: I've been looking at ways to optimize the D lexer's operation using SIMD Brian, i just found the lexer code repo (and fixed the broken code link on the wiki), but the review thread on the wiki looks very old. The wiki page: http://wiki.dlang.org/Review_Queue The review thread link: http://forum.dlang.org/post/aiipgaqyddjijpjiu...@forum.dlang.org Thanks. Best regards, -Tom
Re: What's going on with std.experimental.lexer?
On Friday, 13 June 2014 at 12:01:00 UTC, Tom Browder via Digitalmars-d wrote: On Wed, Jun 4, 2014 at 4:12 PM, Brian Schott via Digitalmars-d digitalmars-d@puremagic.com wrote: I've been looking at ways to optimize the D lexer's operation using SIMD Brian, i just found the lexer code repo (and fixed the broken code link on the wiki), but the review thread on the wiki looks very old. The wiki page: http://wiki.dlang.org/Review_Queue The review thread link: http://forum.dlang.org/post/aiipgaqyddjijpjiu...@forum.dlang.org Thanks. Best regards, -Tom Last formal review has happened quite some time ago, this is correct. Much has been done since then. Anyway, I am ready to proceed with it as soon as Brian is.
Re: What's going on with std.experimental.lexer?
Please no. See: javax Spelling out 'experimental' is probably the best, for all those reasons already stated. What's wrong with javax?
Re: What's going on with std.experimental.lexer?
On Friday, 13 June 2014 at 14:59:55 UTC, Dejan Lekic wrote: Please no. See: javax Spelling out 'experimental' is probably the best, for all those reasons already stated. What's wrong with javax? The fact that it started as same experimental package but stuff there never got moved to main namespace or actually changed in breaking fashion. Because many people missed experimental part and started using it as normal standard library package.
Re: What's going on with std.experimental.lexer?
Am 13.06.2014 16:59, schrieb Dejan Lekic: Please no. See: javax Spelling out 'experimental' is probably the best, for all those reasons already stated. What's wrong with javax? experimental is 100% clear and simple to understand beeing evil javax was interpreted as eXtendet or eXtra or whatever, so people have no problem in using experimental stuff all the while
Re: What's going on with std.experimental.lexer?
On Mon, 09 Jun 2014 14:23:59 -0700 Andrei Alexandrescu via Digitalmars-d digitalmars-d@puremagic.com wrote: On 6/9/14, 2:15 PM, Dejan Lekic wrote: I am more for stdx, which is what some developers already use as package name for experimental stuff. The way I see it is instead of explaining stdx stands for the experimental part of std I'd rather use std.experimental which explains itself. We added immutable after having explained to a million people invariant in this context means immutable. That and the fact that we'd like to avoid the javax disaster. std.experimental _should_ be annoyingly long and obvious. We want people to use it, but it needs to be clear that it's temporary and _experimental_ until it actually ends up in std. - Jonathan M Davis
Re: What's going on with std.experimental.lexer?
On 10 June 2014 07:26, Jonathan M Davis via Digitalmars-d digitalmars-d@puremagic.com wrote: On Mon, 09 Jun 2014 14:23:59 -0700 Andrei Alexandrescu via Digitalmars-d digitalmars-d@puremagic.com wrote: On 6/9/14, 2:15 PM, Dejan Lekic wrote: I am more for stdx, which is what some developers already use as package name for experimental stuff. The way I see it is instead of explaining stdx stands for the experimental part of std I'd rather use std.experimental which explains itself. We added immutable after having explained to a million people invariant in this context means immutable. That and the fact that we'd like to avoid the javax disaster. std.experimental _should_ be annoyingly long and obvious. We want people to use it, but it needs to be clear that it's temporary and _experimental_ until it actually ends up in std. I agree all the way with std.experimental as the package name. Though I might throw in an alternative argument to stdx and instead promote unsafe.* or std.unsafe. ;-)
Re: What's going on with std.experimental.lexer?
On 6/10/14, 3:57 AM, Iain Buclaw via Digitalmars-d wrote: I agree all the way with std.experimental as the package name. Though I might throw in an alternative argument to stdx and instead promote unsafe.* or std.unsafe. ;-) The only issue I see with *.unsafe.* is that it sounds related to @safety. So far I haven't seen a better name than std.experimental.
Re: What's going on with std.experimental.lexer?
On Tue, 10 Jun 2014 10:31:59 -0400, David Gileadi gilea...@nspmgmail.com wrote: On 6/10/14, 3:57 AM, Iain Buclaw via Digitalmars-d wrote: I agree all the way with std.experimental as the package name. Though I might throw in an alternative argument to stdx and instead promote unsafe.* or std.unsafe. ;-) The only issue I see with *.unsafe.* is that it sounds related to @safety. So far I haven't seen a better name than std.experimental. If it weren't a keyword, volatile would be a very good package name for this. -Steve
Re: What's going on with std.experimental.lexer?
On Friday, 6 June 2014 at 23:50:40 UTC, Brian Schott wrote: SIMD reduces execution time by 5.15% with DMD. Compiling the non-SIMD code with GDC reduces execution time by 42.39%. So... There's that. Changing the code generator to output a set of if statements that implements a binary search did more-or-less nothing with the DMD timings, but brought GDC's lead up to 49%. (i.e. the GDC-compiled version executes in 51% of the time that the DMD-compiled version does)
Re: What's going on with std.experimental.lexer?
On Thursday, 5 June 2014 at 10:57:37 UTC, Dicebot wrote: On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote: I've been looking at ways to optimize the D lexer's operation using SIMD instructions. I'm not yet sure if I'll need to change the lexer generator's API to do this. I'm going to wait until I have my proof-of-concept code and some benchmark results before asking for a voting thread or creating a pull request. I still thing we should use it more like `std.staging` - once your updates are ready, go through review/voting and keep module in `std.experimental` for at least one DMD release before adding to Phobos core. This also means relaxing API requirements a lot for initial inclusion. I am more for stdx, which is what some developers already use as package name for experimental stuff.
Re: What's going on with std.experimental.lexer?
On 6/9/14, 2:15 PM, Dejan Lekic wrote: On Thursday, 5 June 2014 at 10:57:37 UTC, Dicebot wrote: On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote: I've been looking at ways to optimize the D lexer's operation using SIMD instructions. I'm not yet sure if I'll need to change the lexer generator's API to do this. I'm going to wait until I have my proof-of-concept code and some benchmark results before asking for a voting thread or creating a pull request. I still thing we should use it more like `std.staging` - once your updates are ready, go through review/voting and keep module in `std.experimental` for at least one DMD release before adding to Phobos core. This also means relaxing API requirements a lot for initial inclusion. I am more for stdx, which is what some developers already use as package name for experimental stuff. The way I see it is instead of explaining stdx stands for the experimental part of std I'd rather use std.experimental which explains itself. We added immutable after having explained to a million people invariant in this context means immutable. Andrei
Re: What's going on with std.experimental.lexer?
On Mon, Jun 9, 2014 at 2:15 PM, Dejan Lekic via Digitalmars-d digitalmars-d@puremagic.com wrote: I am more for stdx, which is what some developers already use as package name for experimental stuff. Please no. See: javax Spelling out 'experimental' is probably the best, for all those reasons already stated.
Re: What's going on with std.experimental.lexer?
Am 09.06.2014 22:21, schrieb Brian Schott: On Friday, 6 June 2014 at 23:50:40 UTC, Brian Schott wrote: SIMD reduces execution time by 5.15% with DMD. Compiling the non-SIMD code with GDC reduces execution time by 42.39%. So... There's that. Changing the code generator to output a set of if statements that implements a binary search did more-or-less nothing with the DMD timings, but brought GDC's lead up to 49%. (i.e. the GDC-compiled version executes in 51% of the time that the DMD-compiled version does) the LLVM optimizer is also very good (sometimes far better then the gcc one) - what are your LDC timings?
Re: What's going on with std.experimental.lexer?
Am 07.06.2014 01:50, schrieb Brian Schott: On Friday, 6 June 2014 at 00:33:23 UTC, Brian Schott wrote: Implementing some SIMD code just in the lexWhitespace function causes a drop in total lexing time of roughly 3.7%. This looks promising so far, so I'm going to implement similar code in lexStringLiteral, lexSlashStarComment, lexSlashSlashComment, and lexSlashPlusComment. Some moe numbers: SIMD reduces execution time by 5.15% with DMD. Compiling the non-SIMD code with GDC reduces execution time by 42.39%. So... There's that. thats why im always puzzled when people start to optimze algorithms based on DMD results - currently one should always compare any results before optimization with GDC/LDC
Re: What's going on with std.experimental.lexer?
On 06/08/2014 08:21 PM, dennis luehring wrote: thats why im always puzzled when people start to optimze algorithms based on DMD results - currently one should always compare any results before optimization with GDC/LDC I second that, dmd results are easily misleading as you often optimize around backend deficiencies.
Re: What's going on with std.experimental.lexer?
On Thu, 05 Jun 2014 10:57:35 + Dicebot via Digitalmars-d digitalmars-d@puremagic.com wrote: On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote: I've been looking at ways to optimize the D lexer's operation using SIMD instructions. I'm not yet sure if I'll need to change the lexer generator's API to do this. I'm going to wait until I have my proof-of-concept code and some benchmark results before asking for a voting thread or creating a pull request. I still thing we should use it more like `std.staging` - once your updates are ready, go through review/voting and keep module in `std.experimental` for at least one DMD release before adding to Phobos core. This also means relaxing API requirements a lot for initial inclusion. Yeah. std.experimental really should be for stuff that's gone through the review process and is essentially ready for inclusion in Phobos but needs more real world usage to make sure that it really is fully ready to be merged into std and then have its API frozen. It really doesn't make any sense to put anything that's in more flux than that in there, because the release cycle is just way too long for it. Now, Brian's lexer might effectively be at that point already, I don't know. It's been reviewed before, but I generally haven't had much time to look it over for those reviews and don't know what state it's currently in. So, if its API is close enough to final at this point, then it could probably be put in std.experimental, but if it's still in a lot of flux, then that probably isn't appropriate. We do need to figure out what the official process is on this though so that it's clear when it's appropriate for something to go in std.experimental, otherwise we'll have folks who keep trying to put stuff in there which really isn't ready yet, and we'll keep end up arguing over what can and can't go in there. - Jonathan M Davis
Re: What's going on with std.experimental.lexer?
On Friday, 6 June 2014 at 00:33:23 UTC, Brian Schott wrote: Implementing some SIMD code just in the lexWhitespace function causes a drop in total lexing time of roughly 3.7%. This looks promising so far, so I'm going to implement similar code in lexStringLiteral, lexSlashStarComment, lexSlashSlashComment, and lexSlashPlusComment. Some moe numbers: SIMD reduces execution time by 5.15% with DMD. Compiling the non-SIMD code with GDC reduces execution time by 42.39%. So... There's that.
Re: What's going on with std.experimental.lexer?
I've been meaning to mention this, but I use std.experimental.lexer in a code generation tool for our project; it's worked well so far, and I'd happily recommend its use. Looking forward to further updates.
Re: What's going on with std.experimental.lexer?
On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote: I've been looking at ways to optimize the D lexer's operation using SIMD instructions. I'm not yet sure if I'll need to change the lexer generator's API to do this. I'm going to wait until I have my proof-of-concept code and some benchmark results before asking for a voting thread or creating a pull request. I still thing we should use it more like `std.staging` - once your updates are ready, go through review/voting and keep module in `std.experimental` for at least one DMD release before adding to Phobos core. This also means relaxing API requirements a lot for initial inclusion.
Re: What's going on with std.experimental.lexer?
On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote: I've been looking at ways to optimize the D lexer's operation using SIMD instructions. I'm not yet sure if I'll need to change the lexer generator's API to do this. I'm going to wait until I have my proof-of-concept code and some benchmark results before asking for a voting thread or creating a pull request. Implementing some SIMD code just in the lexWhitespace function causes a drop in total lexing time of roughly 3.7%. This looks promising so far, so I'm going to implement similar code in lexStringLiteral, lexSlashStarComment, lexSlashSlashComment, and lexSlashPlusComment. So far this only uses SSE2 instructions, so all 64-bit x86 processors can benefit from this. (SSE4.2 would allow us to do more interesting things, but it's not as common)
