Re: Scott Meyers' DConf 2014 keynote The Last Thing D Needs

[Dmitry Olshansky via Digitalmars-d-announce]

29-May-2014 04:58, Walter Bright пишет:

On 5/28/2014 5:35 PM, Brian Rogoff wrote:

Could you elaborate? Using some of the examples Brian gave, which ones
do you
think are are mathematically consistent/human inconsistent and which
the inverse?


Off the top of my head:

 static if (condition)
 else :

 ... declarations ...

All attributes apply to either:

1. the next statement or declaration
2. { ... }
3. : ...

That case is (3), as static if is set up as an attribute.



Static if is certainly NOT an attribute, it doesn't make any sense.
And no, it doesn't matter how the current frontend implements it, 
because you can argue next to any decisions this way.


--
Dmitry Olshansky

Re: Scott Meyers' DConf 2014 keynote The Last Thing D Needs

[Dmitry Olshansky via Digitalmars-d-announce]

29-May-2014 02:10, Jonathan M Davis via Digitalmars-d-announce пишет:

On Tue, 27 May 2014 06:42:41 -1000
Andrei Alexandrescu via Digitalmars-d-announce
digitalmars-d-announce@puremagic.com wrote:

 
http://www.reddit.com/r/programming/comments/26m8hy/scott_meyers_dconf_2014_keynote_the_last_thing_d/
 
  https://news.ycombinator.com/newest (search that page, if not found
  click More and search again)
 
  https://www.facebook.com/dlang.org/posts/855022447844771
 
  https://twitter.com/D_Programming/status/471330026168651777

Fortunately, for the most part, I think that we've avoided the types of
inconsistencies that Scott describes for C++, but we do definitely have some
of our own. The ones that come to mind at the moment are:


Not talking about other moments, but Unicode kind of caught my eye..


6. The situation with ranges and string is kind of ugly, with them being
treated as ranges of code points. I don't know what the correct solution to
this is, since treating them as ranges of code units promotes efficiency but
makes code more error-prone, whereas treating them as ranges of graphemes
would just cost too much.


This is gross oversimplification of the matter. There is no more 
correct, less correct. Each algorithm requires its own level of 
consideration, if there is a simple truism about Unicode it is:


Never operate on a single character, rather operate on slices of text.

To sum up the situation:

Unicode standard defines *all* of its algorithms in terms of code points 
and some use grapheme clusters. It never says anything about code units 
beyond mapping of code units -- code point. So whether or not you 
should actually decode is up to the implementation.




Ranges of code points is _mostly_ correct but
still
incorrect and _more_ efficient than graphemes but still quite a bit less
efficient than code units. So, it's kind of like it's got the best and worst
of both worlds. The current situation causes inconsistencies with everything
else (forcing us to use isNarrowString all over the place) and definitely
requires frequent explaining, but it does prevent some classes of problems.
So, I don't know. I used to be in favor of the current situation, but at
this
point, if we could change it, I think that I'd argue in faver of just
treating
them as ranges of code units and then have wrappers for ranges of code
points
or graphemes.


Agreed. The simple dream of automatically decoding UTF and staying 
Unicode correct is a failure.



It seems like the current situation promotes either using
ubyte[] (if you care about efficiency) or the new grapheme facilities in
std.uni if you care about correctness, whereas just using strings as
ranges of
dchar is probably a bad idea unless you just don't want to deal with any of
the Unicode stuff, don't care all that much about efficiency, and are
willing
have bugs in the areas where operating at the code point level is incorrect.


The worst thing about current situation is any generic code that works 
on UTF ranges has to jump through unbelievable amount of hoops to undo 
string has no length madness.


I think what we should do is define an StringRange or some such, that 
will at least make the current special case of string more generic.


--
Dmitry Olshansky

Re: Scott Meyers' DConf 2014 keynote The Last Thing D Needs

[Dmitry Olshansky via Digitalmars-d-announce]

29-May-2014 23:06, Steven Schveighoffer пишет:

On Thu, 29 May 2014 14:11:27 -0400, Dmitry Olshansky
dmitry.o...@gmail.com wrote:


29-May-2014 04:58, Walter Bright пишет:

On 5/28/2014 5:35 PM, Brian Rogoff wrote:

Could you elaborate? Using some of the examples Brian gave, which ones
do you
think are are mathematically consistent/human inconsistent and which
the inverse?


Off the top of my head:

 static if (condition)
 else :

 ... declarations ...

All attributes apply to either:

1. the next statement or declaration
2. { ... }
3. : ...

That case is (3), as static if is set up as an attribute.



Static if is certainly NOT an attribute, it doesn't make any sense.


Well... it sorta does. static if does not introduce a new scope, even
with {}, and this only happens with attributes.


Let it be just a declaration, as simple as that. Attributes affect other 
declarations in the scope, static if doesn't.




-Steve



--
Dmitry Olshansky

Re: DConf 2014 Day 1 Talk 4: Inside the Regular Expressions in D by Dmitry Olshansky

[Dmitry Olshansky via Digitalmars-d-announce]

11-Jun-2014 22:03, Atila Neves пишет:

On Tuesday, 10 June 2014 at 19:36:57 UTC, bearophile wrote:

At about 40.42 in the Thoughts on static regex there is written
even compile-time printf would be awesome. There is a patch about
__ctWrite in GitHug, it should be fixed and merged.

Bye,
bearophile


I wish I'd taken the mic at the end, and 2 days later Adam D. Ruppe said
what I was thinking of saying: unit test and debug the CTFE function at
runtime and then use it at compile-time when it's ready for production.



Yes, that's a starting point - a function working at R-T.


Yes, Dmitry brought up compiler bugs. But if you write a compile-time UT
and it fails, you'll know it wasn't because of your own code because the
run-time ones still pass.


It doesn't help that it's not your fault :)
And with a bit of __ctfe's to workaround compiler bugs you won't be so 
sure of your code anymore.




Maybe there's still a place for something more than pragma msg, but I'd
definitely advocate for the above at least in the beginning. If
anything, easier ways to write compile-time UTs would be, to me,
preferable to a compile-time printf.



There is nice assertCTFEable written by Kenji in Phobos. I think it's 
our private magic for now but I see no reason not to expose it somewhere.



Atila



--
Dmitry Olshansky

Re: DConf 2014 Day 1 Talk 4: Inside the Regular Expressions in D by Dmitry Olshansky

[Dmitry Olshansky via Digitalmars-d-announce]

12-Jun-2014 03:29, Adam D. Ruppe пишет:

On Wednesday, 11 June 2014 at 18:03:06 UTC, Atila Neves wrote:

I wish I'd taken the mic at the end, and 2 days later Adam D. Ruppe
said what I was thinking of saying: unit test and debug the CTFE
function at runtime and then use it at compile-time when it's ready
for production.


Aye. It wasn't long ago that this wasn't really possible because of how
incomplete and buggy CTFE was, you kinda had to do it with special code,
but now so much of the language works, there's a good chance if it works
at runtime it will work at compile time too.

I was really surprised with CTFE a few months ago when I tried to use my
dom.d with it... and it actually worked. That's amazing to me.

But anyway, in general, the ctfe mixin stuff could be replaced with an
external code generator, so yeah that's the way I write them now - as a
code generator standalone thing then go back and enum it to actually
use. (BTW I also like to generate fairly pretty code, e.g. indentend
properly, just because it makes it easier to read.)



This one thing I'm loosing sleep over - what precisely is so good in 
CTFE code generation in _practical_ context (DSL that is quite stable, 
not just tiny helpers)?


By the end of day it's just about having to write a trivial line in your 
favorite build system (NOT make) vs having to wait for a couple of 
minutes each build hoping the compiler won't hit your system's memory 
limits.


And these couple of minutes are more like 30 minutes at a times. Worse 
yet unlike proper build system it doesn't keep track of actual changes 
(same regex patterns get recompiled over and over), at this point 
seamless integration into the language starts felling like a joke.


And speaking of seamless integration: just generate a symbol name out of 
pattern at CTFE to link to later, at least this much can be done 
relatively fast. And voila even the clunky run-time generation is not 
half-bad at integration.


Unless things improve dramatically CTFE code generation + mixin is just 
our funny painful toy.


--
Dmitry Olshansky

Re: DConf 2014 Day 1 Talk 4: Inside the Regular Expressions in D by Dmitry Olshansky

[Dmitry Olshansky via Digitalmars-d-announce]

12-Jun-2014 16:25, Dicebot пишет:

On Thursday, 12 June 2014 at 09:17:45 UTC, Dmitry Olshansky wrote:

This one thing I'm loosing sleep over - what precisely is so good in
CTFE code generation in _practical_ context (DSL that is quite stable,
not just tiny helpers)?

By the end of day it's just about having to write a trivial line in
your favorite build system (NOT make) vs having to wait for a couple
of minutes each build hoping the compiler won't hit your system's
memory limits.


Oh, this is a very good question :) There are two unrelated concerns here:



It's always nice to ask something on D NG, so many good answers I can 
hardly choose whom to reply ;) So this is kind of broadcast.


Yes, the answer seems spot on - reflection! But allow me to retort.

I'm not talking about completely stand-alone generator. Just as well 
generator tool could be written in D using the same exact sources as 
your D program does. Including the static introspection and 
type-awareness. Then generator itself is a library + an invocation 
script in D.


The Q is specifically of CTFE in this scenario, including not only 
obvious shortcomings of design, but fundamental ones of compilation 
inside of compilation. Unlike proper compilation is has nothing 
persistent to back it up. It feels backwards, a bit like C++ TMP but, of 
course, much-much better.



1)

Reflection. It is less of an issue for pure DSL solutions because those
don't provide any good reflection capabilities anyway, but other code
generation approaches have very similar problems.

By doing all code generation in separate build step you potentially lose
many of guarantees of keeping various parts of your application in sync.



Use the same sources for the generator. In essence all is the same, just 
relying on separate runs and linkage, not mixin. Necessary hooks to 
link to later could indeed be generated with a tiny bit of CTFE.


Yes, deeply embedded stuff might not be that easy. The scope and damage 
is smaller though.



2)

Moving forward. You use traditional reasoning of DSL generally being
something rare and normally stable. This fits most common DSL usage but
tight in-language integration D makes possible brings new opportunities
of using DSL and code generation casually all other your program.



Well, I'm biased by heavy-handed ones. Say I have a (no longer) secret 
plan of doing a next-gen parser generator in D. Needless to say swaths 
of non-trivial code generation. I'm all for embedding nicely but I see 
very little _practical_ gains in CTFE+mixin here EVEN if CTFE wouldn't 
suck. See the point above about using the same metadata and types as the 
user application would.



I totally expect programming culture to evolve to the point where
something like 90% of all application code is being generated in typical
project. D has good base for promoting such paradigm switch and reducing
any unnecessary mental context switches is very important here.

This was pretty much the point I was trying to make with my DConf talk (
and have probably failed :) )


I liked the talk, but you know ... 4th or 5th talk with CTFE/mixin I 
think I might have been distracted :)


More specifically this bright future of 90%+ concise DSL driven programs 
is undermined by the simple truth - no amount of improvement in CTFE 
would make generators run faster then optimized standalone tool 
invocation. The tool (library written in D) may read D metadata just fine.


I heard D builds times are important part of its adoption so...




And these couple of minutes are more like 30 minutes at a times. Worse
yet unlike proper build system it doesn't keep track of actual changes
(same regex patterns get recompiled over and over), at this point
seamless integration into the language starts felling like a joke.

And speaking of seamless integration: just generate a symbol name out
of pattern at CTFE to link to later, at least this much can be done
relatively fast. And voila even the clunky run-time generation is not
half-bad at integration.

Unless things improve dramatically CTFE code generation + mixin is
just our funny painful toy.


Unfortunately current implementation of frontend falls behind language
capabilities a lot. There are no fundamental reasons why it can't work
with better compiler.


It might solve most of _current_ problems, but I foresee fundamental 
issues of no global state in CTFE that in say 10 years from now would 
look a lot like `#include` in C++. A major one is there is no way for 
compiler to not recompile generated code as it has no knowledge of how 
it might have changed from the previous run.



In fact, deadlnix has made a very good case for
SDC taking over as next D frontend exactly because of things like CTFE JIT.


Yeah, we ought to help him!

--
Dmitry Olshansky

Re: Embarrassment of riches: another talk came online today

[Dmitry Olshansky via Digitalmars-d-announce]

10-Jun-2014 20:30, Andrei Alexandrescu пишет:

Leverage - my talk at Lang.NEXT.



Loved the title at first site and indeed the talk was great.
It's definitely something I'd show people to get them into D, honest and 
intriguing.


Seems ironic to say that D has no legacy baggage compared to C++ and 
then have a readily served self-defeat with the goofy 10. and .1 being 
supported for the sake of compatibility with C :)


--
Dmitry Olshansky

Re: dmd front end now switched to Boost license

[Dmitry Olshansky via Digitalmars-d-announce]

13-Jun-2014 04:31, Walter Bright пишет:

https://github.com/D-Programming-Language/dmd/pull/3655


It's probably nice to have less restrictive license, but what we aim to 
achieve with that?


Make commercial companies contribute to DMD more freely?
There is no problem even with GPL.
Let them build and sell their own products out of DMDFE?
	Highly unlikely to be a profitable anyway, and we'd better get back the 
patches.



--
Dmitry Olshansky

Re: dmd front end now switched to Boost license

[Dmitry Olshansky via Digitalmars-d-announce]

14-Jun-2014 04:46, Walter Bright пишет:

On 6/13/2014 4:31 AM, Dmitry Olshansky wrote:

It's probably nice to have less restrictive license, but what we aim
to achieve
with that?




I do not want to come across as rude but from pragmatic standpoint it's 
not interesting. I'm not opposing it (after all I agreed to change it), 
I just don't see any valuable gains.



1. Boost is the least restrictive license


This gains nothing in and by itself. 4 speaks of potential adv, which 
realistically is not something we desperately want. Maybe as a proactive 
move, that I could understand.




2. Minimize friction for adopting D


Let's not deluge ourselves, it does nothing to do that unlike many other 
things. Changing license of G++ frontend to boost won't make people 
adopt C++ any faster.


The only place of friction is backend, and opening FE for commerce 
doesn't help it.



3. Harmonization with usage of Boost in the runtime library



In other words simplify licensing, but again compiler and runtime 
library do not have to have anything in common. There is no issue to 
begin with.



4. Allow commercial use of DMDFE (so what if someone does? It'll drive
even more adoption of D!)


The only strictly valid point. Making commercial compilers and tools on 
D front-end is the only solid result this move enables.



5. Boost is well known and accepted


All of licenses are well known. Again by itself it's not interesting, it 
won't make dmd any more easy to get into FOSS distros.


--
Dmitry Olshansky

Re: DConf 2014 Day 1 Talk 4: Inside the Regular Expressions in D by Dmitry Olshansky

[Dmitry Olshansky via Digitalmars-d-announce]

14-Jun-2014 19:05, Dicebot пишет:

On Thursday, 12 June 2014 at 16:42:38 UTC, Dmitry Olshansky wrote:

[snip]

Well, I'm biased by heavy-handed ones. Say I have a (no longer) secret
plan of doing a next-gen parser generator in D. Needless to say swaths
of non-trivial code generation. I'm all for embedding nicely but I see
very little _practical_ gains in CTFE+mixin here EVEN if CTFE wouldn't
suck. See the point above about using the same metadata and types as
the user application would.


Consider something like REST API generator I have described during
DConf. There is different code generated in different contexts from same
declarative description - both for server and client. Right now simple
fact that you import very same module from both gives solid 100%
guarantee that API usage between those two programs stays in sync.


But let's face it - it's a one-time job to get it right in your favorite 
build tool. Then you have fast and cached (re)build. Comparatively costs 
of CTFE generation are paid in full during _each_ build.



In your proposed scenario there will be two different generated files
imported by server and client respectively. Tiny typo in writing your
build script will result in hard to detect run-time bug while code
itself still happily compiles.


Or a link error if we go a hybrid path where the imported module is 
emitting declarations/hooks via CTFE to be linked to by the proper 
generated code. This is something I'm thinking that could be a practical 
solution.


I.e. currently to get around wasting cycles again and again:

module a;
bool verify(string s){
  static re = ctRegex!; return match(s, re);
}
//
module b;
import a;
void foo(){
...
verify(blah);
...
}

vs would-be hybrid approach:

module gen_re;

void main() //or wrap it in tiny template mixin
{
generateCtRegex(
//all patterns
);
}

module b;
import std.regex;
//notice no import of a

void foo(){
...
static re = ctRegex!(...); //
...
}
and using ctRegex as usual in b, but any miss of compiled cache would 
lead to a link error.


In fact it might be the best of both worlds if there is a switch to try 
full CTFE vs link-time external option.




You may keep convenience but losing guarantees hurts a lot. To be able
to verify static correctness of your program / group of programs type
system needs to be aware how generated code relates to original source.


Build system does it. We have this problem with all of external deps 
anyway (i.e. who verifies the right version of libXYZ is linked not some 
other?)



Also this approach does not scale. I can totally imagine you doing it
for two or three DSL in single program, probably even dozen. But
something like 100+?


Not everything is suitable, of course. Some stuff  is good only inline 
and on spot. But it does use the same sources, it may look a lot like 
this in case of REST generators:


import everything;

void main(){
foreach(m; module){
//... generate client code from meta-data
}
}

Waiting for 100+ DSL compiled in a JIT interpreter that can't optimize a 
thing (pretty much by definition or use separate flags for that?) is not 
going to be fun too.



Huge mess to maintain. According to my experience
all builds systems are incredibly fragile beasts, trusting them
something that impacts program correctness and won't be detected at
compile time is just too dangerous.


Could be, but we have dub which should be simple and nice.
I had very positive experience with scons and half-generated sources.



I heard D builds times are important part of its adoption so...


Adoption - yes. Production usage - less so (though still important).
Difference between 1 second and 5 seconds is very important. Between 10
seconds and 1 minute - not so much.

JIT will be probably slower than stand-alone generators but not that
slower.


It might solve most of _current_ problems, but I foresee fundamental
issues of no global state in CTFE that in say 10 years from now
would look a lot like `#include` in C++.


I hope 10 years ago from now we will consider having global state in
RTFE stone age relict :P


Well, no amount of purity dismisses the point that a cache is a cache. 
When I say global in D I mean thread/fiber local.





A major one is there is no way for compiler to not recompile generated
code as it has no knowledge of how it might have changed from the
previous run.


Why can't we merge basic build system functionality akin to rdmd into
compiler itself? It makes perfect sense to me as build process can
benefit a lot from being semantically aware.


I wouldn't cross my fingers, but yes ideally it would need to have 
powers of a build system making it that much more complicated. Then it 
can cache results including templates instantiations across module and 
separate invocations of the tool. It's a distant dream though.


Currently available caching at the level of object files is very coarse 
grained and 

Re: dmd front end now switched to Boost license

[Dmitry Olshansky via Digitalmars-d-announce]

14-Jun-2014 22:03, Nick Sabalausky пишет:

On 6/14/2014 10:18 AM, Dmitry Olshansky wrote:

14-Jun-2014 04:46, Walter Bright пишет:


3. Harmonization with usage of Boost in the runtime library


In other words simplify licensing, but again compiler and runtime
library do not have to have anything in common. There is no issue to
begin with.



Uhh, *no*.

Scenario A:
--
Them: What license does D use?


Me: WAT? Language is not a product in itself. What license C++ use then?
In short, everything they care about was and is Boost.

--
Dmitry Olshansky

Re: DConf 2014 Day 1 Talk 4: Inside the Regular Expressions in D by Dmitry Olshansky

[Dmitry Olshansky via Digitalmars-d-announce]

15-Jun-2014 20:21, Dicebot пишет:

On Saturday, 14 June 2014 at 16:34:35 UTC, Dmitry Olshansky wrote:

But let's face it - it's a one-time job to get it right in your
favorite build tool. Then you have fast and cached (re)build.
Comparatively costs of CTFE generation are paid in full during _each_
build.


There is no such thing as one-time job in programming unless you work
alone and abandon any long-term maintenance. As time goes any mistake
that can possibly happen will inevitably happen.


The frequency of such event is orders of magnitude smaller. Let's not 
take arguments to supreme as then doing anything is futile due to the 
potential of mistake it introduces sooner or later.



In your proposed scenario there will be two different generated files
imported by server and client respectively. Tiny typo in writing your
build script will result in hard to detect run-time bug while code
itself still happily compiles.


Or a link error if we go a hybrid path where the imported module is
emitting declarations/hooks via CTFE to be linked to by the proper
generated code. This is something I'm thinking that could be a
practical solution.

snip


What is the benefit of this approach over simply keeping all ctRegex
bodies in separate package, compiling it as a static library and
referring from actual app by own unique symbol? This is something that
can does not need any changes in compiler or Phobos, just matter of
project layout.


Automation. Dumping the body of ctRegex is manual work after all, 
including putting it with the right symbol. In proposed scheme it's just 
a matter of copy-pasting a pattern after initial setup has been done.



It does not work for more complicated cases were you actually need
access to generated sources (generate templates for example).


Indeed, this is a limitation, and the import of generated source would 
be required.



You may keep convenience but losing guarantees hurts a lot. To be able
to verify static correctness of your program / group of programs type
system needs to be aware how generated code relates to original source.


Build system does it. We have this problem with all of external deps
anyway (i.e. who verifies the right version of libXYZ is linked not
some other?)


It is somewhat worse because you don't routinely change external
libraries, as opposed to local sources.



But surely we have libraries that are built as separate project and are 
external dependencies, right? There is nothing new here except that 
d--obj--lib file is changed to generator--generated D file---obj 
file.



Huge mess to maintain. According to my experience
all builds systems are incredibly fragile beasts, trusting them
something that impacts program correctness and won't be detected at
compile time is just too dangerous.


Could be, but we have dub which should be simple and nice.
I had very positive experience with scons and half-generated sources.


dub is terrible at defining any complicated build models. Pretty much
anything that is not single step compile-them-all approach can only be
done via calling external shell script.


I'm not going to like dub then ;)


If using external generators is
necessary I will take make over anything else :)


Then I understand your point about inevitable mistakes, it's all in the 
tool.



snip


tl; dr: I believe that we should improve compiler technology to achieve
same results instead of promoting temporary hacks as the true way to do
things. Relying on build system is likely to be most practical solution
today but it is not solution I am satisfied with and hardly one I can
accept as accomplished target.
Imaginary compiler that continuously runs as daemon/service, is capable
of JIT-ing and provides basic dependency tracking as part of compilation
step should behave as good as any external solution with much better
correctness guarantees and overall user experience out of the box.


What I want to point out is to not mistake goals and the means to an 
end. No matter how we call it CTFE code generation is just a means to an 
end, with serious limitations (especially as it stands today, in the 
real world).


Seamless integration is not about packing everything into single 
compiler invocation:


dmd src/*.d

Generation is generation, as long as it's fast and automatic it solves 
the problem(s) meta programming was established to solve.


For instance if D compiler allowed external tools as plugins (just an 
example to show means vs ends distinction) with some form of the 
following construct:


mixin(call_external_tool(args, 3, 14, 15, .92));

it would make any generation totally practical *today*. This was 
proposed before, and dismissed out of fear of security risks, never 
identifying the proper set of restrictions. After all we have textual 
mixins of potential security risk no problem.


Let's focus on the facts that this has the benefits of:
- sane debugging of the plug-in (it's just a program with the usual symbols)
- fast, as the 

Re: DConf 2014 Lightning Talks

[Dmitry Olshansky via Digitalmars-d-announce]

22-Jul-2014 03:20, Dicebot пишет:

Brian WAT slides were absolutely brilliant, I have totally enjoyed
watching those for the second time now :)


Yes, his talk is one of my favorites. So much fun in so little time :)

--
Dmitry Olshansky

Re: Fix #2529: explicit protection package #3651

[Dmitry Olshansky via Digitalmars-d-announce]

On Tuesday, 19 August 2014 at 17:08:25 UTC, Walter Bright wrote:

On 8/19/2014 7:01 AM, Dicebot wrote:
 Walter, now that release is out can you please state your
opinion about
 https://github.com/D-Programming-Language/dmd/pull/3651 ? It
is blocking Phobos
 module split and decoupling.

I keep thinking there's gotta be a way to do this without 
language changes.


For what it's worth I don't care how things turn out w.r.t. 
syntax, language extension or convention but the problem must be 
solved.



The way I see the problem:

There must be nice and clean way to have internal stuff that is 
invisible for user. Preferably this stuff must not be monolithic 
and have form of package.


Example structure:
mod/package.d
mod/internal/bar.d
mod/...

I actually face it in Phobos right now, my blocked pull:
https://github.com/D-Programming-Language/phobos/pull/2412

Options:

1. Use current package attribute.
Disadvantages:
a) Must retain internals in the same package, user may still 
import them no problem at all. At least it does not do much harm 
as nothing is accessible.
b) Can't use std.mod.internal.bar 'package' stuff in std.mod 
(upper one). That makes it next to unusable. Basically it means 
we have to push internals up, instead of pushing them down.


2. Use separate public internal package.
Main disadvantage:
a) Now user may import and actually USE internal stuff.
b) Not only that but also internal methods of equally internal 
structs and classes leak out. In other words: a struct defined in 
internal module may never be exposed without some kind of 
wrapping. Ugly and unnecessary complication.



I'd rather have a language feature, but whatever you guys invent 
as 3rd option that doesn't suck will have my vote.



---
Dmitry Olshansky

Re: Fix #2529: explicit protection package #3651

[Dmitry Olshansky via Digitalmars-d-announce]

23-Aug-2014 12:57, Kagamin пишет:

On Friday, 22 August 2014 at 15:06:14 UTC, Dmitry Olshansky wrote:

Example structure:
mod/package.d
mod/internal/bar.d
mod/...

I actually face it in Phobos right now, my blocked pull:
https://github.com/D-Programming-Language/phobos/pull/2412


I think, it should have structure

std.regex.package;
std.regex.backtracking;
std.regex.ir;

What is the reason to have regex stuff in a remote directory? That would
make things hard to find and navigate.


Yes, that was my initial pull. The stuff inside was marked as `package`, 
as only package.d currently has the same public API.


Then Dicebot suggested that since all modules are for the moment 
internal they have to be moved somewhere until public API is fleshed out 
(as it may change module name etc.).


2 options that both do NOT currently work:
std.regex.internal
std.internal.regex

--
Dmitry Olshansky

Re: Fix #2529: explicit protection package #3651

[Dmitry Olshansky via Digitalmars-d-announce]

23-Aug-2014 13:33, Kagamin пишет:

On Saturday, 23 August 2014 at 09:14:13 UTC, Dmitry Olshansky wrote:

Yes, that was my initial pull. The stuff inside was marked as
`package`, as only package.d currently has the same public API.

Then Dicebot suggested that since all modules are for the moment
internal they have to be moved somewhere until public API is fleshed
out (as it may change module name etc.).

2 options that both do NOT currently work:
std.regex.internal
std.internal.regex


If the user relies that a symbol is found in
std.internal.regex.backtracking, moving it will break the code too. So
what's the difference?


I don't see the difference. In any case user should see public aliases 
and never know where they lead to. Dicebot - any objections?


However since my pull hang there for months, I decide to appease 
puristic requests and move things to internals, only to find out that it 
doesn't work.


For what its worth I'd _prefer_ std.regex.internal as in my opinion 
internals got to be well contained. But as discussed, `package` doesn't 
work this way.



--
Dmitry Olshansky

Re: Multiple alias this is coming.

[Dmitry Olshansky via Digitalmars-d-announce]

19-Sep-2014 02:16, bearophile пишет:

IgorStepanov:


Do you ask about alias this or about it multiple usage. Multiple usage
is similar to single, but multiple:)


I meant the multiple usage. And none of your examples here are use cases
:-( I'd like to see one use case, or more.



For instance one could implement OOP and polymorphism on structs, all in 
library code.


struct Base {   }

struct Itnerface {  }

struct Derived{
Base _super;
Interface _iface;
alias this _super;
alias this _iface;
}

I recall trying something like that but stopping because I needed 
multiple alias this.



--
Dmitry Olshansky

gchunt v0.1.0 is out!

[Dmitry Olshansky via Digitalmars-d-announce]

gchunt is a tool is to help D developers identify and keep in check the 
usage of GC in their projects.


So far it just postprocesses D compiler's -vgc output into a nice Wiki 
table. Results looks like this (Phobos):

http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage#Labeled_data

Notice the source links grouped for each allocation reason per function. 
In short: it's a ready to use spread-sheet for reviewers to work through.


Release notes  Windows binary:
https://github.com/DmitryOlshansky/gchunt/releases/tag/v0.1.0

Past discussions and the origin of the tool:
http://forum.dlang.org/thread/pzyzqwpqdllegxnys...@forum.dlang.org#post-m11hhb:2423nf:241:40digitalmars.com

Happy GC hunting !

--
Dmitry Olshansky

Re: gchunt v0.1.0 is out!

[Dmitry Olshansky via Digitalmars-d-announce]

12-Nov-2014 13:32, Kagamin пишет:

https://github.com/D-Programming-Language/phobos/blob/271c771a57764dcc511ca12ae91d490872d9b500/std/base64.d#L1144


Probably should be an assert?


I'd bet it should. Ranges typically don't `enforce`.

--
Dmitry Olshansky

Re: gchunt v0.1.0 is out!

[Dmitry Olshansky via Digitalmars-d-announce]

21-Nov-2014 05:18, Piotr Szturmaj пишет:

W dniu 2014-11-11 o 23:38, Dmitry Olshansky pisze:

gchunt is a tool is to help D developers identify and keep in check the
usage of GC in their projects.

So far it just postprocesses D compiler's -vgc output into a nice Wiki
table. Results looks like this (Phobos):
http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage#Labeled_data


Nice :)

I think I found a -vgc bug:
https://github.com/D-Programming-Language/phobos/blob/271c771a57764dcc511ca12ae91d490872d9b500/std/array.d#L419
- this line doesn't allocate, unlike the one below.



Can you reduce and file it? Thanks!

--
Dmitry Olshansky

Re: On the meaning of string.length

[Dmitry Olshansky via Digitalmars-d-announce]

20-Nov-2014 16:50, Adam D. Ruppe пишет:

On Wednesday, 19 November 2014 at 21:00:50 UTC, Ary Borenszweig wrote:

In Ruby `length` returns the number of unicode characters


What is a unicode character? Even in utf-32, one printed character might
be made up of two unicode code points. Or sometimes, two printed
characters might come from a single code point.


Perl goes for grapheme cluster as character. I'd say that's probably the 
closest thing to it.


Sadly being systems language we can't go so far as to create a per 
process table of cached graphemes, and then use index in that table as 
character ;)


--
Dmitry Olshansky

Re: D is for Data Science

[Dmitry Olshansky via Digitalmars-d-announce]

25-Nov-2014 00:34, weaselcat пишет:

On Monday, 24 November 2014 at 15:27:19 UTC, Gary Willoughby wrote:

Just browsing reddit and found this article posted about D.
Written by Andrew Pascoe of AdRoll.

From the article:
The D programming language has quickly become our language of choice
on the Data Science team for any task that requires efficiency, and is
now the keystone language for our critical infrastructure. Why?
Because D has a lot to offer.

Article:
http://tech.adroll.com/blog/data/2014/11/17/d-is-for-data-science.html



Quoting the article:

 One of the best things we can do is minimize the amount of memory 
we’re allocating; we allocate a new char[] every time we read a line.


This is wrong. byLine reuses buffer if its mutable which is the case 
with char[]. I recommend authors to always double checking hypothesis 
before stating it in article, especially about performance.


Observe:
https://github.com/D-Programming-Language/phobos/blob/master/std/stdio.d#L1660
https://github.com/D-Programming-Language/phobos/blob/master/std/stdio.d#L1652

And notice a warning about reusing the buffer here:

https://github.com/D-Programming-Language/phobos/blob/master/std/stdio.d#L1741


Reddit:
http://www.reddit.com/r/programming/comments/2n9gfb/d_is_for_data_science/



Why is File.byLine so slow?


Seems to be mostly fixed sometime ago. It's slower then straight fgets 
but it's not that bad.


Also nearly optimal solution using C's fgets with growable buffer is way 
simpler then outlined code in the article. Or we can mmap the file too.



Having to work around the standard library
defeats the point of a standard library.


Truth be told the most of slowdown should be in eager split, notably 
with GC allocation per line. It may also trigger GC collection after 
splitting many lines, maybe even many collections.


The easy way out is to use standard _splitter_ which is lazy and 
non-allocating.  Which is a _2-letter_ change, and still using nice 
clean standard function.


Article was really disappointing for me because I expected to see that 
single line change outlined above to fix the 80% of problem elegantly. 
Instead I observe 100+ spooky lines that needlessly maintain 3 buffers 
at the same time (how scientific) instead of growing single one to 
amortize the cost. And then a claim that's nice to be able to improve 
speed so easily.



--
Dmitry Olshansky

Re: D is for Data Science

[Dmitry Olshansky via Digitalmars-d-announce]

25-Nov-2014 01:28, bearophile пишет:

Dmitry Olshansky:


Why is File.byLine so slow?


Seems to be mostly fixed sometime ago.


Really? I am not so sure.

Bye,
bearophile


I too has suspected it in the past and then I tested it.
Now I test it again, it's always easier to check then to argue.

Two minimal programs
//my.d:
import std.stdio;

void main(string[] args) {
auto file = File(args[1], r);
size_t cnt=0;
foreach(char[] line; file.byLine()) {
cnt++;
}
}
//my2.d
import core.stdc.stdio;

void main(string[] args) {
char[] buf = new char[32768];
size_t cnt;
shared(FILE)* file = fopen(args[1].ptr, r);
while(fgets(buf.ptr, cast(int)buf.length, file) != null){
cnt++;
}
fclose(file);
}

In the below console session, log file - is my dmsg log replicated many 
times (34 megs total).


dmitry@Ubu64 ~ $ wc -l log
522240 log
dmitry@Ubu64 ~ $ du -hs log
34M log

# touch it, to have it in disk cache:
dmitry@Ubu64 ~ $ cat log  /dev/null

dmitry@Ubu64 ~ $ dmd my
dmitry@Ubu64 ~ $ dmd my2

dmitry@Ubu64 ~ $ time ./my2 log

real0m0.062s
user0m0.039s
sys 0m0.023s
dmitry@Ubu64 ~ $ time ./my log

real0m0.181s
user0m0.155s
sys 0m0.025s

~4 time in user mode, okay...
Now with full optimizations, ranges are very sensitive to optimizations:

dmitry@Ubu64 ~ $ dmd -O -release -inline  my
dmitry@Ubu64 ~ $ dmd -O -release -inline  my2
dmitry@Ubu64 ~ $ time ./my2 log

real0m0.065s
user0m0.042s
sys 0m0.023s
dmitry@Ubu64 ~ $ time ./my2 log

real0m0.063s
user0m0.040s
sys 0m0.023s

Which is 1:1 parity. Another myth busted? ;)

--
Dmitry Olshansky

Re: D is for Data Science

[Dmitry Olshansky via Digitalmars-d-announce]

25-Nov-2014 02:43, bearophile пишет:

Dmitry Olshansky:


Which is 1:1 parity. Another myth busted? ;)


 dmitry@Ubu64 ~ $ time ./my2 log

 real0m0.065s
 user0m0.042s
 sys0m0.023s
 dmitry@Ubu64 ~ $ time ./my2 log

 real0m0.063s
 user0m0.040s
 sys0m0.023s


Read the above more carefully.
OMG. I really need to watch my fingers, and double-check:)

dmitry@Ubu64 ~ $ time ./my log

real0m0.156s
user0m0.130s
sys 0m0.026s

dmitry@Ubu64 ~ $ time ./my2 log

real0m0.063s
user0m0.040s
sys0m0.023s

Which is quite bad. Optimizations do help but not much.



There is still an open bug report:
https://issues.dlang.org/show_bug.cgi?id=11810

Do you want also to benchmark that byLineFast that for me is usually
significantly faster than the byLine?



And it seems like byLineFast is indeed fast.

dmitry@Ubu64 ~ $ time ./my3 log

real0m0.056s
user0m0.031s
sys 0m0.025s
dmitry@Ubu64 ~ $ time ./my2 log

real0m0.065s
user0m0.041s
sys 0m0.024s


Now once I was destroyed the question is who is going to make a PR of this?

--
Dmitry Olshansky

Re: This Week in D, issue 2

[Dmitry Olshansky via Digitalmars-d-announce]

On 19-Jan-2015 20:05, Adam D. Ruppe wrote:

http://arsdnet.net/this-week-in-d/jan-18.html

http://www.reddit.com/r/programming/comments/2sy7lg/this_week_in_d_january_18_2015/


For those of you who saw the draft earlier, hit refresh to ensure you
aren't seeing a cached version.

RSS feed:
http://arsdnet.net/this-week-in-d/twid.rss

This week, we got new web style thanks to the folks in the other forum.
Tech speaking, it now serves gzipped files as a test of what I want to
see about putting on dlang.org too.

Email list will be coming next week and hopefully a move to dlang.org too.

Fixed a few bugs in my stats gathering too, hopefully we'll have all the
kinks worked out and on-time release next week!


Falsey values may include any struct with opCast(T:bool) or somehow 
indicate that it's going to be attempted, afterall this is common D idiom.


--
Dmitry Olshansky

Re: This Week in D #23 - Interview with Dmitry Olshansky, dmd beta, std.experimental.color

[Dmitry Olshansky via Digitalmars-d-announce]

On 29-Jun-2015 06:46, Adam D. Ruppe wrote:

http://arsdnet.net/this-week-in-d/jun-28.html


I should have probably said on the day one - AMA.

P.S. Thanks to Joakim for editing my stream of consciousness
into this tidy text ;)

--
Dmitry Olshansky

Re: This Week in D #23 - Interview with Dmitry Olshansky, dmd beta, std.experimental.color

[Dmitry Olshansky via Digitalmars-d-announce]

On 03-Jul-2015 01:45, Mathias Lang via Digitalmars-d-announce wrote:

Very interesting interview !
On the question about love/hate of D, there's this sentence: ' for,
works with any range or something that can be sliced to get a
range.'. Didn't you mean foreach ?



Absolutely.
Adam can we fix that ? Thanks!


2015-06-29 5:46 GMT+02:00 Adam D. Ruppe via Digitalmars-d-announce
digitalmars-d-announce@puremagic.com
mailto:digitalmars-d-announce@puremagic.com:

http://arsdnet.net/this-week-in-d/jun-28.html





--
Dmitry Olshansky

Re: perpetual: new module for mapping file into memory

[Dmitry Olshansky via Digitalmars-d-announce]

On 18-Aug-2015 07:45, Rikki Cattermole wrote:

What I don't like:
- Does not allow for calling the constructor on the type should it have one


Hm it depends - one thing is that after mmaping a file the type may 
already be constructed, if it wasn't then emplacement should be the 
right thing to do. That is there should be a default/string value.



- Does not map to std.mmfile in any way


Which is a good thing. std.mmfile is crappy 8 year old ported crap.


- No way to say an offset in the file to


That may be useful.

 - You implement basically everything in std.mmfile all over again

Low-level API might be in order but it's not std.mmfile which is not 
low-level enough. Separating out a set of portable mmaping primitives 
would be awesome.


What I don't like is perpetual being a class - too bad as it won't honor 
scoping and destructor may or may not be called. Just make it a struct.


Also type should be Perputal, then perpetual is helper function that 
creates it then by means of IFTI type may be deduced. I suggest the 
following API:


// all parameters after file are either T itself
auto p = perpetual(file-name, 42);// would deduce T=int

// or args to T:
struct RGB{ int r,g,b; }
// if not in file use RGB(1,2,3)
auto p = perpetual!RGB(file-name, 1, 2, 3);

--
Dmitry Olshansky

Re: perpetual: new module for mapping file into memory

[Dmitry Olshansky via Digitalmars-d-announce]

On 18-Aug-2015 17:32, Sergei Degtiarev wrote:

On Tuesday, 18 August 2015 at 06:52:47 UTC, Dmitry Olshansky wrote:

What I don't like is perpetual being a class - too bad as it won't
honor scoping and destructor may or may not be called. Just make it a
struct.

Why do you think the destructor may not be called? It is, and it syncs
memory and unmaps the file. Making perpetual a struct would require
implementing reference counting or disabling copying, and all such
staff. In my opinion, perpetual should definitely have reference
semantics, this is exactly a point where D shines.



Class is allocated on GC heap dtor is called on collection in that sense 
it is a finalizer. Just use struct if you don't need inheritance and/or 
do resource management.



--
Dmitry Olshansky

Re: D-Day for DMD is today!

[Dmitry Olshansky via Digitalmars-d-announce]

On 23-Aug-2015 08:17, Walter Bright wrote:

https://github.com/D-Programming-Language/dmd/pull/4923

We have made the switch from C++ DMD to D DMD!

Many, many thanks to Daniel Murphy for slaving away for 2.5 years to
make this happen. More thanks to Martin Nowak for helping shepherd it
through the final stages, and to several others who have pitched in on
this.

This is a HUGE milestone for us.

Much work remains to be done, such as rebasing existing dmd pull
requests. Thanks in advance for the submitters who'll be doing that. I
hope you aren't too unhappy about the extra work - it's in a good cause!



Congratulations!

Looks like my last excuse for not hacking on DMD evaporated.. Ouch ;)

--
Dmitry Olshansky

Re: This Week in D - dconf videos, objc and iOS support improving, interview with Etienne Cimon

[Dmitry Olshansky via Digitalmars-d-announce]

On 13-Jul-2015 07:05, Adam D. Ruppe wrote:

Another nice interview that Joakim did this week and some interesting
announcements about ios related support. And a long, pointless bikeshed
argument, ugh.

http://arsdnet.net/this-week-in-d/jul-12.html


Now that all the videos are out for dconf, in the next week or maybe
two, depending on how busy I am, I'll put what I wrote about it together
with the videos and see about doing a final dconf special edition.


I think archive pannel should list entries in reverse hronologicall 
order, else it would soon be hard to find the first few of the recent 
weekly issues.


--
Dmitry Olshansky

Re: dcrypt - a cryptography library written in D

[Dmitry Olshansky via Digitalmars-d-announce]

On 14-Jul-2015 10:07, Marc =?UTF-8?B?U2Now7x0eiI=?= schue...@gmx.net 
wrote:

On Monday, 13 July 2015 at 15:49:39 UTC, puzzlehawk wrote:

[snip]


Also, in 2.068 there will be an optional enum `blockSize` (in bits) for
the hash algorithms, for use with the newly added std.digest.hmac. Your
library has a runtime method for that. Isn't the block size always known
at compile time? If you could change it to an enum, your library would
be compatible with Phobos.

Maybe a similar interface should be used for the block ciphers?


Would be awesome to have similar API. Even most C API for hashes vs 
cyphers looks almost the same.


--
Dmitry Olshansky

Re: dcrypt - a cryptography library written in D

[Dmitry Olshansky via Digitalmars-d-announce]

On 13-Jul-2015 18:49, puzzlehawk wrote:

dcrypt is a cryptography library written entirely in D and a little bit
D inline assembler. The library is inspired by BouncyCastle.


I sure hope it's not the API part of BC.


Most
important symmetric algorithms are implemented but work on asymmetric
cryptography has not yet started.
dcrypt is not yet ready for daily use. There's a lot of work to be done
and I'll be glad to get some feedback or collaboration.

puzzlehawk



--
Dmitry Olshansky

Re: Moving forward with work on the D language and foundation

[Dmitry Olshansky via Digitalmars-d-announce]

On 24-Aug-2015 21:42, Andrei Alexandrescu wrote:

Hello everyone,


Following an increasing desire to focus on working on the D language and
foundation, I have recently made the difficult decision to part ways
with Facebook, my employer of five years and nine months.

Facebook has impacted my career and life very positively, and I am
grateful to have been a part of it for this long. The time has come for
me, however, to fully focus on pushing D forward. As sorry I am for
leaving a good and secure career behind, I am excited many times over
about the great challenges and opportunities going forward.


Thank you for making this tough choice for all of us. Combined with the 
recent DDMD announcement this weeks turns out to be a huge milestone for 
the D community.


It's sure getting only more exciting in the D world from now on.

--
Dmitry Olshansky

Re: Release D 2.069.0

[Dmitry Olshansky via Digitalmars-d-announce]

On 07-Nov-2015 16:25, Jonathan M Davis via Digitalmars-d-announce wrote:

On Thursday, November 05, 2015 02:38:01 Sebastiaan Koppe via 
Digitalmars-d-announce wrote:

On Wednesday, 4 November 2015 at 17:52:23 UTC, Dmitry Olshansky
wrote:

If host machine is x64 bit windows try setting large address
aware bit on the executable (there are tools to do that IRC),
would allow it to eat up to ~4 gigs.


What can we do when it eats up all 4 gigs? (Windows)




Getting 64-bit packaged should be simplest thing possible. Ask around on
D.learn maybe somebody could either show how to build it manually or at 
least have done so before.


Otherwise - the single biggest problem with memory is CTFE leaking 
memory like crazy. Second to that is templates.


IMHO enabling D's GC in the frontend is better way to fix leaking in the 
CTFE, but there are some issues with that (it segfaults if we enable GC).


Lastly compiler's frontend could use some improvement in general and now 
that it's written in D and I totally expect better memory management to 
come soon.



Phobos has been built in pieces for quite a while now
specifically because it requires too much memory to build it all at once
(std.algorithm alone requires a lot of memory thanks to all of the template
instantiations in its unit tests).



That's only true for unittests, it still builds the library in one go 
(in about 7 seconds for me) on windows.



--
Dmitry Olshansky

Re: Release D 2.069.0

[Dmitry Olshansky via Digitalmars-d-announce]

On 04-Nov-2015 18:25, Márcio Martins wrote:

On Wednesday, 4 November 2015 at 01:50:38 UTC, Martin Nowak wrote:

Glad to announce D 2.069.0.

http://dlang.org/download.html
http://downloads.dlang.org/releases/2.x/2.069.0/

This is the first release with a self-hosted dmd compiler and comes
with even more rangified phobos functions, std.experimental.allocator,
and many other improvements.

See the changelog for more details.
http://dlang.org/changelog/2.069.0.html

-Martin


Seems like memory usage went up by a significant amount, as I can no
longer compile our project on my Windows machine.

DMD 2.068.2 uses up to 1100MB of memory during our build and succeeds
DMD 2.069 used up to 1600MB of memory before crashing with "Error: out
of memory"

Are there any plans to release a dmd64.exe?


If host machine is x64 bit windows try setting large address aware bit 
on the executable (there are tools to do that IRC), would allow it to 
eat up to ~4 gigs.


--
Dmitry Olshansky

Re: Voting for std.experimental.allocator

[Dmitry Olshansky via Digitalmars-d-announce]

Yes


--
Dmitry Olshansky

Re: This Week in D #23 - Interview with Dmitry Olshansky, dmd beta, std.experimental.color

[Dmitry Olshansky via Digitalmars-d-announce]

On 11-Jul-2015 00:26, Joakim wrote:

On Friday, 10 July 2015 at 20:42:02 UTC, Dmitry Olshansky wrote:

On 10-Jul-2015 23:34, Joakim wrote:

On Thursday, 2 July 2015 at 10:26:36 UTC, Dmitry Olshansky wrote:

On 29-Jun-2015 06:46, Adam D. Ruppe wrote:

http://arsdnet.net/this-week-in-d/jun-28.html


I should have probably said on the day one - AMA.

P.S. Thanks to Joakim for editing my stream of consciousness
into this tidy text ;)


Looks like you have a question on reddit, not sure how he reached that
conclusion though:

https://www.reddit.com/r/programming/comments/3ck3ru/interview_with_dmitry_olshansky_author_of_ds/



Answered. Never knew it was there at all.


Oh, he's probably reacting to these two quotes:

In the end, it turned out that UTF decoding had become the bottleneck
and it's soon to be removed.
The key one is to remove decoding of UTF and match directly on the
encoded chars


On second thought should have said it like match directly on encoded 
characters _as if decoding_ them w/o actually going for decoded code 
point values. Still confusing I guess.


--
Dmitry Olshansky

Re: Voting for std.experimental.allocator

[Dmitry Olshansky via Digitalmars-d-announce]

On 11-Jul-2015 01:51, Andrei Alexandrescu wrote:

On 7/10/15 4:29 PM, Alex Parrill wrote:

On Friday, 10 July 2015 at 18:32:04 UTC, Andrei Alexandrescu wrote:

On 7/9/15 5:44 PM, Alex Parrill wrote:

Yes, but the mmap allocator on Windows needs to be fixed.


What is the issue with it? I recall I pulled something recently. --
Andrei


It leaks.
http://forum.dlang.org/post/itmcarskypkuospvf...@forum.dlang.org


I see. Could you or someone with Windows expertise create a PR for my
branch? -- Andrei


This one? :)
https://github.com/andralex/phobos/pull/17

--
Dmitry Olshansky

Re: This Week in D #23 - Interview with Dmitry Olshansky, dmd beta, std.experimental.color

[Dmitry Olshansky via Digitalmars-d-announce]

On 10-Jul-2015 23:34, Joakim wrote:

On Thursday, 2 July 2015 at 10:26:36 UTC, Dmitry Olshansky wrote:

On 29-Jun-2015 06:46, Adam D. Ruppe wrote:

http://arsdnet.net/this-week-in-d/jun-28.html


I should have probably said on the day one - AMA.

P.S. Thanks to Joakim for editing my stream of consciousness
into this tidy text ;)


Looks like you have a question on reddit, not sure how he reached that
conclusion though:

https://www.reddit.com/r/programming/comments/3ck3ru/interview_with_dmitry_olshansky_author_of_ds/


Answered. Never knew it was there at all.

--
Dmitry Olshansky

Re: Firs step of D/Objective-C merged

[Dmitry Olshansky via Digitalmars-d-announce]

On 11-Jul-2015 23:25, Jacob Carlborg wrote:

I just manged to get all tests passing on all platforms. Now the first
step of D/Objective-C has been merged [1][2]. The only thing that's
supported for now is calling Objective-C instance methods from D.

I want to give special thanks to Michel Fortin who did the original
implementation. I also want to thank Walter, Daniel and all others who
reviewed the code.

Thank you.



Awesome news. I admire your persistence :)


--
Dmitry Olshansky

Re: associative arrays with manual memory management

[Dmitry Olshansky via Digitalmars-d-announce]

On 24-Aug-2015 15:01, Ilya Yaroshenko wrote:

http://code.dlang.org/packages/aammm/~master

# aammm
Associative arrays with manual memory management

All enries and buckets would be dealocated and disposed by internal
implementation's destructor.
The destructor is called by garbage collector (by default).



Rox!


 Example
```D
 //std.experimental.allocator is included into `aammm`
 import std.experimental.allocator.mallocator;
 import aammm;

 auto a = AA!(string, int, shared Mallocator)(Mallocator.instance);


Sure hope a factory to do IFTI is available? So that the following works:

auto a  = aa!(string, int)(Mallocator.instance); // 3rd CT param is deduced


 a[foo] = 0;
 a.remove(foo); //dealocates and disposes the entry
 assert(a == null); // should not crash
```




--
Dmitry Olshansky

Re: Moving forward with work on the D language and foundation

[Dmitry Olshansky via Digitalmars-d-announce]

On 25-Aug-2015 23:04, bachmeier wrote:

On Tuesday, 25 August 2015 at 19:29:06 UTC, Daniel Kozák wrote:


I can't agree more. OK maybe I would add this
https://twitter.com/kozzi11/status/636190895856091136 ;-)


This is a big recent development for many:

https://github.com/DlangScience


I just hope our math experts will join this organization even if only to 
bump the numbers. Seeing a one-man shop for D science is kinda 
disappointing.


--
Dmitry Olshansky

Re: Moving forward with work on the D language and foundation

[Dmitry Olshansky via Digitalmars-d-announce]

On 26-Aug-2015 12:20, John Colvin wrote:

On Wednesday, 26 August 2015 at 05:51:06 UTC, Dmitry Olshansky wrote:

On 25-Aug-2015 23:04, bachmeier wrote:

On Tuesday, 25 August 2015 at 19:29:06 UTC, Daniel Kozák wrote:


I can't agree more. OK maybe I would add this
https://twitter.com/kozzi11/status/636190895856091136 ;-)


This is a big recent development for many:

https://github.com/DlangScience


I just hope our math experts will join this organization even if only
to bump the numbers. Seeing a one-man shop for D science is kinda
disappointing.


Not a one man shop. Ilya Yaroshenko is helping a lot, Lars T Kyllingstad
is on board (along with SciD) and there are many others who are getting
involved in some way.

Anyone else who's interested should come to
https://gitter.im/DlangScience/public and get involved in the
conversation there.


All of you guys should be displayed here on the right:
https://github.com/DlangScience

Else it's a very bad marketing.

--
Dmitry Olshansky

Re: D-Day for DMD is today!

[Dmitry Olshansky via Digitalmars-d-announce]

On 02-Sep-2015 16:23, Rory McGuire via Digitalmars-d-announce wrote:

On Wed, Sep 2, 2015 at 12:37 PM, Dmitry Olshansky via
Digitalmars-d-announce <digitalmars-d-announce@puremagic.com
<mailto:digitalmars-d-announce@puremagic.com>> wrote:


Because the compiler and library releases go in lockstep we'd have
to wait for another release cycle to even field-test @nogc. Not
acceptable really.

Phobos ends up battle testing every new feature and if it wasn't new
features will be definitely shipped in a broken state. Let's not
forget that new D frontend features still have many bugs that stay
undetected even after testing in Phobos.

--
Dmitry Olshansky


Surely the compiler should always rely on the previous compiler +
standard library versions.

Is there really a valid reason not to do this? Surely if the dog food is
so bad no one should be eating?


Anyhow I'd let compiler team decide this.

--
Dmitry Olshansky

Re: D-Day for DMD is today!

[Dmitry Olshansky via Digitalmars-d-announce]

On 02-Sep-2015 07:52, Iain Buclaw via Digitalmars-d-announce wrote:

On 2 Sep 2015 5:31 am, "Jonathan M Davis via Digitalmars-d-announce"
> wrote:
 >
 > On Tuesday, September 01, 2015 09:44:17 Steven Schveighoffer via
Digitalmars-d-announce wrote:
 > > On 9/1/15 6:48 AM, "Luís Marques  > wrote:
 > > > On Sunday, 23 August 2015 at 05:17:33 UTC, Walter Bright wrote:
 > > >> We have made the switch from C++ DMD to D DMD!
 > > >
 > > > Is there a rough prediction of when the use of phobos in ddmd
will start
 > > > to be accepted?
 > >
 > > I'm not a dmd dev, but I'm not sure it will be accepted, since
phobos is
 > > very unstable. We have to be cautious about making dmd breakable easily
 > > by a change to phobos.
 > >
 > > Of course, I think there is a baseline dmd/gdc/ldc that must be used to
 > > build dmd, so perhaps as long as you use phobos features that work
 > > there, it will be OK.
 >
 > Plenty of Phobos is stable and hasn't changed in quite a while. We do
 > sometimes deprecate stuff still, but there isn't much that gets
deprecated
 > at this point, and the deprecation cycle is about two years. The common
 > problem would be regressions, and the compiler gets those as much or more
 > often than Phobos does. But it is true that some stuff in Phobos changes
 > occasionally, and that could affect how new a compiler you need to
compile
 > the current dmd.
 >

Don't forget, Phobos library maintainers love to use new features
everywhere.  Tagging many functions with @nogc almost weeks after it was
introduced in master was the last major backwards breaking change you
did that I'm aware of.



Because the compiler and library releases go in lockstep we'd have to 
wait for another release cycle to even field-test @nogc. Not acceptable 
really.


Phobos ends up battle testing every new feature and if it wasn't new 
features will be definitely shipped in a broken state. Let's not forget 
that new D frontend features still have many bugs that stay undetected 
even after testing in Phobos.


--
Dmitry Olshansky

Re: OneDrive Client written in D

[Dmitry Olshansky via Digitalmars-d-announce]

On 26-Sep-2015 11:31, Dmitry Olshansky wrote:

On 25-Sep-2015 22:37, skilion wrote:

On Friday, 25 September 2015 at 07:40:18 UTC, Dmitry Olshansky wrote:

Removing anonymous groups and using raw-literal:

regex(`^\s*(\w+)\s*=\s*"(.*)"\s*$`)

Looks simpler I guess.


Good advice, thanks.


Though if key="value" is expected I'd suggest to use lazy .* -
`"(.*?)"` to stop on first " if any.


After playing a bit with it, I realized that it doesn't work in this
case (a="a"a" still match). The right solution would be "(.[^"])"


I hope that was ([^"]*)

Yeah, .*? won't work because of '$' at the end of the pattern.
Anyway to allow classical escapes in string literal I'd go for:

`"(?:[^"]+|\")*"`



i.e. `"(?:[^"]+|\\")*"` to defeat regex escaping.

--
Dmitry Olshansky

Re: OneDrive Client written in D

[Dmitry Olshansky via Digitalmars-d-announce]

On 25-Sep-2015 22:37, skilion wrote:

On Friday, 25 September 2015 at 07:40:18 UTC, Dmitry Olshansky wrote:

Removing anonymous groups and using raw-literal:

regex(`^\s*(\w+)\s*=\s*"(.*)"\s*$`)

Looks simpler I guess.


Good advice, thanks.


Though if key="value" is expected I'd suggest to use lazy .* -
`"(.*?)"` to stop on first " if any.


After playing a bit with it, I realized that it doesn't work in this
case (a="a"a" still match). The right solution would be "(.[^"])"


I hope that was ([^"]*)

Yeah, .*? won't work because of '$' at the end of the pattern.
Anyway to allow classical escapes in string literal I'd go for:

`"(?:[^"]+|\")*"`

--
Dmitry Olshansky

Re: Scriptlike v0.9.4 - Perl-like interpolated strings, full examples and more.

[Dmitry Olshansky via Digitalmars-d-announce]

On 23-Sep-2015 09:30, Jacob Carlborg wrote:

On 2015-09-22 22:18, Nick Sabalausky wrote:

Big update to Scriptlike, v0.9.4:
https://github.com/Abscissa/scriptlike

Scriptlike is a library to help you write script-like programs in D.


One thing that really bugs me in Phobos, Scriptlike seems to have the
same problem, is that there are three (!!!) different functions to
remove something from the file system. Give me just one function that
removes everything, regardless if it's a file, directory and if it's
empty or not.



Bugzilla issue with this enhancement would help a lot with this ;)

And a PR would make it happen...

--
Dmitry Olshansky

Re: Go 1.5

[Dmitry Olshansky via Digitalmars-d-announce]

On 18-Sep-2015 23:46, Jack Stouffer wrote:

On Friday, 18 September 2015 at 19:26:27 UTC, Rory wrote:

[snip]

The reason Go has a better GC than D is that Go users have no choice but
to use the GC, while D users have a bunch more options.


To put it differently - D is a big language that has lots of things to 
improve/extend whereas Go is simplistic. This means the focus of the Go 
team is on run-time only for a while now, while we keep on improving the 
core language together with the druntime.


--
Dmitry Olshansky

Re: OneDrive Client written in D

[Dmitry Olshansky via Digitalmars-d-announce]

On 24-Sep-2015 16:22, Suliman wrote:

On Tuesday, 22 September 2015 at 20:43:32 UTC, skilion wrote:

I've been waiting for a good sync client for OneDrive (15 GB for
free!) on Linux, but Microsoft seems to have other plans...
So I've decided to write my own, using D. Take a look:

  http://skilion.github.io/onedrive/



What this string in config.d do?
  auto r = regex("(?:^\\s*)(\\w+)(?:\\s*=\\s*\")(.*)(?:\"\\s*$)");




Removing anonymous groups and using raw-literal:

regex(`^\s*(\w+)\s*=\s*"(.*)"\s*$`)

Looks simpler I guess.

Though if key="value" is expected I'd suggest to use lazy .* - `"(.*?)"` 
to stop on first " if any.


--
Dmitry Olshansky

Re: code-d 0.10.1 released (D support for vscode)

[Dmitry Olshansky via Digitalmars-d-announce]

On 11-Mar-2016 23:03, WebFreak001 wrote:

I just released a new version of code-d, it now supports projects
without any dub.json file which some people probably will like. This is
really useful for standalone projects with a custom build system like
writing an OS/Kernel or other projects that don't want to use dub.



It would really help to state first of all that code-d is a D extension 
for Visual Studio Code which is a nice thing even though I never tried it.



--
Dmitry Olshansky

gchunt v0.2.0

[Dmitry Olshansky via Digitalmars-d-announce]

New release of a tool that generates nice summary reports for static 
analysis of GC usage. It's a simple postprocessor for the compiler's 
-vgc output.


https://github.com/DmitryOlshansky/gchunt/releases

Updated to the latest compiler/libarires and also detects dup/idup 
unreported by -vgc switch.


--
Dmitry Olshansky

Re: Google Summer of Code

[Dmitry Olshansky via Digitalmars-d-announce]

On 23-Apr-2016 01:43, CraigDillabaugh wrote:

I am pleased to announce that the D Foundation has been awarded 4 slots
for the 2016 Google Summer of Code.

https://summerofcode.withgoogle.com/organizations/5078256051027968/

Congratulations to

Lodovico Giaretta
A replacement of std.xml for the Phobos standard library

Sebastian Wilzbach
Science for D - a non-uniform RNG

Jeremy DeHaan
Precise Garbage Collector

Wojciech Szęszoł
Improvements for dstep

on their successful proposals.


Congrats fellows!


--
Dmitry Olshansky

Re: LZ4 decompression at CTFE

[Dmitry Olshansky via Digitalmars-d-announce]

On 28-Apr-2016 21:31, deadalnix wrote:

On Thursday, 28 April 2016 at 17:58:50 UTC, Stefan Koch wrote:

On Thursday, 28 April 2016 at 17:29:05 UTC, Dmitry Olshansky wrote:


What's the benefit? I mean after CTFE-decompression they are going to
add weight to the binary as much as decompressed files.

Compression on the other hand might be helpful to avoid
precompressing everything beforehand.


The compiler can load files faster, that are being used by ctfe only.
Which would be stripped out by the linker later.
And keep in mind that it also works at runtime.

Memory is scarce at compiletime and this can help reducing the memory
requirements. When a bit of structure is added on top.


Considering the speed and memory consumption of CTFE, I'd bet on the
exact reverse.


Yeah, the whole CTFE to save compile-time memory sounds like a bad joke 
to me;)


Also, the damn thing is allocation in a loop.




--
Dmitry Olshansky

Re: GSoC 2016 - Precise GC

[Dmitry Olshansky via Digitalmars-d-announce]

On 06-May-2016 05:37, Jeremy DeHaan wrote:

On Wednesday, 4 May 2016 at 12:42:30 UTC, jmh530 wrote:

On Wednesday, 4 May 2016 at 02:50:08 UTC, Jeremy DeHaan wrote:


I'm not sure, but one would think that @safe code wouldn't need any
extra information about the union. I wouldn't know how to
differentiate between them though during runtime. Probably someone
with more experience with the compiler would know more about that
kind of thing.


You can identify safe functions with
https://dlang.org/phobos/std_traits.html#isSafe
or
https://dlang.org/phobos/std_traits.html#functionAttributes


All I meant was that I don't know enough about what the compiler does
with built in types to make this work. It almost sounds like we would
need a safe union and unsafe union type and do some extra stuff for the
unsafe union, but I'm just starting to learn about this stuff.


I'd note that a union without pointers doesn't hurt precise scanner, 
it's only the ones with pointers that are bad.


--
Dmitry Olshansky

Re: DIP: Tail call optimization

[Dmitry Olshansky via Digitalmars-d-announce]

On Sunday, 10 July 2016 at 05:03:46 UTC, Dietrich Daroch wrote:

Hi everyone (=

I've just added a new proposal to add a new attribute to ensure 
TCO is applied.


The proposal is really simple, but I'm clueless on how to 
implement it and also interested on getting feedback on it.




In contrast to what many folks expect, TCO is affecting program 
semantics in a way that changes stack overflow to normal 
execution.


Therefore it's not an optimization but part of semantics, and 
there should be a way to mark a call as a tail-call in any 
optimization level.


It was a big road block for me when I tried to implement 
threaded-code interpreter, because it will stack overflow in 
debug mode.




The proposal it's ready for merge on the new [DIPs 
repo](https://github.com/dlang/DIPs/pull/6)


--
Dietrich

Re: New (page-per-artifact) standard library doc examples are now editable and runnable

[Dmitry Olshansky via Digitalmars-d-announce]

On 2/17/17 6:06 AM, Seb wrote:

On Saturday, 7 January 2017 at 16:12:49 UTC, Andrei Alexandrescu wrote:

Following https://github.com/dlang/dlang.org/pull/1532, the new-style
docs now also allow editing and running examples. Start at
http://dlang.org/library-prerelease/ and go anywhere to check it out.



Broken on the first example I tried :(

http://dlang.org/library-prerelease/std/algorithm/searching/find.html

---
Dmitry Olshansky

Re: Questionnaire

[Dmitry Olshansky via Digitalmars-d-announce]

On 2/8/17 7:27 PM, Ilya Yaroshenko wrote:

1. Why your company uses  D?

  a. D is the best
  b. We like D
  c. I like D and my company allowed me to use D
  d. My head like D
  e. Because marketing reasons
  f. Because my company can be more efficient with D for some tasks then
with any other system language



Will probably stop on this question as the rest is not applicable.
I do not use D at my company and the reason is that any language to get 
a use need to pass stringent sorting criteria that I don't even 
appreciate fully. The language that have green light I believe are


C++, Go, Java, Python (under heavy pressure to switch to Go)

Anyhow when trying to sell D to any company the only case I can make a 
good offer is having these attributes:


1) It's a new project, not extension of an existing behemoth code base
2) Fairly unique - i.e. cannot be just a bunch of existing libraries 
glued together with some business logic.

3) Needs native performance at least in some areas of the project.

Now let's imagine a company considers technology X and I want to propose 
D instead. Let's look at possibilities:


C++ - they are not afraid of creating their own stack and 
performance-minded. This is probably the only case where selling D is 
easy. However these days selling them Rust would be much easier.


Rust - they value native speed, safety and afraid of GC. D's state of GC 
would only confirm their fears and D's safety is mostly opt-in/ relies 
on GC/not supported enough.


Java - they love VM safety and GC, most likely invested in Java 
ecosystem. Here the better sell would be Scala or Kotlin etc.


C# - they are probably hooked on MS technology and tooling 
(VisualStudio). The current state of D's IDE will make them cry like 
little babies, no selling here.


Go - they value simplicity and robust run-time (Go's GC breaks news with 
sub-milisecond pauses on large heaps). The sheer complexity of D is 
enough for it to be a hard sell, D's GC is coup de grace.


Scripting languages - they don't care for elaborate type systems and 
willing to trade performance for flexibility. Selling easy templates to 
them is like giving candies to kids with diabeties. Trying to lure with 
performance hits a brick wall because e.g. NodeJS/LuaJIT have fast JITs 
already and they don't care going beyond that level.


---
Dmitry Olshansky

Re: A New Import Idiom`

[Dmitry Olshansky via Digitalmars-d-announce]

On 2/14/17 3:32 AM, Jerry wrote:

Anyways yes this is kind of cool and fascinating how it works, but that
aside I hope I never see this used in phobos. Does anyone else feel this
way?


+1
Let's not make Phobos as scary as C++ STL.

---
Dmitry Olshansky

Re: Questionnaire

[Dmitry Olshansky via Digitalmars-d-announce]

On 2/11/17 5:04 AM, bachmeier wrote:

On Friday, 10 February 2017 at 23:02:38 UTC, Dmitry Olshansky wrote:

Go - they value simplicity and robust run-time (Go's GC breaks news
with sub-milisecond pauses on large heaps). The sheer complexity of D
is enough for it to be a hard sell, D's GC is coup de grace.


I have never understood the appeal of Go. With respect to the GC,
there's this:
https://blog.plan99.net/modern-garbage-collection-911ef4f8bd8e#.o6pxesvuw



Has nothing new to say, yes GO's GC fragments heap, is slower at 
allocation and adds "read/write barriers from hell". But it does 
optimize for short pauses, which in turn avoids ugly spikes in server 
workloads and that is priceless. I have had the pleasure of trying to 
"tune away" the GC spikes of Java cluster software - it's not pleasant.




With respect to "simplicity", I found it to be easy to learn a language
that makes it hard to get stuff done. I've never understood the argument
that programming in Go is simple. Clearly others have their own view.


I agree with your view on this one. Go puts both advanced and novice 
programmers on the same ground - both have to write dumb code with 
little to no abstraction. In my limited time with Go I found it futile 
to abstract away even the most trivial patterns such as map-reduce with 
concurrency.


---
Dmitry Olshansky

Re: Plugin for 1C:Enterprise in dmd

[Dmitry Olshansky via Digitalmars-d-announce]

On 1/18/17 1:00 PM, MGW wrote:

I managed to create plugin for 1C:Enterprise (http://1c.ru/eng/title.htm).
The major difficulty in realization C++ interfaces is successfully
overcome.
I need a more detailed description of compatibility of structures,
classes and interfaces between C++ and D.

For clarity you can see this short video
https://www.youtube.com/watch?v=apLppufZulI



OMG!
I mean I'm both frightened and excited of the opportunities this create. :)

---
Dmitry Olshansky

Re: Pry v0.3.1 is out!

[Dmitry Olshansky via Digitalmars-d-announce]

On 1/15/17 2:26 AM, Dmitry Olshansky wrote:

Pry is a new pragmatic parser combinators library.


[snip]

Two key areas of focus are (compared to say Pegged):
- performance, on par with hand-written code or die


Actually testing the latest version with LDC I found out that
handwritten code is a bit *slower*. Beats me, as I spent quite some time
laying out that handwritten stuff.

All in all, this makes me confident that I soon will never have to
write parsers by hand, the last nebulous reason is out.

---
Dmitry Olshansky

Re: Pry v0.3.1 is out!

[Dmitry Olshansky via Digitalmars-d-announce]

On 1/17/17 1:16 PM, Bastiaan Veelo wrote:

On Monday, 16 January 2017 at 22:29:01 UTC, Dmitry Olshansky wrote:

I think left-recursion is better handled at the grammar level.
What I currently have is parser combinators level where adding
this transformation is awkward and too much magic IMO.


Handling left-recursion by grammar transformation often has unwanted
side-effects (operator precedence) and eliminating indirect
left-recursion this way can be impossible in practice. Depending on the
complexity of the grammar, even identifying the recursive loop can be a
challenge.


I do not suggest to change the grammar itself, I think that processing 
of the grammar may perform hidden transformations internally.




The trouble is that one can be happily implementing a parser or
designing a grammar when suddenly for some input the parser hangs
indefinitely. Users are likely quick to blame the parser lib, while in
fact it is the grammar that has left-recursion. Hitting that roadblock
is a real bummer.

In some cases the grammar is a given (by a standard for example) and
transforming it to combat left-recursion can obfuscate it beyond
recognition. Hardening Pegged to deal with the various kinds of
left-recursion was very challenging, but easier than transforming the
grammar I was dealing with (ISO 10206).


Interesting, what kind of hardening?

---
Dmitry Olshansky

Re: Pry v0.3.1 is out!

[Dmitry Olshansky via Digitalmars-d-announce]

On 1/16/17 1:29 AM, Bastiaan Veelo wrote:

On Sunday, 15 January 2017 at 01:26:07 UTC, Dmitry Olshansky wrote:

Pry is a new pragmatic parser combinators library.

https://github.com/DmitryOlshansky/pry


Interesting. How about left-recursion? (I added support for
left-recursive grammars to Pegged.)


I think left-recursion is better handled at the grammar level.
What I currently have is parser combinators level where adding
this transformation is awkward and too much magic IMO.

However I plan to add a grammar on top of combinators, and yes handling 
left-recursive grammars is going to be an interesting challenge.


---
Dmitry Olshansky

Re: Pry v0.3.1 is out!

[Dmitry Olshansky via Digitalmars-d-announce]

On 1/17/17 11:52 PM, Bastiaan Veelo wrote:

On Sunday, 15 January 2017 at 01:26:07 UTC, Dmitry Olshansky wrote:

Two key areas of focus are (compared to say Pegged):
- performance, on par with hand-written code or die


I didn't profile, but apart from the time-complexity that is inherent to
straight forward recursive descent parsers like Pegged (improved by
memoization), my impression is that Pegged's main performance problem is
due to a very high constant: it is concatenating strings, all the time.
A LOT of strings, and a LOT of concatenations. Almost all of them are
discarded, of course. Finding a way to do the same without string
concatenation would have a big impact.


I considered to work with Pegged but I decided against it, because my 
priorities are different: get it fast then get it full of features.

IMHO performance is something to think of at design time not afterwards.

---
Dmitry Olshansky

Re: Optimizing std.regex

[Dmitry Olshansky via Digitalmars-d-announce]

On 11/8/16 12:23 PM, Chris wrote:

On Monday, 7 November 2016 at 13:13:24 UTC, Mike Parker wrote:

Dmitry mentioned here in the forums not long ago that he had squeezed
some big performance improvements out of std.regex. His post on the D
Blog describes how he managed to do it through an algorithmic
optimization.

The post:
https://dlang.org/blog/2016/11/07/big-performance-improvement-for-std-regex/


Reddit:
https://www.reddit.com/r/programming/comments/5bm1bc/from_the_d_blog_optimizing_stdregex/



Great job! Would it be possible (or indeed worth it) to create a similar
engine for general text processing / string handling?


Depends on what do you mean general - hard to find something more 
general then regex for text processing. Or do you mean using Bit NFA 
directly without the regex layer? It should be possible, however one 
needs to keep in mind the limitation - 32 or 64 states in total.


---
Dmitry Olshansky

Re: Battle-plan for CTFE

[Dmitry Olshansky via Digitalmars-d-announce]

On 10/16/16 2:27 AM, Uplink_Coder wrote:

On Wednesday, 5 October 2016 at 08:34:06 UTC, Rory McGuire wrote:


No worries, I've been watching this space for over a decade. I really
believe you are working on one of the most important parts of IT for
the next decade. I am planning/making a library that uses CTFE
extensively and feel much more confident about it purely because of
your work on getting CTFE performance to be a non-issue.

R


Little update here:
The LLVM backend is almost on feature parity.
Meaning that that soon the new CTFE engine is a real jit.
In the process I discoverd quite a few horrible bugs and inconsistency
in the API.
I am quite astonished that it ever ran before :)



That LLVM thing is surely nice to have but I highly doubt it will be 
allowed as dependency for DMD.


---
Dmitry Olshansky

Re: New team member: Lucia

[Dmitry Olshansky via Digitalmars-d-announce]

On 10/13/16 8:15 PM, Andrei Alexandrescu wrote:

Hello everyone,


Please join me in welcoming Lucia Lucia Cojocaru to our team. Lucia is a
MSc student in computer security, having Razvan Deaconescu and Razvan
Rughiniș as advisers. She just completed an internship at Bloomberg.

Her BSc thesis work[1] is an educational tool for facilitating better
understanding of dynamic linking and loading. The code is open source[2].

Lucia is interested in D-related projects for her MSc research.

The plan with our new team members Lucia and Alexandru (and hopefully
1-2 more) is to first get them through a month-long bootcamp process
during which they get familiar with the language and toolchain. An
integral part of the bootcamp is to get a few bugs fixed such that the
newcomers make some positive impact from the get-go and get used to the
review process.


Thanks,

Andrei

[1]
https://drive.google.com/file/d/0B_idW6n517Zfb3lLaGJJckp4Y0U/view?usp=sharing

[2] https://github.com/somzzz/dyninspector


Welcome aboard, Lucia!


Dmitry Olshansky

Pry v0.3.1 is out!

[Dmitry Olshansky via Digitalmars-d-announce]

Pry is a new pragmatic parser combinators library.

https://github.com/DmitryOlshansky/pry

(also available on Dub)

It's still in the early stages but I think it might be a good time to 
gauge some interest.


Two key areas of focus are (compared to say Pegged):
- performance, on par with hand-written code or die
- versatility, generating some goofy parse tree is not a goal, the goal 
is extraction of data the way the user specifies


For now it contains a modest example of a calculator app and a benchmark 
inspired by it. The interesting tidbit is that this version already 
includes an optimization that typically prevents recursive descent 
parser from falling into exponential behavior.


Future directions:
- actually add grammar layer on top of combinators, it is fancy but useful
- more built-in parsers to play with, including some stuff inspired by 
std.regex
- some goodies for usability, e.g. shortcuts to avoid constructing 
Stream out of string etc.


---
Dmitry Olshansky

Re: DConf 2017 Schedule

[Dmitry Olshansky via Digitalmars-d-announce]

On 3/15/17 11:07 PM, Bastiaan Veelo wrote:

On Wednesday, 15 March 2017 at 14:06:23 UTC, Yuxuan Shui wrote:

So someone already wrote a parser combinator for D?

I searched code.dlang.org (1.5 years ago?), and there was none, so I
wasted couple weeks writing my own


So, is yours on code.dlang.org? If it would have been, maybe Dmitry
wouldn't have to waste his time. Anyway, Pegged definitely was there
already.

:-)


The more the better ;)

See you at dconf.

---
Dmitry Olshansky

Re: Cap'n Proto for D v0.1.2

[Dmitry Olshansky via Digitalmars-d-announce]

On 4/18/17 9:14 PM, Swoorup Joshi wrote:

On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote:

"Cap’n Proto is an insanely fast data interchange format and
capability-based RPC system. Think JSON, except binary. Or think
Protocol Buffers, except faster."

This is the initial public release of my optimized port of the Java
implementation of Cap'n Proto.

State:
* Passes Cap'n Proto testsuite.
* Optimized. Just a little slower than the official C++ implementation
(see benchmarks on github).
* Missing RPC part of Cap'n Proto.

http://code.dlang.org/packages/capnproto-dlang
https://github.com/ThomasBrixLarsen/capnproto-dlang


Java?? Yikes


Risking a flamewar but what's wrong with Java?

---
Dmitry Olshansky

Re: Release D 2.075.0

[Dmitry Olshansky via Digitalmars-d-announce]

On Saturday, 22 July 2017 at 21:22:00 UTC, Walter Bright wrote:
On 7/22/2017 2:04 AM, Martin Nowak It'll be converted anyway. 
:-)






Putting the entire set in D (C compiler, C++ compiler, C 
preprocessor, htod converter, optimizer, code generator) makes 
the whole thing much more tractable, and who knows what we will 
be able to do with it!


Does that mean that DMC++ will hit Github?
I'm also interested in open-sourced version of snn library, of 
course.

Re: DCompute OpenCL kernels now work

[Dmitry Olshansky via Digitalmars-d-announce]

On Sunday, 8 October 2017 at 04:40:35 UTC, Nicholas Wilson wrote:
I am happy to announce that DCompute will soon[1] support the 
OpenCL 2.1 runtime. I have tested it locally and it works now 
:) I wasted a good deal of time wondering why it wasn't and it 
was a small typo in DerelictCL trying to load OpenCL 2.2 
symbols when loading 2.1.  That along with OpenCL 2.x support 
for DerelictCL will be merged and tagged soon and then [1] can 
be merged.


Most of the hard work is now done, leaving general polish and 
user feedback as the main development tasks. A unified driver 
abstracting over the CUDA and OpenCL drivers is on my todo list.


https://github.com/libmir/dcompute/pull/36


This is awesome! I have some ambitious plans which depend on this 
development.

Re: iopipe alpha 0.0.1 version

[Dmitry Olshansky via Digitalmars-d-announce]

On Monday, 16 October 2017 at 14:45:21 UTC, Steven Schveighoffer 
wrote:

On 10/12/17 8:41 AM, Steven Schveighoffer wrote:

On 10/12/17 1:48 AM, Dmitry Olshansky wrote:
On Thursday, 12 October 2017 at 04:22:01 UTC, Steven 
Schveighoffer wrote:

[...]


Might be able to help you on that using WinAPI for I/O. (I 
assume bypassing libc is one of goals).


That would be awesome! Yes, the idea is to avoid any "extra" 
buffering. So using CreateFile, ReadFile, etc.


Dmitry hold off on this if you were going to do it. I have been 
looking at Jason White's io library, and think I'm going to 
just extract all the low-level types he has there as a basic io 
library, as they are fairly complete, and start from there. His 
library includes the ability to use Windows.



Meh, not that I had mich spare time to actually do anything ;)

Might help by reviewing what you have there.

-Steve

Re: iopipe alpha 0.0.1 version

[Dmitry Olshansky via Digitalmars-d-announce]

On Thursday, 12 October 2017 at 04:22:01 UTC, Steven 
Schveighoffer wrote:
I added a tag for iopipe and added it to the dub registry so 
people can try it out.


I didn't want to add it until I had fully documented and 
unittested it.


http://code.dlang.org/packages/iopipe
https://github.com/schveiguy/iopipe

If you plan on using it, expect some API changes in the near 
future. I think the next step is really to add Windows support 
for the IODev type.


Might be able to help you on that using WinAPI for I/O. (I assume 
bypassing libc is one of goals).

Re: Intellij D Language v1.15.2

[Dmitry Olshansky via Digitalmars-d-announce]

On Thursday, 23 November 2017 at 20:11:01 UTC, singingbush wrote:
Hi all. A new release intellij-dlanguage plugin has been made 
available for download from the Jetbrains repository this week.


The speed at which features and bug fixes are being done has 
picked up recently. We've had 4 releases this month alone.


Then I think you guys should post on Annonce more often :)

It would be really helpful if there are any Intellij users out 
there who don't already use our plugin to install it via the 
plugin repo and try it out (there are 2 D plugins, make sure to 
install the correct one). We now have error reporting built in 
to the plugin so that if anything breaks it's easy to inform 
the team about the problem.


Awesome! Will give it a spin.

Re: Caching D compiler - preview version

[Dmitry Olshansky via Digitalmars-d-announce]

On Wednesday, 1 November 2017 at 19:33:15 UTC, Walter Bright 
wrote:

On 10/29/2017 9:25 AM, Dmitry Olshansky wrote:
On Saturday, 28 October 2017 at 23:18:05 UTC, Martin Nowak 
wrote:

On 10/24/2017 05:02 PM, Dmitry Olshansky wrote:
Experimental std.regex.v2 is sadly broken by a recent change 
to array ops. It would be very interesting to check as it 
eats up to 17Gb of RAM.


What got broken there?


New array ops implemented as __simd, I think it needs a __ctfe 
branch.



Please post a bugzilla issue for it.


https://issues.dlang.org/show_bug.cgi?id=17964

Re: Reorganization and list of D libraries (300+)

[Dmitry Olshansky via Digitalmars-d-announce]

On Monday, 6 November 2017 at 16:59:52 UTC, jmh530 wrote:
On Monday, 6 November 2017 at 16:12:14 UTC, Martin Tschierschke 
wrote:


Even being the wrong Martin :-) I think the DUB registry 
really needs more and better filters, so that the gems inside 
can be found easily. (like: Number of Github stars, number of 
downloads, number of developers and in the future: money 
donated to this project ...).


There has been several attempts but I would like to encourage 
the D Foundation to put more focus on this. Better ecosystem 
around third party software means less work for improvement of 
standard lib.


Regards mt.


Agreed. (IMO more important than Elvis operator)


+111

Re: Release D 2.077.0

[Dmitry Olshansky via Digitalmars-d-announce]

On Friday, 3 November 2017 at 19:46:58 UTC, Walter Bright wrote:

On 11/3/2017 3:02 AM, Mike Parker wrote:
For clarity, where the changeling says that GDC & LDC use 
auto-vectorization, that's actually happening with the array 
operations and core.simd is not required, correct?


I think that GDC and LDC do do auto-vectorization, but I 
haven't verified it myself.


Auto-vectorization is a fundamentally bizarre feature. It takes 
low level code and reverse-engineers it back into a higher 
level construct, and then proceeds to generate code for that 
higher level construct.


Everything else a compiler does is start from a high level 
construct and then generate low level code.


The trouble with AV is whether it succeeds or not depends on 
peculiarities (and I mean peculiarities) of the particular 
vector instruction set target. It can decided to not vectorize 
based on seemingly trivial and innocuous changes to the loop.


I’ll share an anecdotal experience from a time I worked in 
reasearch lab of a well known tech giant. 2 senior researchers 
spent literally 2 weeks trying to coerce compiler into 
vectorizing an inner loop of a non-trivial matrix algorithm.


The only diagnostic from compiler was “loop form is not correct”. 
Thankfully it did tell them it failed, else they’d have to 
disassemble it each time.


I think eventually they either rewritten it to fit heuristic or 
just carried on with explicit intrinsics.



What's needed is a language feature that is straightforwardly 
vectorizable. That would be D's array operations.


Sadly array ops would be insufficient for said problem. It wasn’t 
a direct element wise expression.

Re: Release D 2.077.0

[Dmitry Olshansky via Digitalmars-d-announce]

On Saturday, 4 November 2017 at 08:19:17 UTC, Walter Bright wrote:

On 11/3/2017 1:20 PM, Dmitry Olshansky wrote:
Sadly array ops would be insufficient for said problem. It 
wasn’t a direct element wise expression.


That sounds like that might be why it failed vectorization :-)


As I recall it there were no trivial loops there. Usually these 2 
magicians could make compiler eat it in a few hours of shuffling 
the code. They vectorized about half a dozen loops that way.


The last one took 10 times more then the others taken together ;)



If you recall the expression, it would be interesting to see it.


Even if I had it saved somewhere the place was NDA-ed to death. I 
traded 3 months of intellectual work (and property) for a modest 
amount of money. Interesting experience but no illusions about 
R centers anymore.

Re: GSoC 2018 - Your project ideas

[Dmitry Olshansky via Digitalmars-d-announce]

On Thursday, 7 December 2017 at 22:26:08 UTC, Bastiaan Veelo 
wrote:

On Tuesday, 5 December 2017 at 18:20:40 UTC, Seb wrote:
I am looking forward to hearing (1) what you think can be done 
in three months by a student and (2) will have a huge impact 
on the D ecosystem.


[2] https://wiki.dlang.org/GSOC_2018_Ideas


I see there is a dub section in [2]. Maybe another issue that 
has been brought up repeatedly fits in that category, namely 
extending code.dlang.org in various ways?


+

Indeed enhancing user experience of code.dlang.org such as 
showing github stars and e.g. downloads per month would be way 
more important then build tool itself.

Caching D compiler - preview version

[Dmitry Olshansky via Digitalmars-d-announce]

What is dcache?

It's a patch for dmd that enables a *persistent* shared-memory 
hash-map, protected by a spin-lock from races. Dmd processes with 
-cache flag would detect the following pattern:


enum/static variable = func(args..);

And if mangle of func indicates it is from std.* we use a cache 
to store D source code form of a result of function call (a 
literal) produced by CTFE.


In action:

https://github.com/dlang/dmd/pull/7239

(Watch as 2.8s - 4.4s to compile various ctRegex programs becomes 
constant ~1.0s.)


Caching is done per expression so it stays active even after you 
change various parts of your files.


Broadening the scope to 3rd party libraries is planned but cache 
invalidation is going to be tricky. Likewise there is a trove of 
things aside from CTFE that can be easily cached and shared 
across both parallel and sequental compiler invocations.



Why caching compiler?

It became apparent that CTFE computations could be quite 
time-consuming and memory intensive. The fact that each CTFE 
invocation depends on a set of constant arguments, makes it a 
perfect candidate for caching.


Motivating example is ctRegex, patterns are hardly ever change 
and std.library changes only on compiler upgrade,  yet each 
change to a file causes complete re-evaluation of all patterns in 
a module.


With presistent per-expression cache we can precompile all of 
CTFE evluations for regexes, so we get to use ctRegex and 
maintain sane compile-times.




How to use

Pass new option to dmd:

-cache=mmap

This enables persistent cache using memory-mapped file.
Future backends would take the form of e.g.:

-cache=memcache:memcached.my.network:11211



Implementation

Caveats emptor: this is alpha version, use at your own risk!

https://github.com/DmitryOlshansky/dmd/tree/dcache

Keeping things simple - it's a patch of around 200 SLOCs.
I envision it becoming a hundred lines more if we get to do 
things cleanly.


Instead of going with strangely popular idea of compilation 
servers I opted for simple distributed cache, as it doesn't 
require changing any of the build systems.


Shared memory mapping split in 3 sections: Metadata (spinlock) + 
ToC (hash-table index) + Data (chunks)


For now it's an immutable cache w/o eviction.

A ToC entry is as follows:

hash(64-bit), data index, data size, last_recent_use

Indexes point to Data section of memory map.

Data itself is a linked list of blocks, where a header contains:

(isFree, next, 0-terminated key, padding to 16 bytes)

last_recent_use is a ts of the start of the respective 
compilation.  last_recent < now - 24h is considered unutilized 
and may be reused.


In theory we can cache result of any compilation step with a 
proper key and invalidation strategy.


1. Lexing - key is compiler-version + abs path + timestamp, store 
as is. Lexing from cache is simply taking slices of memory.


2. Parsing to Ast - key is compiler-version + abs path + 
timestamp + version/debug flags


3. CTFE invocations - key is tricky, for now only enabled for 
std.* as follows:


enum/static varname = func(args...);

Use compiler-version + compiler-flags + mangleof + stringof args.

Re: Caching D compiler - preview version

[Dmitry Olshansky via Digitalmars-d-announce]

On Tuesday, 24 October 2017 at 14:17:32 UTC, Dmitry Olshansky 
wrote:

On Tuesday, 24 October 2017 at 13:29:12 UTC, Mike Parker wrote:
On Tuesday, 24 October 2017 at 13:19:15 UTC, Dmitry Olshansky 
wrote:

What is dcache?

It's a patch for dmd that enables a *persistent* 
shared-memory hash-map, protected by a spin-lock from races. 
Dmd processes with -cache flag would detect the following 
pattern:


Blog post or it didn't happen!


Let us at least try it outside of toy examples.

If anybody has std.regex.ctRegex usage I'd be curious to see:

1. Build time w/o -cache=mmap
2. First build time w -cache=mmap
3. Subsequent build times w -cache=mmap

P.S. It's a crude PoC. I think we can do better.


Another caveat: Posix-only for now.

Did a few cleanups and widened the scope a bit.

So here is what happens in my benchmark for std.regex.

-O -inline -release:
88s --> 80s,  memory use ~700Mb -> ~400Mb

-release:
19s -> 12.8s

Experimental std.regex.v2 is sadly broken by a recent change to 
array ops.
It would be very interesting to check as it eats up to 17Gb of 
RAM.

Re: Caching D compiler - preview version

[Dmitry Olshansky via Digitalmars-d-announce]

On Tuesday, 24 October 2017 at 13:29:12 UTC, Mike Parker wrote:
On Tuesday, 24 October 2017 at 13:19:15 UTC, Dmitry Olshansky 
wrote:

What is dcache?

It's a patch for dmd that enables a *persistent* shared-memory 
hash-map, protected by a spin-lock from races. Dmd processes 
with -cache flag would detect the following pattern:


Blog post or it didn't happen!


Let us at least try it outside of toy examples.

If anybody has std.regex.ctRegex usage I'd be curious to see:

1. Build time w/o -cache=mmap
2. First build time w -cache=mmap
3. Subsequent build times w -cache=mmap

P.S. It's a crude PoC. I think we can do better.

---
Dmitry Olshansky

Re: iopipe alpha 0.0.1 version

[Dmitry Olshansky via Digitalmars-d-announce]

On Tuesday, 24 October 2017 at 19:05:02 UTC, Martin Nowak wrote:
On Tuesday, 24 October 2017 at 14:47:02 UTC, Steven 
Schveighoffer wrote:
iopipe provides "infinite" lookahead, which is central to its 
purpose. The trouble with bolting that on top of ranges, as 
you said, is that we have to copy everything out of the range, 
which necessarily buffers somehow (if it's efficient i/o), so 
you are double buffering. iopipe's purpose is to get rid of 
this unnecessary buffering. This is why it's a great fit for 
being the *base* of a range.


In other words, if you want something to have optional 
lookahead and range support, it's better to start out with an 
extendable buffering type like an iopipe, and bolt ranges on 
top, vs. the other way around.


Arguably this it is somewhat hacky to use a range as end marker 
for slicing sth., but you'd get the same benefit, access to the 
random buffer with zero-copying.


auto beg = rng.save; // save current position
auto end = rng.find("bla"); // lookahead using popFront
auto window = beg[0 .. end]; // get a random access window to 
underlying buffer



I had a design like that except save returned a “mark” (not full 
range) and there was a slice primitive. It even worked with 
patched std.regex, but at a non-zero performance penalty.


I think that maintaining the illusion of a full copy of range 
when you do “save” for buffered I/O stream is too costly. Because 
a user can now legally advance both - you need to RC buffers 
behind the scenes with separate “pointers” for each range that 
effectively pin them.



So basically forward ranges with slicing.
At least that would require to extend all algorithms with 
`extend` support, though likely you could have a small extender 
proxy range for IOPipes.


Note that rng could be a wrapper around unbuffered IO reads.

Re: Caching D compiler - preview version

[Dmitry Olshansky via Digitalmars-d-announce]

On Saturday, 28 October 2017 at 23:18:05 UTC, Martin Nowak wrote:

On 10/24/2017 05:02 PM, Dmitry Olshansky wrote:
Experimental std.regex.v2 is sadly broken by a recent change 
to array ops. It would be very interesting to check as it eats 
up to 17Gb of RAM.


What got broken there?


New array ops implemented as __simd, I think it needs a __ctfe 
branch.

Re: LDC 1.10.0 beta

[Dmitry Olshansky via Digitalmars-d-announce]

On Sunday, 13 May 2018 at 18:12:51 UTC, kinke wrote:

Hi everyone,

on behalf of the LDC team, I'm glad to announce the first beta 
for LDC 1.10. The highlights of this version in a nutshell:


* Based on D 2.080.0.
* Supports DragonFly BSD.
* Some fixes, most notably wrt. exception stack traces on Linux.


Fantastic!
And the time flies, wasn’t it 1.4 just recently? ;)



Full release log and downloads: 
https://github.com/ldc-developers/ldc/releases/tag/v1.10.0-beta1


Thanks to all contributors!

Re: serialport v1.0.0

[Dmitry Olshansky via Digitalmars-d-announce]

On Sunday, 6 May 2018 at 22:02:05 UTC, Oleg B wrote:

Stable version of serialport package

* Blocking `SerialPortBlk` for classic usage

* Non-blocking `SerialPortNonBlk` and `SerialPortFR` for usage 
in fibers or in vibe-d


These 3 versions of the same API is precisely the reason for me 
starting the Photon project.


Otherwise - great to see serial ports library and I'd be glad to 
one day test it for compatibility with my approach.




* Variative initialization and configuration

* Hardware flow control config flag

Doc: http://serialport.dpldocs.info/v1.0.0/serialport.html
Dub: http://code.dlang.org/packages/serialport
Git: https://github.com/deviator/serialport

Re: iopipe v0.0.4 - RingBuffers!

[Dmitry Olshansky via Digitalmars-d-announce]

On Saturday, 12 May 2018 at 12:14:28 UTC, Steven Schveighoffer 
wrote:

On 5/11/18 5:42 PM, Joakim wrote:
On Friday, 11 May 2018 at 16:07:26 UTC, Steven Schveighoffer 
wrote:

[...]


What stops you from downloading a linux release from here?

https://github.com/ldc-developers/ldc/releases


So I did that, it's not much faster, a few milliseconds. Still 
about half as fast as GNU grep.


But I am not expecting any miracles here. GNU grep does pretty 
much everything it can to achieve performance -- including 
eschewing the standard library buffering system as I am doing. 
I can probably match the performance at some point, but I doubt 
it's worth worrying about. It's still really really fast 
without trying to do anything crazy.




I could offer a few tricks to fix that w/o getting too dirty. GNU 
grep is fast, but std.regex is faster then that in raw speed on a 
significant class of quite common patterns. But I loaded file at 
once.


I hope at some point, however, to work with Dmitry to add 
iopipe-based regex engine so we can see how much better we can 
make regex.


As such initiative goes it’s either now or never. Please get in 
touch directly over Slack or smth, let’s make it roll. I wanted 
to do grep-like utility since 2012. Now at long last we have all 
the building blocks.




-Steve

Re: iopipe v0.0.4 - RingBuffers!

[Dmitry Olshansky via Digitalmars-d-announce]

On Saturday, 12 May 2018 at 14:48:58 UTC, Joakim wrote:
On Saturday, 12 May 2018 at 12:45:16 UTC, Dmitry Olshansky 
wrote:
On Saturday, 12 May 2018 at 12:14:28 UTC, Steven Schveighoffer 
wrote:

[...]


I could offer a few tricks to fix that w/o getting too dirty. 
GNU grep is fast, but std.regex is faster then that in raw 
speed on a significant class of quite common patterns. But I 
loaded file at once.



[...]


As such initiative goes it’s either now or never. Please get 
in touch directly over Slack or smth, let’s make it roll. I 
wanted to do grep-like utility since 2012. Now at long last we 
have all the building blocks.


If you're talking about writing a grep prototype in D, that's a 
great idea, especially for publicizing D. :)


For shaming others to beat us using some other language. Making 
life better for everyone. Taking a DMD to a gun fight ;)

Re: iopipe v0.0.4 - RingBuffers!

[Dmitry Olshansky via Digitalmars-d-announce]

On Friday, 11 May 2018 at 13:28:58 UTC, Steven Schveighoffer 
wrote:

On 5/11/18 1:30 AM, Dmitry Olshansky wrote:
On Thursday, 10 May 2018 at 23:22:02 UTC, Steven Schveighoffer 
wrote:
OK, so at dconf I spoke with a few very smart guys about how 
I can use mmap to make a zero-copy buffer. And I implemented 
this on the plane ride home.


However, I am struggling to find a use case for this that 
showcases why you would want to use it. While it does work, 
and works beautifully, it doesn't show any measurable 
difference vs. the array allocated buffer that copies data 
when it needs to extend.


I’d start with something clinicaly synthetic.
Say your record size is exactly half of buffer + 1 byte. If 
you were to extend the size of buffer, it would amortize.


Hm.. this wouldn't work, because the idea is to keep some of 
the buffer full. What will happen here is that the buffer will 
extend to be able to accomodate the extra byte, and then you 
are back to having less of the buffer full at once. Iopipe is 
not afraid to increase the buffer :)


Then you cannot test it in such way.





Basically:
16 Mb buffer fixed
vs
16 Mb mmap-ed ring

Where you read pieces in 8M+1 blocks.Yes, we are aiming to 
blow the CPU cache there. Otherwise CPU cache is so fast that 
ocasional copy is zilch, once we hit primary memory it’s not. 
Adjust sizes for your CPU.


This isn't how it will work. The system looks at the buffer and 
says "oh, I can just read 8MB - 1 byte," which gives you 2 
bytes less than you need. Then you need the extra 2 bytes, so 
it will increase the buffer to hold at least 2 records.


I do get the point of having to go outside the cache. I'll look 
and see if maybe specifying a 1000 line context helps ;)


Nope. Consider reading binary records where you know length in 
advance and skip over it w/o need to touch every byte. There it 
might help. If you touch every byte and do something the cost of 
copying the tail is zilch.


One example is net string which is:

13,Hello, world!

Basically length in ascii digits ‘,’ followed by tgat much UTF-8 
codeunits.

No decoding nessary.

Torrent files use that I think, maybe other files. Is a nice 
example that avoids scans to find delimiters.




Update: nope, still pretty much the same.

The amount of work done per byte though has to be minimal to 
actually see anything.


Right, this is another part of the problem -- if copying is so 
rare compared to the other operations, then the difference is 
going to be lost in the noise.


What I have learned here is:

1. Ring buffers are really cool (I still love how it works) and 
perform as well as normal buffers


This is also good. Normal ring buffers usually suck  in speed 
department.



2. The use cases are much smaller than I thought
3. In most real-world applications, they are a wash, and not 
worth the OS tricks needed to use it.
4. iopipe makes testing with a different kind of buffer really 
easy, which was one of my original goals. So I'm glad that 
works!


I'm going to (obviously) leave them there, hoping that someone 
finds a good use case, but I can say that my extreme excitement 
at getting it to work was depressed quite a bit when I found it 
didn't really gain much in terms of performance for the use 
cases I have been doing.
Should be mostly trivial in fact. I mean our first designs for 
IOpipe is where I wanted regex to work with it.


Basically - if we started a match, extend window until we get 
it or lose it. Then release up to the next point of potential 
start.


I'm thinking it's even simpler than that. All matches are dead 
on a line break (it's how grep normally works), so you simply 
have to parse the lines and run each one via regex. What I 
don't know is how much it costs regex to startup and run on an 
individual line.


It is malloc/free/addRange/removeRange for each call. I optimized 
2.080 to reuse last recently used engine w/o these costs but I’ll 
have to check if it covers all cases.




One thing I could do to amortize is keep 2N lines in the 
buffer, and run the regex on a whole context's worth of lines, 
then dump them all.


I believe integrating iopipe awareness it in regex will easily 
make it 50% faster. A guestimate though.




I don't get why grep is so bad at this, since it is supposedly


grep on Mac is a piece of sheat, sadly and I don’t know why 
exactly (too old?). Use some 3-rd party thing like ‘sift’ written 
in Go.




-Steve

Re: iopipe v0.0.4 - RingBuffers!

[Dmitry Olshansky via Digitalmars-d-announce]

On Thursday, 10 May 2018 at 23:22:02 UTC, Steven Schveighoffer 
wrote:
OK, so at dconf I spoke with a few very smart guys about how I 
can use mmap to make a zero-copy buffer. And I implemented this 
on the plane ride home.


However, I am struggling to find a use case for this that 
showcases why you would want to use it. While it does work, and 
works beautifully, it doesn't show any measurable difference 
vs. the array allocated buffer that copies data when it needs 
to extend.


I’d start with something clinicaly synthetic.
Say your record size is exactly half of buffer + 1 byte. If you 
were to extend the size of buffer, it would amortize.


Basically:
16 Mb buffer fixed
vs
16 Mb mmap-ed ring

Where you read pieces in 8M+1 blocks.Yes, we are aiming to blow 
the CPU cache there. Otherwise CPU cache is so fast that 
ocasional copy is zilch, once we hit primary memory it’s not. 
Adjust sizes for your CPU.


The amount of work done per byte though has to be minimal to 
actually see anything.





in the buffer. But alas, it's roughly the same, even with large 
number of lines for context (like 200).


However, this example *does* show the power of iopipe -- it 
handles all flavors of unicode with one template function, is 
quite straightforward (though I want to abstract the line 
tracking code, that stuff is really tricky to get right). Oh, 
and it's roughly 10x faster than grep, and a bunch faster than 
fgrep, at least on my machine ;) I'm tempted to add regex 
processing to see if it still beats grep.


Should be mostly trivial in fact. I mean our first designs for 
IOpipe is where I wanted regex to work with it.


Basically - if we started a match, extend window until we get it 
or lose it. Then release up to the next point of potential start.




Next up (when my bug fix for dmd is merged, see 
https://issues.dlang.org/show_bug.cgi?id=17968) I will be 
migrating iopipe to depend on 
https://github.com/MartinNowak/io, which should unlock Windows 
support (and I will add RingBuffer Windows support at that 
point).


Enjoy!

https://github.com/schveiguy/iopipe
https://code.dlang.org/packages/iopipe
http://schveiguy.github.io/iopipe/

-Steve

Re: iopipe v0.0.4 - RingBuffers!

[Dmitry Olshansky via Digitalmars-d-announce]

On Friday, 11 May 2018 at 09:55:10 UTC, Kagamin wrote:
On Thursday, 10 May 2018 at 23:22:02 UTC, Steven Schveighoffer 
wrote:
However, I am struggling to find a use case for this that 
showcases why you would want to use it. While it does work, 
and works beautifully, it doesn't show any measurable 
difference vs. the array allocated buffer that copies data 
when it needs to extend.


Depends on OS and hardware. I would expect mmap implementation 
to be slower as it reads file in chunks of 4kb and relies on 
page faults.


It doesn’t. Instead it has a buffer mmaped twice side by side. 
Therefore you can avoid copy at the end when it wraps around.


Otherwise it’s the same buffering as usual.

Re: Looks like Digital Mars C++ is on the front page of HN at the moment!

[Dmitry Olshansky via Digitalmars-d-announce]

On Wednesday, 23 May 2018 at 01:18:43 UTC, Walter Bright wrote:


DigitalMars C/C++ Compiler (github.com)
56 points by tomcam 3 hours ago | unvote | flag | hide | 10 
comments


Yay! Any thoughts about opening runtime library?



https://news.ycombinator.com/news


And it’s beyond 100+ now. Also I see that you have quite a 
reputation in compilers ;)

Re: iopipe v0.1.0 - now with Windows support!

[Dmitry Olshansky via Digitalmars-d-announce]

On Sunday, 17 June 2018 at 04:52:07 UTC, Martin Nowak wrote:

On 06/10/2018 10:10 PM, Steven Schveighoffer wrote:
Note that the new io library also supports sockets, which 
IODev did not have support for, AND has a pluggable driver 
system, so you could potentially use fiber-based async io 
without rebuilding. It just makes a lot of sense for D to have 
a standard low-level io library that everything can use 
without having to kludge together multiple types of io 
libraries.


Note that the WIP std.io library is fully @nogc @safe, so it's 
a bit
edgy on using latest features. Soon want to move to use 
DIP10008 instead

of preallocated exceptions.


This is very encouraging. I’d like to see it working well with 
Photon (though my time is very limited atm). Any thoughts on what 
set of syscalls I need to support?


Maybe I could just provide my own “native” driver that fits your 
concept of I/O driver in io library.


With that and @nogc in the Driver interface¹ it's still to be 
seen

whether we can adapt this well with vibe.d or need to adjust the
low-level design.

-Martin

¹: https://martinnowak.github.io/io/std/io/driver/Driver.html

Re: GitHub could be acquired by Microsoft

[Dmitry Olshansky via Digitalmars-d-announce]

On Monday, 4 June 2018 at 03:51:15 UTC, Anton Fediushin wrote:
This is still just a rumour, we'll know the truth on Monday 
(which is today).


Some articles about the topic:

https://fossbytes.com/microsoft-github-aquisition-report/
https://www.theverge.com/2018/6/3/17422752/microsoft-github-acquisition-rumors

What's your opinion about that? Will you continue using GitHub?


Well, MS already contributes big time to many open-source 
projects, including Git.

I do not see immanent  problem with them buying it.



Both GitLab and Bitbucket can be used instead to host your D 
projects - dub registry supported them for a while now.


Both are fine, though Gitlab seems more sexy now.

IMHO Microsoft isn't the type of company I want to see behind 
the GitHub. Maybe I am wrong since Microsoft has both money and 
programmers to improve it further, I just don't trust them too 
much which is the right thing to do when dealing with companies.


Would you trust Google? Me, I’m not.
In fact if we were to place trust, comercial IT companies would 
be pretty down on my list of “trust” in any case.


This means that I will move my repositories elsewhere and use 
GitHub just to contribute to other projects.

Re: Another take on decimal data types

[Dmitry Olshansky via Digitalmars-d-announce]

On Thursday, 11 January 2018 at 21:12:59 UTC, kdevel wrote:
On Thursday, 11 January 2018 at 20:40:01 UTC, Dmitry Olshansky 
wrote:

   printf ("%.2f\n", d);


C’s printf by definition can’t be customized.


Sure.



“”” GNU C Library lets you define “””

Here is your compatibility story.


What did you expect?


To be honest: A compile time error. Modern C compilers can 
check such format strings. Example: GCC 6:


1. It’s a warning.
2. There is no typechecking it’s just a hardcoded linting in the 
compiler.
3. It’s D use writefln or if going for C primitives know their 
quirks.
4. Passing anything but basic types to C vararg is undefined in 
C++ IIRC, dunno what D should do about it.




mis.c
```
#include 

int main ()
{
   double d = 0;
   printf ("%p\n", d);
   return 0;
}
```

$ gcc -Wall mis.c
mis.c: In function 'main':
mis.c:6:14: warning: format '%p' expects argument of type 'void 
*', but argument 2 has type 'double' [-Wformat=]

printf ("%p\n", d);

Re: Another take on decimal data types

[Dmitry Olshansky via Digitalmars-d-announce]

On Thursday, 11 January 2018 at 20:35:03 UTC, kdevel wrote:

Great project!

On Monday, 8 January 2018 at 22:16:25 UTC, rumbu wrote:

- all format specifiers implemented (%f, %e, %g, %a);


Really?

[...]


What's next:
- more tests;


Here you are:

```
import std.stdio;
import decimal;

void main ()
{
   decimal32 d = "0.7";
   d *= decimal32("1.05");
   d.writeln;
   printf ("%.2f\n", d);



C’s printf by definition can’t be customized. What did you expect?



   float f = 0.7f;
   f *= 1.05f;
   f.writeln;
   printf ("%.2f\n", f);

   decimal32 e = 1_000_000_000;
   while (e > 1e-7) {
  e.writeln;
  e /= 10;
   }
}
```

This prints:

   0.735
   0.00  <--- expected: 0.74
   0.735
   0.73
 <--- loop output missing

(DMD64 D Compiler v2.077.1)

Re: Vanquish Forever These Bugs That Blasted Your Kingdom

[Dmitry Olshansky via Digitalmars-d-announce]

On Wednesday, 7 February 2018 at 13:29:04 UTC, Mike Parker wrote:
Walter's got a new post up! It's the first in a new series on 
the benefits of BetterC mode. In this one, he talks about 
solving the fencepost problem (off-by-one errors) with D's 
arrays.


Blog:
https://dlang.org/blog/2018/02/07/vanquish-forever-these-bugs-that-blasted-your-kingdom/



DynamicArray is actually laid out in length, pointer order. 
Thankfully only reaaally low-level code has to know that.




Reddit:
https://www.reddit.com/r/programming/comments/7vw0gj/vanquish_forever_these_bugs_that_blasted_your/