Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread BLM768 via Digitalmars-d

On Thursday, 31 March 2016 at 20:51:53 UTC, Jack Stouffer wrote:


Wouldn't just be better to use a std.allocator backed set 
container instead over special casing the AA syntax like this?


Syntactically, that makes more sense.

Or there's always ubyte[0][T].


Re: GC.malloc is pure - wat

2016-03-31 Thread ag0aep6g via Digitalmars-d

On 01.04.2016 00:52, deadalnix wrote:

Is it not pure, strong or weak. GC.malloc is pure because the memory is
going to be garbage collected. malloc is not pure as the memory need to
be freed. The state in the allocator is exposed to the program.


How does malloc expose its state and GC.malloc doesn't?

* They have the same signature. Both return pointers (to mutable data).
* malloc has free, GC.malloc has GC.free. So you can manually free the 
memory in both cases.

* You don't have to free memory from malloc. You can let it leak.


Lie to the compiler and it will punish you for it.


This also happens with functions that have an implementation in source:


int[] f() pure nothrow {return [0];}

void main()
{
auto a = f();
auto b = f();
a[0] = 1;
b[0] = 2;

import std.stdio;
writeln(a is b); /* should be false */
writeln(a[0]); /* should be 1 */
}


Prints "true" and "2" when compiled with `dmd -O -release`.

So I don't think that lying to the compiler is the problem here.


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread Walter Bright via Digitalmars-d

On 3/31/2016 7:31 PM, Jonathan M Davis via Digitalmars-d wrote:

LOL. Except that as with everything, Murphy is at work. If it's something
that you want to be around forever, it probably won't be, but if it's
something that you don't want to be around, then it probably will be until
the end of time. :)

Either way, there's no running from bad suggestions in the newsgroup. We
have the proof! ;)


Sorta like how a dropped peanut butter and jelly sandwich always lands face 
down!


Re: Policy for exposing range structs

2016-03-31 Thread Jonathan M Davis via Digitalmars-d
On Thursday, March 31, 2016 09:10:49 Steven Schveighoffer via Digitalmars-d 
wrote:
> On 3/30/16 3:19 PM, Liran Zvibel wrote:
> > On Sunday, 27 March 2016 at 17:01:39 UTC, David Nadlinger wrote:
> >> Compression in the usual sense won't help. Sure, it might reduce the
> >> object file size, but the full string will again have to be generated
> >> first, still requiring absurd amounts time and space. The latter is
> >> definitely not negligible for symbol names of several hundreds of
> >> kilobytes; it shows up prominently in compiler profiles of affected
> >> Weka builds.
> >
> > We love Voldemort types at Weka, and use them a lot in our
> > non-gc-allocating ranges and algorithm libraries. Also, we liberally
> > nest templates inside of other templates.
> > I don't think we can do many of the things we do if we had to define
> > everything at module level. This flexibility is amazing for us and part
> > of the reason we love D.
>
> Voldemort types are what cause the bloat, templates inside templates
> aren't as much of a problem. It's because the Voldemort type has to
> include in its symbol name at least twice, and I think 3 times actually
> (given the return type), the template parameter/function parameter types
> of the function it resides in. If the template is just a template, it's
> just included once. This is why moving the type outside the function is
> effective at mitigation. It's linear growth vs. exponential.

This makes me wonder if there would be a sane way to basically treat the
Voldemort type as if it were outside the function it was declared from a
mangling standpoint while still treating its accessibility the same way.
Maybe that wouldn't work due to potential name clashes, but if the problem
is purely because the type is declared inside the function, I would think
that it would make sense to try and find a way to treat it more like it was
declared outside the function when dealing with the name mangling. Then - in
theory - you'd get the benefits of using a Voldemort type while getting the
name mangling cost that you get when you declare a private type outside of
the function.

- Jonathan M Davis



Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread Jonathan M Davis via Digitalmars-d
On Thursday, March 31, 2016 17:44:15 Steven Schveighoffer via Digitalmars-d 
wrote:
> hm... I suppose:
>
> a[x] = void;
>
> could add. Removal is never done by assigning, only by aa.remove.

Well, from the standpoint of it being a map, you could argue that _every_
key is in the set. It's just that some of them map to true and most of them
map to false. But that's just trying to find a way to think about it that
makes Walter's suggestion consistent with what we have now, I guess.

Still, while it's true that aa.remove is how you'd normally do it, I think
that Walter's suggestion of assigning true or false makes by far the most
sense of the ones made thus far - and you could just make

aa.remove(key);

and

aa[key] = false;

equivalent for void[T] to make it more consistent.

- Jonathan M Davis



Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread Jonathan M Davis via Digitalmars-d
On Thursday, March 31, 2016 14:45:18 Walter Bright via Digitalmars-d wrote:
> On 3/31/2016 2:09 PM, Ali Çehreli wrote:
> > Expanding on Walter's idea:
> >a[x] = shared(void);// add
> >a[x] = void;// remove
>
> For shame!
>
> > Ali
> > "ducks and runs for cover" :)
>
> The internet is forever :-)

LOL. Except that as with everything, Murphy is at work. If it's something
that you want to be around forever, it probably won't be, but if it's
something that you don't want to be around, then it probably will be until
the end of time. :)

Either way, there's no running from bad suggestions in the newsgroup. We
have the proof! ;)

- Jonathan M Davis




Re: GC.malloc is pure - wat

2016-03-31 Thread Timon Gehr via Digitalmars-d

On 01.04.2016 00:52, deadalnix wrote:



DMD optimizes malloc because the only reason it would have to return a
different pointer would be internal state (and in fact, it indeed has
internal state).


It's the other way around. Without the "optimization" there is no way 
for a 'pure' function to return a reference to the same mutable memory 
block independent of its arguments more than once. In the end it is up 
to the spec, but IMO compiler optimizations should not be able to 
introduce mutable aliasing that was not there before. (Even for 
immutable references, sharing the same result is moot, as D supports 
checking for reference equality.)


Re: GC.malloc is pure - wat

2016-03-31 Thread Nordlöw via Digitalmars-d

On Thursday, 31 March 2016 at 22:18:03 UTC, ag0aep6g wrote:
As has been explained to me in this very thread, the fact that 
malloc returns a pointer to mutable data makes it not strongly 
pure.


Got it. Sorry for the misleads. I'm tired. Gonna sleep. Thanks.


Re: Cannot use TCMalloc

2016-03-31 Thread Nordlöw via Digitalmars-d

On Thursday, 31 March 2016 at 23:15:18 UTC, Basile B. wrote:


it is indeed: http://code.dlang.org/packages/tcmallocd


Nice! Good work!


Re: Cannot use TCMalloc

2016-03-31 Thread Basile B. via Digitalmars-d

On Thursday, 31 March 2016 at 21:44:21 UTC, Basile B. wrote:

I think it'll be usable in a allocator-like structure.


it is indeed: http://code.dlang.org/packages/tcmallocd


Re: GC.malloc is pure - wat

2016-03-31 Thread deadalnix via Digitalmars-d

On Thursday, 31 March 2016 at 22:18:03 UTC, ag0aep6g wrote:

On 31.03.2016 23:25, Nordlöw wrote:

A solution is to fake purity through

extern(C) pure nothrow @system @nogc
{
 void* malloc(size_t size);
 void* realloc(void* ptr, size_t size);
 void free(void* ptr);
}

Pay attention to the use of malloc() though.

Note that this makes malloc() strongly pure (its single 
parameters is

passed by value).


As has been explained to me in this very thread, the fact that 
malloc returns a pointer to mutable data makes it not strongly 
pure.




Is it not pure, strong or weak. GC.malloc is pure because the 
memory is going to be garbage collected. malloc is not pure as 
the memory need to be freed. The state in the allocator is 
exposed to the program.



Uh, as far as I see, that would be catastrophic.

I feel like I must be missing something here again, but it 
looks like dmd actually gets this wrong:



extern(C) void* malloc(size_t size) pure nothrow;

void main()
{
auto a = cast(ubyte*) malloc(1);
auto b = cast(ubyte*) malloc(1);

import std.stdio;
writeln(a is b); /* should be false */
*a = 1;
*b = 2;
writeln(*a); /* should be 1 */
}


Compiled with dmd and no optimization switches, this prints 
"false" and "1" as expected.
Compiled with `dmd -O -release`, this prints "true" and "2". 
The heck?

With `ldc2 -O5 -release`: "false" and "1". No problem there.

This looks like a serious bug in dmd to me.


Lie to the compiler and it will punish you for it.

DMD optimizes malloc because the only reason it would have to 
return a different pointer would be internal state (and in fact, 
it indeed has internal state).




Re: Policy for exposing range structs

2016-03-31 Thread Anon via Digitalmars-d

On Thursday, 31 March 2016 at 20:40:03 UTC, Adam D. Ruppe wrote:
meh, if it is allowed, it is going to happen. Why make it worse 
when there's so little cost in making it better?


Define "little cost". Whatever compression algorithm chosen will 
need support added to any/all tools that want to demangle D. GDB 
and LLDB currently link to liblzma (on my system, at least. 
Configurations may vary). nm and objdump don't link to any 
compression lib. Good luck convincing binutils to add compression 
dependencies like that for D when they don't need them any other 
mangling schemes.


And no, ddemangle isn't a solution here, as then all those 
projects would need to be able to refer to it, and the best way 
for them to do that is to bundle it. Since ddemangle is written 
in D, that would mean binutils would suddenly depend on having a 
working D compiler. That won't happen in the next decade.


Also, any D code that uses core.demangle for any reason would 
suddenly depend on that compression library.


I'm not even fully convinced that my bootstring idea is low 
enough cost, and it's fairly simple, fast, and memory efficient 
compared to any compression algorithm.


I often don't actually modify the string at all and by putting 
the string as a template argument, it enables a kind of 
compile-time memoization like I talked about here a short while 
ago: http://stackoverflow.com/a/36251271/1457000


The string may be exceedingly if imported from a file or 
generated externally and cached:


MyType!(import("definition.txt")) foo;

enum foo = ctfeFunction();

MyType!foo test;


Just assigning to an enum gets you memoization (tested on LDC w/ 
DFE 2.68.2).

I don't see how the template factors into this.

Now, yes, if you call the function directly multiple times 
assigning to different enums, it won't memoize those. And it 
doesn't work lazily how the SO question asked for, but this does:


enum lazily(string name: "foo") = ctfeFunction();

If you don't refer to lazily!"foo", ctfeFunction() never gets 
called. If you do, it gets called once, regardless of how many 
times you use lazily!"foo".


That gives you lazy memoization of any CTFEable function without 
ever needing the function parameters to become template 
parameters.


I'm not sure what MyType is, but that looks like a prime 
candidate for my previous post's mixin examples. If not, you 
could use "definition.txt" as its parameter, and have it import() 
as an implementation detail.



$ is actually a valid identifier character in C


Nope. $ as an identifier character is a commonly supported 
extension, but code that uses it doesn't compile with `clang 
-std=c11 -Werror -pedantic`.


Re: GC.malloc is pure - wat

2016-03-31 Thread ag0aep6g via Digitalmars-d

On 31.03.2016 23:25, Nordlöw wrote:

A solution is to fake purity through

extern(C) pure nothrow @system @nogc
{
 void* malloc(size_t size);
 void* realloc(void* ptr, size_t size);
 void free(void* ptr);
}

Pay attention to the use of malloc() though.

Note that this makes malloc() strongly pure (its single parameters is
passed by value).


As has been explained to me in this very thread, the fact that malloc 
returns a pointer to mutable data makes it not strongly pure.


See 
http://klickverbot.at/blog/2012/05/purity-in-d/#indirections-in-the-return-type



Therefore successive calls to malloc() with the same
argument will be optimized away and all those calls will return the same
values as the first call. When used in allocators this shouldn't be a
problem, though.


Uh, as far as I see, that would be catastrophic.

I feel like I must be missing something here again, but it looks like 
dmd actually gets this wrong:



extern(C) void* malloc(size_t size) pure nothrow;

void main()
{
auto a = cast(ubyte*) malloc(1);
auto b = cast(ubyte*) malloc(1);

import std.stdio;
writeln(a is b); /* should be false */
*a = 1;
*b = 2;
writeln(*a); /* should be 1 */
}


Compiled with dmd and no optimization switches, this prints "false" and 
"1" as expected.

Compiled with `dmd -O -release`, this prints "true" and "2". The heck?
With `ldc2 -O5 -release`: "false" and "1". No problem there.

This looks like a serious bug in dmd to me.


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread Walter Bright via Digitalmars-d

On 3/31/2016 2:09 PM, Ali Çehreli wrote:

Expanding on Walter's idea:

   a[x] = shared(void);// add
   a[x] = void;// remove


For shame!



Ali
"ducks and runs for cover" :)


The internet is forever :-)


Re: Cannot use TCMalloc

2016-03-31 Thread Basile B. via Digitalmars-d

On Thursday, 31 March 2016 at 21:26:45 UTC, Basile B. wrote:

On Thursday, 31 March 2016 at 20:21:00 UTC, Nordlöw wrote:

[...]


It looks like if you manage to hook default malloc, free etc 
the GC will be affected.


[...]


Actually it works, I forgot a hyphen in the scipt line !
Do you plan to make a package or something ? I think it'll be 
usable in a allocator-like structure.




Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread Steven Schveighoffer via Digitalmars-d

On 3/31/16 5:09 PM, Ali Çehreli wrote:

On 03/31/2016 01:39 PM, Steven Schveighoffer wrote:

 > But how do you add a key to the set? Currently only allowed via:
 >
 > a[x] = ...;

Expanding on Walter's idea:

   a[x] = shared(void);// add
   a[x] = void;// remove

Ali
"ducks and runs for cover" :)


hm... I suppose:

a[x] = void;

could add. Removal is never done by assigning, only by aa.remove.

-Steve


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread Walter Bright via Digitalmars-d

On 3/31/2016 12:58 PM, H. S. Teoh via Digitalmars-d wrote:

How is this different from bool[T] then? Just the fact that you can't
get a reference to the bool?


Differences are:

1. it uses less storage, as the bool is implied
2. you cannot have a key in the set that has an associated value of false
3. as you said, you cannot get a reference to the bool (because it is implied)



Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread tsbockman via Digitalmars-d

On Thursday, 31 March 2016 at 21:09:30 UTC, Ali Çehreli wrote:

Expanding on Walter's idea:

  a[x] = shared(void);// add
  a[x] = void;// remove

Ali
"ducks and runs for cover" :)


And shared(void)[T] declares an add-only set, right?


Re: Cannot use TCMalloc

2016-03-31 Thread Basile B. via Digitalmars-d

On Thursday, 31 March 2016 at 20:21:00 UTC, Nordlöw wrote:

On Thursday, 31 March 2016 at 19:09:20 UTC, Basile B. wrote:
You need to install the "-devel" version to get the it as a 
static library. On OpenSuse it's named gperftools-devel.


Maybe on ubuntu it's this one:

"libgoogle-perftools-dev"

http://packages.ubuntu.com/fr/trusty/amd64/libgoogle-perftools-dev/filelist, because it 
contains the "*.a" static library you wanna link in.


Works. Wonderful. Thanks.

BTW: Will this affect D's builtin GC or does it use mmap 
directly?


It looks like if you manage to hook default malloc, free etc the 
GC will be affected.


But a simple try to call the functions from the static lib fails 
(the first script line is passed to DMD by my editor):



#!runnable flags: -L-ltcmalloc
extern(C) void* tc_malloc(size_t size);
extern(C) void  tc_free(void* ptr);

void main(string[] args)
{
auto p = tc_malloc(16);
tc_free(p);
}

/tmp/temp_7FCC33C16DF0.d:(.text._Dmain+0xa): référence 
indéfinie vers « tc_malloc »
/tmp/temp_7FCC33C16DF0.d:(.text._Dmain+0x12): référence 
indéfinie vers « tc_free »


interface seems to be this:
https://github.com/gperftools/gperftools/blob/master/src/gperftools/tcmalloc.h.in#L88

Do you have the interface or the D sources to call the functions ?


Re: GC.malloc is pure - wat

2016-03-31 Thread Nordlöw via Digitalmars-d

On Friday, 24 April 2015 at 15:05:15 UTC, anonymous wrote:

GC.malloc is marked pure. But it isn't, is it?

This should hold for a pure function:
assert(f(x) == f(x));
This fails with GC.malloc, of course.

Or consider this:
auto v = f(x);
auto w = f(x);
When f is pure, a compiler should be free to reuse the value of 
v for w. That's no good with GC.malloc, obviously.


A solution is to fake purity through

extern(C) pure nothrow @system @nogc
{
void* malloc(size_t size);
void* realloc(void* ptr, size_t size);
void free(void* ptr);
}

Pay attention to the use of malloc() though.

Note that this makes malloc() strongly pure (its single 
parameters is passed by value). Therefore successive calls to 
malloc() with the same argument will be optimized away and all 
those calls will return the same values as the first call. When 
used in allocators this shouldn't be a problem, though.


Used successfully at

https://github.com/nordlow/justd/blob/master/packedarray.d#L6


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread Adam D. Ruppe via Digitalmars-d
On Thursday, 31 March 2016 at 20:39:36 UTC, Steven Schveighoffer 
wrote:

But how do you add a key to the set? Currently only allowed via:

a[x] = ...;


Oh yeah... when I use a built in AA as a set now, I either set it 
to true or to itself:


string[string] lameSet;

lameSet[a] = a;

lameSet.remove(a);

so i guess one of those could happen


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread H. S. Teoh via Digitalmars-d
On Thu, Mar 31, 2016 at 02:09:30PM -0700, Ali Çehreli via Digitalmars-d wrote:
> On 03/31/2016 01:39 PM, Steven Schveighoffer wrote:
> 
> > But how do you add a key to the set? Currently only allowed via:
> >
> > a[x] = ...;
> 
> Expanding on Walter's idea:
> 
>   a[x] = shared(void);// add
>   a[x] = void;// remove
> 
> Ali
> "ducks and runs for cover" :)

Yes you better run, that syntax is so atrocious I'm reaching for my
rotten tomatoes... :-P


T

-- 
Don't modify spaghetti code unless you can eat the consequences.


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread Ali Çehreli via Digitalmars-d

On 03/31/2016 01:39 PM, Steven Schveighoffer wrote:

> But how do you add a key to the set? Currently only allowed via:
>
> a[x] = ...;

Expanding on Walter's idea:

  a[x] = shared(void);// add
  a[x] = void;// remove

Ali
"ducks and runs for cover" :)



Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread Jack Stouffer via Digitalmars-d

On Thursday, 31 March 2016 at 19:24:14 UTC, deadalnix wrote:
Pretty much as per title. I has that in the back of my mind for 
a while. Would that work ?


Wouldn't just be better to use a std.allocator backed set 
container instead over special casing the AA syntax like this?


Re: Policy for exposing range structs

2016-03-31 Thread Adam D. Ruppe via Digitalmars-d

On Thursday, 31 March 2016 at 20:12:34 UTC, Anon wrote:
Having thought about it a bit more, I am now of the opinion 
that super-long strings have no business being in template 
args, so we shouldn't cater to them.


meh, if it is allowed, it is going to happen. Why make it worse 
when there's so little cost in making it better?


The main case with longer strings going into template arguments 
I'm aware of is for when strings will be processed, then fed to 
`mixin()`.


I often don't actually modify the string at all and by putting 
the string as a template argument, it enables a kind of 
compile-time memoization like I talked about here a short while 
ago: http://stackoverflow.com/a/36251271/1457000


The string may be exceedingly if imported from a file or 
generated externally and cached:


MyType!(import("definition.txt")) foo;

enum foo = ctfeFunction();

MyType!foo test;


I've never had a huge problem with this in practice, but I've 
never had a huge problem with big names in practice at all 
either. I can imagine both though.


(what bothers me more than the mangle actually is the compiler 
error messages showing all those strings. Gross. I'd like to see 
XML error messages to make displaying them easier. But that's a 
separate topic.)


The original mangling discussion started from the need to 
either fix a mangling problem or officially discourage 
Voldemort types.



Yes, indeed, that's probably the bigger problem.

* Retains current ability to access D symbols from C (in 
contrast to ideas that would use characters like '$' or '?')


$ is actually a valid identifier character in C (and quite a few 
other languages). You can use it in the linker as well as in C 
source code.




Re: Any usable SIMD implementation?

2016-03-31 Thread Iakh via Digitalmars-d

On Thursday, 31 March 2016 at 08:23:45 UTC, Martin Nowak wrote:
I'm currently working on a templated arrayop implementation 
(using RPN

to encode ASTs).
So far things worked out great, but now I got stuck b/c 
apparently none
of the D compilers has a working SIMD implementation (maybe GDC 
has but

it's very difficult to work w/ the 2.066 frontend).

https://github.com/MartinNowak/druntime/blob/arrayOps/src/core/internal/arrayop.d
 https://github.com/MartinNowak/dmd/blob/arrayOps/src/arrayop.d

I don't want to do anything fancy, just unaligned loads, 
stores, and integral mul/div. Is this really the current state 
of SIMD or am I missing sth.?


-Martin


Unfortunately my one(https://github.com/Iakh/simd) is far from
production code. For now I'm trying to figure out interface common
to all archs/compilers. And its more about SIMD comparison 
operations.


You could do loads, stores and mul with default D SIMD support
but not int div


Re: Tristate - wanna?

2016-03-31 Thread Nordlöw via Digitalmars-d

On Saturday, 26 March 2016 at 22:11:53 UTC, Nordlöw wrote:

Partial implementation at

https://github.com/nordlow/justd/blob/master/fuzzy.d#L15


moved to

https://github.com/nordlow/justd/blob/master/nstate.d#L15


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread Steven Schveighoffer via Digitalmars-d

On 3/31/16 4:11 PM, Adam D. Ruppe wrote:

On Thursday, 31 March 2016 at 19:57:50 UTC, Walter Bright wrote:

On 3/31/2016 12:44 PM, H. S. Teoh via Digitalmars-d wrote:

Ah, makes sense.  But what would aa[x] return?


A bool indicating membership.


Ewww. If it looks like an AA, let's at least keep the AA interface.

aa[x] returns void, which, having no value, would be a compile error.


And how would you add elements to it?


aa[x] = true;  // add member x
aa[x] = false; // remove member x
x in aa; // compile error


aa[x] is a compile error since it is of type void.

x in aa returns void*, null if it is not there, not-null if it is there.
Dereferencing a void* is illegal anyway so its value is irrelevant. (I'd
say just use null and cast(void*) 1)

It is actually just a bool, but with consistent typing to other AAs.

Phobos's RedBlackTree works with a literal bool from opIn:
http://dpldocs.info/experimental-docs/std.container.rbtree.RedBlackTree.opBinaryRight.html



But how do you add a key to the set? Currently only allowed via:

a[x] = ...;

-Steve


Re: Cannot use TCMalloc

2016-03-31 Thread Nordlöw via Digitalmars-d

On Thursday, 31 March 2016 at 19:09:20 UTC, Basile B. wrote:
You need to install the "-devel" version to get the it as a 
static library. On OpenSuse it's named gperftools-devel.


Maybe on ubuntu it's this one:

"libgoogle-perftools-dev"

http://packages.ubuntu.com/fr/trusty/amd64/libgoogle-perftools-dev/filelist, because it 
contains the "*.a" static library you wanna link in.


Works. Wonderful. Thanks.

BTW: Will this affect D's builtin GC or does it use mmap directly?


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread tsbockman via Digitalmars-d

On Thursday, 31 March 2016 at 19:58:54 UTC, H. S. Teoh wrote:
How is this different from bool[T] then? Just the fact that you 
can't get a reference to the bool?


void[T] could be more efficient, since it wouldn't need to 
allocate memory for a bool payload. I expect that alignment 
concerns typically require more than one byte per bool in a 
bool[T]. (I haven't studied the current implementation, though.)


Re: Policy for exposing range structs

2016-03-31 Thread Anon via Digitalmars-d

On Thursday, 31 March 2016 at 17:52:43 UTC, Adam D. Ruppe wrote:
Yeah, but my thought is the typical use case isn't actually the 
problem - it is OK as it is. Longer strings are where it gets 
concerning to me.


Doubling the size of UTF-8 (the effect of the current base16 
encoding) bothers me regardless of string length. Especially when 
use of __MODULE__ and/or __FILE__ as template arguments seems to 
be fairly common.


Having thought about it a bit more, I am now of the opinion that 
super-long strings have no business being in template args, so we 
shouldn't cater to them.


The main case with longer strings going into template arguments 
I'm aware of is for when strings will be processed, then fed to 
`mixin()`. However, that compile time string processing is much 
better served with a CTFE-able function using a Rope for 
manipulating the string until it is finalized into a normal 
string. If you are doing compile-time string manipulation with 
templates, the big symbol is the least of your worries. The 
repeated allocation and reallocation will quickly make your code 
uncompilable due to soaring RAM usage. The same is (was?) true of 
non-rope CTFE string manipulation. Adding a relatively 
memory-intensive operation like compression isn't going to help 
in that case.


Granted, the language shouldn't have a cut-off for string length 
in template arguments, but if you load a huge string as a 
template argument, I think something has gone wrong in your code. 
Catering to that seems to me to be encouraging it, despite the 
existence of much better approaches.


The only other case I can think of where you might want a large 
string as a template argument is something like:


```
struct Foo(string s)
{
mixin(s);
}

Foo!q{...} foo;
```

But that is much better served as something like:

```
mixin template Q()
{
mixin(q{...}); // String doesn't end up in mangled name
}

struct Foo(alias A)
{
mixin A;
}

Foo!Q foo;
```

Or better yet (when possible):

```
mixin template Q()
{
... // No string mixin needed
}

struct Foo(alias A)
{
mixin A;
}

Foo!Q foo;
```

The original mangling discussion started from the need to either 
fix a mangling problem or officially discourage Voldemort types. 
Most of the ideas we've been discussing and/or working on have 
been toward keeping Voldemort types, since many here want them. 
I'm not sure what use case would actually motivate compressing 
strings/symbols.


My motivations for bootstring encoding:

* Mostly care about opDispatch, and use of __FILE__/__MODULE__ as 
compile-time parameters. Symbol bloat from their use isn't 
severe, but it could be better.

* ~50% of the current mangling size for template string parameters
* Plain C identifier strings (so, most identifiers) will end up 
directly readable in the mangled name even without a demangler
* Retains current ability to access D symbols from C (in contrast 
to ideas that would use characters like '$' or '?')
* I already needed bootstring encoding for an unrelated project, 
and figured I could offer to share it with D, since it seems like 
it would fit here, too


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread Adam D. Ruppe via Digitalmars-d

On Thursday, 31 March 2016 at 19:57:50 UTC, Walter Bright wrote:

On 3/31/2016 12:44 PM, H. S. Teoh via Digitalmars-d wrote:

Ah, makes sense.  But what would aa[x] return?


A bool indicating membership.


Ewww. If it looks like an AA, let's at least keep the AA 
interface.


aa[x] returns void, which, having no value, would be a compile 
error.



And how would you add elements to it?


aa[x] = true;  // add member x
aa[x] = false; // remove member x
x in aa; // compile error


aa[x] is a compile error since it is of type void.

x in aa returns void*, null if it is not there, not-null if it is 
there. Dereferencing a void* is illegal anyway so its value is 
irrelevant. (I'd say just use null and cast(void*) 1)


It is actually just a bool, but with consistent typing to other 
AAs.


Phobos's RedBlackTree works with a literal bool from opIn: 
http://dpldocs.info/experimental-docs/std.container.rbtree.RedBlackTree.opBinaryRight.html


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread H. S. Teoh via Digitalmars-d
On Thu, Mar 31, 2016 at 12:57:50PM -0700, Walter Bright via Digitalmars-d wrote:
> On 3/31/2016 12:44 PM, H. S. Teoh via Digitalmars-d wrote:
> >Ah, makes sense.  But what would aa[x] return?
> 
> A bool indicating membership.
> 
> >And how would you add elements to it?
> 
> aa[x] = true;  // add member x
> aa[x] = false; // remove member x
> x in aa; // compile error

How is this different from bool[T] then? Just the fact that you can't
get a reference to the bool?


T

-- 
Don't throw out the baby with the bathwater. Use your hands...


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread Walter Bright via Digitalmars-d

On 3/31/2016 12:44 PM, H. S. Teoh via Digitalmars-d wrote:

Ah, makes sense.  But what would aa[x] return?


A bool indicating membership.


And how would you add elements to it?


aa[x] = true;  // add member x
aa[x] = false; // remove member x
x in aa; // compile error



Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread H. S. Teoh via Digitalmars-d
On Thu, Mar 31, 2016 at 09:39:53PM +0200, Jacob Carlborg via Digitalmars-d 
wrote:
> On 2016-03-31 21:29, H. S. Teoh via Digitalmars-d wrote:
> >On Thu, Mar 31, 2016 at 07:24:14PM +, deadalnix via Digitalmars-d wrote:
> >>Pretty much as per title. I has that in the back of my mind for a
> >>while.  Would that work ?
> >
> >What's a "builtin set of T"?
> 
> int[string] is a built-in associative array, void[string] would be the same
> but a set [1] instead. "T" in his example of be "some type".
> 
> [1] https://en.wikipedia.org/wiki/Set_%28abstract_data_type%29
[...]

Ah, makes sense.  But what would aa[x] return?  And how would you add
elements to it? The current syntax doesn't seem to make sense for sets.


T

-- 
Bare foot: (n.) A device for locating thumb tacks on the floor.


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread tsbockman via Digitalmars-d

On Thursday, 31 March 2016 at 19:24:14 UTC, deadalnix wrote:
Pretty much as per title. I has that in the back of my mind for 
a while. Would that work ?


I like this idea. I actually thought of it myself before, which 
suggests that the syntax would be somewhat intuitive and 
therefore easy to remember.


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread Jacob Carlborg via Digitalmars-d

On 2016-03-31 21:29, H. S. Teoh via Digitalmars-d wrote:

On Thu, Mar 31, 2016 at 07:24:14PM +, deadalnix via Digitalmars-d wrote:

Pretty much as per title. I has that in the back of my mind for a
while.  Would that work ?


What's a "builtin set of T"?


int[string] is a built-in associative array, void[string] would be the 
same but a set [1] instead. "T" in his example of be "some type".


[1] https://en.wikipedia.org/wiki/Set_%28abstract_data_type%29

--
/Jacob Carlborg


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread deadalnix via Digitalmars-d

On Thursday, 31 March 2016 at 19:29:19 UTC, H. S. Teoh wrote:
On Thu, Mar 31, 2016 at 07:24:14PM +, deadalnix via 
Digitalmars-d wrote:
Pretty much as per title. I has that in the back of my mind 
for a while.  Would that work ?


What's a "builtin set of T"?


T


https://en.wikipedia.org/wiki/Set_%28abstract_data_type%29


Re: Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread H. S. Teoh via Digitalmars-d
On Thu, Mar 31, 2016 at 07:24:14PM +, deadalnix via Digitalmars-d wrote:
> Pretty much as per title. I has that in the back of my mind for a
> while.  Would that work ?

What's a "builtin set of T"?


T

-- 
Век живи - век учись. А дураком помрёшь.


Could we reserve void[T] for builtin set of T ?

2016-03-31 Thread deadalnix via Digitalmars-d
Pretty much as per title. I has that in the back of my mind for a 
while. Would that work ?


Re: Cannot use TCMalloc

2016-03-31 Thread Basile B. via Digitalmars-d

On Thursday, 31 March 2016 at 15:28:47 UTC, Nordlöw wrote:
Has anybody compiled and run a D program with TCMalloc instead 
of glibc's own PTMalloc?


The performance, especially multi-thread allocation, looks very 
promising:

http://goog-perftools.sourceforge.net/doc/tcmalloc.html

I tried adding either

-L-ltcmalloc
-L-ltcmalloc_minimal

to DMD but all these errors as

/usr/bin/ld: cannot find -ltcmalloc
/usr/bin/ld: cannot find -ltcmalloc_minimal

none of them works on my Ubuntu 15.10.

It's installed on my system via

sudo apt-get install libtcmalloc-minimal4

and placed at

/usr/lib/libtcmalloc_minimal_debug.so.4.2.6
/usr/lib/libtcmalloc_minimal.so.4.2.6
/usr/lib/libtcmalloc_minimal_debug.so.4
/usr/lib/libtcmalloc_minimal.so.4

What's wrong?

Please help.


You need to install the "-devel" version to get the it as a 
static library. On OpenSuse it's named gperftools-devel.


Maybe on ubuntu it's this one:

"libgoogle-perftools-dev"

http://packages.ubuntu.com/fr/trusty/amd64/libgoogle-perftools-dev/filelist, because it 
contains the "*.a" static library you wanna link in.




Re: Any usable SIMD implementation?

2016-03-31 Thread Johan Engelen via Digitalmars-d

On Thursday, 31 March 2016 at 08:23:45 UTC, Martin Nowak wrote:


I don't want to do anything fancy, just unaligned loads, 
stores, and integral mul/div. Is this really the current state 
of SIMD or am I missing sth.?


I think you want to write your code using SIMD primitives.
But in case you want the compiler to generate SIMD instructions, 
perhaps @ldc.attributes.target may help you:

http://wiki.dlang.org/LDC-specific_language_changes#.40.28ldc.attributes.target.28.22feature.22.29.29

I have not checked what LDC does with SIMD with default 
commandline parameters.


Cheers,
  Johan



Re: Policy for exposing range structs

2016-03-31 Thread Adam D. Ruppe via Digitalmars-d

On Thursday, 31 March 2016 at 17:30:44 UTC, Anon wrote:

My encoding is shorter in the typical use case



Yeah, but my thought is the typical use case isn't actually the 
problem - it is OK as it is. Longer strings are where it gets 
concerning to me.


Would a hybrid approach (my encoding, optionally using 
compression when it would be advantageous) make sense?


Yeah, that might be cool too.

Alternately, we could do the compression on whole mangled 
names, not just the string values, but I don't know how 
desirable that is.


There are often a lot of repeats in there... so maybe.


Re: Policy for exposing range structs

2016-03-31 Thread Anon via Digitalmars-d

On Thursday, 31 March 2016 at 16:46:42 UTC, Adam D. Ruppe wrote:

On Thursday, 31 March 2016 at 16:38:59 UTC, Anon wrote:
I've been spending my D time thinking about potential changes 
to how template string value parameters are encoded.



How does it compare to simply gzipping the string and writing 
it out with base62?


My encoding is shorter in the typical use case, at least when 
using xz instead gzip. (xz was quicker/easier to get raw 
compressed data without a header.)


1= Raw UTF-8, 2= my encoder, 3= `echo -n "$1" | xz -Fraw | base64`

---
1. some_identifier
2. some_identifier_
3. AQA0c29tZV9pZGVudGlmaWVyAA==

1. /usr/include/d/std/stdio.d
2. usrincludedstdstdiod_jqacdhbd
3. AQAZL3Vzci9pbmNsdWRlL2Qvc3RkL3N0ZGlvLmQa

1. Hello, World!
2. HelloWorld_0far4i
3. AQAMSGVsbG8sIFdvcmxkIQA=

1. こんにちは世界
2. XtdCDr5mL02g3rv
3. AQAU44GT44KT44Gr44Gh44Gv5LiW55WMAA==
---

The problem is that compression isn't magical, and a string needs 
to be long enough and have enough repetition to compress well. If 
it isn't, compression causes the data to grow, and base64 
compounds that. For the sake of fairness, let's also do a larger 
(compressible) string.


Input: 1000 lines, each with the text "Hello World"

1. 12000 bytes
2. 12008 bytes
3. 94 bytes

However, my encoding is still fairly compressible, so we *could* 
route it through the same compression if/when a symbol is 
determined to be compressible. That yields 114 bytes.


The other thing I really like about my encoder is that plain C 
identifiers are left verbatim visible in the result. That would 
be especially nice with, e.g., opDispatch.


Would a hybrid approach (my encoding, optionally using 
compression when it would be advantageous) make sense? My encoder 
already has to process the whole string, so it could do some sort 
of analysis to estimate how compressible the result would be. I 
don't know what that would look like, but it could work.


Alternately, we could do the compression on whole mangled names, 
not just the string values, but I don't know how desirable that 
is.


Re: Policy for exposing range structs

2016-03-31 Thread Adam D. Ruppe via Digitalmars-d

On Thursday, 31 March 2016 at 16:38:59 UTC, Anon wrote:
I've been spending my D time thinking about potential changes 
to how template string value parameters are encoded.



How does it compare to simply gzipping the string and writing it 
out with base62?




Re: Policy for exposing range structs

2016-03-31 Thread Anon via Digitalmars-d

On Thursday, 31 March 2016 at 11:15:18 UTC, Johan Engelen wrote:

Hi Anon,
  I've started implementing your idea. But perhaps you already 
have a beginning of an implementation? If so, please contact me 
:)

https://github.com/JohanEngelen

Thanks,
  Johan


No, I haven't started implemented things for that idea. The 
experiments I did with it were by manually altering mangled names 
in Vim.


I've been spending my D time thinking about potential changes to 
how template string value parameters are encoded. My code is a 
bit messy (and not integrated with the compiler at all), but I 
use a bootstring technique (similar to Punycode[1]) to encode 
Unicode text using only [a-zA-Z0-9_].


The results are always smaller than base16 and base64 encodings. 
For plain ASCII text, the encoding tends to grow by a small 
amount. For text containing larger UTF-8 code points, the 
encoding usually ends up smaller than the raw UTF-8 string.


A couple examples of my encoder at work:

---
some_identifier
some_identifier_

/usr/include/d/std/stdio.d
usrincludedstdstdiod_jqacdhbd

Hello, World!
HelloWorld_0far4i


こんにちは世界 (UTF-8: 21 bytes)
XtdCDr5mL02g3rv (15 bytes)
---

I still need to clean up the encoder/decoder and iron out some 
specifics on how this could fit into the mangling, but I should 
have time to work on this some more later today/tomorrow.


[1]: https://en.wikipedia.org/wiki/Punycode


Re: Mindset of the D team vs the Rust team

2016-03-31 Thread Jérôme M . Berger via Digitalmars-d
On 03/24/2016 05:50 PM, Bruno Medeiros wrote:
> Using old communication software like NNTP is one example of that. 
> Compare with Rust's Discourse.
> 
Funny, that's the main reason why I still lurk around the D forums even
though I haven't written a line of D in years, while I don't follow the
Rust forums even though I do most of my experimenting in Rust these days...

Jerome
-- 
mailto:jeber...@free.fr
http://jeberger.free.fr
Jabber: jeber...@jabber.fr



signature.asc
Description: OpenPGP digital signature


Cannot use TCMalloc

2016-03-31 Thread Nordlöw via Digitalmars-d
Has anybody compiled and run a D program with TCMalloc instead of 
glibc's own PTMalloc?


The performance, especially multi-thread allocation, looks very 
promising:

http://goog-perftools.sourceforge.net/doc/tcmalloc.html

I tried adding either

-L-ltcmalloc
-L-ltcmalloc_minimal

to DMD but all these errors as

/usr/bin/ld: cannot find -ltcmalloc
/usr/bin/ld: cannot find -ltcmalloc_minimal

none of them works on my Ubuntu 15.10.

It's installed on my system via

sudo apt-get install libtcmalloc-minimal4

and placed at

/usr/lib/libtcmalloc_minimal_debug.so.4.2.6
/usr/lib/libtcmalloc_minimal.so.4.2.6
/usr/lib/libtcmalloc_minimal_debug.so.4
/usr/lib/libtcmalloc_minimal.so.4

What's wrong?

Please help.


Re: Policy for exposing range structs

2016-03-31 Thread Steven Schveighoffer via Digitalmars-d

On 3/31/16 10:30 AM, Adam D. Ruppe wrote:

On Thursday, 31 March 2016 at 14:00:38 UTC, Steven Schveighoffer wrote:

Ugh, let's try the huffman coding thing first :)


Do that after and along with!


Stack traces would be unusable.


That's why I'd keep the function name outside the hash. The inner spam
wouldn't be readable (not like a megabyte long name is readable
anyway...), but the function name (which includes template arguments*)
still is and that's probably the most useful part anyway.


Moving types outside the function actually results in a pretty readable 
stack trace.


I noticed the stack trace printer just gives up after so many characters 
anyway.


-Steve


Re: foreach_reverse and lockstep.

2016-03-31 Thread Jack Stouffer via Digitalmars-d

On Thursday, 31 March 2016 at 03:12:34 UTC, Sean Campbell wrote:
Why doesn't reverse iteration of lockstep work? It does for 
zip. Is this intended or is it a bug?


Please file an enhancement request at issues.dlang.org


Re: Policy for exposing range structs

2016-03-31 Thread jmh530 via Digitalmars-d
On Thursday, 31 March 2016 at 13:10:49 UTC, Steven Schveighoffer 
wrote:


I too like Voldemort types, but I actually found moving the 
types outside the functions quite straightforward. It's just 
annoying to have to repeat the template parameters. If you make 
them private, then you can simply avoid all the constraints. 
It's a bad leak of implementation, since now anything in the 
file has access to that type directly, but it's better than the 
issues with voldemort types.




If you move anything with a Voldemort type to their own modules, 
then do what you say, then there is no longer an access issue. 
Leads to a proliferation of modules.


I can think of another alternative, but it is probably a needless 
complexity. Suppose there is a protection attribute with the 
property that things in the module can only access it if given 
permission explicitly. For instance, taking the D wiki Voldemort 
type example and modifying it to your approach would give


struct TheUnnameable
{
int value;
this(int x) {value = x;}
int getValue() { return value; }
}

auto createVoldemortType(int value)
{
return TheUnnameable(value);
}

The Unnameable would then be changed to

explicit struct TheUnnameable
{
explicit(createVoldemortType);
int value;
this(int x) {value = x;}
int getValue() { return value; }
}

where explicit used as a protection attribute would restrict 
TheUnnameable to only things where the explicit function gives 
explicit permission.


Re: Policy for exposing range structs

2016-03-31 Thread Adam D. Ruppe via Digitalmars-d
On Thursday, 31 March 2016 at 14:00:38 UTC, Steven Schveighoffer 
wrote:

Ugh, let's try the huffman coding thing first :)


Do that after and along with!


Stack traces would be unusable.


That's why I'd keep the function name outside the hash. The inner 
spam wouldn't be readable (not like a megabyte long name is 
readable anyway...), but the function name (which includes 
template arguments*) still is and that's probably the most useful 
part anyway.


* I just thought of another thing though string arguments to 
templates are included in the mangle, and with CTFE mixin stuff, 
they can become VERY long.


void foo(string s)() {}

pragma(msg, foo!"hi there, friend".mangleof);

_D1p48__T3fooVAyaa16_68692074686572652c20667269656e64Z3fooFNaNbNiNfZv

That "68692074686572652c20667269656e64" portion of it is the 
string represented as hexadecimal.


If you do a CTFE thing, the code string you pass in may be kept 
in ALL the names generated from it.


We should probably do something about these too. Simply gzipping 
before converting to hex is a possible option if we want it 
reversible. Or, of course, hashing too if we don't care about 
that.


And IMO a huge string in a stack trace is unreadable anyway... 
I'd be tempted to say if the string is longer than like 64 chars, 
just hash it. But this is debatable.


A possible thing the compiler *could* do is place inside the 
binary a hash-to-actual-symbol table that the exception printer 
can utilize to print a nicer stack trace...


Indeed. I actually just emailed Liran with this suggestion as 
well. I don't think it is ideal for D in general, but for an 
internal project, it might solve some problems.


Re: Policy for exposing range structs

2016-03-31 Thread Steven Schveighoffer via Digitalmars-d

On 3/31/16 9:38 AM, Adam D. Ruppe wrote:


Of course, a chain of voldemort types will still include that type as a
template argument in the next call, so names as a whole can still be
very long, but how long are your chains? I can see this becoming a 10 KB
long name in extreme circumstances (still yikes) but not megabytes.


If you are hashing anyway, just hash the hash :) In other words, this 
function:


auto foo(T)(T t)
{
   static struct R {}
   return R;
}

would result in a return type of a hash, with no care about whether T 
was a hashed symbol or not. This puts a cap on the size of the name, it 
would never get that big.


-Steve


Re: Policy for exposing range structs

2016-03-31 Thread Steven Schveighoffer via Digitalmars-d

On 3/31/16 9:38 AM, Adam D. Ruppe wrote:

On Thursday, 31 March 2016 at 13:10:49 UTC, Steven Schveighoffer wrote:

Voldemort types are what cause the bloat, templates inside templates
aren't as much of a problem.



So here's an idea of other things don't work out: voldemort types don't
have a name that can be said... that could be true in the mangle too.

We could potentially just hash it to some fixed length. Take the
existing name, SHA1 it, and call the mangle
function_containing_type$that_hash. Demangling the inside is useless
anyway, so we lose nothing from that.


Ugh, let's try the huffman coding thing first :) Stack traces would be 
unusable.


However, this does seem promising if that doesn't work out. The hash 
would definitely solve the linking and binary size issue.


A possible thing the compiler *could* do is place inside the binary a 
hash-to-actual-symbol table that the exception printer can utilize to 
print a nicer stack trace...


-Steve


Re: Policy for exposing range structs

2016-03-31 Thread Adam D. Ruppe via Digitalmars-d
On Thursday, 31 March 2016 at 13:10:49 UTC, Steven Schveighoffer 
wrote:
Voldemort types are what cause the bloat, templates inside 
templates aren't as much of a problem.



So here's an idea of other things don't work out: voldemort types 
don't have a name that can be said... that could be true in the 
mangle too.


We could potentially just hash it to some fixed length. Take the 
existing name, SHA1 it, and call the mangle 
function_containing_type$that_hash. Demangling the inside is 
useless anyway, so we lose nothing from that.


For example, a function foo.bar() returning a voldemort type 
might have the mangle:


_D3foo3barv$55ca6286e3e4f4fba5d0448333fa99fc5a404a73

where that's the hash of whatever filth the compiler makes up for 
it.


and methods on the voldemort type would start with that too:

_D503foo3barv$55ca6286e3e4f4fba5d0448333fa99fc5a404a735empty

That's getting messy to see as an example, but it took the name 
of the function + the voldemort as a whole to be the public name 
of the type.


A demangler could scan that for the $ and recognize it as 
ugliness and slice it off, printing the name as a somewhat 
human-readable


foo.bar().empty

perhaps, so we can see it is a method on the return value of that 
function. It isn't completely impossible like demanging a whole 
hash; should be short in the binary and readable enough to make 
sense of in a stack trace.




I actually recall something like this being done at some point in 
the past, but I don't remember the details. I think it just 
truncated the name though, it didn't attempt to remain 
semi-reversible.




Of course, a chain of voldemort types will still include that 
type as a template argument in the next call, so names as a whole 
can still be very long, but how long are your chains? I can see 
this becoming a 10 KB long name in extreme circumstances (still 
yikes) but not megabytes.



And, still, having to run over the names to build the hash is 
going to be a memory/speed thing in the compiler, but SHA1ing a 
few megabytes isn't as slow as writing it out to disk over and 
over again.




Another thing to consider is what my D to Javascript thing did: 
abbreviate EVERYTHING in the object file. You can't demangle that 
at all, but it leads to much smaller binaries. This probably 
isn't that useful for D in general, but might solve Weka's 
problem since they can use internal hacks.


Re: Policy for exposing range structs

2016-03-31 Thread Steven Schveighoffer via Digitalmars-d

On 3/30/16 3:19 PM, Liran Zvibel wrote:

On Sunday, 27 March 2016 at 17:01:39 UTC, David Nadlinger wrote:

Compression in the usual sense won't help. Sure, it might reduce the
object file size, but the full string will again have to be generated
first, still requiring absurd amounts time and space. The latter is
definitely not negligible for symbol names of several hundreds of
kilobytes; it shows up prominently in compiler profiles of affected
Weka builds.


We love Voldemort types at Weka, and use them a lot in our
non-gc-allocating ranges and algorithm libraries. Also, we liberally
nest templates inside of other templates.
I don't think we can do many of the things we do if we had to define
everything at module level. This flexibility is amazing for us and part
of the reason we love D.


Voldemort types are what cause the bloat, templates inside templates 
aren't as much of a problem. It's because the Voldemort type has to 
include in its symbol name at least twice, and I think 3 times actually 
(given the return type), the template parameter/function parameter types 
of the function it resides in. If the template is just a template, it's 
just included once. This is why moving the type outside the function is 
effective at mitigation. It's linear growth vs. exponential.


I too like Voldemort types, but I actually found moving the types 
outside the functions quite straightforward. It's just annoying to have 
to repeat the template parameters. If you make them private, then you 
can simply avoid all the constraints. It's a bad leak of implementation, 
since now anything in the file has access to that type directly, but 
it's better than the issues with voldemort types.


See the update to my iopipe library here: 
https://github.com/schveiguy/iopipe/commit/1b0696dc82fce500c6b314ec3d8e5e11e0c1bcd7


This one commit made my example program 'convert' 
(https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d) 
save over 90% binary size (went from 10MB to <1MB).


This also calmed down some REALLY horrible stack traces when I was 
debugging. As in, I could actually understand what function it was 
talking about, and it didn't take 10 seconds to print stack trace.




But, as David said -- it comes with a great price for us.

I just processed our biggest executable, and came up with the following
numbers:
total symbols: 99649
Symbols longer than 1k: 9639
Symbols longer than 500k: 102
Symbols longer than 1M: 62. The longest symbols are about 5M bytes!

This affects our exe sizes in a terrible way, and also increases our
compile and link times considerably. I will only be able to come up with
statistics of how much time was wasted due to too-long-symbols after we
fix it, but obviously this is a major problem for us.


From my testing, it doesn't take much to get to the point where the 
linker is unusable. A simple struct when nested in 15 calls to a 
function makes the linker take an unreasonable amount of time (over 1.5 
minutes, I didn't wait to see how long). See my bug report for details.


Another factor in the name length is the module name which is included 
in every type and function. So you have a factor like 3^15 for the name, 
but then you multiply this by the module names as well.



I think we should try the solution proposed by Anon, as it has a good
possibility of saving quite a bit.
It's important to make sure that when a template is given as a template
parameter, the complete template is treated as the LName.


I hope this is given serious thought, looks like someone has already 
started implementation.


Anon, it appears that your mechanism has been well received by a few 
knowledgeable people here. I encourage you to solidify your proposal in 
a DIP (D improvement proposal) here: http://wiki.dlang.org/DIPs.


-Steve


Re: Any usable SIMD implementation?

2016-03-31 Thread John Colvin via Digitalmars-d

On Thursday, 31 March 2016 at 08:23:45 UTC, Martin Nowak wrote:
I'm currently working on a templated arrayop implementation 
(using RPN

to encode ASTs).
So far things worked out great, but now I got stuck b/c 
apparently none
of the D compilers has a working SIMD implementation (maybe GDC 
has but

it's very difficult to work w/ the 2.066 frontend).

https://github.com/MartinNowak/druntime/blob/arrayOps/src/core/internal/arrayop.d
 https://github.com/MartinNowak/dmd/blob/arrayOps/src/arrayop.d

I don't want to do anything fancy, just unaligned loads, 
stores, and integral mul/div. Is this really the current state 
of SIMD or am I missing sth.?


-Martin


Am I being stupid or is core.simd what you want?


Re: Policy for exposing range structs

2016-03-31 Thread Johan Engelen via Digitalmars-d

On Saturday, 26 March 2016 at 17:42:06 UTC, Anon wrote:


The (conceptually) simple change I suggested brings the mangled 
name length down to O(n).


Hi Anon,
  I've started implementing your idea. But perhaps you already 
have a beginning of an implementation? If so, please contact me :)

https://github.com/JohanEngelen

Thanks,
  Johan


Re: Any usable SIMD implementation?

2016-03-31 Thread ZombineDev via Digitalmars-d

On Thursday, 31 March 2016 at 08:23:45 UTC, Martin Nowak wrote:
I'm currently working on a templated arrayop implementation 
(using RPN

to encode ASTs).
So far things worked out great, but now I got stuck b/c 
apparently none
of the D compilers has a working SIMD implementation (maybe GDC 
has but

it's very difficult to work w/ the 2.066 frontend).

https://github.com/MartinNowak/druntime/blob/arrayOps/src/core/internal/arrayop.d
 https://github.com/MartinNowak/dmd/blob/arrayOps/src/arrayop.d

I don't want to do anything fancy, just unaligned loads, 
stores, and integral mul/div. Is this really the current state 
of SIMD or am I missing sth.?


-Martin


I don't know how far has Ilya's work [1] advanced, but you may 
want to join efforts with him. There are also two std.simd 
packages [2] [3].


BTW, I looked at your code a couple of days ago and I thought 
that it is a really interesting approach to encode operations 
like that. I'm just wondering if pursuing this approach is a good 
idea in the long run, i.e. is it expressible enough to cover the 
use cases of HPC which would also need something similar, but for 
custom linear algebra types.


Here's an interesting video about approaches to solving this 
problem in C++: https://www.youtube.com/watch?v=hfn0BVOegac


[1]: 
http://forum.dlang.org/post/nilhvnqbsgqhxdshp...@forum.dlang.org


[2]: https://github.com/D-Programming-Language/phobos/pull/2862

[3]: https://github.com/Iakh/simd


Re: foreach_reverse and lockstep.

2016-03-31 Thread ZombineDev via Digitalmars-d

On Thursday, 31 March 2016 at 03:12:34 UTC, Sean Campbell wrote:
Why doesn't reverse iteration of lockstep work? It does for 
zip. Is this intended or is it a bug?


Lockstep is actually not a range. It overloads the opApply[1] 
operator to support foreach. To support foreach_reverse, it would 
also have to override opApplyReverse. I'd say that it's not a bug 
but still a valid enhancement request. BTW the upcomming release 
has better documentation on the differences between lockstep [2] 
and zip [3].


[1]: 
https://github.com/D-Programming-Language/phobos/blob/master/std/range/package.d#L4153


[2]: http://dlang.org/phobos-prerelease/std_range#lockstep

[3]: http://dlang.org/phobos-prerelease/std_range#zip


Any usable SIMD implementation?

2016-03-31 Thread Martin Nowak via Digitalmars-d
I'm currently working on a templated arrayop implementation (using RPN
to encode ASTs).
So far things worked out great, but now I got stuck b/c apparently none
of the D compilers has a working SIMD implementation (maybe GDC has but
it's very difficult to work w/ the 2.066 frontend).

https://github.com/MartinNowak/druntime/blob/arrayOps/src/core/internal/arrayop.d
https://github.com/MartinNowak/dmd/blob/arrayOps/src/arrayop.d

I don't want to do anything fancy, just unaligned loads, stores, and
integral mul/div. Is this really the current state of SIMD or am I
missing sth.?

-Martin


Re: Pre-alpha D language online tour

2016-03-31 Thread André via Digitalmars-d

On Wednesday, 30 March 2016 at 19:58:32 UTC, Mark Isaacson wrote:
This is awesome! My one complaint is that the section of the 
screen that contains the code doesn't scale well vertically 
when my browser window is large. The written explanations scale 
to fit the content, the code editor does not. I'd just make it 
stretch vertically to fill the screen :). At that point you 
might also consider putting the Run/Reset buttons at the top.


Yeah I am not satisfied either. I have some issues open at the 
GitHub page regarding the layout and polishing. I will have your 
suggestions in mind! Thanks.

- André


Re: Pre-alpha D language online tour

2016-03-31 Thread André via Digitalmars-d

On Thursday, 31 March 2016 at 06:32:00 UTC, Jacob Carlborg wrote:

On 2016-03-29 18:26, André wrote:


http://dlang-tour.steinsoft.net


The example at [1] doesn't compile. "f" is declared twice and 
there's no built in property "name".


[1] http://dlang-tour.steinsoft.net/tour/basics/2


Thank you for finding this one! Fixed and will be online sonn.
- André