Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Fri, 13 Jul 2018 18:14:35 +
schrieb iris :

> Any idea about the performance of this json parser? 
> https://jsonformatter.org/json-parser ?

That one is implemented in client side JavaScript. I didn't
measure it, but the closest match in Kostya's benchmark could
be the Node JS entry that is an order of magnitude slower.

-- 
Marco

Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side

[Marco Leise via Digitalmars-d-announce]

I understand your opinion and I think it is all reasonable.
You talk about longer compile times since every D module is
like a C++ header. That touches one of my pet peeves with the
language or eco system as it stands and I wonder if you would
agree with me on the following:

Libraries should be tied into applications using interface
files (*.di) that are auto-generated by the compiler for the
_library author_ with inferred function attributes. If after
a code change, a declaration in the *.di file changes, the
library's interface changed and a new minor version must be
released. The language must allow to explicitly declare a
function or method as @gc, impure, etc. so the auto-inferred
attributes don't later become an issue when the implementation
changes from e.g. a pure to an impure one.
Opaque struct pointers as seen in most C APIs should also be
considered for *.di files to reduce the number of imports for
member fields.

That means:
* No more fuzzyness about whether a library function
  will remain @nogc, @safe, etc. in the next update.
* Explicit library boundaries that don't recursively import the
  world.

-- 
Marco

Re: Beta 2.076.1

[Marco Leise via Digitalmars-d-announce]

P.S.: The directory layout could be improved as well.
Currently there is:

src\
   +-dmd
   +-druntime
   +-phobos

But druntime in posix.mak:10 expects a src directory inside
the dmd directory:

dmd\
   +-src

So effectively the directory names have to be swapped for
that to work. At that point the superfluous directories for
the other operating systems, containing only dmd.conf could be
removed as well.

Other than that I'm happy with the package, as it provides the
man pages, pre-built HTML documentation and a binary to
bootstrap dmd on systems that lack a D compiler. (The use case
being compilation from source for Gentoo Linux.)

-- 
Marco

Re: Beta 2.076.1

[Marco Leise via Digitalmars-d-announce]

Could you include the "default_ddoc_theme.ddoc" and
"config.sh" in the source releases? The sources cannot be
compiled without them.

-- 
Marco

Re: new cpuid is ready for comments

[Marco Leise via Digitalmars-d-announce]

Am Mon, 11 Jul 2016 16:30:44 +
schrieb Ilya Yaroshenko :

> Please report your CPU (GitHub/Gist):

Instead of yet another Haswell/Broadwell, here is an oldie but
goldie: A 16 year old Pentium-III-M from an IBM T23 notebook:

https://gist.github.com/mleise/4ed5637371ca5cea07bef8241adc7d80

I repost this in your bug tracker momentarily.

-- 
Marco

Re: QtE5 - is a wrapping of Qt-5 for D

[Marco Leise via Digitalmars-d-announce]

Very nice!

-- 
Marco

Re: LZ4 decompression at CTFE

[Marco Leise via Digitalmars-d-announce]

Am Tue, 26 Apr 2016 23:55:46 -0700
schrieb Walter Bright :

> On 4/26/2016 3:05 PM, Stefan Koch wrote:
> > Hello,
> >
> > originally I want to wait with this announcement until DConf.
> > But since I working on another toy. I can release this info early.
> >
> > So as per title. you can decompress .lz4 flies created by the standard lz4hc
> > commnadline tool at compile time.
> >
> > No github link yet as there is a little bit of cleanup todo :)
> >
> > Please comment.  
> 
> Sounds nice. I'm curious how it would compare to:
> 
> https://www.digitalmars.com/sargon/lz77.html
> 
> https://github.com/DigitalMars/sargon/blob/master/src/sargon/lz77.d

There exist some comparisons for the C++ implementations
(zlib's DEFLATE being a variation of lz77):
http://catchchallenger.first-world.info//wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO
https://pdfs.semanticscholar.org/9b69/86f2fff8db7e080ef8b02aa19f3941a61a91.pdf 
(pg.9)

The high compression variant of lz4 basically like gzip with
9x faster decompression. That makes it well suited for use
cases where you compress once, decompress often and I/O
sequential reads are fast e.g. 200 MB/s or the program does
other computations meanwhile and one doesn't want
decompression to use a lot of CPU time.

-- 
Marco

Re: Blog post: PGO: Optimizing D's virtual calls

[Marco Leise via Digitalmars-d-announce]

Am Fri, 15 Apr 2016 09:04:21 +
schrieb Johan Engelen :

> On Wednesday, 13 April 2016 at 11:34:26 UTC, Johan Engelen wrote:
> > Hi all,
> >   I've written an article about how I implemented 
> > profile-guided optimization (PGO) of virtual calls to direct 
> > calls (a micro-micro-optimization, expected performance gain of 
> > just a few percent if any!).  
> 
> I've updated the article with performance measurements of just 
> the D code of LDC: the improvement with PGO is about 7% !

Compilers are full of logic that makes some code 0.5% faster.
The sum of all these little bits is what makes up an
optimizing compiler and if today you still find an optimization
that's worth 1 or 2% you have a jackpot!

7% is quite a bit more than any other option like -cpu=native
could achieve. If feasible I'd enjoy if ldc2 shipped with a
pgo configuration option. If you use the compiler in a lot of
edit/compile cycles it will pay off.

-- 
Marco

Re: Release D 2.071.0

[Marco Leise via Digitalmars-d-announce]

Am Thu, 07 Apr 2016 10:13:35 +
schrieb Dicebot :

> On Tuesday, 5 April 2016 at 22:43:05 UTC, Martin Nowak wrote:
> > Glad to announce D 2.071.0.
> >
> > http://dlang.org/download.html
> >
> > This release fixes many long-standing issues with imports and 
> > the module
> > system.
> > See the changelog for more details.
> >
> > http://dlang.org/changelog/2.071.0.html
> >
> > -Martin  
> 
> It is second time git tag for DMD release refers to commit with 
> VERSION file content which doesn't match the tag. May indicate 
> something is wrong in release procedure.

Or maybe something is wrong with source based Linux
distributions in this time and age. I don't know. But I'm glad
that you fire proof the source bundles first, before I move my
lazy ass to update DMD packages for Gentoo. I hope to start
from a good, clean 2.071.0/2.071.1 tar ball. :D

Nice work on the import bugs. There is so much less on the
attic now.

-- 
Marco

Re: D Profile Viewer

[Marco Leise via Digitalmars-d-announce]

Am Thu, 24 Mar 2016 20:34:07 +
schrieb Andrew :

> Hi
> 
> I wrote a program to turn the non-human-readable trace.log into 
> an interactive HTML file that can be used to help profile a D 
> program.
> 
> Its here: https://bitbucket.org/andrewtrotman/d-profile-viewer
> 
> There's also a readme that (hopefully) explains how to use it.
> 
> Please let me know if you find any bugs.
> 
> Andrew.

Sexy pie charts! Although I'm using OProfile since it works
without instrumenting the code.

-- 
Marco

Re: Graillon 1.0, VST effect fully made with D

[Marco Leise via Digitalmars-d-announce]

We can probably agree that we don't know about the impact on a
large multimedia application written in D. What you can
communicate is: Create a @nogc thread routine and don't
register it with the GC to write real-time VSTs.

Guillaume did a good job, taking the GC out of the real-time
thread. It's D, it is a bit of a hack, it's the correct way to
do it and works. But I don't see it debunking any myths about
GC and real time...
A) It doesn't mix them to begin with.
B) "Realtime GCs" are a thing. D's GC is not optimized for
   such a use case.
C) With a small heap it doesn't matter. (We need more complex
   multimedia projects.)

What I've seen is a program, a non-linear video editor, called
PowerDirector that pauses for seconds every now and then.
These pauses reminded me a lot of GC pauses, but I can't be
sure. Although memory use is less after the pause, it could
also be a cleaning of caches. In any case quite a few of these
applications try to make "good use" of available RAM, causing
constant memory pressure.

Now there has been so much talk about the GC that I don't even
know what the filter does!

-- 
Marco

Re: Tonight: Introduction to D at Codeaholics (HK)

[Marco Leise via Digitalmars-d-announce]

Am Thu, 12 Nov 2015 01:30:06 +0800
schrieb Lionello Lunesu :

> * Why doesn't D explicitly specify the exceptions that can be thrown? 
> (To which I answered that I never saw the point in Java and only found 
> it tedious. This did not convince the person.)

Maybe that's your point of view or maybe you were just
undecided. When you write a library it is sometimes better to
be explicit about your interface and that includes any
exceptions. This not only enables users of the library to
selectively catch exceptions they can handle at layer X, but
facilitates static checks:

* Are any exceptions missing from DDoc/@throws that are
  thrown in the code? (Potential for auto-generating the DDoc.)
* A function is nothrow, but the user only catches, e.g.
  UtfException explicitly. Is that the only exception type
  that could occur?

There were some more nice points that I don't remember from
when I failed at implementing this many months ago. The
questioner has my sympathy in any case, but it's certainly not
a priority.

The way I wanted to implement it was by making attribute-less
functions map to @throws(Exception), which implicitly makes
the feature opt-in: It is always correct to state a super set
of the actual thrown exceptions in an API to have room for
extensions. Thrown exceptions would be collected much like
nothrow is deduced right now, but as a list with respect to the
hierarchical nature of Exceptions.

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Tue, 27 Oct 2015 14:00:06 +
schrieb Martin Nowak :

> On Tuesday, 27 October 2015 at 13:14:36 UTC, wobbles wrote:
> >> How can `coordinates` member be known at compile-time when the 
> >> input argument is a run-time string?
> >
> > I suspect through the opDispatch operator overload.
> >
> > http://dlang.org/operatoroverloading.html#dispatch
> 
> Yikes, this is such an anti-pattern.
> https://github.com/rejectedsoftware/vibe.d/issues/634

For my defense I can say that the JSON parser is not a range
and thus less likely to be used in UFCS chains. It can be
replaced with .singleKey!"coordinates"()

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Thu, 22 Oct 2015 06:10:56 -0700
schrieb Walter Bright :

> On 10/21/2015 3:40 PM, Laeeth Isharc wrote:
> > Have you thought about writing up your experience with writing fast json?  
> > A bit
> > like Walter's Dr Dobbs's article on wielding a profiler to speed up dmd.
> 
> Yes, Marco, please. This would make an awesome article, and we need articles 
> like that!
> 
> You've already got this:
> 
>  https://github.com/kostya/benchmarks/pull/46#issuecomment-147932489
> 
> so most of it is already written.

There is at least one hurdle. I don't have a place to publish
articles, no personal blog or site I contribute articles to
and I don't feel like creating a one-shot one right now. :)

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Wed, 21 Oct 2015 04:17:16 +
schrieb Laeeth Isharc :

> Very impressive.
> 
> Is this not quite interesting ?  Such a basic web back end 
> operation, and yet it's a very different picture from those who 
> say that one is I/O or network bound.  I already have JSON files 
> of a couple of gig, and they're only going to be bigger over 
> time, and this is a more generally interesting question.
> 
> Seems like you now get 2.1 gigbytes/sec sequential read from a 
> cheap consumer SSD today...

You have this huge amount of Reddit API JSON, right?
I wonder if your processing could benefit from the fast
skipping routines or even reading it as "trusted JSON".

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Wed, 21 Oct 2015 17:00:39 +
schrieb Suliman :

> >> > Nice! I see you are using bitmasking trickery in multiple 
> >> > places. stdx.data.json is mostly just the plain lexing 
> >> > algorithm, with the exception of whitespace skipping. It was 
> >> > already very encouraging to get those benchmark numbers that 
> >> > way. Good to see that it pays off to go further.
> >>
> >> Is there any chance that new json parser can be include in 
> >> next versions of vibed? And what need to including its to 
> >> Phobos?
> >
> > It's already available on code.dlang.org:
> > http://code.dlang.org/packages/std_data_json
> 
> 
> Jonatan, I mean https://github.com/mleise/fast :)

That's nice, but it has a different license and I don't think
Phobos devs would be happy to see all the inline assembly I
used and duplicate functionality like the number parsing and
UTF-8 validation and missing range support.

-- 
Marco

Re: Beta D 2.069.0-b2

[Marco Leise via Digitalmars-d-announce]

Am Tue, 20 Oct 2015 19:26:13 +0200
schrieb Martin Nowak <code+news.digitalm...@dawg.eu>:

> On 10/17/2015 09:05 PM, Marco Leise wrote:
> > Oh wait, false alert. That was a relic from older days. My
> > build script placed a dummy dmd.conf there.
> > 
> > I do seem to get problems with ldc2-0.16.0:
> 
> Are you using something befor 0.16.0-beta2, b/c I thought the problem
> was resolved.
> https://github.com/D-Programming-Language/dmd/pull/5025#issuecomment-142143727

Indeed I should have checked that. I'm using 0.16.0_alpha4.
Alright then. Everything works as designed now. :)

-- 
Marco

Re: Beta D 2.069.0-b2

[Marco Leise via Digitalmars-d-announce]

Oh wait, false alert. That was a relic from older days. My
build script placed a dummy dmd.conf there.

I do seem to get problems with ldc2-0.16.0:

  make -C druntime -f posix.mak MODEL=32 
  ../dmd/src/dmd -conf= -c -o- -Isrc -Iimport -Hfimport/core/sync/barrier.di 
src/core/sync/barrier.d
  core.exception.AssertError@expression.d(4369): Assertion failure

That is this line of code:
https://github.com/D-Programming-Language/dmd/blob/v2.069.0-b2/src/expression.d#L4369

Stack trace (with file + line numbers now, hey!):
#0  StringExp::compare(RootObject*) (this=0xb66e30, obj=0xb65c80) at 
expression.d:4341
#1  0x004fb6ed in StringExp::equals(RootObject*) (this=0xb66e30, 
o=0xb65c80) at expression.d:4175
#2  0x004c4fe9 in match(RootObject*, RootObject*) (o1=0xb66e30, 
o2=0xb65c80) at dtemplate.d:246
#3  0x004c51c6 in arrayObjectMatch(Array*, 
Array*) (oa1=0x764aca98, oa2=0x764ac898) at dtemplate.d:290
#4  0x004cddd7 in TemplateInstance::compare(RootObject*) 
(this=0x764aca00, o=0x764ac800) at dtemplate.d:6232
#5  0x004cdaf8 in 
TemplateDeclaration::findExistingInstance(TemplateInstance*, 
Array*) (this=0x764ac600, tithis=0x764aca00, fargs=0x0) at 
dtemplate.d:2039
#6  0x004d2f24 in TemplateInstance::semantic(Scope*, 
Array*) (this=0x764aca00, sc=0x765dfc00, fargs=0x0) at 
dtemplate.d:5583
#7  0x00406877 in TemplateInstance::semantic(Scope*) 
(this=0x764aca00, sc=0x765dfc00) at dtemplate.d:5967
#8  0x0057a03a in TypeInstance::resolve(Loc, Scope*, Expression**, 
Type**, Dsymbol**, bool) (this=0x764ae100, loc=..., sc=0x765dfc00, 
pe=0x7fffcec0, pt=0x7fffcec8, ps=0x7fffceb8, intypeid=false) at 
mtype.d:7412
#9  0x0057a327 in TypeInstance::toDsymbol(Scope*) (this=0x764ae100, 
sc=0x765dfc00) at mtype.d:7459
#10 0x0043c5d6 in AliasDeclaration::semantic(Scope*) 
(this=0x764ae200, sc=0x765dfc00) at .:598
#11 0x004897f9 in Module::semantic() (this=0x76378400) at 
dmodule.d:976
#12 0x00488e0f in Import::semantic(Scope*) (this=0x76a82800, 
sc=0x76aa9c00) at dimport.d:258
#13 0x0042759a in AttribDeclaration::semantic(Scope*) 
(this=0x76a82900, sc=0x76aa9600) at attrib.d:168
#14 0x004897f9 in Module::semantic() (this=0x76a7fe00) at 
dmodule.d:976
#15 0x00488e0f in Import::semantic(Scope*) (this=0x77eca100, 
sc=0x77ed2200) at dimport.d:258
#16 0x0042759a in AttribDeclaration::semantic(Scope*) 
(this=0x77eca200, sc=0x77ecee00) at attrib.d:168
#17 0x004897f9 in Module::semantic() (this=0x77ec9000) at 
dmodule.d:976
#18 0x00567de5 in tryMain(unsigned long, char const**) (argc=8, 
argv=0x7fffe978) at mars.d:1485
#19 0x0056a567 in D main () at mars.d:1695

`sz` is 0, `string` and `len` seem to be ok.

-- 
Marco

Re: Beta D 2.069.0-b2

[Marco Leise via Digitalmars-d-announce]

Am Wed, 14 Oct 2015 15:52:57 +0200
schrieb Martin Nowak :

> Second beta for the 2.069.0 release.
> 
> http://dlang.org/download.html#dmd_beta
> http://dlang.org/changelog/2.069.0.html
> 
> Please report any bugs at https://issues.dlang.org
> 
> -Martin

When I use a specific host compiler, it still picks up the
dmd.conf provided in the package and doesn't find object.d.
Should I manually delete dmd.conf before building?

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Sat, 17 Oct 2015 16:27:06 +
schrieb Sean Kelly :

> On Saturday, 17 October 2015 at 16:14:01 UTC, Andrei Alexandrescu 
> wrote:
> > On 10/17/15 6:43 PM, Sean Kelly wrote:
> >> If this is the benchmark I'm remembering, the bulk of the time 
> >> is spent
> >> parsing the floating point numbers. So it isn't a test of JSON 
> >> parsing
> >> in general so much as the speed of scanf.
> >
> > In many cases the use of scanf can be replaced with drastically 
> > faster methods, as I discuss in my talks on optimization 
> > (including Brasov recently). I hope they'll release the videos 
> > soon. -- Andrei
> 
> Oh absolutely. My issue with the benchmark is just that it claims 
> to be a JSON parser benchmark but the bulk of CPU time is 
> actually spent parsing floats. I'm on my phone though so perhaps 
> this is a different benchmark--I can't easily check. The one I 
> recall came up a year or so ago and was discussed on D.general.

1/4 to 1/3 of the time is spent parsing numbers in highly
optimized code. You see that in a profiler the number parsing
shows up on top, but the benchmark also exercises the
structural parsing a lot. It is not a very broad benchmark
though, lacking serialization, UTF-8 decoding, validation of
results etc. I believe the author didn't realize how over time
it became the go-to performance test. The author of RapidJSON
has a very in-depth benchmark suite, but it would be a bit of
work to get something non-C++ integrated:
https://github.com/miloyip/nativejson-benchmark
It includes conformance tests as well.

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Thu, 15 Oct 2015 18:46:12 +0200
schrieb Sönke Ludwig <slud...@rejectedsoftware.com>:

> Am 14.10.2015 um 09:01 schrieb Marco Leise:
> > […]
> > stdx.data.json:  2.76s,  207.1Mb  (LDC)
> >
> > Yep, that's right. stdx.data.json's pull parser finally beats
> > the dynamic languages with native efficiency. (I used the
> > default options here that provide you with an Exception and
> > line number on errors.)
> 
>  From when are the numbers for stdx.data.json? The latest results for 
> the pull parser that I know of were faster than RapidJson:
> http://forum.dlang.org/post/wlczkjcawyteowjbb...@forum.dlang.org

You know, I'm not surprised at the "D new lazy Ldc" result,
which is in the ball park figure of what I measured without
exceptions & line-numbers, but the Rapid C++ result seems way
off compared to kostya's listing. Or maybe that Core i7 doesn't
work well with RapidJSON.

I used your fork of the benchmark, made some modifications
like adding taggedalgebraic and what else was needed to make
it compile with vanilla ldc2 0.16.0. Then I removed the flags
that disable exceptions and line numbers. Compilation options
are the same as for the existing gdc and ldc2 entries. I did
not add " -partial-inliner -boundscheck=off -singleobj ".

> Judging by the relative numbers, your "fast" result is still a bit 
> faster, but if that doesn't validate, it's hard to make an objective 
> comparison.

Every value that is read (as opposed to skipped) is validated
according to RFC 7159. That includes UTF-8 validation. Full
validation (i.e. readJSONFile!validateAll(…);) may add up to
14% overhead here.

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Thu, 15 Oct 2015 18:17:07 +0200
schrieb Sönke Ludwig :

> Am 15.10.2015 um 13:06 schrieb Rory McGuire via Digitalmars-d-announce:
> > In browser JSON.serialize is the usual way to serialize JSON values.
> > The problem is that on D side if one does deserialization of an object
> > or struct. If the types inside the JSON don't match exactly then vibe
> > freaks out.
> 
> For float and double fields, the serialization code should actually 
> accept both, floating point and integer numbers:
> 
> https://github.com/rejectedsoftware/vibe.d/blob/2fffd94d8516cd6f81c75d45a54c655626d36c6b/source/vibe/data/json.d#L1603
> https://github.com/rejectedsoftware/vibe.d/blob/2fffd94d8516cd6f81c75d45a54c655626d36c6b/source/vibe/data/json.d#L1804
> 
> Do you have a test case for your error?
 
Well it is not an error. Rory originally wrote about
conversions between "1" and 1 happening on the browser side.
That would mean adding a quirks mode to any well-behaving JSON
parser. In this case: "read numbers as strings". Hence I was
asking if the data on the client could be fixed, e.g. the json
number be turned into a string first before serialization.

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Fri, 16 Oct 2015 11:09:37 +
schrieb Per Nordlöw <per.nord...@gmail.com>:

> On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote:
> > https://github.com/kostya/benchmarks#json
> 
> Does fast.json use any non-standard memory allocation patterns or 
> plain simple GC-usage?

Plain GC.  I found no way in D to express something as
"borrowed", but the GC removes the ownership question from the
picture. That said the only thing that I allocate in this
benchmark is the dynamic array of coordinates (1_000_000 * 3 *
double.sizeof), which can be replaced by lazily iterating over
the array to remove all allocations.

Using these lazy techniques for objects too and
calling .borrow() instead of .read!string() (which .idups)
should allow GC free usage. (Well, except for the one in
parseJSON, where I append 16 zero bytes to the JSON string to
make it SSE safe.)

-- 
Marco

Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

fast.json usage:

UTF-8 and JSON validation of used portions by default:

auto json = parseJSONFile("data.json");

Known good file input:

auto json = parseTrustedJSONFile("data.json");
auto json = parseTrustedJSON(`{"x":123}`);

Work with a single key from an object:

json.singleKey!"someKey"
json.someKey

Iteration:

foreach (key; json.byKey)  // object by key
foreach (idx; json)// array by index

Remap member names:

@JsonRemap(["clazz", "class"])
struct S { string clazz; }

@JsonRemap(["clazz", "class"])
enum E { clazz; }

Example:

double x = 0, y = 0, z = 0;
auto json = parseTrustedJSON(`{ "coordinates": [ { "x": 1, "y": 2, "z": 3 
}, … ] }`);

foreach (idx; json.coordinates)
{
// Provide one function for each key you are interested in
json.keySwitch!("x", "y", "z")(
{ x += json.read!double; },
{ y += json.read!double; },
{ z += json.read!double; }
);
}

Features:
  - Loads double values in compliance with IEEE round-to-nearest
(no precision loss in serialization->deserialization round trips)
  - UTF-8 validation of non-string input (file, ubyte[])
  - Currently fastest JSON parser money can buy
  - Reads strings, enums, integral types, double, bool, POD
structs consisting of those and pointers to such structs

Shortcomings:
  - Rejects numbers with exponents of huge magnitude (>=10^28)
  - Only works on Posix x86/amd64 systems
  - No write capabilities
  - Data size limited by available contiguous virtual memory

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Wed, 14 Oct 2015 07:55:18 +
schrieb Idan Arye <generic...@gmail.com>:

> On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise wrote:
> > auto json = parseTrustedJSON(`{ "coordinates": [ { "x": 1, 
> > "y": 2, "z": 3 }, … ] }`);
> 
> I assume parseTrustedJSON is not validating? Did you use it in 
> the benchmark? And were the competitors non-validating as well?

That is correct. For the benchmark parseJSONFile was used
though, which validates UTF-8 and JSON in the used portions.
That probably renders your third question superfluous. I
wouldn't know anyways, but am inclined to think they all
validate the entire JSON and some may skip UTF-8 validation,
which is a low cost operation in this ASCII file anyways.

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Wed, 14 Oct 2015 10:22:37 +0200
schrieb Rory McGuire via Digitalmars-d-announce
:

> Does this version handle real world JSON?
> 
> I've keep getting problems with vibe and JSON because web browsers will
> automatically make a "1" into a 1 which then causes exceptions in vibe.
> 
> Does yours do lossless conversions automatically? 

No I don't read numbers as strings. Could the client
JavaScript be fixed? I fail to see why the conversion would
happen automatically when the code could explicitly check for
strings before doing math with the value "1". What do I miss?

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Wed, 14 Oct 2015 08:19:52 +
schrieb Per Nordlöw <per.nord...@gmail.com>:

> On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote:
> > https://github.com/kostya/benchmarks#json
> 
> I can't find fast.json here. Where is it?

»»» D Gdc Fast  0.34226.7 «««
C++ Rapid   0.79687.1

Granted if he wrote "D fast.json" it would have been easier to
identify.

-- 
Marco

Re: Walter and I talk about D in Romania

[Marco Leise via Digitalmars-d-announce]

Am Fri, 2 Oct 2015 07:25:44 -0400
schrieb Andrei Alexandrescu :

> Walter and I will travel to Brasov, Romania to hold an evening-long 
> event on the D language. There's been strong interest in the event with 
> over 300 registrants so far.
> 
> http://curiousminds.ro
> 
> Scott Meyers will guest star in a panel following the talks. We're all 
> looking forward to it!
> 
> 
> Andrei

That's a lot of people. You must be some kind of programming
national hero in Romania. Good luck and watch out for those
C++ moroi in the audience!

-- 
Marco

Re: Release D 2.068.1

[Marco Leise via Digitalmars-d-announce]

Am Tue, 15 Sep 2015 08:39:41 +
schrieb John Colvin :

> On Monday, 14 September 2015 at 23:53:16 UTC, Martin Nowak wrote:
> > On Monday, 14 September 2015 at 20:14:45 UTC, Jack Stouffer 
> > wrote:
> >> On Monday, 14 September 2015 at 17:51:59 UTC, Martin Nowak 
> >> wrote:
> >>> What platform are you on?
> >>
> >> I'm on OS X, using the homebrew version of DMD. And homebrew 
> >> is telling me that I have 2.068.1 installed
> >
> > Well I guess it's a bug in the homebrew script then.
> > Nobody is setting the VERSION file and there is no git repo to 
> > query.
> > https://github.com/Homebrew/homebrew/blob/f8b0ff3ef63e60a1da17ec8d8e68d949b1cebc27/Library/Formula/dmd.rb#L50
> 
> Where is the VERSION file documented? Why does it need manual 
> intervention only for patch releases and pre-releases?

I noticed this as well. There are different VERSION files
depending on where you download DMD source code from: The
releases on GitHub have 2.068 while the releases on dlang.org
have 2.068.1 in VERSION. That's easy to miss when you build
from source.
(Also the dlang.org downloads add to the DMD download
statistics, which is why I use those even though they
include extra binaries for all OSs.)

-- 
Marco

Re: Beta D 2.068.2-b1

[Marco Leise via Digitalmars-d-announce]

And I found another regression in 2.068.1:
https://issues.dlang.org/show_bug.cgi?id=15056

-- 
Marco

Re: Beta D 2.068.2-b1

[Marco Leise via Digitalmars-d-announce]

Am Thu, 10 Sep 2015 05:37:48 +0200
schrieb Martin Nowak :

> Due to a regression in 2.068.1 we'll directly follow up with an
> unplanned point release 2.068.2.
> This is the beta for that point release.
> 
> http://downloads.dlang.org/pre-releases/2.x/2.068.2/
> 
> Please test any of your code against this beta to help finding bugs.
> 
> https://issues.dlang.org/
> 
> -Martin

I tested some code with 2.068.x today and filed this one:

https://issues.dlang.org/show_bug.cgi?id=15042

Kudos to CyberShadow at this point for the excellent "digger"
tool.

-- 
Marco

Re: Arch Linux D package update

[Marco Leise via Digitalmars-d-announce]

Am Fri, 12 Jun 2015 06:26:10 +
schrieb Dicebot pub...@dicebot.lv:

 I will look into making it user-local service 
 (https://wiki.archlinux.org/index.php/Systemd/User), that is 
 simply I currently don't know much about and didn't want to delay 
 update much longer.

I switched back to system wide for now. If dcd-server
evolves further and starts reading not only includes but also
the port number from a user local file (or uses per-user named
sockets like some other services), I'll look into it again.
Anyways, I learned a lot about systemd during this exercise,
that I should have learned earlier. The Arch Linux Wiki is a
great source of information. :)

-- 
Marco

Re: Arch Linux D package update

[Marco Leise via Digitalmars-d-announce]

Am Thu, 04 Jun 2015 14:46:38 +
schrieb Dicebot pub...@dicebot.lv:

  - provides systemd service : `sudo systemctl enable dcd.service` to 
 start automatically upon system startup

Is it really a system-wide server? I can see how it might be
technically necessary due to how systemd works and how you
would need to pass a different IP port for every user.
On the other hand, every user can add imports at runtime and
shut the server down. They might even have to, if they want to
switch projects and clean up the import path list.

  - provides default /etc/dcd.conf with stdlib paths for Arch Linux

Should work so long noone gives root a XDG_CONFIG_HOME or
writes to /root/.config/dcd/dcd.conf. Just contemplating if
when running as system wide service it should generally read
from /etc/[dcd/]dcd.conf instead of user directories. Maybe
via a --system option to dcd-server.

https://github.com/Hackerpilot/DCD/blob/master/src/server.d#L248

-- 
Marco

Re: New D tool releases

[Marco Leise via Digitalmars-d-announce]

Am Mon, 08 Jun 2015 19:18:02 +
schrieb Brian Schott briancsch...@gmail.com:

 Dfix 0.2.2:
 https://github.com/Hackerpilot/dfix/releases/tag/v0.2.2
 * Fixed a bug that caused the string concatenation fix to be 
 applied
inside of deprecated attributes. I plan to revert this fix if
dmd starts accepting `deprecated(string1 ~ string2)`
 
 Dfmt 0.4.0-beta1 and 0.3.6
 https://github.com/Hackerpilot/dfmt/releases/tag/v0.3.6
 * Fix a spacing bug with function contracts
 
 https://github.com/Hackerpilot/dfmt/releases/tag/v0.4.0-beta1
 * Introduced support for `// dfmt off` and `// dfmt on` comments
 * Added the ability to configure spacing around the `:` in 
 selective
imports.
 * Added the ability to configure indentation of case statements.
 * Added the ability to configure indentation of attribute 
 declarations.
 * Added the ability to configure formatting of labeled loops.
 
 D-Scanner 0.2.0-beta1
 https://github.com/Hackerpilot/Dscanner/releases/tag/0.2.0-beta1
 * Fixed several bugs in the lint checks.
 * Added support for generating Emacs etags files.
 * `--imports` option can now work on more than one file at a time
 * Added undocumented public declaration check.
 * Added unused label check.
 * Added check for variables with the same name as labels.
 * Added check for redundant parenthesis.
 * Added line field to ctags output to work with the Tagbar 
 plugin better.
 * Unused variable declaration check is now disabled inside of 
 __traits
expressions.
 * Undocumented declaration check is disabled for things marked
deprecated or @disable.
 * Undocumented declaration check ignores @property functions.

Sounds like a big step forward! One question though, what's
the rationale behind @properties not needing documentation?

-- 
Marco

Re: Arch Linux D package update

[Marco Leise via Digitalmars-d-announce]

Am Thu, 04 Jun 2015 14:46:38 +
schrieb Dicebot pub...@dicebot.lv:

  - provides systemd service : `sudo systemctl enable dcd.service` to 
 start automatically upon system startup
  - provides default /etc/dcd.conf with stdlib paths for Arch Linux

Would you mind if I copy this idea?

-- 
Marco

Re: DUB 0.9.23 released

[Marco Leise via Digitalmars-d-announce]

Am Mon, 06 Apr 2015 10:21:45 +0200
schrieb Sönke Ludwig slud...@rejectedsoftware.com:

 The new version contains some important bug fixes for sub modules and 
 overridden string imports, as well as some other major fixes. Apart from 
 that, the major additions are:
 
   - copyFiles now get hard linked instead of copied and whole
 directories can be copied in addition to individual files
   - The dub init command has been extended to take an optional list of
 dependencies that is added to the generated dub.json
   - A new deimos template has been been implemented
 (dub init --type=deimos)
   - The default compiler is now chosen based on what is found in PATH

How does it resolve preferences when 3 comilers are on PATH ? =)
dmd first?

   - New project generators for Sublime Text and CMake
   - In single file build mode, --parallel can now be used to compile
 with multiple compiler instances at once
 
 Find the full list of changes in the change log [1] and download at:
 
 http://code.dlang.org/download
 
 [1]: https://github.com/D-Programming-Language/dub/blob/master/CHANGELOG.md

And one idea that recently came to my mind since I'm a fan of
shared libraries. Would it work in D to say compile a shared
library and generate a number of .di files required to use it?
Similar to druntime that provides only a hand full of its
modules as .di or where necessary as .d files.

I'm looking for a perspective out of the compile the world
model in Dlang, since as far as I understand it, the D
front-end will always parse all imported modules recursively.
Even in single-file compilation mode, when only app.d
changed it would parse the whole thing down to the last corner
of all used D libraries like GtkD or vibe.d.

With the minimum amount of required .di files for libraries,
further private imports inside their modules are hidden away
from the compiler, thereby short-circuiting the import the
world recursion.

-- 
Marco

Re: Standardpaths library

[Marco Leise via Digitalmars-d-announce]

Am Tue, 07 Apr 2015 11:58:58 +
schrieb Kagamin s...@here.lot:

 On Monday, 6 April 2015 at 21:40:28 UTC, Marco Leise wrote:
  So if some configuration is portable and you want to create 
  something
  really fine grained you could offer that directory as an
  alternative roaming config dir (returning null or
  ~/.config on Linux).
 
 Though it's unusual for an average programmer to figure out the 
 correct usage of roaming profiles :)

All they need to know is that Roaming must only contain small
and machine independent files.

(Why? Because in environments where users profiles are stored
on servers, this is the part that gets synced with those
servers. So e.g. a browser cache should not end up there, nor
should configuration that makes sense only on the current
workstation.)

Frankly, I didn't know until FreeSlave's library brought it
up. :D
Anyways my idea of portable libraries is that they should not
implement the least common denominator, but the full width of
what is available or not across different systems and offer a
common API for that, so you have the option to write well
behaving applications while rewriting as little code as
possible.

Two examples:
There is an application menu in OS X and in
Gnome 3, but other desktops don't provide it. I would support
it, so applications integrate nicely with these desktops and
turn it into another main menu item or taskbar right-click
menu on other desktops.
POSIX doesn't close file handles for spawned child-processes
by default, which is now considered a bad default by most.
I think a portable library should try to provide the same
functionality across systems and open files in close on exec
mode by default on POSIX so it is similar to Windows.

-- 
Marco

Re: Standardpaths library

[Marco Leise via Digitalmars-d-announce]

Am Tue, 07 Apr 2015 17:27:34 +
schrieb FreeSlave freeslav...@gmail.com:

 On Monday, 6 April 2015 at 21:40:28 UTC, Marco Leise wrote:
 
  I believe modern desktops offer enough granularity to cover
  each of those. For example if I was playing a game on Linux
  files would go here:
 
  /usr/share/[games/] - read-only data files.
  ~/.cache/ - downloaded archives, precompiled scripts, browser
  caches and other files that can be recreated or
  fetched again if they were to be deleted
  ~/.config/ - the user's personal configuration; may be
   overriding something in a system directory if
   desired
  ~/.local/share/ - pretty much a catch all for save games,
user created content that goes beyond the
scope of config files, highscores,
highlighting schemes in an IDE, Steam, ...
 
  Or the other way around:
 
  data   = /usr/share
  save games = ~/.local/share
  downloaded content = ~/.local/share (or ~/.cache)
  configs= ~/.config
 
  Windows has the Local and Roaming directories, which
  serve similar but different purposes. E.g. anything machine
  specific or big must not be in Roaming.
  So if I generally asked for the config dir, I'd probably expect
  AppData/Local on Windows and ~/.config on Linux (because I
  might write a configuration that only works for this machine.)
  Roaming is interesting for users that have their profiles on
  servers and might switch to another workstation. So if some
  configuration is portable and you want to create something
  really fine grained you could offer that directory as an
  alternative roaming config dir (returning null or
  ~/.config on Linux).
  In any case there will be multiple results for some
  directories (/usr/share, /usr/local/share) and also
  several standard paths mapping to the same physical directory
  (user data and user cache both map to AppData/Local on
  Windows). The user needs to be made aware of this so (s)he
  doesn't overwrite files in one standard path with files in a
  supposedly different one.
 
  So much for my thoughts on standard paths extreme edition. ;)
 
 Just found out, there's also special Saved Games directory on 
 Windows. But only starting with Vista, therefore it's not CSIDL, 
 but KNOWNFOLDERID (same as for Downloads). Still I did not find 
 fairly new winapi headers for D. Seems like the most use 
 translation of MinGW headers which stuck at, likely, Windows XP 
 era. Although we could just read some paths from registry 
 avoiding calling SHGetKnownFolderPath, it's way too hacky.

The druntime Windows headers are written on a as needed
basis. I don't think anyone would object if you added
something you need, except maybe if it stops druntime from
working in WinXP in general. Don't know about enums that
are only meaningful on Vista and later...

 Currently the only Roaming directories returned by standardpaths 
 library on Windows are Templates and Applications, which is fine. 
 The whole roaming thing is specific to Windows, but it would be 
 useful to add separate function probably.

What do you think about OS agnostic code?

  string configPath =
standardPath(StandardPath.config, appName);
  string roamingConfigPath =
standardPath(StandardPath.roamingConfig, appName);

If this worked on Linux it would return the same path for both
calls, but you would avoid introducing version(Windows) switches
in user code.
The appName is there for the common use-case to get the path in
the context of the calling application. If I write a program an
want to open some images/splash.png in the installation
directory, I'd expect just the executable directory on Windows,
but /usr/share/app name/ (or one of the overrides) on Linux.
Then again, maybe opening files is outside the scope of your
library. I could imagine it would become a full file system
abstraction layer just to support correct file lookup with all
these /usr/share, /usr/local/share, ~/.local/share and manual
overrides. :p

 The same directory for configs, data and cache locations is the 
 problem. I should mention it in documentation. Though for cache 
 directory it returns AppData/Local/cache the same way as Qt does.

-- 
Marco

Re: Standardpaths library

[Marco Leise via Digitalmars-d-announce]

Am Sun, 05 Apr 2015 12:31:23 +
schrieb FreeSlave freeslav...@gmail.com:

 On Sunday, 5 April 2015 at 11:42:42 UTC, Marco Leise wrote:
  On another note when I ran your 'printdirs' it didn't list a
  user Fonts or Applications directory. The Applications
  directory is ok, but I do have a ~/.fonts/ directory and
  /etc/fonts/fonts.conf says:
!-- the following element will be removed in the future --
dir~/.fonts/dir
  Fonts in ~/.fonts are listed in LibreOffice. It seems like you
  do parse /etc/fonts/fonts.conf. Maybe there is a bug in the
  parser?
 
  The whole Applications thing doesn't make much sense on Linux,
  right? Is that a directory where applications are installed to
  including their assets?
 
 Probably you don't have local 
 $XDG_CONFIG_DIR/fontconfig/fonts.conf file.
 
 I've opened issue 
 https://github.com/MyLittleRobo/standardpaths/issues/8
 
 About Applications: on my Windows 7 it returns 
 C:/Users/Username/Application Data/Microsoft/Windows/Start 
 Menu/ProgramsC:/ProgramData/Microsoft/Windows/Start Menu/Programs 
 where .lnk files are stored (I believe these are used in the 
 start  menu). Since freedesktop systems use .desktop files it 
 would be sane to return paths which contain them 
 (~/.local/share/applications, /usr/local/share/applications and 
 /usr/share/applications). I just have not implemented it yet. 
 Though not sure it the whole thing can be useful, since things 
 are not the same on Windows and freedesktop: Windows uses 
 directories to make menu hierarchy, while freedesktop for the 
 same purpose use Categories field in .desktop files. Also .lnk 
 and .desktop are different things themselves.

You are right, the two are very different. One has to write OS
specific code to use them. The funny thing is, D as a systems
programming language could actually be used by someone to
write a Linux package manager or Windows installer. :p

Do as you see fit. Qt as an inspiration is a good thing I
believe. Some classes I had a look at were intuitive and well
thought out.

-- 
Marco

Re: Standardpaths library

[Marco Leise via Digitalmars-d-announce]

Am Sun, 05 Apr 2015 12:39:04 +
schrieb FreeSlave freeslav...@gmail.com:

 On Sunday, 5 April 2015 at 11:42:42 UTC, Marco Leise wrote:
 is this Windows?
  return executable_path
 
 That depends on what do you understand by data.

The data I was referring to is the stuff that is installed with
the program executable. Files that are usually the same across
installations.

 Are game's saves data too? Or content downloaded while playing
 (server-specific assets or new levels).
 In the past it was ok to write configs and 
 data to the same path where the game (or application) originally 
 installed. But starting with Vista or Windows 7 it's not the case 
 since Program Files folder become write-protected by default.

I believe modern desktops offer enough granularity to cover
each of those. For example if I was playing a game on Linux
files would go here:

/usr/share/[games/] - read-only data files.
~/.cache/ - downloaded archives, precompiled scripts, browser
caches and other files that can be recreated or
fetched again if they were to be deleted
~/.config/ - the user's personal configuration; may be
 overriding something in a system directory if
 desired
~/.local/share/ - pretty much a catch all for save games,
  user created content that goes beyond the
  scope of config files, highscores,
  highlighting schemes in an IDE, Steam, ...

Or the other way around:

data   = /usr/share
save games = ~/.local/share
downloaded content = ~/.local/share (or ~/.cache)
configs= ~/.config

Windows has the Local and Roaming directories, which
serve similar but different purposes. E.g. anything machine
specific or big must not be in Roaming.
So if I generally asked for the config dir, I'd probably expect
AppData/Local on Windows and ~/.config on Linux (because I
might write a configuration that only works for this machine.)
Roaming is interesting for users that have their profiles on
servers and might switch to another workstation. So if some
configuration is portable and you want to create something
really fine grained you could offer that directory as an
alternative roaming config dir (returning null or
~/.config on Linux).
In any case there will be multiple results for some
directories (/usr/share, /usr/local/share) and also
several standard paths mapping to the same physical directory
(user data and user cache both map to AppData/Local on
Windows). The user needs to be made aware of this so (s)he
doesn't overwrite files in one standard path with files in a
supposedly different one.

So much for my thoughts on standard paths extreme edition. ;)

-- 
Marco

Re: Standardpaths library

[Marco Leise via Digitalmars-d-announce]

Am Sun, 05 Apr 2015 09:08:12 +
schrieb FreeSlave freeslav...@gmail.com:

 I wrote small library for getting standard paths (like Pictures, 
 Music)
 Here's dub package http://code.dlang.org/packages/standardpaths
 And github repo https://github.com/MyLittleRobo/standardpaths
 
 You can see open issues on github. Please, participate in 
 discussions if you're interested. The biggest problem now is OS X 
 support https://github.com/MyLittleRobo/standardpaths/issues/4
 
 You can generate documentation with dub build --build=docs.

This is indeed useful. I once had a specific use case where I
needed the default location for where the assets of my
application is stored. On Windows applications come with an
installer and install the assets right into the installation
directory (usually current directory). On Linux applications
store their data in /usr/share/appname/. So with a
compile-time only template I accessed these as:

string asset1 = dirs!myapp.staticData ~ /image.png;

Now I realize this is a bit simplistic for several reasons:

* On Windows, one might still want to create a subdirectory to
  separate assets from program code, e.g. data.

* On Linux, as your library shows, data directories are
  layered like this:
  ~/.local/share, /usr/local/share, /usr/share

* The _actual_ data directory is often passed into the
  configure script on Linux and no general algorithm could
  guess it.

* An application name would typically become lower-case on
  Linux while keeping its casing on Windows in e.g.
  AppData/appname.

Unsure if that's it, but that could make data file lookup for
application SomeTool go like this:

was the data directory explicitly overridden (e.g.
through ./configure or by the programmer)?
  return overridden_dir;  // note: relative directories expand
as based on executable directory
is this Linux?
  for (dir in
  [${XDG_DATA_HOME}/sometool, /usr/local/share/sometool, /usr/share/sometool])
is the file in dir ?
  return dir
is this Windows?
  return executable_path

This goes a bit into heuristics and best practices, so it
probably doesn't fit with your library that provides
clearly defined standard paths from the desktop environment.

Aside from that I think it is a common enough use case to
lookup assets that ship with your program in the typical
installation directories.


On another note when I ran your 'printdirs' it didn't list a
user Fonts or Applications directory. The Applications
directory is ok, but I do have a ~/.fonts/ directory and
/etc/fonts/fonts.conf says:
  !-- the following element will be removed in the future --
  dir~/.fonts/dir
Fonts in ~/.fonts are listed in LibreOffice. It seems like you
do parse /etc/fonts/fonts.conf. Maybe there is a bug in the
parser?

The whole Applications thing doesn't make much sense on Linux,
right? Is that a directory where applications are installed to
including their assets?

-- 
Marco

Re: Devisualization.Image

[Marco Leise via Digitalmars-d-announce]

Am Tue, 18 Nov 2014 12:37:54 +0300
schrieb Timur Gafarov gecko0...@gmail.com:

 15.11.2014 07:48, Rikki Cattermole пишет:
  To further Devisualization, I have got the start of an image library.
  It should be fairly interface complete now.
 
  For this I really could use help from anyone with experience with PNG
  especially with Adam7 interlacing and color correction such as gamma.
  Currently missing an exporter. Only imports. Does not import grayscale
  or palleted.
 
  Goal: to act as the image representation within memory (including colors).
  Usage:
  Devisualization.Window uses it for window icon's (untested for x11 can
  somebody confirm this does indeed work?).
  Devisualization.Font heavily uses it for glyphs and the output of its
  rasterization.
 
  [0] https://github.com/Devisualization/image
 
 I have working PNG export in my image processing package, part of dlib: 
 https://github.com/gecko0307/dlib
 There're also importers for BMP and TGA.

I also just wrote a TGA importer and found RLE and 16-bit in
particular badly supported on Linux. E.g. eye-of-gnome
(Gnome's picture viewer) doesn't correctly restore the last
pixel's color in an RLE image and most software doesn't handle
16-bit at all. Add to that, that the specs are not very
specific as to what allowed values are and you get all
sorts of funny implementations quirks.

-- 
Marco

Re: Devisualization.Image

[Marco Leise via Digitalmars-d-announce]

Am Wed, 19 Nov 2014 00:35:09 +0100
schrieb Marco Leise marco.le...@gmx.de:

  I have working PNG export in my image processing package, part of dlib: 
  https://github.com/gecko0307/dlib
  There're also importers for BMP and TGA.
 
 I also just wrote a TGA importer and found RLE and 16-bit in
 particular badly supported on Linux. E.g. eye-of-gnome
 (Gnome's picture viewer) doesn't correctly restore the last
 pixel's color in an RLE image and most software doesn't handle
 16-bit at all. Add to that, that the specs are not very
 specific as to what allowed values are and you get all
 sorts of funny implementations quirks.
 
For example:
- the original Targa paint software TIPS used to store a
  palette even for TrueColor images, meaning you should
  actually calculate the data offset as header + id + palette
  even for 24- and 32-bit images.
- the number of channels for a TrueColor TGA is always 3, plus
  as many attribute bits as given in the descriptor.

If you only support 8-bits per color channel, you should check
if bitsPerPixel is = 24. Otherwise, bitsPerPixel == 16 would
be interpreted as a two color channel image instead of three
channels à 5-bits. And bitsPerPixel == 15 would become a one
channel image. :p
The 15-/16-bit variants were used quite a bit in the early
days of 3D games, like in the menu graphics for Trespasser.
They are the same except that 16-bit allows for an additional
attribute bit that could be used for a transparency mask.

-- 
Marco

Re: Atmosphere GM - Statistical package

[Marco Leise via Digitalmars-d-announce]

Am Fri, 24 Oct 2014 17:54:26 +
schrieb Ilya Yaroshenko ilyayaroshe...@gmail.com:

 Hello!
 
 link http://9il.github.io/atmosphere_gm
 
 You are welcome to suggest required algorithms!
 
 Best Regards,
 Ilya

Your English is a bit confusing :) What can I use this
package for? Let's say I measure some value over a period
of time where the distance between the samples is between
100 to 500 ms and there is an error of +/- 2%. Does Atmosphere
GM provide algorithms to filter the error so I can derive from
the graph without huge error spikes?
Or is it more for card games and dice rolls?

-- 
Marco

Re: Digger 1.0

[Marco Leise via Digitalmars-d-announce]

Am Mon, 22 Sep 2014 18:59:12 +
schrieb Vladimir Panteleev vladi...@thecybershadow.net:

 On Monday, 22 September 2014 at 17:28:50 UTC, ketmar via 
 Digitalmars-d-announce wrote:
  On Mon, 22 Sep 2014 15:24:55 +0200
  simendsjo via Digitalmars-d-announce
  digitalmars-d-announce@puremagic.com wrote:
 
  My guess is the average for developers is ~8GB. 2GB RAM is 
  really not
  enough for pretty much anything these days - the browser alone 
  easily
  chews 3-4GB on moderate use. I recommend you just upgrade your
  computer (or compile on a better iron).
  are your primary language is java? i'm asking just out of 
  curiousity.
 
 That was uncalled for.
 
 Firefox requires 4GB of memory to build.
 Chromium requires 8GB of memory to build.
 Android requires 16GB of memory to build.

I've been compiling Firefox and Chromium with 4GB of RAM + 4GB
of swap on Linux 64-bit in the past. I don't know if I had to
close any applications for that.

 I think 4GB for a modern programming language's implementation is 
 not an unreasonable requirement, even if it could be brought down 
 in the future. Especially considering that you can't even buy a 
 new laptop today with less than 4GB of RAM, and 3GB is becoming 
 the norm for smartphones.

So why would Apple be able to get away with 1GB on its just
released iPhone 6? Maybe 1048576 kilobytes is enough for
everyone?

-- 
Marco

Re: New Beta: 2.066.1-rc1 is Now Available

[Marco Leise via Digitalmars-d-announce]

Am Fri, 12 Sep 2014 01:51:33 +0900
schrieb Andrew Edwards rid...@yahoo.com:

 DMD 2.066.1-rc1 is now available for testing. Visit 
 http://wiki.dlang.org/Beta_Testing to download. Please submit all bugs 
 to bugzilla at https://issues.dlang.org/.
 
 Happy testing

I followed the first LINUX link on
http://wiki.dlang.org/DMD_v2.066.1-rc1
and got this:

http://downloads.dlang.org/pre-releases/2014/dmd.2.066.1-rc1.linux.zip

  404 Not Found
  Code: NoSuchKey
  Message: The specified key does not exist.
  Key: pre-releases/2014/dmd.2.066.1-rc1.linux.zip
  RequestId: 041CB7ED85E79D22
  HostId: 77PqxexMbECV1cBnYfWlCZhxrKtBgH2VhFhP8v2tlG6kpALNX3JGPYXbzhmYIx7w

-- 
Marco

Re: New Beta: 2.066.1-rc1 is Now Available

[Marco Leise via Digitalmars-d-announce]

P.S. Is this supposed to contain all 2.066 regression
fixes pulled into master up to this point?
As far as I can tell at least
https://github.com/D-Programming-Language/dmd/pull/3961
from 4 days ago didn't get cherry picked yet.
Or are these RCs more of a preliminary version?

-- 
Marco

Re: New Beta: 2.066.1-rc1 is Now Available

[Marco Leise via Digitalmars-d-announce]

Am Fri, 12 Sep 2014 03:16:52 +0900
schrieb Andrew Edwards rid...@yahoo.com:

 Clear cache and try again.

Thx, try again now was enough.

-- 
Marco

Re: 438-byte Hello, world Win32 EXE in D

[Marco Leise via Digitalmars-d-announce]

Am Tue, 09 Sep 2014 10:20:43 +
schrieb Don x...@nospam.com:

 On Monday, 8 September 2014 at 08:18:32 UTC, Ola Fosheim Grøstad 
 wrote:
  On Monday, 8 September 2014 at 08:08:23 UTC, Kagamin wrote:
  But that downloaded file is bloatware, because it has to 
  implement functionality, which is not provided by the system. 
  That tiny pe file doesn't download anything, it's completely 
  done by the system.
 
  Yeah…
 
  http://stackoverflow.com/questions/284797/hello-world-in-less-than-20-bytes
 
 My personal best --
 
 At my first job, a customer once made a request for a very simple 
 DOS utility. They did mention that they didn't have much disk 
 space on their machine, so they asked me to try to make the 
 program small.
 That was a once-in-a-lifetime opportunity. Naturally, I wrote it 
 in asm.
 The final executable size was 15 bytes. g
 The customer loved it.

Vladimir: Good job!
Don: Nice story. What did it do?

During my time at a vocation school I wrote some stuff like a
tiny Windows media player with some of the ASM in the DOS/PE
header area. And an animated GIF player in ASM as a .com
executable with the GIF included in it. (Easy since GIF
algorithms are 16-bit and they use 8-bit color palettes)

-- 
Marco

Re: Dlang on Gentoo (update)

[Marco Leise via Digitalmars-d-announce]

Added ebuild:

* dub 0.9.21

Updated ebuilds:

* GDC based on GCC 4.8.3 and DMD FE 2.065
* DMD 2.066.0 and tools

I wonder if I can switch to linking with libphobos2.so now
by default for dmd? Any outstanding issues/objections?

-- 
Marco

Dlang on Gentoo (update)

[Marco Leise via Digitalmars-d-announce]

I moved Dlang related ebuild to a new repository under the
Gentoo organization on GitHub a while ago. It is available
through layman as dlang. (Not to be confused with the
outdated d overlay there.)

Information in the Gentoo Wiki:
https://wiki.gentoo.org/wiki/Dlang

Link to the repository on GitHub:
https://github.com/gentoo/dlang/

-- 
Marco

Re: legacy code retreat's triva game : the D version

[Marco Leise]

Am Sun, 22 Dec 2013 02:12:51 +0100
schrieb Timon Gehr timon.g...@gmx.ch:

 On 12/22/2013 02:09 AM, Timon Gehr wrote:
 
  The morale is that uniform random numbers doesn't imply that
  every value in the range will eventually be generated once!
 
 
  Yes it does. (The probability that some value is never generated is 0.)
  The actual morale is that random number generators do not generate true
  randomness, and poor random number generators may generate sequences
  that do not look remotely random.
 
 'pseudo random number generators' would be a more accurate term.

Can you elaborate a bit? How do you know that the Java LCG
can produce every 32-bit integer once? If that's true then
the problem with the Java code was something different and I
was just biased, because I was already expecting the code to
fail before the fact. (Expectations can do strange things to
your perception.)

-- 
Marco

Re: legacy code retreat's triva game : the D version

[Marco Leise]

Am Sun, 22 Dec 2013 09:19:48 +
schrieb Chris Cain clc...@uncg.edu:

 On Sunday, 22 December 2013 at 08:06:30 UTC, Marco Leise wrote:
  Can you elaborate a bit? How do you know that the Java LCG
  can produce every 32-bit integer once? If that's true then
  the problem with the Java code was something different and I
  was just biased, because I was already expecting the code to
  fail before the fact. (Expectations can do strange things to
  your perception.)
 
 If I may,
 
 http://en.wikipedia.org/wiki/Linear_congruential_generator
 
 Definition of an LCG:
 ```
 Xnext = (a * Xprev + c) % m
 ```
 
 An LCG is said to have a full period if the length of the 
 period is m. If the period is m, we know the LCG must produce 
 every number between 0 and m because if there was even one 
 repeated number then the generator as defined above would repeat 
 the entire sequence up to that point and, thus, the period would 
 not be m, which is a contradiction.
 
 According to the Hull-Dobell Theorem, an LCG will have a full 
 period iff:
 1. `c` and `m` are relatively prime.
 For Java, c = 11 and m = 2^48
 This condition applies.
 2. `(a - 1)` is divisible by all prime factors of m`
 For Java, a = 25214903917 and thus a-1 is even which means the 
 prime factors of m (just 2) do divide it.
 This condition applies.
 3. `a - 1` is a multiple of 4 if `m` is a multiple of 4.
 For Java, m is a multiple of 4.
 `(a - 1)/4` is 6303725979, so it's also a multiple of 4.
 This condition applies as well.
 
 Since Java's LCG has a full period over 2^48, we know that taking 
 the top 32 bits (which is what Java does to get better 
 randomness) would also all be represented.


Am Sun, 22 Dec 2013 13:09:51 +0100
schrieb Timon Gehr timon.g...@gmx.ch:

 On 12/22/2013 09:06 AM, Marco Leise wrote:
  Am Sun, 22 Dec 2013 02:12:51 +0100
  schrieb Timon Gehr timon.g...@gmx.ch:
 
  On 12/22/2013 02:09 AM, Timon Gehr wrote:
 
  The morale is that uniform random numbers doesn't imply that
  every value in the range will eventually be generated once!
 
 
  Yes it does. (The probability that some value is never generated is 0.)
  The actual morale is that random number generators do not generate true
  randomness, and poor random number generators may generate sequences
  that do not look remotely random.
 
  'pseudo random number generators' would be a more accurate term.
 
  Can you elaborate a bit?
 
 The probability that a certain number does not occur in one round is 
 (n-1)/n.
 
 ((n-1)/n)^k goes to 0 rather fast as k goes to infinity.
 
 In fact, the expected number of trials until all numbers are covered is 
 ~ n log n, and the probability that the process runs significantly 
 longer is very small.
 
 See also: http://en.wikipedia.org/wiki/Coupon_collector%27s_problem
 
 
  How do you know that the Java LCG can produce every 32-bit integer once?
 
 Typically constants are chosen such that this holds, but your code would 
 require something stronger to fail, namely, that a certain congruence 
 class does not occur. Typically pseudo random number generators are 
 chosen such that the generated sequences look close to true randomness. 
 If such a simple process can be used to reliably distinguish true 
 randomness and the pseudo random number generator, then the pseudo 
 random number generator is not very good.

Thank you two for explaining LCGs to me. That's good
information for reasoning about code. Every good (full period)
LCG is a specific permutation of the numbers [0..m). The next
time I wonder how I can iterate in random order over a list of
length n^2, I know what I'll use ;)

-- 
Marco

Re: legacy code retreat's triva game : the D version

[Marco Leise]

Am Fri, 20 Dec 2013 15:53:08 +0100
schrieb marcpmichel marc.p.mic...@gmail.com:

 
 I participated in the global day of code retreat 2013, and we 
 had to do refactoring on a very ugly piece of code which was 
 available on many languages.
 But there was no D version, so I made one (based on the java 
 version) and pull-requested it.
 
 Here is the ugly thing :
 https://github.com/jbrains/trivia/tree/master/d
 
 EOT

bool notAWinner;
do {
game.roll(rand.front() % 5 + 1);
rand.popFront();

if (rand.front() % 9 == 7) {// -- WARNING! WARNING!
notAWinner = game.wrongAnswer();
} else {
notAWinner = game.wasCorrectlyAnswered();
}
rand.popFront();
} while (notAWinner);

This kind of code is a dangerous gamble.

This is a story about my student time: I once sat in a Java
class and one of the students had an issue with their code not
outputting anything and not quitting either. When the teacher
came around, we found only one obvious point for an infinite
loop could occur and it looked like this:

  Random rng = new Random();
  int count = 0;

  // Visit all items once
  while (count  list.size()) {
bool found = false;
while (!found) {
  int idx = rng.nextInt() % list.size();
  if (list[idx].visited == false) {
list[idx].visited = true;
found = true;
count++;
  }
}
  }

[I don't remember the exact lines, but this is the gist of it.]
The teacher himself wrote this code and presented it to the
class as a simple way to iterate over a list in random order
which was part of todays programming task.
It didn't cause issues for any of the other students, but on
this particular computer the random seed that the Random ctor
chose caused a degenerate case where it never hit any of the 3
remaining indexes of the list.

The morale is that uniform random numbers doesn't imply that
every value in the range will eventually be generated once!

-- 
Marco

Gentoo overlay online

[Marco Leise]

I have completed what I wanted to achieve for the new overlay
and have uploaded it here: https://github.com/gentoo-dlang

Features include:
* Installation of DMD, GDC and LDC in parallel
* Customizable »CFLAGS« for each D compiler
* GDC is integrated with GCC for the best compatibility
* Slotted installation of previous D compiler versions
* Shared library support when using DMD
* Easily compile debug builds with DMD and release builds with
  LDC/GDC even when they depend on libraries like GtkD.
  (Note: This depends on availability of libraries in the
 repository.)

Try e.g.: ACCEPT_KEYWORDS=~amd64 USE=ldc2-0.12.0 emerge GtkD

-- 
Marco

Re: The D in Novosibirsk State University

[Marco Leise]

Am Mon, 11 Nov 2013 00:35:09 +0100
schrieb Dicebot pub...@dicebot.lv:

 On Sunday, 10 November 2013 at 23:19:22 UTC, Froglegs wrote:
   Slides are in English, do most Russian programmers speak 
  English?
 
 No. However, most really interested in programming technology do. 
 Pretty much as in any other part of the world.

Reminds me how I ended up in a 100% Russian bug tracker once,
when I wanted to report a bug for a big, world-wide online game
with localizations into 8 or more languages. In the end I used
Google translate. (They do offer technical support in English,
but I didn't want to deal with 1st level support and have
mails going forth and back just to report an obvious bug. :p)

-- 
Marco

Re: dunit 0.7.0 released

[Marco Leise]

Am Mon, 21 Oct 2013 21:47:22 +0200
schrieb ilya-stromberg ilya-stromberg-2...@yandex.ru:

 On Monday, 21 October 2013 at 19:02:45 UTC, Gary Willoughby wrote:
  On Monday, 21 October 2013 at 11:58:14 UTC, Jonathan M Davis 
  wrote:
  On Monday, October 21, 2013 13:48:16 Dicebot wrote:
  On Monday, 21 October 2013 at 11:09:25 UTC, ilya-stromberg 
  wrote:
   Guys, we have at least 5 (!) different unit test projects!
   Can you cooperate your efforts to create one, but wonderful?
  
  ...and add it to Phobos review queue ;)
 
  I confess that I don't understand why anyone is creating any 
  unit test
  projects for D, and I'd likely vote against any attempt to add 
  such a thing to
  Phobos. D has built in unit testing functionality, and it 
  works great. Maybe
  some additional assert-like functions could be useful (similar 
  to assertThrown
  or assertNotThrown), but we really don't need much beyond what 
  the language
  provides.
 
  - Jonathan m Davis
 
  Seriously though, unit testing is a major tool that all 
  languages should have access to. The built-in stuff is adequate 
  for very simple testing on simple types such as asserting 
  something is true, etc. But when you're writing tests for a 
  class that need dependencies then the built-in stuff won't cut 
  it. The framework i'm currently working allows for mocking of 
  dependencies which in itself is a big deal. Replacing 
  dependencies with 'fake' stubs is something which makes unit 
  tests a lot clearer and less complicated. It also means i can 
  override the return value of any method at runtime to pretend 
  returned data is real there too.
 
  I've also extended the assert mechanism in the D runtime to 
  create better errors. Instead of just getting:
 
  core.exception.AssertError@file(57): Assertion failure
 
  You now get:
 
  
  +--
  | Failed asserting equal
  
  +--
  | File: file.d
  | Line: 57
  
  +--
  | ✓ Expected value: (int[]) [1, 2, 3, 4, 5]
  | ✗ Actual value: (int[]) [1, 2, 4, 5]
 
  This is way more clear exactly what and where failed. This is 
  essential when unit testing using structs, arrays, classes, 
  etc.. as you need this information to truly narrow down exactly 
  what failed. Extending stuff like this also helps with overall 
  project reporting, writing to a file or drawing pretty graphs 
  etc..
 
 +1

We had a discussion about verbose contract assertion failures
a while ago. I did something similar, with an API like that:

ensure!==(x, [1, 2, 3, 4, 5], x isn't the ordered list of integers from 1 to 
5);

It has its shortcomings though, especially when you need x to
be one of a set of values or a power of 2 etc...

Btw: Is ✓ and ✗ supported on Windows Vista and XP ?

-- 
Marco

Re: dmd 2.064.2

[Marco Leise]

Am Tue, 05 Nov 2013 23:24:02 +0100
schrieb Orvid King blah38...@gmail.com:

 On Tuesday, 5 November 2013 at 22:08:48 UTC, Walter Bright wrote:
  Ok, this is it:
 
  http://ftp.digitalmars.com/dmd_2.064.2-0_amd64.deb
  http://ftp.digitalmars.com/dmd-2.064.2-0.fedora.i386.rpm
  http://ftp.digitalmars.com/dmd-2.064.2-0.fedora.x86_64.rpm
  http://ftp.digitalmars.com/dmd_2.064.2-0_i386.deb
  http://ftp.digitalmars.com/dmd-2.064.2-0.openSUSE.i386.rpm
  http://ftp.digitalmars.com/dmd-2.064.2-0.openSUSE.x86_64.rpm
  http://ftp.digitalmars.com/dmd-2.064.2.exe
  http://ftp.digitalmars.com/dmd.2.064.2.zip
  http://ftp.digitalmars.com/dmd.2.064.2.dmg
  http://ftp.digitalmars.com/libphobos2-64_2.064.2-0_amd64.deb
  http://ftp.digitalmars.com/libphobos2-64_2.064.2-0_i386.deb
 
 Dear world: DO NOT use the windows installer if you have ANYTHING
 other than the default installed files in your previous install
 folder, because it will delete EVERYTHING. (and this isn't an
 understatement, my C:/D folder is now entirely empty but for a
 single git index which was locked by my IDE), this means I have
 now lost my local checkouts of the D repos, my git-head dmd
 install location, as well as my auto-build-install-test scripts,
 and my local copy of my JSON work. Thankfully my JSON work wasn't 
 even the latest copy anyways. Deleting everything IS NOT 
 uninstalling.

Stunned silence...
You seem to have gotten away with only few losses. Thanks for
sharing. I can only imagine what that would have done to
someone who has no backups or online repositories for their
code.
I have to say though that I'm sometimes annoyed by accurate
uninstallers that keep a directory because of a log file or
modified configuration. I doesn't hurt to ask for a recursive
deletion of the install directory.

-- 
Marco

Re: Article: D Exceptions and C Callbacks

[Marco Leise]

Am Wed, 07 Aug 2013 12:08:44 +0200
schrieb Mike Parker aldac...@gmail.com:

 On Wednesday, 7 August 2013 at 07:10:21 UTC, Johannes Pfau wrote:
  Am Wed, 07 Aug 2013 03:33:28 +0200
  schrieb Mike Parker aldac...@gmail.com:
 
  
   Maybe modifying glfwSetWindowCloseCallback and similar 
   functions to only accept nothrow functions is a good idea?
  
  I'm not sure what you mean. That's a C function, which means 
  it doesn't have any concept of D's nothrow. And, actually, the 
  callbacks should be declared as nothrow anyway. Like:
 
  He meant declaring glfwSetWindowCloseCallback like this:
  alias extern(C) nothrow void function() callback;
  extern(C) void glfwSetWindowCloseCallback(callback cb);
 
  http://dpaste.dzfl.pl/0f23146f
  There's no difference in the generated code, but this way the 
  compiler
  will complain if you pass a non-nothrow function to
  glfwSetWindowCloseCallback.
 
 Yeah, I see. I misunderstood. I agree that the callbacks should 
 be declared as nothrow. In Derelict, all the C function pointers 
 are declared as such. Apparently, I overlooked adding it to the 
 callbacks.

I recently proposed using nothrow in GtkD as well. This
article was kind of a coincidence now. Yes, it is a real issue
with binding to C code unfortunately. Especially on amd64 with
GCC compiled stuff since it omits the stack frame that DMD
requires to unwind the stack resulting in immediate crashes
(druntime HLTs the execution) as soon as you throw an
exception from a callback.

-- 
Marco

Re: Article: D Exceptions and C Callbacks

[Marco Leise]

Am Thu, 08 Aug 2013 11:33:28 -0700
schrieb Walter Bright newshou...@digitalmars.com:

 On 8/8/2013 6:48 AM, Marco Leise wrote:
  I recently proposed using nothrow in GtkD as well. This
  article was kind of a coincidence now. Yes, it is a real issue
  with binding to C code unfortunately. Especially on amd64 with
  GCC compiled stuff since it omits the stack frame that DMD
  requires to unwind the stack resulting in immediate crashes
  (druntime HLTs the execution) as soon as you throw an
  exception from a callback.
 
 
 I agree that C callbacks called from C code should be nothrow.
 
 At a minimum, the C code that calls the callback is not going to be expecting 
 an 
 exit via exception, and so may leave things in an indeterminate state.

Just for completeness: In a recent discussion on IRC I learned
that vendor specific attributes exist for C compilers to mark
functions as ok to use with exceptions.

-- 
Marco

Re: My first email to Walter, ever

[Marco Leise]

Am Sat, 06 Jul 2013 20:02:16 -0700
schrieb Andrei Alexandrescu seewebsiteforem...@erdani.org:

  * The venerable typeof

check

  * For a class, enumerate all of its members, and figure out their
  attributes (protection level, static or not, type...)

check

  * For a module/namespace, enumerate all of its symbols and figure
  out their attributes.

check

  * For a function, figure out its return type and the type of each of
  its formal arguments.

check

  * Figure out if a certain function exists (that should be easy if
  you have the tools above). That will be useful to decide, for
  example, if an iterator supports random access.

check

  * And if you really want to go meta, define metacode that can take
  an AST node as a parameter and can visit the AST and figure out what
  each node is. That would allow things such as loop fusion and other
  advanced stuff. But for now, let's leave those aside.

che... oh _that_ topic. I think you have done some really good
team work there. And I may add Walter's good intuition to make
the front end reusable for GDC and LDC.
Btw, you didn't want to start a new discussion about AST
manipulation through the back door, did you? ;)

-- 
Marco

Re: Announcing bottom-up-build - a build system for C/C++/D

[Marco Leise]

Am Thu, 27 Jun 2013 10:26:01 +0200
schrieb eles e...@eles.com:

 On Thursday, 27 June 2013 at 07:32:32 UTC, eles wrote:
  CTRL-Z works for me. I think it expects input.
 
 Ignore it. It just suspends it.

You might want to check how many programs you thought to have
killed like this so far in your running session. It might
eat up your RAM.

-- 
Marco

Re: Announcing bottom-up-build - a build system for C/C++/D

[Marco Leise]

How does this build tool handle projects with multiple
executables ? For example the util-linux package contains
dozens of utilities or a project might have a CLI and a GUI
version. Or there might be slight alterations like setting
a version or debug flag: -debug=threading -version=demo

-- 
Marco

Re: Announcing bottom-up-build - a build system for C/C++/D

[Marco Leise]

Am Fri, 28 Jun 2013 07:03:27 + (UTC)
schrieb Graham St Jack graham.stj...@internode.on.net:

 On Fri, 28 Jun 2013 08:28:03 +0200, Marco Leise wrote:
 
  How does this build tool handle projects with multiple executables ? For
  example the util-linux package contains dozens of utilities or a project
  might have a CLI and a GUI version. Or there might be slight alterations
  like setting a version or debug flag: -debug=threading -version=demo
 
 
 A central theme in bottom-up-build is to produce any number of build 
 artifacts: libraries, executables, tests, etc.
 
 Different versions are catered for with:
   bub-config --mode=whatever build-dir
 
 The mode is described in your configuration file, and results in a build 
 directory set up specifically for that mode.
 
 If you want multiple modes, use multiple build directories.
 
 Take a look in example/bub.cfg.

Sounds good!

-- 
Marco

Re: [Phoronix] D Language Still Showing Promise, Advancements

[Marco Leise]

Am Fri, 21 Jun 2013 14:11:17 +0200
schrieb qznc q...@web.de:

 The only-thread-local-garbage-collection of Rust is quite 
 interesting in my opinion. Since many-cores (e.g. Xeon Phi) are 
 coming, a stop-the-world garbage collector might become 
 unacceptable. If this is a good solution will be seen (maybe). I 
 certainly do not want D to adopt this experimental feature. Let 
 them do the research. ;)

Rust is called rust exactly because they don't use bleeding
edge features in the language.

-- 
Marco

Re: [Phoronix] D Language Still Showing Promise, Advancements

[Marco Leise]

Am Thu, 20 Jun 2013 22:53:13 +0200
schrieb Adam D. Ruppe destructiona...@gmail.com:

 On Thursday, 20 June 2013 at 20:47:19 UTC, Michael wrote:
  Also 3 types of pointers scares me.
 
 This actually doesn't scare me because it is kinda useful for 
 certain situations. However, I don't think it needs to be built 
 into the language because library types can do the same kind of 
 thing.

Yes, once you know about UniquePtr, NotNullable, RefCounted
and such, 3 pointer types isn't scary are all.
But Rust pointers don't map 100% to library types. They have
compiler support which removes the syntactical noise of
templates and makes them behave more natural. For example in D
everything becomes a struct once you add functionality around
it and creates corner cases that aren't supported.
I wonder how many D programmers actually use Phobos' library
pointer types.

-- 
Marco

Re: DConf 2013 Day 3 Talk 4: LDC by David Nadlinger

[Marco Leise]

Am Mon, 17 Jun 2013 22:03:09 +0200
schrieb Joseph Rushton Wakeling
joseph.wakel...@webdrake.net:

 On Monday, 17 June 2013 at 15:19:27 UTC, Andrej Mitrovic wrote:
  There seems to be some audio glitching every couple of seconds 
  (at the
  beginning). I've noticed this in other videos as well. It's 
  mostly
  minimal though, not much harm done.
 
 https://www.youtube.com/watch?v=m6jsXQm5IrM#t=106s  :-)

What's this wire doing here? It's dangerous! *pulls*

-- 
Marco

Re: DConf 2013 Day 2 Talk 6: Higgs, an experimental JIT compiler written in D by Maxime Chevalier-Boisvert

[Marco Leise]

Am Sun, 9 Jun 2013 01:53:23 +0200
schrieb Andrej Mitrovic andrej.mitrov...@gmail.com:

 On 6/9/13, bearophile bearophileh...@lycos.com wrote:
  The size of byte is easy, it's 1 byte, but if you ask me a byte
  is unsigned. I have learnt to be careful with byte/ubyte in D
 
 You, me, and Don, and probably others as well. It's easy to forget
 though that bytes are signed.

I found the integer names and sizes intuitive from the
beginning! And there was no confusion for me ever.

It starts with a 'u', it is unsigned, signed otherwise.
And the sizes are common in other programming languages, too.
long being the only confusing one, but since int is  already
the 32-bit version it should be clear that it must be 64-bit.

Though I can understand people who think of a byte as an
unsigned type, I was much more confused by C's signed chars.
Oh really? A character can be negative? Is this programming or
sociology?

-- 
Marco

Re: dmd 2.063 released with 260 bugfixes and enhancements

[Marco Leise]

Am Thu, 06 Jun 2013 14:44:57 +0200
schrieb Dicebot m.stras...@gmail.com:

 On Thursday, 6 June 2013 at 10:50:30 UTC, Marco Leise wrote:
  Yes please, this is holding me back from updating the Gentoo
  package for dmd 2.063. (Unless I want to add that missing file
  as a patch.)
 
 Why not use git tag instead?

The license doesn't allow redistribution of dmd, probably in
order to have a download statistic. I don't want to circumvent
that by creating my own .zip file or using an auto-generated
archive. (Although I wouldn't mind getting rid of the
pre-built binaries. :) )

-- 
Marco

Re: simpledisplay.d now works on as 64 bit on X

[Marco Leise]

Am Thu, 06 Jun 2013 01:27:58 +0200
schrieb Adam D. Ruppe destructiona...@gmail.com:

 cool... I found one on dsource but not github yet. The dsource 
 one transforms the XML but doesn't seem to implement all needed 
 functions. Shouldn't be hard to complete anyway though.

Does that mirror XCB (independent of XCB, implements RPC) or
add a wrapper layer around the C bindings? Because it would be
kinda cool to have a real X sever binding in D and not just
D bindings to the C bindings to the X server. :p

-- 
Marco

Re: dmd 2.063 released with 260 bugfixes and enhancements

[Marco Leise]

Am Sun, 2 Jun 2013 07:40:27 +0200
schrieb Marco Leise marco.le...@gmx.de:

 For some reason I still cannot build dmd 2.063 from the zip
 release. I mentioned it before the release on the beta and
 internals mailing lists and maybe I'm just overlooking
 something trivial, but when I run make I get:
 
 make: *** Keine Regel vorhanden, um das Target »irstate.h«,
   benötigt von »irstate.o«, zu erstellen.  Schluss.
 
 irstate.c is there, but irstate.h is missing. Is that like
 last time when VERSION was missing?

Ok, a pull request about that got merged. Let's see if it ends
up in the .zip ;)


-- 
Marco

Re: simpledisplay.d now works on as 64 bit on X

[Marco Leise]

I'm sorry to hear that you ran into the unsigned long problem.
I noticed it in time and asked about it on the NG and was told
that the difference is between Windows and Posix. One takes
long as int32, the other as the native machine word size on
64-bit. We do the same stupid mistakes over and over. :p

Also eventually we should generate X bindings for all
platforms (including Windows) from the XML definitions like
XCB does. That means a small core written in D that enables
RPC to the X Server and a generator tool that creates
functions, structs and DDoc from what is the XML files.
(See /usr/share/xcb.)

-- 
Marco

Re: simpledisplay.d now works on as 64 bit on X

[Marco Leise]

Am Mon, 03 Jun 2013 01:02:12 +0200
schrieb Adam D. Ruppe destructiona...@gmail.com:

 On Sunday, 2 June 2013 at 21:53:56 UTC, Marco Leise wrote:
  Also eventually we should generate X bindings for all
  platforms (including Windows) from the XML definitions like
  XCB does.
 
 hmm I don't see an xml file for Xlib.h on my system, do you know 
 if there is supposed to be one? Though I'm pretty sure I have 
 most of it correct now, so I guess it isn't really necessary.

XCB is generated from these XML files. They have nothing to do
with Xlib. You can find the XML files here:
http://cgit.freedesktop.org/xcb/proto/tree/src
A generator tool for the target language (C in this case)
parses them and creates include files from them that define
all the remote procedure calls to the X Server. Internally
those calls rely on a small generic set of functions that
serializes the requests for transmission.

What is needed are those generic functions written in D to
establish the connection to the server and handling
serialization, probably relying on std.socket.
And then an XML parser that would write out .d files for the
different modules of the X Server, like xrandr, xkb and so on,
which are all available as separate XML files.

This is what a RPC from glx.xml looks like:

request name=GenTextures opcode=145
field type=CONTEXT_TAG name=context_tag /
field type=INT32 name=n /
reply
pad bytes=1 /
pad bytes=24 /
list type=CARD32 name=data
fieldreflength/fieldref
/list
/reply
/request

In general they also include documentation tags which can be
made available through DDoc automatically and displayed in
IDEs like Mono-D already does.

 I know everybody says Xlib is dying but I still use it. It is 
 simple enough and gets the job done without having to learn a 
 whole thing library...

The big deal about it is asynchronous RCP especially over
internet connections to X Servers, since with Xlib you have to
way for each response before the next request.

-- 
Marco

Re: dmd 2.063 beta 5

[Marco Leise]

Am Thu, 23 May 2013 13:06:44 -0400
schrieb Andrei Alexandrescu seewebsiteforem...@erdani.org:

 TDPL 8.4 discusses a raw/cooked model of construction. The same 
 principle should apply to const/immutable member construction: you get 
 to cook the field, but you can't taste it while raw.

You are not making friends with Japanese D users.

-- 
Marco

Re: D 2.062 release

[Marco Leise]

Am Sun, 17 Feb 2013 17:02:20 -0800
schrieb Walter Bright newshou...@digitalmars.com:

 http://digitalmars.com/d/download.html
 
 The dlang.org site isn't updated yet, but the downloads are there.

I didn't read the whole thread now. Just reporting that
the .zip package cannot be compiled due to missing VERSION
file and that all the Phobos documentation now has empty
Examples: sections (which I thought was already fixed).

-- 
Marco

Re: Another opportunity for a major design win has presented itself

[Marco Leise]

Am Thu, 07 Feb 2013 22:01:10 +0100
schrieb Maxim Fomin ma...@maxim-fomin.ru:

 On Thursday, 7 February 2013 at 20:16:03 UTC, Walter Bright wrote:
  (a) get dynamic linking and loading to work
 
 Wasn't this realized before? By the way, last weeks there seems 
 to be increasing dynamic linking  loading buzz like it was not 
 an issue for ages.

It is like the news: Things boil up, get covered for a week and
disappear from public sight for a year.
It has been an issue but people have probably arranged with
static linking for now or not started to write a
particular program in D.
I remember I've tried to use structs with const members for a
while for example, but arranged with making every field
mutable. Now the topic showed up again after months.

-- 
Marco

Re: Mono-D v0.4.2 Project building fixes

[Marco Leise]

Am Thu, 15 Nov 2012 19:46:15 +0100
schrieb alex i...@alexanderbothe.com:

 ..and the round goes on with a new bunch of fundamental fixes and 
 improvements :)
 
 http://mono-d.alexanderbothe.com
 Issues go here https://github.com/aBothe/Mono-D/issues as usual,
 
 and the direct download for all Mac/Windows users that have 
 MonoDevelop 3.0.4.7:
 http://mono-d.alexanderbothe.com/repo/MonoDevelop.D_0.4.2.0_MD3.0.4.7.mpack

Installed!

-- 
Marco

Re: Wiki page for C bindings / wrappers and reimplementations

[Marco Leise]

Moved: http://mleise.dnsd.info/

(Note: This uses a free DynDNS service and the only free VPS host in the world. 
I expect occasional downtimes. ;-) )

-- 
Marco

Re: dmd 1.075 and 2.060 release

[Marco Leise]

Am Fri, 03 Aug 2012 22:27:27 +0200
schrieb Jordi Sayol g.sa...@yahoo.es:

 Al 03/08/12 21:22, En/na Jacob Carlborg ha escrit:
  On 2012-08-03 14:47, Jordi Sayol wrote:
  
  https://github.com/D-Programming-Language/installer/blob/master/linux/dmd-completion
 
  It's based on d-completion.sh at https://github.com/Lutger/d_utils
  
  How about that. It should absolutely be bundled with the .zip.
  
 
 dmd deb and rpm packages include it, but this script do not works out of the 
 box. bash-completion package must be installed, and the script should be 
 placed in a specific folder.

Yeah, and the Windows executables don't work out of the box on Linux either; 
Wine has to be installed. :-] Aside from that the specific folder likely 
depends on the used OS. (Gentoo uses /usr/share/bash-completion/ and symlinks 
in /etc/bash_completion.d/)
But I think it is like rdmd and should be in the .zip file as dmd.bashcomp 
or similar, maybe in some sort of extras folder with a README, that explains 
what it is and where to put it. For the package I maintain I kept downloading 
the .zip and the .dep, just to integrate the bash completion file. It is good 
to know where I find the latest version from now on!

-- 
Marco

Re: dmd 1.075 and 2.060 release

[Marco Leise]

Am Thu, 02 Aug 2012 12:18:37 -0700
schrieb Walter Bright newshou...@digitalmars.com:

 Another big pile of bug fixes. More contributors than ever!
 
 http://www.digitalmars.com/d/1.0/changelog.html
 http://ftp.digitalmars.com/dmd.1.075.zip
 
 http://www.digitalmars.com/d/2.0/changelog.html
 https://github.com/downloads/D-Programming-Language/dmd/dmd.2.060.zip

A new Gentoo ebuild is submitted for review as well. Thanks for the posix.mak 
changes and zlib update, I could delete quite a bit of my patch. By the way, it 
would be great if the bash completion script was also available in the .zip 
distribution.

-- 
Marco

Re: dmd 1.075 and 2.060 release

[Marco Leise]

Am Thu, 02 Aug 2012 14:54:01 -0700
schrieb Walter Bright newshou...@digitalmars.com:

 On 8/2/2012 1:46 PM, Marco Leise wrote:
  By the way, it would be great if the bash completion script was also 
  available in the .zip distribution.
 
 Please submit a pull request.
 

I'm not its maintainer. I'd like the person who updates it, to know and/or do 
the pull request.

-- 
Marco

Re: dmd 1.075 and 2.060 release

[Marco Leise]

Am Thu, 02 Aug 2012 19:23:15 -0700
schrieb Walter Bright newshou...@digitalmars.com:

 The beauty of git is you don't have to be the maintainer. Anyone can submit 
 pull requests for any project.

I did use GitHub fork  pull request before to fix a small bug in Phobos.
What I mean is, the person who wrote the bash completion script will not know 
that the file moved over to GitHub and keep updating whatever is the original 
source. Don't you know by chance who is responsible for the Ubuntu packages?

-- 
Marco

Re: Dscanner - It exists

[Marco Leise]

Am Wed, 01 Aug 2012 19:58:46 +0200
schrieb Brian Schott briancsch...@gmail.com:

 On Wednesday, 1 August 2012 at 17:36:16 UTC, Walter Bright wrote:
 
  I suggest proposing the D lexer as an addition to Phobos. But 
  if that is done, its interface would need to accept a range as 
  input, and its output should be a range of tokens.
 
 It used to be range-based, but the performance was terrible. The 
 inability to use slicing on a forward-range of characters and the 
 gigantic block on KCachegrind labeled std.utf.decode were the 
 reasons that I chose this approach. I wish I had saved the 
 measurements on this

I can understand you. I was reading a dictionary file with 
readText().splitLines(); and wondering why a unicode decoding was performed. 
Unfortunately ranges work on Unicode units and all structured text files are 
structured by ASCII characters. While these file formats probably just old or 
done with some false sense of compatibility in mind, it is also clear to their 
inventors, that parsing them is easier and faster with single-byte characters 
to delimit tokens.
But we have talked about UTF-8 vs. ASCII and foreach vs. ranges before. I still 
hope for some super-smart solution, that doesn't need a book of documentation 
and allows some kind of ASCII-equivalent range. I've heard that foreach over 
UTF-8 with a dchar loop variable, does an implicit decoding of the UTF-8 
string. While this is useful it is also not self-explanatory and needs some 
reading into the topic.

-- 
Marco

Re: Dscanner - It exists

[Marco Leise]

Am Wed, 1 Aug 2012 22:39:41 +0200
schrieb Philippe Sigaud philippe.sig...@gmail.com:

 I just tested the JSON output and it works nicely. Finally, a way to
 get imports!

What does it do if you import from _inside_ a function ?
Not that this would happen often, but it can. :-]

-- 
Marco

Re: Wiki page for C bindings / wrappers and reimplementations

[Marco Leise]

Am Wed, 01 Aug 2012 21:35:32 +0200
schrieb David d...@dav1d.de:

 You could make it output json, and someone else does the fancy html?

Maybe I'll make it a little more fancy, settle for a practical database table 
structure and then output JSON.

-- 
Marco

(OT: wiki4d, was: dnotify)

[Marco Leise]

Am Mon, 30 Jul 2012 01:37:30 -0400
schrieb Nick Sabalausky seewebsitetocontac...@semitwist.com:

 On Sun, 29 Jul 2012 20:56:48 +0200
 Marco Leise marco.le...@gmx.de wrote:
  
  if I had access to a Wiki about D...
  
 
 You do!
 
 http://www.prowiki.org/wiki4d/wiki.cgi

I knew of this, but it didn't come to my head that I had to edit a non-existing 
page or save an existing page under a new name. I can add a page about C 
bindings now, but it conflicts with the category approach that is used there 
(e.g. Database Bindings). Also, I wouldn't count Deimos bindings as projects. 
Technically I'd prefer a database of existing D stuff, and dynamic web pages 
that filter by category/C bindings/last update etc.

-- 
Marco

Wiki page for C bindings / wrappers and reimplementations

[Marco Leise]

I find that we lack a way to search for ported C headers or D alternatives that 
do the same. So I started a Wiki page here: 
http://prowiki.org/wiki4d/wiki.cgi?CLibraryBindings

Not sure if it is useful there. I'd like to see dynamic pages on dlang.org in a 
distant future, which make searching by category or compatibility easier.

-- 
Marco

Re: Wiki page for C bindings / wrappers and reimplementations

[Marco Leise]

Am Mon, 30 Jul 2012 20:30:21 +0200
schrieb David d...@dav1d.de:

 Am 30.07.2012 16:50, schrieb Marco Leise:
  I find that we lack a way to search for ported C headers or D alternatives 
  that do the same. So I started a Wiki page here: 
  http://prowiki.org/wiki4d/wiki.cgi?CLibraryBindings
 
  Not sure if it is useful there. I'd like to see dynamic pages on dlang.org 
  in a distant future, which make searching by category or compatibility 
  easier.
 
 
 This great and yeah on dlang.org it would be even better.

Hey you just say that because Glamour is already in the list ;)

-- 
Marco

Re: dnotify

[Marco Leise]

Am Sun, 29 Jul 2012 01:04:35 +0200
schrieb David d...@dav1d.de:

 https://github.com/Dav1dde/dnotify
 
 Nothing special, this is just an abstraction for the libnotify binding 
 in deimos (pull request not yet merged)

This is nice, if I had access to a Wiki about D, I'd add a list of libraries 
that can be used (name, C binding, D abstraction). I have to say that I was a 
bit confused at first, thinking you created a wrapper for the filesystem 
monitor 'dnotify' http://en.wikipedia.org/wiki/Dnotify

-- 
Marco

Re: dnotify

[Marco Leise]

P.S.:
NotificationError should probably be NotificationException, since the term 
Error is used for unrecoverable exceptions in D and it does in fact inherit 
from Exception.

Re: Coming Soon: Stable D Releases!

[Marco Leise]

Am Mon, 16 Jul 2012 00:51:16 -0700
schrieb Adam Wilson flybo...@gmail.com:

For the 2.059 release I waited a while until apparent regressions and oddities 
in Phobos (deprecation warnings) were fixed in the main repository before 
updating the dmd package for Gentoo. This was quite a bit of manual work 
collecting, applying and testing patches from GitHub and the bug tracker.
Will this project be an option for me as a package maintainer? What I need is 
basically the official dmd zip file (sources, html docs, man pages, bash 
completion, rdmd binary  co.), but with hotfixes applied at about a week or 
two after release.
It seems like your work could make this possible and also allow me to issue a 
new revision of the package in case of, say a security fix that comes a month 
after release.
Last not least, will we get notified about critical/important fixes in the 
stable branch? 'Cause polling is so old-school ;). RSS, news group, email, 
phone call, anything will do. It looks like GitHub has a commit-RSS-feed, but 
it would be very noisy on most repositories with messages like forgot this in 
last check-in or deleted unused file/code, and not every commit warrants a 
new package.

As it shows, the beta phase doesn't always catch all regressions in people's 
code, so I encourage you to do this project and eventually it will be used by 
GDC and other major from-source projects. By the way: Should this also later 
become the base for the official zip file download? IIRC Walter wanted to keep 
track of the DMD downloads from the main web site (no redistribution) and 
hotfixed versions of D could become increasingly popular.

-- 
Marco

Re: Coming Soon: Stable D Releases!

[Marco Leise]

Am Mon, 16 Jul 2012 17:21:39 +0100
schrieb Iain Buclaw ibuc...@ubuntu.com:

 On 16 July 2012 14:00, Marco Leise marco.le...@gmx.de wrote:
  Am Mon, 16 Jul 2012 00:51:16 -0700
  schrieb Adam Wilson flybo...@gmail.com:
 
  As it shows, the beta phase doesn't always catch all regressions in 
  people's code, so I encourage you to do this project and eventually it will 
  be used by GDC and other major from-source projects. By the way: Should 
  this also later become the base for the official zip file download? IIRC 
  Walter wanted to keep track of the DMD downloads from the main web site (no 
  redistribution) and hotfixed versions of D could become increasingly 
  popular.
 
  --
  Marco
 
 
 And what benefits would GDC get from opting to use this rather than
 the normal releases?

What he said, [regression] fixes that didn't make it into the initial release. 
I don't know about GDC's 'patch level', but for 2.059 I applied patches for the 
following issues after release, to have it feel as solid as good old 2.058:
- issue-7907
- issue-7911
- issue-7922
- outOfMemoryError-undeprecation
- std-path-sep-deprecation

In case crypto algorithms become part of Phobos, some patches may improve 
security as well. Didn't you say you work only with the GitHub release tags for 
stability?

-- 
Marco

Re: D:GameVFS 0.1

[Marco Leise]

Am Fri, 20 Jan 2012 00:36:11 +0100
schrieb Kiith-Sa 4...@theanswer.com:

 You can get D:GameVFS 0.1 here: 
 https://github.com/kiith-sa/D-GameVFS/downloads 
 For basic introduction, see the tutorial in the README at
 https://github.com/kiith-sa/D-GameVFS and the API documentation in the 
 package.

I didn't check your code, but does it handle case-insensitive Windows OS as 
well as Unix like? Looking at the archive files of the game F.E.A.R. I noticed 
that at some point they started using lower case exclusively (and that game 
only runs on Windows). Maybe it was easier to compare directories and patches 
that way.
The stacking of directories can also be used to patch the game, of course. What 
striked me as strange in said game, was that they added the game server/client 
code (for single player as well) in archives, but the main executable, DB 
driver or video codec where plain files in the base directory. So the latter 
got overwritten with every update, while the former just added up with each 
patch level, so at patch 1.08, there were 9 version of them - irreversible yet, 
unless you archived the old main executables as well.
I'm not trying to say that an update should generally overwrite previous files, 
though. Just that their system didn't make sense in every bit. The separate 
archives for each update had one clear benefit: The first expansion was created 
at the time of 1.07, and they could just link in the base archives + updates up 
to 1.07 without running the risk that later updates to the main game would 
interfere with that expansion.

They used a config file in the main executable's directory that listed in 
ascending priority, the directories or archives that should be used. They 
didn't reuse the main executable for expansions though, so those went into sub 
directories with their own configuration file that linked in base game and 
patches using ../some_archive.

Also they split their archives, both to stay below 4GB and to have 
localizations (voices) in one easily replaceable file, which may be have been 
handy at release time. Did you think of ways to probably even switch the 
language from a menu?

From your example in the readme, it isn't immediately clear what the names 
'user', 'main' and 'root' are used for. I assume that StackDir inherits from 
FSDir which is a nice design, but I'd probably expect things to work out as in 
F.E.A.R. or Quake where these things are implementation details. In other 
words, the game *never* attempts a write access into it's data files, and there 
is only one file-system root, not necessarily a singleton. This root object 
then offers methods to mount directories or archives.

  auto fs = new FileSystem();
  fs.mount(main); // can be directory main or main.zip
  fs.mount(user); // overrides main now

Something like this would be nice to 'seamlessly' extract an archive to a 
directory with the same name. But you may have your own ideas there. Writing 
archive files can be difficult as some archives may require recompression or 
check sum creation over the whole archive. I guess most developers just release 
their game with read-only archives and development happens with full write 
access on the extracted directories through external tools (unless the game is 
the level editor).

-- 
Marco

Re: Robert Schadek's Distributed Multithreaded Caching D Compiler

[Marco Leise]

Am Wed, 23 May 2012 16:26:23 -0700
schrieb Walter Bright newshou...@digitalmars.com:

 presentation has been accepted for the D Conference 2012.
 
 Congratulations, Robert!
 
 http://astoriaseminar.com/sessions.html

What, still no caching at finer granularity than modules? Meh. :p
Besides my reservation, it is good to see at least one 'independent' speaker. A 
lot of the news group topics would also make for interesting sessions if they 
are important enough.

-- 
Marco

Re: DCT: D compiler as a collection of libraries

[Marco Leise]

Am Sun, 20 May 2012 10:09:34 +0200
schrieb Roman D. Boiko r...@d-coding.com:

 Could you name a few specific concerns?

Mostly my own gut feeling, that things that sound great in my head turn out to 
bite me in the end. Things that one just doesn't think of because of the 
limited horizon everyone has and probably a lack of experience in the field of 
compiler/tools writing.
On the other hand I have good experience with working by community feedback. So 
the rough edges in the design may already be ironed out by now.

 I'm going to pick up several use cases and prioritize them 
 according to my judgement. Feel free to suggest any cases that 
 you think are needed (with motivation). Prioritizing is necessary 
 to define what is out of scope and plan work into milestones, in 
 order to ensure the project is feasible.

There is one feature I remember caused some head-aches for Code::Blocks. They 
used a separate parser instance for every project in the IDE, which meant that 
all the standard include files would be parsed and kept in memory multiple 
times. When they later switched to a common parser for all projects they ran 
into stability issues. If you can arrange it, it would be great for 
multi-project IDEs to be able to add and remove projects to your parser without 
reparsing Phobos/druntime (which may have its .di files replaced by .d files, 
looking at the past discussion).

C bindings could be an option. (As in: the smallest common denominator.) They 
allow existing tools (written in Java, C#, Python, ...) to use your library.

  Since assembly code is usually small I just preallocate an 
  array of sourceCode.length tokens and realloc it to the correct 
  size when I'm done parsing. Nothing pretty, but simple and I am 
  sure it won't get any faster ;).
 I'm sure it will :) (I'm going to elaborate on this some time 
 later).

I'm curious.

 There are several EoF conditions: \0, \x1A, __EOF__ and physicas 
 EOF. And any loop would need to check for all. Preallocation 
 could eliminate the need to check for physical EoF, but would 
 make it impossible to avoid input string copying and also would 
 not allow passing a function a slice of string to be lexed.

I found that I usually either load from file into a newly allocated buffer 
(where a copy occurs, only because I forgot about assumeUnique in 
std.exception) or I am editing the file in which case I recreate the source 
string after every key stroke anyway. I can still pass slices of that string to 
functions though. Not sure what you mean.
It probably doesn't work for D as well as for ASM code, but I could also check 
for \x1A and __EOF__ in the same fashion. (By the way, is it \x1A (substitute, 
^Z) or did you mean \0x04 (end-of-transmission, ^D)?) The way it works is: 
Parser states like 'in expression' can safely peek at the next character at any 
time. If it doesn't match what they expect they emit an error and drop back to 
the surrounding parser state. When they reach the file level, that's the 
only place where an EOF (which will only occur once per file anyway) will be 
consumed.
In theory one can drop all string length checks and work on char* directly with 
a known terminator char that is distinct from anything else.

  ** Errors  **
  I generally keep the start and end column, in case someone 
  wants that.

 This functionality has been the priority from the beginning. Not 
 implemented yet but designed for. Evaluation of column and line 
 only on demand is caused by the assumption that such information 
 is needed rarely (primarily to display information to the user). 
 My new design also addresses the use cases when it is needed 
 frequently.

 [...] 

 Incremental changes are the key to efficiency, and I'm going to 
 invest a lot of effort into making them. Also immutability of 
 data structures will enable many optimizations.

No more questions!

-- 
Marco

Re: DCT: D compiler as a collection of libraries

[Marco Leise]

Am Sun, 20 May 2012 20:37:07 +0200
schrieb Roman D. Boiko r...@d-coding.com:

   Since assembly code is usually small I just preallocate an 
   array of sourceCode.length tokens and realloc it to the 
   correct size when I'm done parsing. Nothing pretty, but 
   simple and I am sure it won't get any faster ;).
  I'm sure it will :) (I'm going to elaborate on this some time 
  later).
 
  I'm curious.
 Maybe I'm don't understand your use case, but the idea is that if
 you parse as you type it should be possible to avoid parsing and
 allocating memory for those lines which have not changed. But
 that is not compatible with storing tokens in the array, since it
 would cause reallocating memory each time, so the other data
 structure should be used (e.g., a linked list, or, if efficient
 lookup is needed, a red-black tree). Only benchmarks can show
 whether (and how much) my approach would be faster for specific
 situation (input patterns like average size of data, complexity
 of parsing algorithms, requirements, etc.).

I just only thought about the single-use situation, not the situation when 
editing files. Now the idea has evolved a bit and you probably thought about 
storing the scope hierarchy in a tree, too. It is still difficult to write a 
fast parser when someone can add/remove a '{' somewhere at the top of 
datetime.d, which changes the interpretation of the rest of the file. Mono-D 
has some hickups worse than those (several seconds even) - maybe a bug. I'll 
keep my fingers crossed.

 If you want to pass a slice of input string to a function, you
 cannot append \0 to it without copying data. If you don't append
 some pre-defined character, you must check for length *and* all
 supported terminating characters. On the contrary, your design
 might not require passing slices, and if language syntax allows
 deterministic parsing (when you know what to expect next), there
 is no need for checking EoF.

Now I get it!

-- 
Marco

Re: DCT: D compiler as a collection of libraries

[Marco Leise]

Am Fri, 11 May 2012 10:01:28 +0200
schrieb Roman D. Boiko r...@d-coding.com:

 There were several discussions about the need for a D compiler 
 library.
 
 I propose my draft implementation of lexer for community review:
 https://github.com/roman-d-boiko/dct
 
 Lexer is based on Brian Schott's project 
 https://github.com/Hackerpilot/Dscanner, but it has been 
 refactored and extended (and more changes are on the way).
 
 The goal is to have source code loading, lexer, parser and 
 semantic analysis available as parts of Phobos. These libraries 
 should be designed to be usable in multiple scenarios (e.g., 
 refactoring, code analysis, etc.).
 
 My commitment is to have at least front end built this year (and 
 conforming to the D2 specification unless explicitly stated 
 otherwise for some particular aspect).
 
 Please post any feed here. A dedicated project web-site will be 
 created later.

A general purpose D front-end library has been discussed several times. So at 
least for some day-dreaming about better IDE support and code analysis tools it 
has a huge value. It is good to see that someone finally takes the time to 
implement one. My only concern is that not enough planing went into the design. 
I think about things like brainstorming and collecting possible use cases from 
the community or looking at how some C++ tools do their job and what 
infrastructure they are built on. Although it is really difficult to tell from 
other people's code which decision was 'important' and what was just the 
author's way to do it.

Inclusion into Phobos I would not see as a priority. As Jonathan said, there 
are already some clear visions of how such modules would look like. Also if any 
project seeks to replace the DMD front-end, Walter should be the technical 
advisor. Like anyone putting a lot of time and effort into a design, he could 
have strong feelings about certain decisions and implementing them in a 
seemingly worse way.
That said, you make the impression of being really dedicated to the project, 
even giving yourself a time line, which is a good sign. I wish you all the best 
for the project and hope that - even without any 'official' recognition - it 
will see a lot of use as the base of D tools.

To learn about parsing I wrote a syntax highlighter for the DCPU-16 assembly of 
Minecraft author Markus Persson. (Which was a good exercise.) Interestingly I 
ended up with a similar design for the Token struct like yours:
- separate array for line # lookup
- TokenType/TokenKind enum
- Trie for matching token kinds (why do you expand it into nested switch-case 
through CTFE mixins though?)
Since assembly code is usually small I just preallocate an array of 
sourceCode.length tokens and realloc it to the correct size when I'm done 
parsing. Nothing pretty, but simple and I am sure it won't get any faster ;).
One notable difference is that I only check for isEoF in one location, since I 
append \0 to the source code as a stop token (that can also serve as an 
end-of-line indicator). (The general idea is to move checks outside of loops.)

** Errors  **
I generally keep the start and end column, in case someone wants that. A real 
world example:

  ubyte x = ...;
  if (x = 0  x = 8) ...
  Line 2, warning: Comparison is always true.

At first glace you say No, a byte can become larger than 8., but the compiler 
is just complaining about the first part of the expression here, which would be 
clarified by e.g. some ASCII/Unicode art:

  Line 2, warning: Comparison is always true:
  if (x = 0  x = 8) ...
  ^^

** Code highlighting **
Especially Gtk's TextView (which GtkSourceView is base on) isn't made for 
several thousand tokens. The text view will take a second per 2 tokens or 
so. The source view works around that by highlighting in chunks, but you can 
still fell the delay when you jump to the end of the file in gedit and even 
MonoDevelop suffers from the bad performance. If I understood your posts 
correctly, you are already planning for minimal change sets. It is *very* 
important to only update changed parts in a syntax highlighting editor. So for 
now I ended up writing a 'insertText' and 'deleteText' method for the parser 
that take 'start index', 'text' and 'start index', 'end index' respectively and 
return a list of removed and added tokens.
If possible, make sure that your library can be used with GtkSourceView, 
Scintilla and QSyntaxHighlighter. Unfortunately the bindings for the last two 
could use an update, but the API help should get you an idea of how to use it 
most efficiently.

-- 
Marco

Re: Native GTK bindings v2

[Marco Leise]

I did a test run with a system profiler and I see this:

--8-

CPU: Core 2, speed 2001 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask 
of 0x00 (Unhalted core cycles) count 10

%symbol name
24.5102  void gc.gcx.Pool.__invariant()
10.4862  void gc.gcx.Gcx.__invariant()

--8-

invariant()? Wouldn't that be removed in a release build? Then it came to my 
head, that maybe, just maybe, setting
export DFLAGS=-ffunction-sections -fdata-sections
before running make would disable all other flags. ... Yes that was the 
problem. Now it is working as expected.

-- 
Marco

Gentoo ebuild for 2.059

[Marco Leise]

DMD 2.059 is now available in the sunrise overlay. It took a little longer, 
but includes fixes for the regressions that occurred shortly after the initial 
release:

- move the import of onOutOfMemoryError out of the deprecated section
  
https://github.com/D-Programming-Language/phobos/commit/55d89dcad1629a2a1a9165ffd5a91aee588fd8a1

- sep (deprecated) - dirSeparator
  
https://github.com/D-Programming-Language/phobos/commit/dfe83f155d92c98e08b9c896f6312d96794b39f9

- Issue 7907 - [ICE] invalid expression on template argument crashes dmd
  https://github.com/D-Programming-Language/dmd/pull/879

- Issue 7911 - Nested static if failing to execute
  http://d.puremagic.com/issues/show_bug.cgi?id=7911

- Issue 7922 - alias this causes weird formatting issues for strings
  https://github.com/D-Programming-Language/dmd/pull/885

So it should be safe to upgrade on Gentoo now.

-- 
Marco

Re: Native GTK bindings v2

[Marco Leise]

Am Sun, 22 Apr 2012 00:54:41 +0200
schrieb Artur Skawina art.08...@gmail.com:

 On 04/22/12 00:06, Marco Leise wrote:
  Am Sat, 21 Apr 2012 21:46:18 +0200
  schrieb Artur Skawina art.08...@gmail.com:
  
  On 04/21/12 19:16, Marco Leise wrote:
  I just noticed cairo.d is still a dummy. I am using image surfaces. 
 
  Yes, cairo.d is generated from GI data too and only contains the few 
  symbols
  and types required to use the other libs. There are other cairo D 
  bindings, which
  probably could be used with a small glue layer. Making sane cairo bindings 
  is
  on my to-do list, but I won't have the time for that in the next few 
  weeks. 
  
  No hurry, I have GtkD. It's biggest win is that it is considered stable and 
  usable by many for a long time.
  
  -ffunction-sections -fdata-sections -Wl,--gc-sections is a good idea, 
  but I found it to break exception throwing for my programs.
 
  GDC/DMD? Would you happen to have a small contained sample that breaks?
  Not garbage collecting sections means executables that are several times
  larger (IIRC for the small gtk example the difference was 1.2M vs 0.3M).
 
  artur
  
  No example. I don't know what I tried it on, but I remember that I was 
  hunting a bug that occurred, because an important exception wasn't thrown. 
  I think it was a plain old throw  It was with GDC, since DMD doesn't 
  offer a one-section-per-function flag. I can give it a second try. I didn't 
  post any bug reports, since gc-sections is a difficult beast:
  https://bitbucket.org/goshawk/gdc/issue/293/ffunction-sections-fdata-sections-for
  https://bugzilla.redhat.com/show_bug.cgi?id=788107
  Without support from Iain, I don't expect a bug would be fixed by adding 
  some hack to keep a function from being garbage collected (or whatever 
  caused me problems).
 
 The first bug is about GDC having gc-sections as a *default*, which Iain 
 seems to think
 isn't important because a) phobos will be a shared library soon and b) it 
 needs testing.
 I agree with the latter, but the former is wrong - it will be many, many 
 years before
 even considering using a phobos DLL will be an option (for reasons that i 
 mentioned in
 #293).
 The second ticket is about some already fixed upstream binutils bug.
 
 I've been running with phobos built using -ffunction-sections 
 -fdata-sections since ~
 the time of #293 and so far haven't seen any problems (which of course 
 doesn't mean that
 there aren't any). 
 
 Preventing a section from being garbage collected could be as simple as 
 adding KEEP()
 around its name in the linker script. But i've failed to reproduce any 
 problems.
 I'll need to remember to keep adding the options to every app makefile, 
 because so far i
 often didn't bother to do it... But, at least for exceptions, i wouldn't 
 expect problems,
 as the C++ case is already handled and GDC uses a similar scheme.
 
 artur

Ok, my trust in gc-sections is slowly returning. So far it works as advertised. 
:) It shaved off ~900 KB, so I am at acceptable 2.2 MB now. LTO is not working 
for me due to 
https://bitbucket.org/goshawk/gdc/issue/284/lto-undefined-reference-to .

-- 
Marco