Re: Fastest JSON parser in the world is a D project

[mw via Digitalmars-d-announce]

On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise wrote:

fast.json usage:

auto json = parseTrustedJSON(`{"x":123}`);

Work with a single key from an object:

json.singleKey!"someKey"
json.someKey



Newbie Q: how to get the value?

---
import std.stdio;
import fast.json;

void main() {
auto json = parseTrustedJSON(`{"x":123}`);
writeln(json.x);  // shall I get 123 here?
}
---

core.exception.AssertError@/home/xxx/.dub/packages/fast-0.3.5/fast/source/fast/json.d(1208):
 Assertion failure


dmd v2.092.0

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Fri, 13 Jul 2018 18:14:35 +
schrieb iris :

> Any idea about the performance of this json parser? 
> https://jsonformatter.org/json-parser ?

That one is implemented in client side JavaScript. I didn't
measure it, but the closest match in Kostya's benchmark could
be the Node JS entry that is an order of magnitude slower.

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[iris via Digitalmars-d-announce]

Any idea about the performance of this json parser? 
https://jsonformatter.org/json-parser ?

Re: Fastest JSON parser in the world is a D project

[Mir Al Monsor via Digitalmars-d-announce]

you could check it out on http://jsontuneup.com for treeview your 
json object and wrong inside your json.

Re: Fastest JSON parser in the world is a D project

[Suliman via Digitalmars-d-announce]

What about data validation? Does it's fast complete full 
validation of data, and what about other parsers? Are they 
complete full validation?

Re: Fastest JSON parser in the world is a D project

[Suliman via Digitalmars-d-announce]

Marco, could you add your lib to review or do any steps that will 
help to include it's in Phobos? I think not only I interesting in 
good base JSON lib in base distribution.

Re: Fastest JSON parser in the world is a D project

[Jack Applegame via Digitalmars-d-announce]

On Thursday, 29 October 2015 at 12:11:54 UTC, Suliman wrote:
Marco, could you add your lib to review or do any steps that 
will help to include it's in Phobos? I think not only I 
interesting in good base JSON lib in base distribution.


Marco's json library doesn't meet requirements for inclusion in 
Phobos and should stay separately in DUB registry.
Phobos needs much more generic library with support for streaming 
and ranges. I believe, that at the moment the best candidate is 
std.data.json by Sönke Ludwig.

Re: Fastest JSON parser in the world is a D project

[wobbles via Digitalmars-d-announce]

On Wednesday, 28 October 2015 at 11:26:59 UTC, wobbles wrote:


So yes - opDispatch is cool but should be used VERY sparingly.


I just had a thought, I could check if dataName is in 
[__traits(allMembers ... )]. That would at least ensure I'm 
referencing something that exists. Maybe that'd be useful in 
vibes Bson/Json code. (Except the opposite, you want to check 
you're referencing something that DOESN'T exist, so you can be 
sure it's not 'remove' for example).

Re: Fastest JSON parser in the world is a D project

[wobbles via Digitalmars-d-announce]

On Tuesday, 27 October 2015 at 14:00:07 UTC, Martin Nowak wrote:

On Tuesday, 27 October 2015 at 13:14:36 UTC, wobbles wrote:
How can `coordinates` member be known at compile-time when 
the input argument is a run-time string?


I suspect through the opDispatch operator overload.

http://dlang.org/operatoroverloading.html#dispatch


Yikes, this is such an anti-pattern.
https://github.com/rejectedsoftware/vibe.d/issues/634


Heh - yeah it is quite problematic.

The only time I've needed to use it was when I was reading in 
Json with some structure like

{
  [
{  "timestamp" : { ... timestamp info ... },
   "info1" : { ... info ...},
   "info2" : { ... info ...},
.
.
   "info 23" : { ... info  ...}
},
{ < more of the above >}
  ]
}

and I wanted to be able get a Json[timestamp] map, where the Json 
is either a info1, info2 etc etc.
I didn't want to write 23 different functions "hash_info1", 
"hash_info2" etc etc.

So, opDispatch!

Basically I wanted to hash the timestamp and some data. My 
opDispatch became:


@ignore auto opDispatch(string name)(){
		static assert(name.startsWith("hash_"), "Error, use 
StatHosts.hash_XYZ to gather XYZ[timestamp] info");

static assert(name.length > 5);
enum dataName = name[5..$];
typeof(mixin("StatDetail."~dataName))[StatTimestampDetail] data;

foreach(stat; statistics){
data[stat.timestamp] = mixin("stat."~dataName);
}
return data;
}

23 functions merged into 1...

The static assert reduces the number of places it can break 
things at least, still some weird things can happen but for the 
most part it's ok.


So yes - opDispatch is cool but should be used VERY sparingly.

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Tue, 27 Oct 2015 14:00:06 +
schrieb Martin Nowak :

> On Tuesday, 27 October 2015 at 13:14:36 UTC, wobbles wrote:
> >> How can `coordinates` member be known at compile-time when the 
> >> input argument is a run-time string?
> >
> > I suspect through the opDispatch operator overload.
> >
> > http://dlang.org/operatoroverloading.html#dispatch
> 
> Yikes, this is such an anti-pattern.
> https://github.com/rejectedsoftware/vibe.d/issues/634

For my defense I can say that the JSON parser is not a range
and thus less likely to be used in UFCS chains. It can be
replaced with .singleKey!"coordinates"()

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Adam D. Ruppe via Digitalmars-d-announce]

On Tuesday, 27 October 2015 at 14:00:07 UTC, Martin Nowak wrote:

Yikes, this is such an anti-pattern.
https://github.com/rejectedsoftware/vibe.d/issues/634


Every time I use opDispatch, I add an if(name != "popFront") 
constraint, at least (unless it is supposed to be forwarding). It 
helps with this a lot and think everyone should do it.

Re: Fastest JSON parser in the world is a D project

[Meta via Digitalmars-d-announce]

On Wednesday, 28 October 2015 at 13:56:27 UTC, Adam D. Ruppe 
wrote:

On Tuesday, 27 October 2015 at 14:00:07 UTC, Martin Nowak wrote:

Yikes, this is such an anti-pattern.
https://github.com/rejectedsoftware/vibe.d/issues/634


Every time I use opDispatch, I add an if(name != "popFront") 
constraint, at least (unless it is supposed to be forwarding). 
It helps with this a lot and think everyone should do it.


I would go even farther and say that one should never define 
opDispatch without a template constraint limiting which members 
can be dispatched on. It may be a bit radical but we could even 
go as far as outright deprecating unconstrained opDispatch.


//Okay
auto opDispatch(string member)()
if (member == "get" || isVectorSwizzle!member)
{
//...
}

//Deprecated
auto opDispatch(string member)()
{
//...
}

Re: Fastest JSON parser in the world is a D project

[Martin Nowak via Digitalmars-d-announce]

On Tuesday, 27 October 2015 at 13:14:36 UTC, wobbles wrote:
How can `coordinates` member be known at compile-time when the 
input argument is a run-time string?


I suspect through the opDispatch operator overload.

http://dlang.org/operatoroverloading.html#dispatch


Yikes, this is such an anti-pattern.
https://github.com/rejectedsoftware/vibe.d/issues/634

Re: Fastest JSON parser in the world is a D project

[Nordlöw via Digitalmars-d-announce]

On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise wrote:

Example:

double x = 0, y = 0, z = 0;
auto json = parseTrustedJSON(`{ "coordinates": [ { "x": 1, 
"y": 2, "z": 3 }, … ] }`);


foreach (idx; json.coordinates)
{
// Provide one function for each key you are interested 
in

json.keySwitch!("x", "y", "z")(
{ x += json.read!double; },
{ y += json.read!double; },
{ z += json.read!double; }
);
}


How can `coordinates` member be known at compile-time when the 
input argument is a run-time string?

Re: Fastest JSON parser in the world is a D project

[Laeeth Isharc via Digitalmars-d-announce]

On Thursday, 22 October 2015 at 20:10:36 UTC, rsw0x wrote:
On Thursday, 22 October 2015 at 19:16:00 UTC, Laeeth Isharc 
wrote:
On Thursday, 22 October 2015 at 18:23:08 UTC, Andrei 
Alexandrescu wrote:

On 10/22/2015 09:08 AM, Walter Bright wrote:

[...]


This has been a homerun. Congratulations for this work and 
also for publicizing it! (Consider it might have remained 
just one forum discussion read by all of 80 persons...) -- 
Andrei


We really do need to stop hiding our light under a bushel.  
Thinking in marketing terms doesn't always come easy to 
technically minded people, and I understand why, but 
ultimately the community benefits a great deal from people 
becoming aware of the very real benefits D has to offer (alas 
people won't just get it, even if you think they should), and 
there are personal career benefits too from helping 
communicate how you have applied D to do useful work.  It's 
hard to find great programmers and showing what you can do 
will pay off over time.


D has no well defined area to be used in. Everyone knows D, 
when written in a very specific C-mimicking way, is performant. 
But nobody is using C# or Scala or Python for performance.


You reply to my post, but I don't entirely see how it relates.  D 
is very flexible, and that's its virtue.  Because splitting a 
codebase across multiple languages does have a cost, even if it's 
often worth paying the cost in order to use the right till for 
the job when those tools are by their nature specialised.  I 
don't think everyone knows D is performant, and I wouldn't say 
fast JSON is written in a C mimicking way, taken as a whole.


Choices are based on making trade-offs, and the relevant data are 
not static, but constantly shifting.  When an SSD in 2015 that 
isn't especially pricey gives 2.1 Gig a sec throughput and one 
has many terabytes of text data a month to get through, and 
that's today and datasets keep growing and what I write today may 
be in use for years then the right decision will be a very 
different one to that five years ago.   That's not just my 
perception, but those in other fields where the problems are 
similar - bioinformatics and advertising data being some of the 
many others.  AdRoll is known for their Python work, but their 
data scientists use D.


And my point, which you didn't really reply to, is that as a 
community we should do a bit more to share our experiences on how 
D can be useful in doing real work.  As Walter observes, that's 
also something that pays off personally too.

Re: Fastest JSON parser in the world is a D project

[Walter Bright via Digitalmars-d-announce]

On 10/22/2015 1:53 PM, Marco Leise wrote:

There is at least one hurdle. I don't have a place to publish
articles, no personal blog or site I contribute articles to
and I don't feel like creating a one-shot one right now. :)


You can publish it on my site:

http://digitalmars.com/articles/index.html

But I highly recommend that you create your own web site. It's great for your 
professional career.

Re: Fastest JSON parser in the world is a D project

[Walter Bright via Digitalmars-d-announce]

On 10/22/2015 9:29 PM, Joakim wrote:

The main D forum is as good a place as any.  Just start a thread there.


No, articles should be more than postings.

Re: Fastest JSON parser in the world is a D project

[Jacob Carlborg via Digitalmars-d-announce]

On 2015-10-22 22:53, Marco Leise wrote:


There is at least one hurdle. I don't have a place to publish
articles, no personal blog or site I contribute articles to
and I don't feel like creating a one-shot one right now. :)


You could have a look at this blog implementation by Dicebot [1]. You 
still need to host it though.


[1] https://github.com/Dicebot/mood

--
/Jacob Carlborg

Re: Fastest JSON parser in the world is a D project

[Laeeth Isharc via Digitalmars-d-announce]

On Friday, 23 October 2015 at 19:48:31 UTC, Jacob Carlborg wrote:

On 2015-10-22 22:53, Marco Leise wrote:


There is at least one hurdle. I don't have a place to publish
articles, no personal blog or site I contribute articles to
and I don't feel like creating a one-shot one right now. :)


You could have a look at this blog implementation by Dicebot 
[1]. You still need to host it though.


[1] https://github.com/Dicebot/mood


Mood is very nice, and I plan on using it in the medium term 
(made a pull request so it would compile using gdc or ldc).  But 
you might want to wait a little while as you want a blog to be 
stable, and I think there is a problem with segfaulting right now 
- perhaps to do with the caching of posts, although it shouldn't 
be hard either to fix that or rewrite it your own way (as I 
started doing).


It's worth setting one up though - what you use doesn't matter 
(look at Nikola or one of the other static site generators) - and 
Walter is right.

Re: Fastest JSON parser in the world is a D project

[Meta via Digitalmars-d-announce]

On Thursday, 22 October 2015 at 19:16:00 UTC, Laeeth Isharc wrote:
We really do need to stop hiding our light under a bushel.  
Thinking in marketing terms doesn't always come easy to 
technically minded people, and I understand why, but ultimately 
the community benefits a great deal from people becoming aware 
of the very real benefits D has to offer (alas people won't 
just get it, even if you think they should), and there are 
personal career benefits too from helping communicate how you 
have applied D to do useful work.  It's hard to find great 
programmers and showing what you can do will pay off over time.


Yeah, we don't want to repeat Lisp's mistake.

Re: Fastest JSON parser in the world is a D project

[Laeeth Isharc via Digitalmars-d-announce]

On Thursday, 22 October 2015 at 18:23:08 UTC, Andrei Alexandrescu 
wrote:

On 10/22/2015 09:08 AM, Walter Bright wrote:

On 10/21/2015 1:38 PM, Laeeth Isharc wrote:

On Wednesday, 21 October 2015 at 19:03:56 UTC, Suliman wrote:

Could anybody reddit this benchmark?


done
https://www.reddit.com/r/programming/comments/3pojrz/the_fastest_json_parser_in_the_world/





It's item 9 on the front page of https://news.ycombinator.com/ 
too!


This has been a homerun. Congratulations for this work and also 
for publicizing it! (Consider it might have remained just one 
forum discussion read by all of 80 persons...) -- Andrei


We really do need to stop hiding our light under a bushel.  
Thinking in marketing terms doesn't always come easy to 
technically minded people, and I understand why, but ultimately 
the community benefits a great deal from people becoming aware of 
the very real benefits D has to offer (alas people won't just get 
it, even if you think they should), and there are personal career 
benefits too from helping communicate how you have applied D to 
do useful work.  It's hard to find great programmers and showing 
what you can do will pay off over time.

Re: Fastest JSON parser in the world is a D project

[rsw0x via Digitalmars-d-announce]

On Thursday, 22 October 2015 at 19:16:00 UTC, Laeeth Isharc wrote:
On Thursday, 22 October 2015 at 18:23:08 UTC, Andrei 
Alexandrescu wrote:

On 10/22/2015 09:08 AM, Walter Bright wrote:

[...]


This has been a homerun. Congratulations for this work and 
also for publicizing it! (Consider it might have remained just 
one forum discussion read by all of 80 persons...) -- Andrei


We really do need to stop hiding our light under a bushel.  
Thinking in marketing terms doesn't always come easy to 
technically minded people, and I understand why, but ultimately 
the community benefits a great deal from people becoming aware 
of the very real benefits D has to offer (alas people won't 
just get it, even if you think they should), and there are 
personal career benefits too from helping communicate how you 
have applied D to do useful work.  It's hard to find great 
programmers and showing what you can do will pay off over time.


D has no well defined area to be used in. Everyone knows D, when 
written in a very specific C-mimicking way, is performant. But 
nobody is using C# or Scala or Python for performance.

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Thu, 22 Oct 2015 06:10:56 -0700
schrieb Walter Bright :

> On 10/21/2015 3:40 PM, Laeeth Isharc wrote:
> > Have you thought about writing up your experience with writing fast json?  
> > A bit
> > like Walter's Dr Dobbs's article on wielding a profiler to speed up dmd.
> 
> Yes, Marco, please. This would make an awesome article, and we need articles 
> like that!
> 
> You've already got this:
> 
>  https://github.com/kostya/benchmarks/pull/46#issuecomment-147932489
> 
> so most of it is already written.

There is at least one hurdle. I don't have a place to publish
articles, no personal blog or site I contribute articles to
and I don't feel like creating a one-shot one right now. :)

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[bachmeier via Digitalmars-d-announce]

On Thursday, 22 October 2015 at 20:54:01 UTC, Marco Leise wrote:

Am Thu, 22 Oct 2015 06:10:56 -0700
schrieb Walter Bright :


On 10/21/2015 3:40 PM, Laeeth Isharc wrote:
> Have you thought about writing up your experience with 
> writing fast json?  A bit like Walter's Dr Dobbs's article 
> on wielding a profiler to speed up dmd.


Yes, Marco, please. This would make an awesome article, and we 
need articles like that!


You've already got this:

 
https://github.com/kostya/benchmarks/pull/46#issuecomment-147932489


so most of it is already written.


There is at least one hurdle. I don't have a place to publish 
articles, no personal blog or site I contribute articles to and 
I don't feel like creating a one-shot one right now. :)


That's why we need an official D blog. Perhaps you could publish 
it in TWID.

Re: Fastest JSON parser in the world is a D project

[Joakim via Digitalmars-d-announce]

On Thursday, 22 October 2015 at 20:54:01 UTC, Marco Leise wrote:

Am Thu, 22 Oct 2015 06:10:56 -0700
schrieb Walter Bright :


On 10/21/2015 3:40 PM, Laeeth Isharc wrote:
> Have you thought about writing up your experience with 
> writing fast json?  A bit like Walter's Dr Dobbs's article 
> on wielding a profiler to speed up dmd.


Yes, Marco, please. This would make an awesome article, and we 
need articles like that!


You've already got this:

 
https://github.com/kostya/benchmarks/pull/46#issuecomment-147932489


so most of it is already written.


There is at least one hurdle. I don't have a place to publish 
articles, no personal blog or site I contribute articles to and 
I don't feel like creating a one-shot one right now. :)


The main D forum is as good a place as any.  Just start a thread 
there.

Re: Fastest JSON parser in the world is a D project

[Walter Bright via Digitalmars-d-announce]

On 10/21/2015 1:38 PM, Laeeth Isharc wrote:

On Wednesday, 21 October 2015 at 19:03:56 UTC, Suliman wrote:

Could anybody reddit this benchmark?


done
https://www.reddit.com/r/programming/comments/3pojrz/the_fastest_json_parser_in_the_world/




It's item 9 on the front page of https://news.ycombinator.com/ too!

Link to actual article (don't click on this link, or your upvote will not be 
counted):


https://news.ycombinator.com/item?id=10430951

Re: Fastest JSON parser in the world is a D project

[Walter Bright via Digitalmars-d-announce]

On 10/21/2015 3:40 PM, Laeeth Isharc wrote:

Have you thought about writing up your experience with writing fast json?  A bit
like Walter's Dr Dobbs's article on wielding a profiler to speed up dmd.


Yes, Marco, please. This would make an awesome article, and we need articles 
like that!


You've already got this:

https://github.com/kostya/benchmarks/pull/46#issuecomment-147932489

so most of it is already written.

Re: Fastest JSON parser in the world is a D project

[Jonathan M Davis via Digitalmars-d-announce]

On Wednesday, October 21, 2015 06:36:31 Suliman via Digitalmars-d-announce 
wrote:
> On Monday, 19 October 2015 at 07:48:16 UTC, Sönke Ludwig wrote:
> > Am 16.10.2015 um 18:04 schrieb Marco Leise:
> >> Every value that is read (as opposed to skipped) is validated
> >> according to RFC 7159. That includes UTF-8 validation. Full
> >> validation (i.e. readJSONFile!validateAll(…);) may add up to
> >> 14% overhead here.
> >>
> >
> > Nice! I see you are using bitmasking trickery in multiple
> > places. stdx.data.json is mostly just the plain lexing
> > algorithm, with the exception of whitespace skipping. It was
> > already very encouraging to get those benchmark numbers that
> > way. Good to see that it pays off to go further.
>
> Is there any chance that new json parser can be include in next
> versions of vibed? And what need to including its to Phobos?

It's already available on code.dlang.org:

http://code.dlang.org/packages/std_data_json

For it to get into Phobos, it has to get through the review process and be
voted in. It was put up for formal review two or three months ago, but that
didn't get to the point that it was voted on (I assume that there was more
work that needed to be done on it first; I haven't really read through that
thread though, so I don't know - I was too busy when the review started to
get involved in it). So, whatever needs to be done for it to be ready for a
formal vote needs to be done, and then it can be voted in, but all of that
takes time, so if you want to use it soon, you might as well just grab it
from code.dlang.org - and it will make it so that you're in a better
position to give feedback on it as well so that it will be that much better
if/when it makes it into Phobos.

- Jonathan M Davis

Re: Fastest JSON parser in the world is a D project

[Suliman via Digitalmars-d-announce]

> Nice! I see you are using bitmasking trickery in multiple 
> places. stdx.data.json is mostly just the plain lexing 
> algorithm, with the exception of whitespace skipping. It was 
> already very encouraging to get those benchmark numbers that 
> way. Good to see that it pays off to go further.


Is there any chance that new json parser can be include in 
next versions of vibed? And what need to including its to 
Phobos?


It's already available on code.dlang.org:
http://code.dlang.org/packages/std_data_json



Jonatan, I mean https://github.com/mleise/fast :)

Re: Fastest JSON parser in the world is a D project

[Laeeth Isharc via Digitalmars-d-announce]

On Wednesday, 21 October 2015 at 09:59:09 UTC, Kapps wrote:
On Wednesday, 21 October 2015 at 04:17:19 UTC, Laeeth Isharc 
wrote:


Seems like you now get 2.1 gigbytes/sec sequential read from a 
cheap consumer SSD today...


Not many consumer drives give more than 500-600 MB/s (SATA3 
limit) yet. There are only a couple that I know of that reach 
2000 MB/s, like Samsung's SM951, and they're generally a fair 
bit more expensive than what most consumers tend to buy (but at 
about $1 / GB, still affordable for businesses certainly).


Yes - that's the one I had in mind.  It's not dirt cheap, but at 
GBP280 if you have some money and want speed, the price is hardly 
an important factor.  I should have said consumer grade rather 
than consumer, but anyway you get my point.


That's today, in 2015.  Maybe one can do even better than that by 
striping data, although it sounds like it's not that easy, but 
still.  "The future is here already; just unevenly distributed".



Seems like if you're processing JSON, which is not the most 
difficult task one might reasonably want to be doing, then 
CPU+memory is the bottleneck more than the SSD.  I don't know 
what outlook is for drive speeds (except they probably won't go 
down), but data sets are certainly not shrinking.  So I am 
intrigued by the difference between what people say is typical 
and what seems to be the case, certainly in what I want to do.

Re: Fastest JSON parser in the world is a D project

[Suliman via Digitalmars-d-announce]

Could anybody reddit this benchmark?

Re: Fastest JSON parser in the world is a D project

[Laeeth Isharc via Digitalmars-d-announce]

On Wednesday, 21 October 2015 at 19:03:56 UTC, Suliman wrote:

Could anybody reddit this benchmark?


done
https://www.reddit.com/r/programming/comments/3pojrz/the_fastest_json_parser_in_the_world/

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Wed, 21 Oct 2015 04:17:16 +
schrieb Laeeth Isharc :

> Very impressive.
> 
> Is this not quite interesting ?  Such a basic web back end 
> operation, and yet it's a very different picture from those who 
> say that one is I/O or network bound.  I already have JSON files 
> of a couple of gig, and they're only going to be bigger over 
> time, and this is a more generally interesting question.
> 
> Seems like you now get 2.1 gigbytes/sec sequential read from a 
> cheap consumer SSD today...

You have this huge amount of Reddit API JSON, right?
I wonder if your processing could benefit from the fast
skipping routines or even reading it as "trusted JSON".

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Wed, 21 Oct 2015 17:00:39 +
schrieb Suliman :

> >> > Nice! I see you are using bitmasking trickery in multiple 
> >> > places. stdx.data.json is mostly just the plain lexing 
> >> > algorithm, with the exception of whitespace skipping. It was 
> >> > already very encouraging to get those benchmark numbers that 
> >> > way. Good to see that it pays off to go further.
> >>
> >> Is there any chance that new json parser can be include in 
> >> next versions of vibed? And what need to including its to 
> >> Phobos?
> >
> > It's already available on code.dlang.org:
> > http://code.dlang.org/packages/std_data_json
> 
> 
> Jonatan, I mean https://github.com/mleise/fast :)

That's nice, but it has a different license and I don't think
Phobos devs would be happy to see all the inline assembly I
used and duplicate functionality like the number parsing and
UTF-8 validation and missing range support.

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Laeeth Isharc via Digitalmars-d-announce]

On Wednesday, 21 October 2015 at 22:24:30 UTC, Marco Leise wrote:

Am Wed, 21 Oct 2015 04:17:16 +
schrieb Laeeth Isharc :


Very impressive.

Is this not quite interesting ?  Such a basic web back end 
operation, and yet it's a very different picture from those 
who say that one is I/O or network bound.  I already have JSON 
files of a couple of gig, and they're only going to be bigger 
over time, and this is a more generally interesting question.


Seems like you now get 2.1 gigbytes/sec sequential read from a 
cheap consumer SSD today...


You have this huge amount of Reddit API JSON, right?
I wonder if your processing could benefit from the fast
skipping routines or even reading it as "trusted JSON".


The couple of gig were just Quandl metadata for one provider, but 
you're right I have that Reddit data too.  And that's just a 
beginning.  What some have been doing for a while, I'm beginning 
to do now, and many others will be doing in the next few years - 
just as soon as they have finished having meetings about what to 
do...  I don't suppose they'll be using python, at least not for 
long.


I am sure it could benefit - I kind of need to get some other 
parts going first.  (For once it truly is a case of Knuth's 97%). 
 But I'll be coming back to look at best way, for json, but text 
files more generally.


Have you thought about writing up your experience with writing 
fast json?  A bit like Walter's Dr Dobbs's article on wielding a 
profiler to speed up dmd.


And actually if you have time, would you mind dropping me an 
email?  laeeth at


kaledicassociates.com

Thanks.


Laeeth.

Re: Fastest JSON parser in the world is a D project

[Andrei Alexandrescu via Digitalmars-d-announce]

On 10/21/2015 04:38 PM, Laeeth Isharc wrote:

On Wednesday, 21 October 2015 at 19:03:56 UTC, Suliman wrote:

Could anybody reddit this benchmark?


done
https://www.reddit.com/r/programming/comments/3pojrz/the_fastest_json_parser_in_the_world/


Getting good press. Congratulations! -- Andrei

Re: Fastest JSON parser in the world is a D project

[Suliman via Digitalmars-d-announce]

On Monday, 19 October 2015 at 07:48:16 UTC, Sönke Ludwig wrote:

Am 16.10.2015 um 18:04 schrieb Marco Leise:

Every value that is read (as opposed to skipped) is validated
according to RFC 7159. That includes UTF-8 validation. Full
validation (i.e. readJSONFile!validateAll(…);) may add up to
14% overhead here.



Nice! I see you are using bitmasking trickery in multiple 
places. stdx.data.json is mostly just the plain lexing 
algorithm, with the exception of whitespace skipping. It was 
already very encouraging to get those benchmark numbers that 
way. Good to see that it pays off to go further.


Is there any chance that new json parser can be include in next 
versions of vibed? And what need to including its to Phobos?

Re: Fastest JSON parser in the world is a D project

[Kapps via Digitalmars-d-announce]

On Wednesday, 21 October 2015 at 04:17:19 UTC, Laeeth Isharc 
wrote:


Seems like you now get 2.1 gigbytes/sec sequential read from a 
cheap consumer SSD today...


Not many consumer drives give more than 500-600 MB/s (SATA3 
limit) yet. There are only a couple that I know of that reach 
2000 MB/s, like Samsung's SM951, and they're generally a fair bit 
more expensive than what most consumers tend to buy (but at about 
$1 / GB, still affordable for businesses certainly).

Re: Fastest JSON parser in the world is a D project

[Laeeth Isharc via Digitalmars-d-announce]

On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote:

The test is pretty simple: Parse a JSON object, containing an 
array of 1_000_000 3D coordinates in the range [0..1) and 
average them.


The performance of std.json in parsing those was horrible still 
in the DMD 2.066 days*:


DMD : 41.44s,  934.9Mb
Gdc : 29.64s,  929.7Mb
Python  : 12.30s, 1410.2Mb
Ruby: 13.80s, 2101.2Mb

Then with 2.067 std.json got a major 3x speed improvement and 
rivaled the popular dynamic languages Ruby and Python:


DMD : 13.02s, 1324.2Mb

In the mean time several other D JSON libraries appeared with 
varying focus on performance or API:


Medea : 56.75s, 1753.6Mb  (GDC)
libdjson  : 24.47s, 1060.7Mb  (GDC)
stdx.data.json:  2.76s,  207.1Mb  (LDC)

Yep, that's right. stdx.data.json's pull parser finally beats 
the dynamic languages with native efficiency. (I used the 
default options here that provide you with an Exception and 
line number on errors.)


A few days ago I decided to get some practical use out of my 
pet project 'fast' by implementing a JSON parser myself, that 
could rival even the by then fastest JSON parser, RapidJSON. 
The result can be seen in the benchmark results right now:


https://github.com/kostya/benchmarks#json

fast:  0.34s, 226.7Mb (GDC)
RapidJSON: 0.79s, 687.1Mb (GCC)

(* Timings from my computer, Haswell CPU, Linux


Very impressive.

Is this not quite interesting ?  Such a basic web back end 
operation, and yet it's a very different picture from those who 
say that one is I/O or network bound.  I already have JSON files 
of a couple of gig, and they're only going to be bigger over 
time, and this is a more generally interesting question.


Seems like you now get 2.1 gigbytes/sec sequential read from a 
cheap consumer SSD today...

Re: Fastest JSON parser in the world is a D project

[Ola Fosheim Grøstad via Digitalmars-d-announce]

On Saturday, 17 October 2015 at 16:27:08 UTC, Sean Kelly wrote:
Oh absolutely. My issue with the benchmark is just that it 
claims to be a JSON parser benchmark but the bulk of CPU time 
is actually spent parsing floats.


Well, most of such language-comparison benchmarks are just for 
fun/marketing. In the real world big JSON files would be 
compressed and most likely retrieved over a network connection 
(like a blob from a database). Pull-parsing of mmap'ed memory is 
a rather unusual scenario for JSON.

Re: Fastest JSON parser in the world is a D project

[Sönke Ludwig via Digitalmars-d-announce]

Am 16.10.2015 um 18:04 schrieb Marco Leise:

Every value that is read (as opposed to skipped) is validated
according to RFC 7159. That includes UTF-8 validation. Full
validation (i.e. readJSONFile!validateAll(…);) may add up to
14% overhead here.



Nice! I see you are using bitmasking trickery in multiple places. 
stdx.data.json is mostly just the plain lexing algorithm, with the 
exception of whitespace skipping. It was already very encouraging to get 
those benchmark numbers that way. Good to see that it pays off to go 
further.

Re: Fastest JSON parser in the world is a D project

[rsw0x via Digitalmars-d-announce]

On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote:
JSON parsing in D has come a long way, especially when you look 
at it from the efficiency angle as a popular benchmark does 
that has been forked by well known D contributers like Martin 
Nowak or Sönke Ludwig.


[...]


Slightly OT:
You have a std.simd file in your repo, was this written by you or 
is there a current std.simd proposal that I'm unaware of?

Re: Fastest JSON parser in the world is a D project

[Brad Anderson via Digitalmars-d-announce]

On Saturday, 17 October 2015 at 09:35:47 UTC, Sönke Ludwig wrote:

Am 17.10.2015 um 13:16 schrieb Marco Leise:

Am Sat, 17 Oct 2015 09:27:46 +0200
schrieb Sönke Ludwig :
Okay, I obviously misread that as a once familiar issue. 
Maybe it indeed
makes sense to add a "JavaScript" quirks mode that behaves 
exactly like

a JavaScript interpreter would.


Ok, but remember: https://www.youtube.com/watch?v=20BySC_6HyY
And then think again. :D



What about just naming it SerializationMode.WAT?


At the very least that needs to be an undocumented alias easter 
egg. :)

Re: Fastest JSON parser in the world is a D project

[Martin Nowak via Digitalmars-d-announce]

On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise wrote:

  - Data size limited by available contiguous virtual memory


Mmaping files for sequential reading is a very debatable choice, 
b/c the common use case is to read a file once. You should at 
least compare the numbers w/ drop_caches between each run.

https://github.com/mleise/fast/blob/69923d5a69f67c21a37e5e2469fc34d60c9ec3e1/source/fast/json.d#L1441

Re: Fastest JSON parser in the world is a D project

[Ola Fosheim Grøstad via Digitalmars-d-announce]

On Saturday, 17 October 2015 at 08:20:33 UTC, Daniel N wrote:
On Saturday, 17 October 2015 at 08:07:57 UTC, Martin Nowak 
wrote:
On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise 
wrote:

  - Data size limited by available contiguous virtual memory


Mmaping files for sequential reading is a very debatable 
choice, b/c the common use case is to read a file once. You 
should at least compare the numbers w/ drop_caches between 
each run.




It's a sensible choice together with appropriate madvise().


Mmap is very expensive, as it affects all cores, you need a 
realistic multithreaded aync benchmark on smaller files to see 
the effect.

Re: Fastest JSON parser in the world is a D project

[Daniel N via Digitalmars-d-announce]

On Saturday, 17 October 2015 at 08:07:57 UTC, Martin Nowak wrote:
On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise 
wrote:

  - Data size limited by available contiguous virtual memory


Mmaping files for sequential reading is a very debatable 
choice, b/c the common use case is to read a file once. You 
should at least compare the numbers w/ drop_caches between each 
run.




It's a sensible choice together with appropriate madvise().

Re: Fastest JSON parser in the world is a D project

[Sönke Ludwig via Digitalmars-d-announce]

Am 17.10.2015 um 13:16 schrieb Marco Leise:

Am Sat, 17 Oct 2015 09:27:46 +0200
schrieb Sönke Ludwig :

Okay, I obviously misread that as a once familiar issue. Maybe it indeed
makes sense to add a "JavaScript" quirks mode that behaves exactly like
a JavaScript interpreter would.


Ok, but remember: https://www.youtube.com/watch?v=20BySC_6HyY
And then think again. :D



What about just naming it SerializationMode.WAT?

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Sat, 17 Oct 2015 16:27:06 +
schrieb Sean Kelly :

> On Saturday, 17 October 2015 at 16:14:01 UTC, Andrei Alexandrescu 
> wrote:
> > On 10/17/15 6:43 PM, Sean Kelly wrote:
> >> If this is the benchmark I'm remembering, the bulk of the time 
> >> is spent
> >> parsing the floating point numbers. So it isn't a test of JSON 
> >> parsing
> >> in general so much as the speed of scanf.
> >
> > In many cases the use of scanf can be replaced with drastically 
> > faster methods, as I discuss in my talks on optimization 
> > (including Brasov recently). I hope they'll release the videos 
> > soon. -- Andrei
> 
> Oh absolutely. My issue with the benchmark is just that it claims 
> to be a JSON parser benchmark but the bulk of CPU time is 
> actually spent parsing floats. I'm on my phone though so perhaps 
> this is a different benchmark--I can't easily check. The one I 
> recall came up a year or so ago and was discussed on D.general.

1/4 to 1/3 of the time is spent parsing numbers in highly
optimized code. You see that in a profiler the number parsing
shows up on top, but the benchmark also exercises the
structural parsing a lot. It is not a very broad benchmark
though, lacking serialization, UTF-8 decoding, validation of
results etc. I believe the author didn't realize how over time
it became the go-to performance test. The author of RapidJSON
has a very in-depth benchmark suite, but it would be a bit of
work to get something non-C++ integrated:
https://github.com/miloyip/nativejson-benchmark
It includes conformance tests as well.

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Ola Fosheim Grøstad via Digitalmars-d-announce]

On Saturday, 17 October 2015 at 13:09:45 UTC, Marco Leise wrote:

Am Sat, 17 Oct 2015 11:12:08 +
schrieb Ola Fosheim Grøstad
:

[…] you could try to construct a simple benchmark where you 
iterate over memory (M*cache 3 size) using a "realistic" 
pattern like brownian motion in N threads and also 
repeatedly/concurrently load JSON code for different file 
sizes so that the CPUs page table mechanisms are stressed by 
mmap, cache misses and (possibly) page faults.


O.O   Are you kidding me? Just give me the correct value
already.


:-P

Re: Fastest JSON parser in the world is a D project

[Sean Kelly via Digitalmars-d-announce]

If this is the benchmark I'm remembering, the bulk of the time is 
spent parsing the floating point numbers. So it isn't a test of 
JSON parsing in general so much as the speed of scanf.

Re: Fastest JSON parser in the world is a D project

[Andrei Alexandrescu via Digitalmars-d-announce]

On 10/17/15 6:43 PM, Sean Kelly wrote:

If this is the benchmark I'm remembering, the bulk of the time is spent
parsing the floating point numbers. So it isn't a test of JSON parsing
in general so much as the speed of scanf.


In many cases the use of scanf can be replaced with drastically faster 
methods, as I discuss in my talks on optimization (including Brasov 
recently). I hope they'll release the videos soon. -- Andrei

Re: Fastest JSON parser in the world is a D project

[Sean Kelly via Digitalmars-d-announce]

On Saturday, 17 October 2015 at 16:14:01 UTC, Andrei Alexandrescu 
wrote:

On 10/17/15 6:43 PM, Sean Kelly wrote:
If this is the benchmark I'm remembering, the bulk of the time 
is spent
parsing the floating point numbers. So it isn't a test of JSON 
parsing

in general so much as the speed of scanf.


In many cases the use of scanf can be replaced with drastically 
faster methods, as I discuss in my talks on optimization 
(including Brasov recently). I hope they'll release the videos 
soon. -- Andrei


Oh absolutely. My issue with the benchmark is just that it claims 
to be a JSON parser benchmark but the bulk of CPU time is 
actually spent parsing floats. I'm on my phone though so perhaps 
this is a different benchmark--I can't easily check. The one I 
recall came up a year or so ago and was discussed on D.general.

Re: Fastest JSON parser in the world is a D project

[Piotrek via Digitalmars-d-announce]

On Friday, 16 October 2015 at 10:08:06 UTC, Andrei Alexandrescu 
wrote:

On 10/15/15 10:40 PM, Jacob Carlborg wrote:

On 2015-10-15 14:51, Johannes Pfau wrote:

Doesn't the GPL force everybody _using_ fast.json to also use 
the GPL

license?


Yes, it does have that enforcement.


Then we'd need to ask Marco if he's willing to relicense the 
code with Boost. -- Andrei


I've just crossed my fingers.

Piotrek

Re: Fastest JSON parser in the world is a D project

[Steven Schveighoffer via Digitalmars-d-announce]

On 10/16/15 2:24 PM, Ola Fosheim Grøstad wrote:

On Friday, 16 October 2015 at 17:38:01 UTC, Steven Schveighoffer wrote:

For example, let's say you have a product that doesn't use JSON. It's
proprietary, and you distribute it under a proprietary license. You
want to include JSON parsing, so you incorporate this GPL'd library.
Then you distribute it under your proprietary license.

Recipient says "Wait, you used fast.json! That means this is now GPL,
I want the source". Then what?


The recipient has no say in this, but the original author can demand
that you either stop distribution or purchase a compatible license.


Exactly my point.


Being able to use GPL on SAAS doesn't satisfy the use case here. This
is a compiled library, it can be used in any piece of software.


My point was that you can use GPLed code in a proprietary service. But
you can also ship propritary code separately that the end user links
with the GPLed code. It is only when you bundle the two that you get a
derived work.


And I don't disagree with your point, just that it was not a correct 
response to "but you definitely can't link any proprietary code aganist 
[sic] it."


-Steve

Re: Fastest JSON parser in the world is a D project

[Steven Schveighoffer via Digitalmars-d-announce]

On 10/16/15 11:56 AM, Ola Fosheim Grøstad wrote:

On Friday, 16 October 2015 at 15:36:26 UTC, Steven Schveighoffer wrote:

You certainly can link with it, and then your code becomes GPL.


No, the code is code. It is an artifact. The GPL is a legal document.
The legal document says what rights you have to the copy you received
and what requirements that follows it. You are allowed to modify it and
do anything you want with it that is covered under fair use. This varies
between jurisdictions.

The license primarily comes into effect  when you _distribute_ or
_publish_, because the legal precedent for putting restrictions on
distribution and publishing is much stronger. And WIPO is much more
clear there.


Right, so what happens when you accidentally distribute it? What license 
is it under?


For example, let's say you have a product that doesn't use JSON. It's 
proprietary, and you distribute it under a proprietary license. You want 
to include JSON parsing, so you incorporate this GPL'd library. Then you 
distribute it under your proprietary license.


Recipient says "Wait, you used fast.json! That means this is now GPL, I 
want the source". Then what?



So, if you build websites for a third party you can use GPL without
redistribution by writing the contract in such a way that the third
party is using your service. Meaning, you run the software. So
circumventing the GPL isn't all that hard if you want to.


Being able to use GPL on SAAS doesn't satisfy the use case here. This is 
a compiled library, it can be used in any piece of software.


-Steve

Re: Fastest JSON parser in the world is a D project

[Ola Fosheim Grøstad via Digitalmars-d-announce]

On Friday, 16 October 2015 at 17:38:01 UTC, Steven Schveighoffer 
wrote:
For example, let's say you have a product that doesn't use 
JSON. It's proprietary, and you distribute it under a 
proprietary license. You want to include JSON parsing, so you 
incorporate this GPL'd library. Then you distribute it under 
your proprietary license.


Recipient says "Wait, you used fast.json! That means this is 
now GPL, I want the source". Then what?


The recipient has no say in this, but the original author can 
demand that you either stop distribution or purchase a compatible 
license.


Being able to use GPL on SAAS doesn't satisfy the use case 
here. This is a compiled library, it can be used in any piece 
of software.


My point was that you can use GPLed code in a proprietary 
service. But you can also ship propritary code separately that 
the end user links with the GPLed code. It is only when you 
bundle the two that you get a derived work.

Re: Fastest JSON parser in the world is a D project

[Ola Fosheim Grøstad via Digitalmars-d-announce]

On Friday, 16 October 2015 at 18:53:39 UTC, Steven Schveighoffer 
wrote:
And I don't disagree with your point, just that it was not a 
correct response to "but you definitely can't link any 
proprietary code aganist [sic] it."


That I don't understand. You can indeed build your executable 
from a mix of proprietary third party libraries and GPL code, 
that means you definetively can link. You cannot distribute it 
together to another third party, but your employer can use it and 
run a service with it.


People attribute way too many limitations to GPL codebases. For 
many organizations the GPL would be perfectly ok for their 
software stack.

Re: Fastest JSON parser in the world is a D project

[Jacob Carlborg via Digitalmars-d-announce]

On 2015-10-16 00:14, Jonathan M Davis via Digitalmars-d-announce wrote:


I thought that http://code.dlang.org/packages/std_data_json was the json
implementation we were looking at adding to Phobos. Or did that fall
through? I haven't paid much attention to the discussion on that, though I
have used it in one of my own projects.


Yes, that was the plan. But if a better alternative shows up, should we 
look at that as well?


--
/Jacob Carlborg

Re: Fastest JSON parser in the world is a D project

[Jacob Carlborg via Digitalmars-d-announce]

On 2015-10-16 00:12, Jonathan M Davis via Digitalmars-d-announce wrote:


I think that you might be able to link code with various other compatible,
open source licenses against it, but you definitely can't link any
proprietary code aganist it. GPL really makes more sense for programs than
for libraries for precisely that reason. And most D libraries are likely to
be Boost licensed, since that's the license used by Phobos and generally
favored by the D community. There's nothing wrong with releasing a library
under the GPL if you really want to, but it seriously limits its usefulness.


Yes, that's correct. It would be fine if everything used GPL, but that's 
not the world we live in. Which makes the license not very practical.


--
/Jacob Carlborg

Re: Fastest JSON parser in the world is a D project

[Andrei Alexandrescu via Digitalmars-d-announce]

On 10/15/15 10:40 PM, Jacob Carlborg wrote:

On 2015-10-15 14:51, Johannes Pfau wrote:


Doesn't the GPL force everybody _using_ fast.json to also use the GPL
license?


Yes, it does have that enforcement.


Then we'd need to ask Marco if he's willing to relicense the code with 
Boost. -- Andrei

Re: Fastest JSON parser in the world is a D project

[Ola Fosheim Grøstad via Digitalmars-d-announce]

On Thursday, 15 October 2015 at 22:13:07 UTC, Jonathan M Davis 
wrote:
On Thursday, October 15, 2015 14:51:58 Johannes Pfau via 
Digitalmars-d-announce wrote:
BTW: Is there a reason why the code is GPL licensed? I 
understand that people might want to use more restrictive 
licenses, but isn't LGPL a better replacement for GPL when 
writing library code? Doesn't the GPL force everybody _using_ 
fast.json to also use the GPL license?


See: http://stackoverflow.com/a/10179181/471401


I think that you might be able to link code with various other 
compatible, open source licenses against it, but you definitely 
can't link any proprietary code aganist it.


Yes, you can. GPL only affects distribution of executables to 
third party, it doesn't affect services. Maybe you are thinking 
of AGPL, which also affects services. But even AGPL allows 
internal usage.

Re: Fastest JSON parser in the world is a D project

[Per Nordlöw via Digitalmars-d-announce]

On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote:

https://github.com/kostya/benchmarks#json


Does fast.json use any non-standard memory allocation patterns or 
plain simple GC-usage?

Re: Fastest JSON parser in the world is a D project

[Steven Schveighoffer via Digitalmars-d-announce]

On 10/16/15 6:20 AM, Ola Fosheim Grøstad wrote:

On Thursday, 15 October 2015 at 22:13:07 UTC, Jonathan M Davis wrote:

On Thursday, October 15, 2015 14:51:58 Johannes Pfau via
Digitalmars-d-announce wrote:

BTW: Is there a reason why the code is GPL licensed? I understand
that people might want to use more restrictive licenses, but isn't
LGPL a better replacement for GPL when writing library code? Doesn't
the GPL force everybody _using_ fast.json to also use the GPL license?

See: http://stackoverflow.com/a/10179181/471401


I think that you might be able to link code with various other
compatible, open source licenses against it, but you definitely can't
link any proprietary code aganist it.


Yes, you can. GPL only affects distribution of executables to third
party, it doesn't affect services. Maybe you are thinking of AGPL, which
also affects services. But even AGPL allows internal usage.



No, you cannot link against GPL library without making your code also 
GPL. "Services" I don't think have anything to do with this, we are 
talking about binary linking.


-Steve

Re: Fastest JSON parser in the world is a D project

[Mike Parker via Digitalmars-d-announce]

On Friday, 16 October 2015 at 12:53:09 UTC, Steven Schveighoffer 
wrote:


Yes, you can. GPL only affects distribution of executables to 
third
party, it doesn't affect services. Maybe you are thinking of 
AGPL, which

also affects services. But even AGPL allows internal usage.



No, you cannot link against GPL library without making your 
code also GPL. "Services" I don't think have anything to do 
with this, we are talking about binary linking.




There was something called the "server loophole." The language of 
GPLv2 only requires source to be distributed if the binaries are 
distributed. The Affero GPL was created to close the loophole, 
requiring source to be made available even if the binaries aren't 
distributed. IIRC, GPLv3 requires the same.

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Thu, 15 Oct 2015 18:46:12 +0200
schrieb Sönke Ludwig :

> Am 14.10.2015 um 09:01 schrieb Marco Leise:
> > […]
> > stdx.data.json:  2.76s,  207.1Mb  (LDC)
> >
> > Yep, that's right. stdx.data.json's pull parser finally beats
> > the dynamic languages with native efficiency. (I used the
> > default options here that provide you with an Exception and
> > line number on errors.)
> 
>  From when are the numbers for stdx.data.json? The latest results for 
> the pull parser that I know of were faster than RapidJson:
> http://forum.dlang.org/post/wlczkjcawyteowjbb...@forum.dlang.org

You know, I'm not surprised at the "D new lazy Ldc" result,
which is in the ball park figure of what I measured without
exceptions & line-numbers, but the Rapid C++ result seems way
off compared to kostya's listing. Or maybe that Core i7 doesn't
work well with RapidJSON.

I used your fork of the benchmark, made some modifications
like adding taggedalgebraic and what else was needed to make
it compile with vanilla ldc2 0.16.0. Then I removed the flags
that disable exceptions and line numbers. Compilation options
are the same as for the existing gdc and ldc2 entries. I did
not add " -partial-inliner -boundscheck=off -singleobj ".

> Judging by the relative numbers, your "fast" result is still a bit 
> faster, but if that doesn't validate, it's hard to make an objective 
> comparison.

Every value that is read (as opposed to skipped) is validated
according to RFC 7159. That includes UTF-8 validation. Full
validation (i.e. readJSONFile!validateAll(…);) may add up to
14% overhead here.

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Thu, 15 Oct 2015 18:17:07 +0200
schrieb Sönke Ludwig :

> Am 15.10.2015 um 13:06 schrieb Rory McGuire via Digitalmars-d-announce:
> > In browser JSON.serialize is the usual way to serialize JSON values.
> > The problem is that on D side if one does deserialization of an object
> > or struct. If the types inside the JSON don't match exactly then vibe
> > freaks out.
> 
> For float and double fields, the serialization code should actually 
> accept both, floating point and integer numbers:
> 
> https://github.com/rejectedsoftware/vibe.d/blob/2fffd94d8516cd6f81c75d45a54c655626d36c6b/source/vibe/data/json.d#L1603
> https://github.com/rejectedsoftware/vibe.d/blob/2fffd94d8516cd6f81c75d45a54c655626d36c6b/source/vibe/data/json.d#L1804
> 
> Do you have a test case for your error?
 
Well it is not an error. Rory originally wrote about
conversions between "1" and 1 happening on the browser side.
That would mean adding a quirks mode to any well-behaving JSON
parser. In this case: "read numbers as strings". Hence I was
asking if the data on the client could be fixed, e.g. the json
number be turned into a string first before serialization.

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Nick Sabalausky via Digitalmars-d-announce]

On 10/16/2015 08:53 AM, Steven Schveighoffer wrote:

On 10/16/15 6:20 AM, Ola Fosheim Grøstad wrote:

On Thursday, 15 October 2015 at 22:13:07 UTC, Jonathan M Davis wrote:

On Thursday, October 15, 2015 14:51:58 Johannes Pfau via
Digitalmars-d-announce wrote:

BTW: Is there a reason why the code is GPL licensed? I understand
that people might want to use more restrictive licenses, but isn't
LGPL a better replacement for GPL when writing library code? Doesn't
the GPL force everybody _using_ fast.json to also use the GPL license?

See: http://stackoverflow.com/a/10179181/471401


I think that you might be able to link code with various other
compatible, open source licenses against it, but you definitely can't
link any proprietary code aganist it.


Yes, you can. GPL only affects distribution of executables to third
party, it doesn't affect services. Maybe you are thinking of AGPL, which
also affects services. But even AGPL allows internal usage.



No, you cannot link against GPL library without making your code also
GPL. "Services" I don't think have anything to do with this, we are
talking about binary linking.



This is the real reason I'm not a huge fan of *GPL. Nobody can 
understand it!

Re: Fastest JSON parser in the world is a D project

[Ola Fosheim Grøstad via Digitalmars-d-announce]

On Friday, 16 October 2015 at 15:07:17 UTC, Ola Fosheim Grøstad 
wrote:
GPL, on the other hand, gives the same right to users of a 
service.


Typo, "AGPL", not "GPL"...

Re: Fastest JSON parser in the world is a D project

[Mike Parker via Digitalmars-d-announce]

On Friday, 16 October 2015 at 14:05:50 UTC, Mike Parker wrote:
On Friday, 16 October 2015 at 12:53:09 UTC, Steven 
Schveighoffer wrote:


Yes, you can. GPL only affects distribution of executables to 
third
party, it doesn't affect services. Maybe you are thinking of 
AGPL, which

also affects services. But even AGPL allows internal usage.



No, you cannot link against GPL library without making your 
code also GPL. "Services" I don't think have anything to do 
with this, we are talking about binary linking.




There was something called the "server loophole." The language 
of GPLv2 only requires source to be distributed if the binaries 
are distributed. The Affero GPL was created to close the 
loophole, requiring source to be made available even if the 
binaries aren't distributed. IIRC, GPLv3 requires the same.


Looks like I got my versions wrong. AGPL is a modified GPLv3.

http://www.gnu.org/licenses/why-affero-gpl.en.html

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Fri, 16 Oct 2015 11:09:37 +
schrieb Per Nordlöw :

> On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote:
> > https://github.com/kostya/benchmarks#json
> 
> Does fast.json use any non-standard memory allocation patterns or 
> plain simple GC-usage?

Plain GC.  I found no way in D to express something as
"borrowed", but the GC removes the ownership question from the
picture. That said the only thing that I allocate in this
benchmark is the dynamic array of coordinates (1_000_000 * 3 *
double.sizeof), which can be replaced by lazily iterating over
the array to remove all allocations.

Using these lazy techniques for objects too and
calling .borrow() instead of .read!string() (which .idups)
should allow GC free usage. (Well, except for the one in
parseJSON, where I append 16 zero bytes to the JSON string to
make it SSE safe.)

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Ola Fosheim Grøstad via Digitalmars-d-announce]

On Friday, 16 October 2015 at 15:36:26 UTC, Steven Schveighoffer 
wrote:

You certainly can link with it, and then your code becomes GPL.


No, the code is code. It is an artifact. The GPL is a legal 
document. The legal document says what rights you have to the 
copy you received and what requirements that follows it. You are 
allowed to modify it and do anything you want with it that is 
covered under fair use. This varies between jurisdictions.


The license primarily comes into effect  when you _distribute_ or 
_publish_, because the legal precedent for putting restrictions 
on distribution and publishing is much stronger. And WIPO is much 
more clear there.


So, if you build websites for a third party you can use GPL 
without redistribution by writing the contract in such a way that 
the third party is using your service. Meaning, you run the 
software. So circumventing the GPL isn't all that hard if you 
want to.


The AGPL also affects publishing as a service, so it makes such 
arrangements much more difficult.

Re: Fastest JSON parser in the world is a D project

[Ola Fosheim Grøstad via Digitalmars-d-announce]

On Friday, 16 October 2015 at 14:09:13 UTC, Nick Sabalausky wrote:
This is the real reason I'm not a huge fan of *GPL. Nobody can 
understand it!


It is really simple!! The basic idea is that people shouldn't 
have to reverse engineer software they use in order to fix 
it/modify it, so when you receive software you should get the 
right to the means to modify it (the source code).


In addition GPL also gives you the right to distribute copies (if 
you want to) so that you can let others enjoy your improved 
version of the program.


It doesn't give the public the right to demand source code to be 
made available, only owners of legally obtained copies get the 
right to demand the full source to be available for them.


It also does not forbid linking against anything, it requires the 
copyright holder to grant rights to the receiver of the copy 
(access to source code and making copies to distribute under the 
same terms).


As long as you keep your modifications/derived works for 
yourself, the only party that has been granted GPL for the 
derived work is yourself. One dilemma here is that a company with 
a million employees is treated like a single entity legally. So 
big companies can embrace the GPL freely for internal use and 
services without the redistribution GPL clauses coming into 
effect, whereas smaller companies that exchange software between 
them cannot restrict redistribution...

Re: Fastest JSON parser in the world is a D project

[Rory McGuire via Digitalmars-d-announce]

In browser JSON.serialize is the usual way to serialize JSON values.
The problem is that on D side if one does deserialization of an object or
struct. If the types inside the JSON don't match exactly then vibe freaks
out.

Another problem with most D JSON implementations is that they don't support
proper JSON, e.g. outputting nan as though it was a valid value etc...
browsers don't like that stuff.

For me the best D JSON implementation at the moment is actually jsvar by
Adam and its not even a JSON parser, it just has that as a needed feature.
It feels slow though I haven't benchmarked, but if I run it over a couple
of Gigs of data(Paged by 1000) it takes a long while.



On Thu, Oct 15, 2015 at 3:42 AM, Marco Leise via Digitalmars-d-announce <
digitalmars-d-announce@puremagic.com> wrote:

> Am Wed, 14 Oct 2015 10:22:37 +0200
> schrieb Rory McGuire via Digitalmars-d-announce
> :
>
> > Does this version handle real world JSON?
> >
> > I've keep getting problems with vibe and JSON because web browsers will
> > automatically make a "1" into a 1 which then causes exceptions in vibe.
> >
> > Does yours do lossless conversions automatically?
>
> No I don't read numbers as strings. Could the client
> JavaScript be fixed? I fail to see why the conversion would
> happen automatically when the code could explicitly check for
> strings before doing math with the value "1". What do I miss?
>
> --
> Marco
>
>

Re: Fastest JSON parser in the world is a D project

[Gary Willoughby via Digitalmars-d-announce]

On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote:

fast:  0.34s, 226.7Mb (GDC)
RapidJSON: 0.79s, 687.1Mb (GCC)

(* Timings from my computer, Haswell CPU, Linux amd64.)


Where's the code?

Re: Fastest JSON parser in the world is a D project

[Daniel Kozak via Digitalmars-d-announce]

Daniel Kozak via Digitalmars-d-announce píše v Čt 15. 10. 2015 v 11:07
+0200:
> 
> 
> Gary Willoughby via Digitalmars-d-announce  remagic.com> napsal Čt, říj 15, 2015 v 10∶08 :
> > On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote:
> > fast:      0.34s, 226.7Mb (GDC)
> > RapidJSON: 0.79s, 687.1Mb (GCC)
> > 
> > (* Timings from my computer, Haswell CPU, Linux amd64.)
> > 
> > Where's the code?
> code.dlang.org

https://github.com/mleise/fast

Re: Fastest JSON parser in the world is a D project

[Jack Stouffer via Digitalmars-d-announce]

On Thursday, 15 October 2015 at 12:51:58 UTC, Johannes Pfau wrote:
BTW: Is there a reason why the code is GPL licensed? I 
understand that people might want to use more restrictive 
licenses, but isn't LGPL a better replacement for GPL when 
writing library code? Doesn't the GPL force everybody _using_ 
fast.json to also use the GPL license?


See: http://stackoverflow.com/a/10179181/471401


It also precludes any of this code being used in Phobos :/

Re: Fastest JSON parser in the world is a D project

[Sönke Ludwig via Digitalmars-d-announce]

Am 15.10.2015 um 13:06 schrieb Rory McGuire via Digitalmars-d-announce:

In browser JSON.serialize is the usual way to serialize JSON values.
The problem is that on D side if one does deserialization of an object
or struct. If the types inside the JSON don't match exactly then vibe
freaks out.


For float and double fields, the serialization code should actually 
accept both, floating point and integer numbers:


https://github.com/rejectedsoftware/vibe.d/blob/2fffd94d8516cd6f81c75d45a54c655626d36c6b/source/vibe/data/json.d#L1603
https://github.com/rejectedsoftware/vibe.d/blob/2fffd94d8516cd6f81c75d45a54c655626d36c6b/source/vibe/data/json.d#L1804

Do you have a test case for your error?

Re: Fastest JSON parser in the world is a D project

[Jonathan M Davis via Digitalmars-d-announce]

On Thursday, October 15, 2015 09:40:05 Per Nordlöw via Digitalmars-d-announce 
wrote:
> On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote:
> > fast:  0.34s, 226.7Mb (GDC)
> > RapidJSON: 0.79s, 687.1Mb (GCC)
>
> Why not add this to std.experimental?

I thought that http://code.dlang.org/packages/std_data_json was the json
implementation we were looking at adding to Phobos. Or did that fall
through? I haven't paid much attention to the discussion on that, though I
have used it in one of my own projects.

- Jonathan M Davis

Re: Fastest JSON parser in the world is a D project

[Jack Stouffer via Digitalmars-d-announce]

On Thursday, 15 October 2015 at 19:40:16 UTC, Jacob Carlborg 
wrote:

On 2015-10-15 14:51, Johannes Pfau wrote:

Doesn't the GPL force everybody _using_ fast.json to also use 
the GPL license?


Yes, it does have that enforcement.


Then this is practically useless for the vast majority of 
programmers.

Re: Fastest JSON parser in the world is a D project

[Jonathan M Davis via Digitalmars-d-announce]

On Thursday, October 15, 2015 14:51:58 Johannes Pfau via Digitalmars-d-announce 
wrote:
> BTW: Is there a reason why the code is GPL licensed? I understand that
> people might want to use more restrictive licenses, but isn't LGPL a
> better replacement for GPL when writing library code? Doesn't the GPL
> force everybody _using_ fast.json to also use the GPL license?
>
> See: http://stackoverflow.com/a/10179181/471401

I think that you might be able to link code with various other compatible,
open source licenses against it, but you definitely can't link any
proprietary code aganist it. GPL really makes more sense for programs than
for libraries for precisely that reason. And most D libraries are likely to
be Boost licensed, since that's the license used by Phobos and generally
favored by the D community. There's nothing wrong with releasing a library
under the GPL if you really want to, but it seriously limits its usefulness.

- Jonathan M Davis

Re: Fastest JSON parser in the world is a D project

[wobbles via Digitalmars-d-announce]

On Thursday, 15 October 2015 at 10:34:16 UTC, Andrei Alexandrescu 
wrote:

On 10/15/15 12:40 PM, Per Nordlöw wrote:
On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise 
wrote:

fast:   0.34s, 226.7Mb (GDC)
RapidJSON: 0.79s, 687.1Mb (GCC)


Why not add this to std.experimental?


Sure seems like a good question! At the least a more generic 
generalization (more character and range types etc) should 
start from Marco's core implementation. -- Andrei


Would it not be a better use of effort to attempt to merge the 
efforts here over to Sonkes new stdx.json? I didn't look at 
either codebase, so I dont know how difficult that'll be.

Re: Fastest JSON parser in the world is a D project

[Jacob Carlborg via Digitalmars-d-announce]

On 2015-10-15 14:51, Johannes Pfau wrote:


Doesn't the GPL force everybody _using_ fast.json to also use the GPL license?


Yes, it does have that enforcement.

--
/Jacob Carlborg

Re: Fastest JSON parser in the world is a D project

[Idan Arye via Digitalmars-d-announce]

On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise wrote:
auto json = parseTrustedJSON(`{ "coordinates": [ { "x": 1, 
"y": 2, "z": 3 }, … ] }`);


I assume parseTrustedJSON is not validating? Did you use it in 
the benchmark? And were the competitors non-validating as well?

Re: Fastest JSON parser in the world is a D project

[Per Nordlöw via Digitalmars-d-announce]

On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote:

https://github.com/kostya/benchmarks#json


I can't find fast.json here. Where is it?

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Wed, 14 Oct 2015 07:55:18 +
schrieb Idan Arye :

> On Wednesday, 14 October 2015 at 07:35:49 UTC, Marco Leise wrote:
> > auto json = parseTrustedJSON(`{ "coordinates": [ { "x": 1, 
> > "y": 2, "z": 3 }, … ] }`);
> 
> I assume parseTrustedJSON is not validating? Did you use it in 
> the benchmark? And were the competitors non-validating as well?

That is correct. For the benchmark parseJSONFile was used
though, which validates UTF-8 and JSON in the used portions.
That probably renders your third question superfluous. I
wouldn't know anyways, but am inclined to think they all
validate the entire JSON and some may skip UTF-8 validation,
which is a low cost operation in this ASCII file anyways.

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Wed, 14 Oct 2015 10:22:37 +0200
schrieb Rory McGuire via Digitalmars-d-announce
:

> Does this version handle real world JSON?
> 
> I've keep getting problems with vibe and JSON because web browsers will
> automatically make a "1" into a 1 which then causes exceptions in vibe.
> 
> Does yours do lossless conversions automatically? 

No I don't read numbers as strings. Could the client
JavaScript be fixed? I fail to see why the conversion would
happen automatically when the code could explicitly check for
strings before doing math with the value "1". What do I miss?

-- 
Marco

Re: Fastest JSON parser in the world is a D project

[Marco Leise via Digitalmars-d-announce]

Am Wed, 14 Oct 2015 08:19:52 +
schrieb Per Nordlöw :

> On Wednesday, 14 October 2015 at 07:01:49 UTC, Marco Leise wrote:
> > https://github.com/kostya/benchmarks#json
> 
> I can't find fast.json here. Where is it?

»»» D Gdc Fast  0.34226.7 «««
C++ Rapid   0.79687.1

Granted if he wrote "D fast.json" it would have been easier to
identify.

-- 
Marco