On Tuesday, 28 February 2017 at 20:48:33 UTC, Petar Kirov
[ZombineDev] wrote:
On Tuesday, 28 February 2017 at 20:27:25 UTC, Alexey H wrote:
Hello, guys!
I'm working on a project that involves parsing of huge JSON
datasets in real-time.
Just an example of what i'm dealing with is here:
https://gist.githubusercontent.com/gdmka/125014058bb7d7f01b867fac56300a61/raw/f0c6b5be5fb01b16dd83f07c577b72f76f72c855/data.json
Can't think of any tools other that D or Go to solve this
problem.
My experience of solving the problem with Go has led me to
Stackoverflow and the community out there seemed too reluctant
to help so i assumed that the language cannot handle such a
set of operations on complex datastructures.
My experience with D was like a charm. Where i have had ~100
lines of Go code i did the same with 12 in with D. But,
nevertheless, i did some profiling (unfortunately on OS X) and
saw much heavier CPU usage with D than Go. Probably because
the Go solution was unpacking all the data strictly to struct.
So, my real question is: can i actually, by any change, get
the description of an underlying struct that the call to
parseJSON generates?
The goers have this thing https://mholt.github.io/json-to-go/
to generate structs from JSON automatically.
Since D easily parses JSON by type inference, i assume it
builds a JSONValue struct which holds all the fields and the
data.
If it is possible, then i can build a similar JSON to D tool
just for the sake of saving people's time and patience.
If you really care about performance, have a look this:
http://forum.dlang.org/post/20151014090114.60780ad6@marco-toshiba
std.json is not tuned for performance, so don't expect good
results from it.
I am not expecting good results from stdlib's json. As for now i
just need a concise way to get 1.2-1.5 MB JSON and dump it into a
struct to perform numeric computations. Since it's not the only
data source i will be parsing, i need a straightforward way to
generate D structs right out of the predefined JSON schema. So i
am willing to sacrifice some speed for convenience at this point.
Fastjson might be good when dealing with trusted input, but this
is not my case.
