On Wednesday, 21 October 2015 at 22:24:30 UTC, Marco Leise wrote:
Am Wed, 21 Oct 2015 04:17:16 +0000
schrieb Laeeth Isharc <laeeth.nos...@nospam-laeeth.com>:

Very impressive.

Is this not quite interesting ? Such a basic web back end operation, and yet it's a very different picture from those who say that one is I/O or network bound. I already have JSON files of a couple of gig, and they're only going to be bigger over time, and this is a more generally interesting question.

Seems like you now get 2.1 gigbytes/sec sequential read from a cheap consumer SSD today...

You have this huge amount of Reddit API JSON, right?
I wonder if your processing could benefit from the fast
skipping routines or even reading it as "trusted JSON".

The couple of gig were just Quandl metadata for one provider, but you're right I have that Reddit data too. And that's just a beginning. What some have been doing for a while, I'm beginning to do now, and many others will be doing in the next few years - just as soon as they have finished having meetings about what to do... I don't suppose they'll be using python, at least not for long.

I am sure it could benefit - I kind of need to get some other parts going first. (For once it truly is a case of Knuth's 97%). But I'll be coming back to look at best way, for json, but text files more generally.

Have you thought about writing up your experience with writing fast json? A bit like Walter's Dr Dobbs's article on wielding a profiler to speed up dmd.

And actually if you have time, would you mind dropping me an email? laeeth at



Reply via email to