Thanks for your comments, John, that is very interesting. I'll have to check out your db code.
I'll drop a penny in the bucket with this: http://github.com/alphapapa/org-agenda-ng I spent a few hours trying an alternative approach that uses org-element-parse-buffer to parse each file, then operates on the parse tree it produces. I was hoping it might have a chance to be an improvement, but unfortunately it's not: org-element-parse-buffer is very slow in comparison to the existing agenda code, and it eats up almost all the time spent building the (very primitive) agenda views. So, I think all we need is for someone to write a version of org-element-parse-buffer in C or CL or whatever their favorite language that compiles to a fast binary may be. Then we can read that into Emacs and operate on the resulting tree quickly. Yep, that's ALL we need... ;) Seriously, though, it would be amazing if the new library support in Emacs could be used to interface with a fast Org parser that provided the same output as org-element-parse-buffer. Maybe it could even be added to Emacs core, as the Org file format doesn't change much. All that's way over my head, I'm afraid. Would some kind of bounty work for a project like this? I'd be willing to contribute a few dollars and some testing. :) I also gave a cursory look at profiling org-element-parse-buffer, but I don't think Nicolas left any low-hanging fruit there. But I'm not experienced with profiling code like that, so don't take my word for it. :)
