On Wed, Apr 2, 2014 at 6:09 PM, Kevin Xu <accol...@gmail.com> wrote: > Yes, I have tried various combinations of WAL/OFF/MEMORY and they do not > appear to affect the insert speed in any meaningful manner (e.g. +/- 0.3s on > 500K inputs). > > [...] Thus, I am trying to see if there is anything I can do to improve on > this.
As Simon already wrote, you seem to have covered all the bases, and your posts indicate you are a sophisticated developer, so notion obvious comes to mind. As DRH suggested, you could try the vtable approach. I'm not sure it will make much difference, given recent discussions on this list about the overhead from all the callbacks SQLite makes into the vtable code, but that's purely speculative on my part. The only way to know it to try it out for yourself. You can try different optimization build settings (-O3?, LTO?, WPO?), and see if that helps any. One thing you might try is to get a baseline for your program w/o SQLite, i.e. do all the parsing, and simply write all your fields to a file as (non-portable) binary, not meant to be re-read. The delta between the runtime of this program (which I suspect you can easily cobble together from your existing code) and your current SQLite-based one, will give you the "overhead" SQLite adds up over plain binary IO. You might find that the overhead is not that big compared to the whole runtime, and parsing differently, avoid std::string, might reap more benefits. Finally, a bit from left field, would be for you to format and write the data directly into the disk format SQLite expects yourself... You'd bypass the binding/VDBE/pager/etc... code from SQLite (i.e what I qualified as the "overhead" above), and provided that "overhead" was large enough, and you'd manage to format the data fast enough, I don't see any other "faster" way to generate an SQLite database file. That's by far the least practical approach, but also the only one that will allow "maximum" performance I believe. It all depends on what you are after. My $0.02. --DD _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
