Thanks for the help. I eventually got it working. And I'm working on the next stage of my project.
On Fri, Nov 27, 2009 at 06:31:03PM -0800, Roger Binns wrote > Alternatively use split to make the input a series of files each > less than 4GB in size. Or use a 64 bit host. [...] > If using a file smaller than 4GB fixes the issue for you then please > let me know and I'll add a ticket about large file support too. First, I tried... [d531][waltdnes][~/SQLite] split -l 18000000 dly04.csv [d531][waltdnes][~/SQLite] ll xa* -rw-r--r-- 1 waltdnes users 3780000000 Nov 27 19:49 xaa -rw-r--r-- 1 waltdnes users 3780000000 Nov 27 19:52 xab -rw-r--r-- 1 waltdnes users 3583911240 Nov 27 19:54 xac ...and import failed with the same error. Then I tried... [d531][waltdnes][~/SQLite] split -l 9000000 dly04.csv [d531][waltdnes][~/SQLite] ll xa* -rw-r--r-- 1 waltdnes users 1890000000 Nov 27 20:06 xaa -rw-r--r-- 1 waltdnes users 1890000000 Nov 27 20:07 xab -rw-r--r-- 1 waltdnes users 1890000000 Nov 27 20:09 xac -rw-r--r-- 1 waltdnes users 1890000000 Nov 27 20:10 xad -rw-r--r-- 1 waltdnes users 1890000000 Nov 27 20:12 xae -rw-r--r-- 1 waltdnes users 1693911240 Nov 27 20:13 xaf ...and import worked. So it appears that the maximum csv file is probably around 2 gigabytes. > You can recompile the shell setting the flags (-D or #define): > _LARGE_FILE, _FILE_OFFSET_BITS=64 and _LARGEFILE_SOURCE I'm not a C programmer. And I'd have to repeat the changes each time there's an update in the Gentoo linux ebuild. Even something minor like 3.6.17 ==> 3.6.18. I've got 2 choices... 1) Pass your comments on to the Gentoo bugzilla, asking the Gentoo SQLite ebuild maintainer to put in a patch with your suggested change. 2) If the patching is going to be done upstream (i.e. by the SQLite people themselves) soon, then don't do anything. The Gentoo version will automatically inherit the large file support with the new version. -- Walter Dnes <waltd...@waltdnes.org> _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
