On Fri, 2005-11-25 at 09:14, Lars Clausen wrote: > Trying to import a 10GB text file (about 50x10^6 entries) into a single > Derby table, I got the following error: > > ij> connect 'jdbc:derby:cdxdb'; ij> elapsedtime on; ij> CALL > SYSCS_UTIL.SYSCS_IMPORT_DATA ( null, 'CDX', > 'URL,IP,MIMETYPE,LENGTH,ARCFILE,OFFSET', '1,2,4,5,6,7', > '/home/lc/index-backping.cdx', '`', null, null, 1); > ERROR 38000: The exception 'SQL Exception: Exception during creation of > file > /home/lc/projects/webarkivering/scripts/sql/cdxdb/tmp/T1132842374093.tmp > for container' was thrown while evaluating an expression. > ERROR XSDF1: Exception during creation of > file > /home/lc/projects/webarkivering/scripts/sql/cdxdb/tmp/T1132842374093.tmp > for container > ERROR XJ001: Java exception: > '/home/lc/projects/webarkivering/scripts/sql/cdxdb/tmp/T1132842374093.tmp > (Too many open files): java.io.FileNotFoundException'.
It turns out that this happens during index creation. I was able to import the text file and run selects on it, but when I try to create an index: ij> select count(*) from cdx; 1 ----------- 50000000 1 row selected ELAPSED TIME = 320818 milliseconds ij> create index cdxurl on cdx(url); ERROR XSDF1: Exception during creation of file /home/lc/projects/webarkivering/scripts/sql/cdxdb/tmp/T1132927896412.tmp for container ERROR XJ001: Java exception: '/home/lc/projects/webarkivering/scripts/sql/cdxdb/tmp/T1132927896412.tmp (Too many open files): java.io.FileNotFoundException'. ij> Derby creates files in the tmp directory at a rate of about 8 per second. If it doesn't close all of these, it would run out of FDs (ulimit 1024) before long. I would file a bug report, but db.apache.org isn't responding. -Lars
