The other problem is that after some time we get a "Too Many Open Files"
error when autocommit fires.
Have you checked your ulimit settings?
http://wiki.apache.org/lucene-java/LuceneFAQ#head-48921635adf2c968f7936dc07d51dfb40d638b82
ulimit -n <number>.
As mike mentioned, you may also want to use 'single' as the lockType.
In solrconfig set:
<indexDefaults>
...
<lockType>single</lockType>
</indexDefaults>
I could of course switch to using the Solr webapp since we're running in
Tomcat anyway, however I really like the ability to have a single WAR
file that contains everything and also not have to worry about actually
making HTTP requests and the complexity that adds.
This sounds like a good candidate to try solrj:
http://wiki.apache.org/solr/Solrj
This way you write your app independent of how you connect to solr. It
also takes care of the XML parsing for you and lets you work with
objects rather then strings.
ryan