Any tips for indexing large amounts of data?

[Brendan Grainger]

Hi,

I am creating an index of approx 500K documents. I wrote an indexing  
program using embeded solr: http://wiki.apache.org/solr/EmbeddedSolr  
and am seeing probably a 10 fold increase in indexing speeds. My  
problem is though, that if I try to reindex say 20K docs at a time it  
slows down considerably. I currently batch my updates in lots of 100  
and between batches I close and reopen the "connection" to solr like so:

    private void openConnection(String environment) throws  
ParserConfigurationException, IOException, SAXException {
        System.setProperty("solr.solr.home", SOLR_HOME);
        solrConfig = new SolrConfig("solrconfig.xml");
        solrCore = new SolrCore(SOLR_HOME + "data/" + environment,  
solrConfig, new IndexSchema(solrConfig, "schema.xml"));
        logger.debug("Opened solr connection");
    }

    private void closeConnection() {
        solrCore.close();
        solrCore = null;
        logger.debug("Closed solr connection");
    }

Does anyone have any pointers or see anything obvious I'm doing wrong?

Thanks


PS Sorry if this is posted twice.