On 3/27/2017 7:13 PM, santosh sidnal wrote:
> i am facing closed connection issue while doing dataimporter, any solution
> to this> stack trace is as below
>
>
> [3/27/17 8:54:41:399 CDT] 000000b4 OracleDataSto > findMappingClass for :
> Entry
> java.sql.SQLRecoverableException: Closed
> Connection
Does the import appear to work correctly at first, then stop before it's
done with this exception after a few hours? If so, then I think you may
be running into a situation where the merge scheduler has multiple
merges scheduled and stops the incoming indexing thread until the
largest merge is done. If the indexing thread is stopped for long
enough, JDBC will disconnect the database, and when the indexing thread
finally starts back up, the dataimporter finds that it cannot read from
the database any more.
If this is what's happening, then the solution is to allow the merge
scheduler to schedule more merges simultaneously. Here's the
indexConfig that I use in solrconfig.xml:
<indexConfig>
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
<int name="maxMergeAtOnce">35</int>
<int name="segmentsPerTier">35</int>
<int name="maxMergeAtOnceExplicit">105</int>
</mergePolicy>
<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
<int name="maxThreadCount">1</int>
<int name="maxMergeCount">6</int>
</mergeScheduler>
</indexConfig>
You don't need the mergePolicy part of this config to solve this issue.
That is the settings that I use, but this greatly increases the number
of files that can be in an index, which may require OS tuning for max
open files.
The mergeScheduler is the important part, and specifically
maxMergeCount. These settings are for standard spinning-magnetic
disks. I increase the maxMergeCount to 6, so more merges can be
scheduled without halting the indexing thread. The maxThreadCount value
should be set to 1 if you're using standard disks. If you're using SSD,
then you can bump it a little bit, because SSD can easily handle
multiple threads randomly writing to the disk. I don't know what values
are appropriate for SSD, but I would probably start with 2-4.
Thanks,
Shawn
