[
https://issues.apache.org/jira/browse/SOLR-9389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451626#comment-15451626
]
Tim Owen commented on SOLR-9389:
--------------------------------
[~dsmiley] We're now running 6 Solr JVMs per box, as the machines in production
have 6 SSDs installed, so yeah it works out at around 200 Solr cores being
served by each Solr JVM. That seems to run fine, and we've had our staging
environment for another Solr installation with hundreds of cores per JVM for
several years. The reason for many shards is that we do frequent updates and
deletes, and want to keep the Lucene index size below a manageable level e.g.
5GB, to avoid a potentially slow merge that would block writes for too long.
With composite routing, our queries never touch all shards in a collection -
just a few.
The problem still is with SolrCloud and the Overseer/Zookeeper, which become
overloaded with traffic once there's any kind of problem e.g. machine failure,
or worse an entire rack losing power - this causes a flood of overseer queue
events and all the nodes feverishly downloading state.json repeatedly. Happy to
talk to anyone who's working on that problem!
> HDFS Transaction logs stay open for writes which leaks Xceivers
> ---------------------------------------------------------------
>
> Key: SOLR-9389
> URL: https://issues.apache.org/jira/browse/SOLR-9389
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Hadoop Integration, hdfs
> Affects Versions: 6.1, master (7.0)
> Reporter: Tim Owen
> Assignee: Mark Miller
> Fix For: master (7.0), 6.3
>
> Attachments: SOLR-9389.patch
>
>
> The HdfsTransactionLog implementation keeps a Hadoop FSDataOutputStream open
> for its whole lifetime, which consumes two threads on the HDFS data node
> server (dataXceiver and packetresponder) even once the Solr tlog has finished
> being written to.
> This means for a cluster with many indexes on HDFS, the number of Xceivers
> can keep growing and eventually hit the limit of 4096 on the data nodes. It's
> especially likely for indexes that have low write rates, because Solr keeps
> enough tlogs around to contain 100 documents (up to a limit of 10 tlogs).
> There's also the issue that attempting to write to a finished tlog would be a
> major bug, so closing it for writes helps catch that.
> Our cluster during testing had 100+ collections with 100 shards each, spread
> across 8 boxes (each running 4 solr nodes and 1 hdfs data node) and with 3x
> replication for the tlog files, this meant we hit the xceiver limit fairly
> easily and had to use the attached patch to ensure tlogs were closed for
> writes once finished.
> The patch introduces an extra lifecycle state for the tlog, so it can be
> closed for writes and free up the HDFS resources, while still being available
> for reading. I've tried to make it as unobtrusive as I could, but there's
> probably a better way. I have not changed the behaviour of the local disk
> tlog implementation, because it only consumes a file descriptor regardless of
> read or write.
> nb We have decided not to use Solr-on-HDFS now, we're using local disk (for
> various reasons). So I don't have a HDFS cluster to do further testing on
> this, I'm just contributing the patch which worked for us.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
