Hey Erick,
Thanks for the reply.
I plan on rebuilding my cluster soon with more nodes so that the index size
(including tlogs) is under 50% of the available disk at a minimum, ideally we
will shoot for under 33% budget permitting. I think I now understand the
problem that managing this resource will solve and I appreciate your (and
Shawn's) feedback.
I would still like to increase the number of transaction logs retained so that
shard recovery (outside of long term failures) is faster than replicating the
entire shard from the leader. I understand that this is an optimization and
not a
solution for replication. If I'm being thick about this call me out :)
Cheers,
Brian
> On Nov 19, 2015, at 11:30, Erick Erickson <erickerick...@gmail.com> wrote:
>
> First, every time you autocommit there _should_ be a new
> tlog created. A hard commit truncates the tlog by design.
>
> My guess (not based on knowing the code) is that
> Real Time Get needs file handle open to the tlog files
> and you'll have a bunch of them. Lots and lots and lots. Thus
> the too many file handles is just waiting out there for you.
>
> However, this entire approach is, IMO, not going to solve
> anything for you. Or rather other problems will come out
> of the woodwork.
>
> To whit: At some point, you _will_ need to have at least as
> much free space on your disk as your current index occupies,
> even without recovery. Background merging of segments can
> effectively do the same thing as an optimize step, which rewrites
> the entire index to new segments before deleting the old
> segments. So far you haven't hit that situation in steady-state,
> but you will.
>
> Simply put, I think you're wasting your time pursuing the tlog
> option. You must have bigger disks or smaller indexes such
> that there is at least as much free disk space at all times as
> your index occupies. In fact if the tlogs are on the same
> drive as your index, the tlog option you're pursuing is making
> the situation _worse_ by making running out of disk space
> during a merge even more likely.
>
> So unless there's a compelling reason you can't use bigger
> disks, IMO you'll waste lots and lots of valuable
> engineering time before... buying bigger disks.
>
> Best,
> Erick
>
> On Thu, Nov 19, 2015 at 6:21 AM, Brian Scholl <bsch...@legendary.com> wrote:
>> I have opted to modify the number and size of transaction logs that I keep
>> to resolve the original issue I described. In so doing I think I have
>> created a new problem, feedback is appreciated.
>>
>> Here are the new updateLog settings:
>>
>> <updateLog>
>> <str name="dir">${solr.ulog.dir:}</str>
>> <int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
>> <int name="numRecordsToKeep">10000000</int>
>> <int name="maxNumLogsToKeep">5760</int>
>> </updateLog>
>>
>> First I want to make sure I understand what these settings do:
>> numRecordsToKeep: per transaction log file keep this number of
>> documents
>> maxNumLogsToKeep: retain this number of transaction log files total
>>
>> During my testing I thought I observed that a new tlog is created every time
>> auto-commit is triggered (every 15 seconds in my case) so I set
>> maxNumLogsToKeep high enough to contain an entire days worth of updates.
>> Knowing that I could potentially need to bulk load some data I set
>> numRecordsToKeep higher than my max throughput per replica for 15 seconds.
>>
>> The problem that I think this has created is I am now running out of file
>> descriptors on the servers. After indexing new documents for a couple hours
>> a some servers (not all) will start logging this error rapidly:
>>
>> 73021439 WARN
>> (qtp1476011703-18-acceptor-0@6d5514d9-ServerConnector@6392e703{HTTP/1.1}{0.0.0.0:8983})
>> [ ] o.e.j.s.ServerConnector
>> java.io.IOException: Too many open files
>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>> at
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>> at
>> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
>> at
>> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> The output of ulimit -n for the user running the solr process is 1024. I am
>> pretty sure I can prevent this error from occurring by increasing the limit
>> on each server but it isn't clear to me how high it should be or if raising
>> the limit will cause new problems.
>>
>> Any advice you could provide in this situation would be awesome!
>>
>> Cheers,
>> Brian
>>
>>
>>
>>> On Oct 27, 2015, at 20:50, Jeff Wartes <jwar...@whitepages.com> wrote:
>>>
>>>
>>> On the face of it, your scenario seems plausible. I can offer two pieces
>>> of info that may or may not help you:
>>>
>>> 1. A write request to Solr will not be acknowledged until an attempt has
>>> been made to write to all relevant replicas. So, B won’t ever be missing
>>> updates that were applied to A, unless communication with B was disrupted
>>> somehow at the time of the update request. You can add a min_rf param to
>>> your write request, in which case the response will tell you how many
>>> replicas received the update, but it’s still up to your indexer client to
>>> decide what to do if that’s less than your replication factor.
>>>
>>> See
>>> https://cwiki.apache.org/confluence/display/solr/Read+and+Write+Side+Fault+
>>> Tolerance for more info.
>>>
>>> 2. There are two forms of replication. The usual thing is for the leader
>>> for each shard to write an update to all replicas before acknowledging the
>>> write itself, as above. If a replica is less than N docs behind the
>>> leader, the leader can replay those docs to the replica from its
>>> transaction log. If a replica is more than N docs behind though, it falls
>>> back to the replication handler recovery mode you mention, and attempts to
>>> re-sync the whole shard from the leader.
>>> The default N for this is 100, which is pretty low for a high-update-rate
>>> index. It can be changed by increasing the size of the transaction log,
>>> (via numRecordsToKeep) but be aware that a large transaction log size can
>>> delay node restart.
>>>
>>> See
>>> https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConf
>>> ig#UpdateHandlersinSolrConfig-TransactionLog for more info.
>>>
>>>
>>> Hope some of that helps, I don’t know a way to say
>>> delete-first-on-recovery.
>>>
>>>
>>>
>>> On 10/27/15, 5:21 PM, "Brian Scholl" <bsch...@legendary.com> wrote:
>>>
>>>> Whoops, in the description of my setup that should say 2 replicas per
>>>> shard. Every server has a replica.
>>>>
>>>>
>>>>> On Oct 27, 2015, at 20:16, Brian Scholl <bsch...@legendary.com> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I am experiencing a failure mode where a replica is unable to recover
>>>>> and it will try to do so forever. In writing this email I want to make
>>>>> sure that I haven't missed anything obvious or missed a configurable
>>>>> option that could help. If something about this looks funny, I would
>>>>> really like to hear from you.
>>>>>
>>>>> Relevant details:
>>>>> - solr 5.3.1
>>>>> - java 1.8
>>>>> - ubuntu linux 14.04 lts
>>>>> - the cluster is composed of 1 SolrCloud collection with 100 shards
>>>>> backed by a 3 node zookeeper ensemble
>>>>> - there are 200 solr servers in the cluster, 1 replica per shard
>>>>> - a shard replica is larger than 50% of the available disk
>>>>> - ~40M docs added per day, total indexing time is 8-10 hours spread
>>>>> over the day
>>>>> - autoCommit is set to 15s
>>>>> - softCommit is not defined
>>>>>
>>>>> I think I have traced the failure to the following set of events but
>>>>> would appreciate feedback:
>>>>>
>>>>> 1. new documents are being indexed
>>>>> 2. the leader of a shard, server A, fails for any reason (java crashes,
>>>>> times out with zookeeper, etc)
>>>>> 3. zookeeper promotes the other replica of the shard, server B, to the
>>>>> leader position and indexing resumes
>>>>> 4. server A comes back online (typically 10s of seconds later) and
>>>>> reports to zookeeper
>>>>> 5. zookeeper tells server A that it is no longer the leader and to sync
>>>>> with server B
>>>>> 6. server A checks with server B but finds that server B's index
>>>>> version is different from its own
>>>>> 7. server A begins replicating a new copy of the index from server B
>>>>> using the (legacy?) replication handler
>>>>> 8. the original index on server A was not deleted so it runs out of
>>>>> disk space mid-replication
>>>>> 9. server A throws an error, deletes the partially replicated index,
>>>>> and then tries to replicate again
>>>>>
>>>>> At this point I think steps 6 => 9 will loop forever
>>>>>
>>>>> If the actual errors from solr.log are useful let me know, not doing
>>>>> that now for brevity since this email is already pretty long. In a
>>>>> nutshell and in order, on server A I can find the error that took it
>>>>> down, the post-recovery instruction from ZK to unregister itself as a
>>>>> leader, the corrupt index error message, and then the (start - whoops,
>>>>> out of disk- stop) loop of the replication messages.
>>>>>
>>>>> I first want to ask if what I described is possible or did I get lost
>>>>> somewhere along the way reading the docs? Is there any reason to think
>>>>> that solr should not do this?
>>>>>
>>>>> If my version of events is feasible I have a few other questions:
>>>>>
>>>>> 1. What happens to the docs that were indexed on server A but never
>>>>> replicated to server B before the failure? Assuming that the replica on
>>>>> server A were to complete the recovery process would those docs appear
>>>>> in the index or are they gone for good?
>>>>>
>>>>> 2. I am guessing that the corrupt replica on server A is not deleted
>>>>> because it is still viable, if server B had a catastrophic failure you
>>>>> could pick up the pieces from server A. If so is this a configurable
>>>>> option somewhere? I'd rather take my chances on server B going down
>>>>> before replication finishes than be stuck in this state and have to
>>>>> manually intervene. Besides, I have disaster recovery backups for
>>>>> exactly this situation.
>>>>>
>>>>> 3. Is there anything I can do to prevent this type of failure? It
>>>>> seems to me that if server B gets even 1 new document as a leader the
>>>>> shard will enter this state. My only thought right now is to try to
>>>>> stop sending documents for indexing the instant a leader goes down but
>>>>> on the surface this solution sounds tough to implement perfectly (and it
>>>>> would have to be perfect).
>>>>>
>>>>> If you got this far thanks for sticking with me.
>>>>>
>>>>> Cheers,
>>>>> Brian
>>>>>
>>>>
>>>
>>
