I forgot to mention the ES version that we are running. It is 0.90.3
On Wednesday, March 12, 2014 10:54:46 PM UTC+5:30, Nikhil Singh wrote:
>
> Hi all,
>
> We have a 5 node elasticsearch setup as follows:
>
> - 3 nodes running in data mode
> - 2 nodes running in master only mode
>
> Each of the nodes have the following configuration options set in
> elasticsearch.yml file
>
>> index.number_of_shards: 6
>> index.number_of_replicas: 0
>> index.mapper.dynamic: true
>> action.auto_create_index: true
>> action.disable_delete_all_indices: true
>
>
> What I understand from the above config is that each index will have 6
> shards and there will not be any replica of each of the shards. Also, we
> have the following discovery options set
>
>> discovery.zen.minimum_master_nodes: 1
>> discovery.zen.ping.multicast.enabled: false
>> discovery.zen.ping.unicast.hosts: ["10.101.55.93[9300-9400]",
>> "10.242.22.126[9300-9400]", "10.144.77.94[9300-9400]",
>> "10.116.173.148[9300-9400]", "10.224.42.205[9300-9400]"]
>
>
> We are also running logstash which is indexing documents in this
> elasticsearch setup. Yesterday, I started noticing the following errors in
> the logstash indexer log file
>
>> {:timestamp=>"2014-03-10T11:05:44.453000+0000", :message=>"Failed to
>> index an event, will retry", :exception=>
>> *org.elasticsearch.action.UnavailableShardsException*:
>> [logstash-2014.03.10][1] [1] shardIt, [0] active : Timeout waiting for
>> [1m], request: index
>> {[logstash-2014.03.10][application][uMrD19E4QuKepoBq17ZKQA],
>> source[{"@source":"file://ip-10-118-115-235//mnt/deploy/apache-tomcat/logs/application.log","@tags":["shipped"],"@fields":{"environment":["production"],"service":["application"],"machine":["proxy1"],"timestamp":"03/10/14
>>
>> 06:03:07.292","thread":"http-8080-126","severity":"DEBUG","message":"ources.AccountServicesResource
>>
>> idx=0"},"@timestamp":"2014-03-10T06:03:07.292Z","@source_host":"ip-10-118-115-235","@source_path":"//mnt/deploy/apache-tomcat/logs/application.log","@message":"03/10/14
>>
>> 06:03:07.292 [http-8080-126] DEBUG ources.AccountServicesResource
>> idx=0","@type":"application"}]},
>> :event=>{"@source"=>"file://ip-10-118-115-235//mnt/deploy/apache-tomcat/logs/application.log",
>>
>> "@tags"=>["shipped"], "@fields"=>{"environment"=>["production"],
>> "service"=>["application"], "machine"=>["proxy1"], "timestamp"=>"03/10/14
>> 06:03:07.292", "thread"=>"http-8080-126", "severity"=>"DEBUG",
>> "message"=>"ources.AccountServicesResource idx=0"},
>> "@timestamp"=>"2014-03-10T06:03:07.292Z",
>> "@source_host"=>"ip-10-118-115-235",
>> "@source_path"=>"//mnt/deploy/apache-tomcat/logs/application.log",
>> "@message"=>"03/10/14 06:03:07.292 [http-8080-126] DEBUG
>> ources.AccountServicesResource idx=0", "@type"=>"application"},
>> :level=>:warn}
>
>
> While I was debugging this issue, I noticed that the first IP in the
> *discovery.zen.ping.unicast.hosts
> *was wrong. That IP did not point to any of the 5 nodes in the cluster. I
> realized that the first IP should be of one of the nodes configured as
> master and I changed that IP to the correct one on all of the nodes and
> restarted ES. After that change, I no longer see the above error.
>
> I have a question - considering the first IP was wrong, the cluster would
> have elected the only other node configured as master as the *master*.
> This means that there was atleast one master in the cluster. So, for this
> exception to happen, could the other master's metadata about shards be
> wrong?
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f5f785a1-6b91-4ff6-b9f9-ed3580c6e510%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
