Please see attached files. I got the elasticsearch.log file from
/var/log/elasticsearch on both nodes. Additionally I got graylog.log from
the same location on both nodes. Even though node 2 doesn't have graylog
installed it had a log file for it. Not sure why that is.
Thanks!
Nathan
On Tuesday, August 2, 2016 at 11:10:49 AM UTC-4, Jochen Schalanda wrote:
>
> Hi Nathan,
>
> please post the *complete* log files of your Elasticsearch and Graylog
> nodes.
>
> Cheers,
> Jochen
>
> On Tuesday, 2 August 2016 16:56:58 UTC+2, Nathan Mace wrote:
>>
>> Removing the leading whitespaces didn't help.
>>
>> However in looking through the logs I found this in the primary node's
>> graylog.log file:
>>
>> ConnectTransportException[[ansted-search-01][x.x.x.149:9300]
>> connect_timeout[30s]]; nested: ConnectException[Connection refused:
>> /x.x.x.149:9300];
>> at
>> org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:987)
>> at
>> org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:920)
>> at
>> org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:893)
>> at
>> org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:260)
>> at
>> org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:434)
>> at
>> org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:386)
>> at
>> org.elasticsearch.discovery.zen.ZenDiscovery.access$4800(ZenDiscovery.java:91)
>> at
>> org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1237)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> It was repeated several times. That is it trying to connect to the
>> second node on port 9300 and not being able to. I see in the documentation
>> that 9300 is the default port and I have nothing in either of the ES YML
>> files referencing that port number, so it seems to be all default. If I do
>> a netstat on both hosts they are both listening on port 9200 and 9300. It
>> would seem that it is listening, but only allowing connections to 9300 from
>> localhost? What would I need to change to allow a connect from the other
>> node?
>>
>> Nathan
>>
>> On Tuesday, August 2, 2016 at 10:22:44 AM UTC-4, Jochen Schalanda wrote:
>>>
>>> Hi Nathan,
>>>
>>> I'm not sure how Elasticsearch handles leading whitespace in their
>>> configuration file. I'd recommend making sure that the configuration
>>> settings really start at the beginning of a line.
>>>
>>> Additionally, please post the complete log files of your Elasticsearch
>>> and Graylog nodes.
>>>
>>> Cheers,
>>> Jochen
>>>
>>> On Tuesday, 2 August 2016 16:00:47 UTC+2, Nathan Mace wrote:
>>>>
>>>> Oh good grief! Clearly been staring at this problem to long, I
>>>> completely missed those hash signs.
>>>>
>>>> OK, now ES is happily running on the proper IP addresses. I can access
>>>> it via curl from other hosts. So that's a large improvement. However
>>>> Graylog still only reports 1 node in the web interface. I've attached the
>>>> current versions of the config files (vs copy/paste). Given my tunnel
>>>> vision on the hash signs, this seems like it will be something obvious but
>>>> I can't find it.
>>>>
>>>> Thank you so much for the help!
>>>>
>>>> Nathan
>>>>
>>>> On Tuesday, August 2, 2016 at 9:30:58 AM UTC-4, Jochen Schalanda wrote:
>>>>>
>>>>> Hi Nathan,
>>>>>
>>>>> leading hash signs (the # character) mean that the line is commented
>>>>> out.
>>>>>
>>>>> For example the following line is completely ignored:
>>>>>
>>>>> # discovery.zen.ping.unicast.hosts: ["x.x.x.146", "x.x.x.149"]
>>>>>
>>>>>
>>>>> While this line is "active" and will be obeyed:
>>>>>
>>>>> cluster.name: graylog
>>>>>
>>>>>
>>>>> Maybe you've only copy & pasted your configuration files in a strange
>>>>> way (which is why I would always recommend to send them as attachments),
>>>>> but that's how it looks like.
>>>>>
>>>>> Cheers,
>>>>> Jochen
>>>>>
>>>>> On Tuesday, 2 August 2016 15:23:22 UTC+2, Nathan Mace wrote:
>>>>>>
>>>>>> Thanks Jochen. I will make the changes. However I am very confused
>>>>>> by your comment about the second node having the cluster.name
>>>>>> setting unset. I'm showing that it is set to "graylog" just like the
>>>>>> first
>>>>>> node. I'm not sure at all what you mean.
>>>>>>
>>>>>> Nathan
>>>>>>
>>>>>> On Tuesday, August 2, 2016 at 6:38:45 AM UTC-4, Jochen Schalanda
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Nathan,
>>>>>>>
>>>>>>> check the elasticsearch_network_host setting of your Graylog nodes.
>>>>>>> It should be set to one (and only one!) public IP address of the
>>>>>>> Graylog
>>>>>>> node which can be accessed by all other Elasticsearch nodes in the
>>>>>>> cluster. elasticsearch_discovery_zen_ping_unicast_hosts should be
>>>>>>> a comma-separated list of host/port pairs containing the addresses of
>>>>>>> the
>>>>>>> Elasticsearch nodes, for example:
>>>>>>>
>>>>>>> elasticsearch_discovery_zen_ping_unicast_hosts = x.x.x.146:9300,
>>>>>>> x.x.x.149
>>>>>>>
>>>>>>>
>>>>>>> See
>>>>>>> http://docs.graylog.org/en/2.0/pages/configuration/elasticsearch.html#network-setup
>>>>>>>
>>>>>>> for details.
>>>>>>>
>>>>>>> Additionally, the cluster.name of your second Elasticsearch node is
>>>>>>> unset, which makes it default to "elasticsearch". The logs of that
>>>>>>> Elasticsearch node should show this pretty clearly.
>>>>>>>
>>>>>>> Also take a look at the network.host settings of both your
>>>>>>> Elasticsearch nodes. This setting must be customized to your network
>>>>>>> setup,
>>>>>>> otherwise they'll only bind to the local network interface (i. e.
>>>>>>> 127.0.0.1 or ::1). See
>>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-network.html#common-network-settings
>>>>>>>
>>>>>>> for details.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Jochen
>>>>>>>
>>>>>>> On Monday, 1 August 2016 22:15:32 UTC+2, Nathan Mace wrote:
>>>>>>>>
>>>>>>>> Primary node (MonoDB, Graylog, and ES): IP Address: x.x.x.146
>>>>>>>> Secondary Node (ES Only): IP Address: x.x.x.149
>>>>>>>>
>>>>>>>> Both on the same subnet. Can ping each other.
>>>>>>>> […]
>>>>>>>>
>>>>>>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/a896acb7-4fd8-41a1-9221-a4a924474262%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[2016-08-02 09:19:13,539][INFO ][node ] [Betty Ross Banner]
version[2.3.4], pid[21914], build[e455fd0/2016-06-30T11:24:31Z]
[2016-08-02 09:19:13,539][INFO ][node ] [Betty Ross Banner]
initializing ...
[2016-08-02 09:19:14,081][INFO ][plugins ] [Betty Ross Banner]
modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-08-02 09:19:14,105][INFO ][env ] [Betty Ross Banner]
using [1] data paths, mounts [[/ (rootfs)]], net usable_space [11.9gb], net
total_space [13.8gb], spins? [unknown], types [rootfs]
[2016-08-02 09:19:14,105][INFO ][env ] [Betty Ross Banner]
heap size [1007.3mb], compressed ordinary object pointers [true]
[2016-08-02 09:19:14,105][WARN ][env ] [Betty Ross Banner]
max file descriptors [65535] for elasticsearch process likely too low, consider
increasing to at least [65536]
[2016-08-02 09:19:16,064][INFO ][node ] [Betty Ross Banner]
initialized
[2016-08-02 09:19:16,064][INFO ][node ] [Betty Ross Banner]
starting ...
[2016-08-02 09:19:16,184][INFO ][transport ] [Betty Ross Banner]
publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2016-08-02 09:19:16,189][INFO ][discovery ] [Betty Ross Banner]
elasticsearch/Jimo-TTbTsWrMv5um2_TFQ
[2016-08-02 09:19:19,261][INFO ][cluster.service ] [Betty Ross Banner]
new_master {Betty Ross
Banner}{Jimo-TTbTsWrMv5um2_TFQ}{127.0.0.1}{127.0.0.1:9300}, reason:
zen-disco-join(elected_as_master, [0] joins received)
[2016-08-02 09:19:19,320][INFO ][http ] [Betty Ross Banner]
publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2016-08-02 09:19:19,320][INFO ][node ] [Betty Ross Banner]
started
[2016-08-02 09:19:19,329][INFO ][gateway ] [Betty Ross Banner]
recovered [0] indices into cluster_state
[2016-08-02 09:39:35,266][INFO ][node ] [Betty Ross Banner]
stopping ...
[2016-08-02 09:39:35,280][INFO ][node ] [Betty Ross Banner]
stopped
[2016-08-02 09:39:35,280][INFO ][node ] [Betty Ross Banner]
closing ...
[2016-08-02 09:39:35,286][INFO ][node ] [Betty Ross Banner]
closed
[2016-08-02 09:19:12,594][INFO ][node ] [Wiccan] stopping
...
[2016-08-02 09:19:12,727][INFO ][node ] [Wiccan] stopped
[2016-08-02 09:19:12,728][INFO ][node ] [Wiccan] closing ...
[2016-08-02 09:19:12,737][INFO ][node ] [Wiccan] closed
[2016-08-02 09:39:36,018][INFO ][node ] [ansted]
version[2.3.4], pid[22525], build[e455fd0/2016-06-30T11:24:31Z]
[2016-08-02 09:39:36,019][INFO ][node ] [ansted]
initializing ...
[2016-08-02 09:39:36,555][INFO ][plugins ] [ansted] modules
[reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-08-02 09:39:36,577][INFO ][env ] [ansted] using [1]
data paths, mounts [[/ (rootfs)]], net usable_space [11.9gb], net total_space
[13.8gb], spins? [unknown], types [rootfs]
[2016-08-02 09:39:36,578][INFO ][env ] [ansted] heap size
[1007.3mb], compressed ordinary object pointers [true]
[2016-08-02 09:39:36,578][WARN ][env ] [ansted] max file
descriptors [65535] for elasticsearch process likely too low, consider
increasing to at least [65536]
[2016-08-02 09:39:38,565][INFO ][node ] [ansted] initialized
[2016-08-02 09:39:38,565][INFO ][node ] [ansted] starting
...
[2016-08-02 09:39:38,648][INFO ][transport ] [ansted]
publish_address {10.100.180.146:9300}, bound_addresses {10.100.180.146:9300}
[2016-08-02 09:39:38,652][INFO ][discovery ] [ansted]
graylog/3AAf8AliSu-GCi7vAPx5_Q
[2016-08-02 09:39:41,713][INFO ][cluster.service ] [ansted]
detected_master
{ansted-search-01}{eau3bPLITyOEhqUDZYFkuw}{10.100.180.149}{10.100.180.149:9300},
added
{{ansted-search-01}{eau3bPLITyOEhqUDZYFkuw}{10.100.180.149}{10.100.180.149:9300},{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{cNjaFEcJSl2ejnAYKTsPCA}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-receive(from master
[{ansted-search-01}{eau3bPLITyOEhqUDZYFkuw}{10.100.180.149}{10.100.180.149:9300}])
[2016-08-02 09:39:41,860][INFO ][http ] [ansted]
publish_address {10.100.180.146:9200}, bound_addresses {10.100.180.146:9200}
[2016-08-02 09:39:41,860][INFO ][node ] [ansted] started
[2016-08-02 09:39:42,112][WARN ][index ] [ansted]
[graylog_0] [graylog_0][0] found shard on path:
[/var/lib/elasticsearch/graylog/nodes/0/indices/graylog_0/0] with a different
index UUID - this shard seems to be leftover from a different index with the
same name. Remove the leftover shard in order to reuse the path with the
current index
[2016-08-02 09:39:42,157][WARN ][index ] [ansted]
[graylog_0] [graylog_0][0] failed to load shard path, trying to remove leftover
[2016-08-02 09:39:42,157][WARN ][index ] [ansted]
[graylog_0] [graylog_0][0] deleting leftover shard on path:
[/var/lib/elasticsearch/graylog/nodes/0/indices/graylog_0/0] with a different
index UUID
[2016-08-02 09:39:54,129][INFO ][cluster.service ] [ansted] removed
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{cNjaFEcJSl2ejnAYKTsPCA}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-receive(from master
[{ansted-search-01}{eau3bPLITyOEhqUDZYFkuw}{10.100.180.149}{10.100.180.149:9300}])
[2016-08-02 09:40:07,928][INFO ][cluster.service ] [ansted] added
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{meza650RQ7am6OhjpbKc6A}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-receive(from master
[{ansted-search-01}{eau3bPLITyOEhqUDZYFkuw}{10.100.180.149}{10.100.180.149:9300}])
[2016-08-02 09:44:37,979][INFO ][node ] [ansted] stopping
...
[2016-08-02 09:44:38,099][INFO ][node ] [ansted] stopped
[2016-08-02 09:44:38,099][INFO ][node ] [ansted] closing ...
[2016-08-02 09:44:38,116][INFO ][node ] [ansted] closed
[2016-08-02 09:44:38,885][INFO ][node ] [ansted]
version[2.3.4], pid[22903], build[e455fd0/2016-06-30T11:24:31Z]
[2016-08-02 09:44:38,886][INFO ][node ] [ansted]
initializing ...
[2016-08-02 09:44:39,450][INFO ][plugins ] [ansted] modules
[reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-08-02 09:44:39,473][INFO ][env ] [ansted] using [1]
data paths, mounts [[/ (rootfs)]], net usable_space [11.9gb], net total_space
[13.8gb], spins? [unknown], types [rootfs]
[2016-08-02 09:44:39,473][INFO ][env ] [ansted] heap size
[1007.3mb], compressed ordinary object pointers [true]
[2016-08-02 09:44:39,474][WARN ][env ] [ansted] max file
descriptors [65535] for elasticsearch process likely too low, consider
increasing to at least [65536]
[2016-08-02 09:44:41,551][INFO ][node ] [ansted] initialized
[2016-08-02 09:44:41,551][INFO ][node ] [ansted] starting
...
[2016-08-02 09:44:41,640][INFO ][transport ] [ansted]
publish_address {10.100.180.146:9300}, bound_addresses {10.100.180.146:9300}
[2016-08-02 09:44:41,645][INFO ][discovery ] [ansted]
graylog/mhgQ99LVR_e1YgU7kQsQTg
[2016-08-02 09:44:44,722][INFO ][cluster.service ] [ansted]
detected_master
{ansted-search-01}{eau3bPLITyOEhqUDZYFkuw}{10.100.180.149}{10.100.180.149:9300},
added
{{ansted-search-01}{eau3bPLITyOEhqUDZYFkuw}{10.100.180.149}{10.100.180.149:9300},{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{meza650RQ7am6OhjpbKc6A}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-receive(from master
[{ansted-search-01}{eau3bPLITyOEhqUDZYFkuw}{10.100.180.149}{10.100.180.149:9300}])
[2016-08-02 09:44:44,819][INFO ][http ] [ansted]
publish_address {10.100.180.146:9200}, bound_addresses {10.100.180.146:9200}
[2016-08-02 09:44:44,820][INFO ][node ] [ansted] started
[2016-08-02 09:45:07,735][INFO ][discovery.zen ] [ansted]
master_left
[{ansted-search-01}{eau3bPLITyOEhqUDZYFkuw}{10.100.180.149}{10.100.180.149:9300}],
reason [shut_down]
[2016-08-02 09:45:07,737][WARN ][discovery.zen ] [ansted] master
left (reason = shut_down), current nodes:
{{ansted}{mhgQ99LVR_e1YgU7kQsQTg}{10.100.180.146}{10.100.180.146:9300},{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{meza650RQ7am6OhjpbKc6A}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}
[2016-08-02 09:45:07,738][INFO ][cluster.service ] [ansted] removed
{{ansted-search-01}{eau3bPLITyOEhqUDZYFkuw}{10.100.180.149}{10.100.180.149:9300},},
reason: zen-disco-master_failed
({ansted-search-01}{eau3bPLITyOEhqUDZYFkuw}{10.100.180.149}{10.100.180.149:9300})
[2016-08-02 09:45:19,251][INFO ][cluster.routing.allocation] [ansted] Cluster
health status changed from [GREEN] to [RED] (reason: [nodes joined]).
[2016-08-02 09:45:19,251][INFO ][cluster.service ] [ansted] new_master
{ansted}{mhgQ99LVR_e1YgU7kQsQTg}{10.100.180.146}{10.100.180.146:9300}, added
{{ansted-search-01}{ApmM7aokRfqW6Xz2Tf5sbg}{10.100.180.149}{10.100.180.149:9300},},
reason: zen-disco-join(elected_as_master, [1] joins received)
[2016-08-02 09:45:19,280][DEBUG][action.admin.indices.alias.exists] [ansted] no
known master node, scheduling a retry
[2016-08-02 09:45:19,281][DEBUG][action.admin.cluster.health] [ansted] no known
master node, scheduling a retry
[2016-08-02 09:45:20,009][INFO ][cluster.routing.allocation] [ansted] Cluster
health status changed from [RED] to [GREEN] (reason: [shards started
[[graylog_0][2], [graylog_0][3]] ...]).
[2016-08-02 09:45:37,412][INFO ][cluster.metadata ] [ansted]
[graylog_0] update_mapping [message]
[2016-08-02 09:45:37,421][INFO ][cluster.metadata ] [ansted]
[graylog_0] update_mapping [message]
[2016-08-02 09:45:37,466][INFO ][cluster.metadata ] [ansted]
[graylog_0] update_mapping [message]
[2016-08-02 09:45:37,616][INFO ][cluster.metadata ] [ansted]
[graylog_0] update_mapping [message]
[2016-08-02 09:45:37,674][INFO ][cluster.metadata ] [ansted]
[graylog_0] update_mapping [message]
[2016-08-02 09:45:37,691][INFO ][cluster.metadata ] [ansted]
[graylog_0] update_mapping [message]
[2016-08-02 09:45:37,783][INFO ][cluster.metadata ] [ansted]
[graylog_0] update_mapping [message]
[2016-08-02 09:45:37,832][INFO ][cluster.metadata ] [ansted]
[graylog_0] update_mapping [message]
[2016-08-02 09:49:04,371][INFO ][cluster.service ] [ansted] removed
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{meza650RQ7am6OhjpbKc6A}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason:
zen-disco-node_left({graylog-591312dc-6d8c-4695-9ae2-877950b92131}{meza650RQ7am6OhjpbKc6A}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false})
[2016-08-02 09:49:18,480][INFO ][cluster.service ] [ansted] added
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{4UvP3MGbSi-dACauYdsiJw}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-join(join from
node[{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{4UvP3MGbSi-dACauYdsiJw}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false}])
[2016-08-02 10:08:35,064][INFO ][cluster.metadata ] [ansted]
[graylog_0] update_mapping [message]
[2016-08-02 10:14:57,053][INFO ][cluster.metadata ] [ansted]
[graylog_0] update_mapping [message]
[2016-08-02 10:15:18,062][INFO ][cluster.metadata ] [ansted]
[graylog_0] update_mapping [message]
[2016-08-02 10:18:26,050][INFO ][cluster.metadata ] [ansted]
[graylog_0] update_mapping [message]
[2016-08-02 10:24:37,117][INFO ][node ] [ansted] stopping
...
[2016-08-02 10:24:37,217][INFO ][node ] [ansted] stopped
[2016-08-02 10:24:37,217][INFO ][node ] [ansted] closing ...
[2016-08-02 10:24:37,223][INFO ][node ] [ansted] closed
[2016-08-02 10:24:37,995][INFO ][node ] [ansted]
version[2.3.4], pid[23335], build[e455fd0/2016-06-30T11:24:31Z]
[2016-08-02 10:24:37,995][INFO ][node ] [ansted]
initializing ...
[2016-08-02 10:24:38,546][INFO ][plugins ] [ansted] modules
[reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-08-02 10:24:38,570][INFO ][env ] [ansted] using [1]
data paths, mounts [[/ (rootfs)]], net usable_space [11.9gb], net total_space
[13.8gb], spins? [unknown], types [rootfs]
[2016-08-02 10:24:38,570][INFO ][env ] [ansted] heap size
[1007.3mb], compressed ordinary object pointers [true]
[2016-08-02 10:24:38,570][WARN ][env ] [ansted] max file
descriptors [65535] for elasticsearch process likely too low, consider
increasing to at least [65536]
[2016-08-02 10:24:40,577][INFO ][node ] [ansted] initialized
[2016-08-02 10:24:40,577][INFO ][node ] [ansted] starting
...
[2016-08-02 10:24:40,671][INFO ][transport ] [ansted]
publish_address {10.100.180.146:9300}, bound_addresses {10.100.180.146:9300}
[2016-08-02 10:24:40,676][INFO ][discovery ] [ansted]
graylog/EWLDMtHpTEKbcb-73PMDqg
[2016-08-02 10:24:43,734][WARN ][discovery.zen ] [ansted] failed to
connect to master
[{ansted-search-01}{ApmM7aokRfqW6Xz2Tf5sbg}{10.100.180.149}{10.100.180.149:9300}],
retrying...
ConnectTransportException[[ansted-search-01][10.100.180.149:9300]
connect_timeout[30s]]; nested: ConnectException[Connection refused:
/10.100.180.149:9300];
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:987)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:920)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:893)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:260)
at
org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:434)
at
org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:386)
at
org.elasticsearch.discovery.zen.ZenDiscovery.access$4800(ZenDiscovery.java:91)
at
org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1237)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.100.180.149:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-08-02 10:24:46,760][WARN ][discovery.zen ] [ansted] failed to
connect to master
[{ansted-search-01}{ApmM7aokRfqW6Xz2Tf5sbg}{10.100.180.149}{10.100.180.149:9300}],
retrying...
ConnectTransportException[[ansted-search-01][10.100.180.149:9300]
connect_timeout[30s]]; nested: ConnectException[Connection refused:
/10.100.180.149:9300];
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:987)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:920)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:893)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:260)
at
org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:434)
at
org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:386)
at
org.elasticsearch.discovery.zen.ZenDiscovery.access$4800(ZenDiscovery.java:91)
at
org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1237)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.100.180.149:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-08-02 10:24:49,773][WARN ][discovery.zen ] [ansted] failed to
connect to master
[{ansted-search-01}{ApmM7aokRfqW6Xz2Tf5sbg}{10.100.180.149}{10.100.180.149:9300}],
retrying...
ConnectTransportException[[ansted-search-01][10.100.180.149:9300]
connect_timeout[30s]]; nested: ConnectException[Connection refused:
/10.100.180.149:9300];
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:987)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:920)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:893)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:260)
at
org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:434)
at
org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:386)
at
org.elasticsearch.discovery.zen.ZenDiscovery.access$4800(ZenDiscovery.java:91)
at
org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1237)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /10.100.180.149:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more
[2016-08-02 10:24:54,153][INFO ][cluster.service ] [ansted] new_master
{ansted}{EWLDMtHpTEKbcb-73PMDqg}{10.100.180.146}{10.100.180.146:9300}, added
{{ansted-search-01}{fVxnXEzrTyCj-Lwm6yaOvQ}{10.100.180.149}{10.100.180.149:9300},{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{4UvP3MGbSi-dACauYdsiJw}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-join(elected_as_master, [1]
joins received)
[2016-08-02 10:24:54,241][INFO ][http ] [ansted]
publish_address {10.100.180.146:9200}, bound_addresses {10.100.180.146:9200}
[2016-08-02 10:24:54,242][INFO ][node ] [ansted] started
[2016-08-02 10:24:54,308][INFO ][gateway ] [ansted] recovered
[1] indices into cluster_state
[2016-08-02 10:24:55,149][INFO ][cluster.routing.allocation] [ansted] Cluster
health status changed from [RED] to [GREEN] (reason: [shards started
[[graylog_0][0], [graylog_0][0]] ...]).
[2016-08-02 10:25:37,289][INFO ][cluster.service ] [ansted] removed
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{4UvP3MGbSi-dACauYdsiJw}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason:
zen-disco-node_left({graylog-591312dc-6d8c-4695-9ae2-877950b92131}{4UvP3MGbSi-dACauYdsiJw}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false})
[2016-08-02 10:25:52,021][INFO ][cluster.service ] [ansted] added
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{znxEOrq4QEiMVjctbytmNA}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-join(join from
node[{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{znxEOrq4QEiMVjctbytmNA}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false}])
[2016-08-02 09:19:07,983][INFO ][node ] [Corona] stopping
...
[2016-08-02 09:19:08,003][INFO ][node ] [Corona] stopped
[2016-08-02 09:19:08,003][INFO ][node ] [Corona] closing ...
[2016-08-02 09:19:08,010][INFO ][node ] [Corona] closed
[2016-08-02 09:19:13,785][INFO ][node ] [Invisible Woman]
version[2.3.4], pid[5489], build[e455fd0/2016-06-30T11:24:31Z]
[2016-08-02 09:19:13,786][INFO ][node ] [Invisible Woman]
initializing ...
[2016-08-02 09:19:14,263][INFO ][plugins ] [Invisible Woman]
modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-08-02 09:19:14,283][INFO ][env ] [Invisible Woman]
using [1] data paths, mounts [[/ (rootfs)]], net usable_space [1013.8gb], net
total_space [1015.1gb], spins? [unknown], types [rootfs]
[2016-08-02 09:19:14,284][INFO ][env ] [Invisible Woman]
heap size [1007.3mb], compressed ordinary object pointers [true]
[2016-08-02 09:19:14,284][WARN ][env ] [Invisible Woman]
max file descriptors [65535] for elasticsearch process likely too low, consider
increasing to at least [65536]
[2016-08-02 09:19:15,925][INFO ][node ] [Invisible Woman]
initialized
[2016-08-02 09:19:15,925][INFO ][node ] [Invisible Woman]
starting ...
[2016-08-02 09:19:16,064][INFO ][transport ] [Invisible Woman]
publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2016-08-02 09:19:16,068][INFO ][discovery ] [Invisible Woman]
elasticsearch/UwtYGgApTYeK_tb2iKjlpg
[2016-08-02 09:19:19,141][INFO ][cluster.service ] [Invisible Woman]
new_master {Invisible
Woman}{UwtYGgApTYeK_tb2iKjlpg}{127.0.0.1}{127.0.0.1:9300}, reason:
zen-disco-join(elected_as_master, [0] joins received)
[2016-08-02 09:19:19,213][INFO ][gateway ] [Invisible Woman]
recovered [0] indices into cluster_state
[2016-08-02 09:19:19,216][INFO ][http ] [Invisible Woman]
publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2016-08-02 09:19:19,216][INFO ][node ] [Invisible Woman]
started
[2016-08-02 09:38:43,636][INFO ][node ] [Invisible Woman]
stopping ...
[2016-08-02 09:38:43,651][INFO ][node ] [Invisible Woman]
stopped
[2016-08-02 09:38:43,651][INFO ][node ] [Invisible Woman]
closing ...
[2016-08-02 09:38:43,658][INFO ][node ] [Invisible Woman]
closed
[2016-08-02 09:38:49,518][INFO ][node ] [ansted-search-01]
version[2.3.4], pid[5595], build[e455fd0/2016-06-30T11:24:31Z]
[2016-08-02 09:38:49,520][INFO ][node ] [ansted-search-01]
initializing ...
[2016-08-02 09:38:50,099][INFO ][plugins ] [ansted-search-01]
modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-08-02 09:38:50,130][INFO ][env ] [ansted-search-01]
using [1] data paths, mounts [[/ (rootfs)]], net usable_space [1013.8gb], net
total_space [1015.1gb], spins? [unknown], types [rootfs]
[2016-08-02 09:38:50,130][INFO ][env ] [ansted-search-01]
heap size [1007.3mb], compressed ordinary object pointers [true]
[2016-08-02 09:38:50,131][WARN ][env ] [ansted-search-01]
max file descriptors [65535] for elasticsearch process likely too low, consider
increasing to at least [65536]
[2016-08-02 09:38:52,066][INFO ][node ] [ansted-search-01]
initialized
[2016-08-02 09:38:52,066][INFO ][node ] [ansted-search-01]
starting ...
[2016-08-02 09:38:52,158][INFO ][transport ] [ansted-search-01]
publish_address {10.100.180.149:9300}, bound_addresses {10.100.180.149:9300}
[2016-08-02 09:38:52,163][INFO ][discovery ] [ansted-search-01]
graylog/eau3bPLITyOEhqUDZYFkuw
[2016-08-02 09:38:55,266][INFO ][cluster.service ] [ansted-search-01]
new_master
{ansted-search-01}{eau3bPLITyOEhqUDZYFkuw}{10.100.180.149}{10.100.180.149:9300},
added
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{cNjaFEcJSl2ejnAYKTsPCA}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-join(elected_as_master, [0]
joins received)
[2016-08-02 09:38:55,294][DEBUG][action.admin.cluster.health]
[ansted-search-01] no known master node, scheduling a retry
[2016-08-02 09:38:55,359][INFO ][http ] [ansted-search-01]
publish_address {10.100.180.149:9200}, bound_addresses {10.100.180.149:9200}
[2016-08-02 09:38:55,360][INFO ][node ] [ansted-search-01]
started
[2016-08-02 09:38:55,390][INFO ][gateway ] [ansted-search-01]
recovered [0] indices into cluster_state
[2016-08-02 09:39:00,543][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] creating index, cause [api], templates [graylog-internal], shards
[4]/[0], mappings [message]
[2016-08-02 09:39:00,860][INFO ][cluster.routing.allocation] [ansted-search-01]
Cluster health status changed from [RED] to [GREEN] (reason: [shards started
[[graylog_0][1], [graylog_0][0], [graylog_0][2], [graylog_0][2],
[graylog_0][1], [graylog_0][0]] ...]).
[2016-08-02 09:39:01,344][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:39:01,354][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:39:01,473][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:39:01,489][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:39:01,507][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:39:01,538][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:39:01,561][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:39:28,112][INFO ][cluster.service ] [ansted-search-01]
added {{ansted}{3AAf8AliSu-GCi7vAPx5_Q}{10.100.180.146}{10.100.180.146:9300},},
reason: zen-disco-join(join from
node[{ansted}{3AAf8AliSu-GCi7vAPx5_Q}{10.100.180.146}{10.100.180.146:9300}])
[2016-08-02 09:39:40,533][INFO ][cluster.service ] [ansted-search-01]
removed
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{cNjaFEcJSl2ejnAYKTsPCA}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason:
zen-disco-node_left({graylog-591312dc-6d8c-4695-9ae2-877950b92131}{cNjaFEcJSl2ejnAYKTsPCA}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false})
[2016-08-02 09:39:54,329][INFO ][cluster.service ] [ansted-search-01]
added
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{meza650RQ7am6OhjpbKc6A}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-join(join from
node[{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{meza650RQ7am6OhjpbKc6A}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false}])
[2016-08-02 09:42:03,689][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:03,778][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:03,782][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:03,790][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:03,843][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:03,883][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:03,886][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:03,900][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:03,932][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:03,980][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:03,989][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:03,992][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:04,052][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:04,113][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:04,120][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:04,169][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:13,660][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:13,693][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:42:13,736][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:44:03,707][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:44:03,711][INFO ][cluster.metadata ] [ansted-search-01]
[graylog_0] update_mapping [message]
[2016-08-02 09:44:24,396][INFO ][cluster.routing.allocation] [ansted-search-01]
Cluster health status changed from [GREEN] to [RED] (reason:
[[{ansted}{3AAf8AliSu-GCi7vAPx5_Q}{10.100.180.146}{10.100.180.146:9300}] left]).
[2016-08-02 09:44:24,396][INFO ][cluster.service ] [ansted-search-01]
removed
{{ansted}{3AAf8AliSu-GCi7vAPx5_Q}{10.100.180.146}{10.100.180.146:9300},},
reason:
zen-disco-node_left({ansted}{3AAf8AliSu-GCi7vAPx5_Q}{10.100.180.146}{10.100.180.146:9300})
[2016-08-02 09:44:31,116][INFO ][cluster.service ] [ansted-search-01]
added {{ansted}{mhgQ99LVR_e1YgU7kQsQTg}{10.100.180.146}{10.100.180.146:9300},},
reason: zen-disco-join(join from
node[{ansted}{mhgQ99LVR_e1YgU7kQsQTg}{10.100.180.146}{10.100.180.146:9300}])
[2016-08-02 09:44:31,743][INFO ][cluster.routing.allocation] [ansted-search-01]
Cluster health status changed from [RED] to [GREEN] (reason: [shards started
[[graylog_0][0], [graylog_0][0]] ...]).
[2016-08-02 09:44:54,135][INFO ][node ] [ansted-search-01]
stopping ...
[2016-08-02 09:44:54,211][INFO ][node ] [ansted-search-01]
stopped
[2016-08-02 09:44:54,211][INFO ][node ] [ansted-search-01]
closing ...
[2016-08-02 09:44:54,226][INFO ][node ] [ansted-search-01]
closed
[2016-08-02 09:45:00,006][INFO ][node ] [ansted-search-01]
version[2.3.4], pid[5699], build[e455fd0/2016-06-30T11:24:31Z]
[2016-08-02 09:45:00,007][INFO ][node ] [ansted-search-01]
initializing ...
[2016-08-02 09:45:00,552][INFO ][plugins ] [ansted-search-01]
modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-08-02 09:45:00,576][INFO ][env ] [ansted-search-01]
using [1] data paths, mounts [[/ (rootfs)]], net usable_space [1013.8gb], net
total_space [1015.1gb], spins? [unknown], types [rootfs]
[2016-08-02 09:45:00,576][INFO ][env ] [ansted-search-01]
heap size [1007.3mb], compressed ordinary object pointers [true]
[2016-08-02 09:45:00,577][WARN ][env ] [ansted-search-01]
max file descriptors [65535] for elasticsearch process likely too low, consider
increasing to at least [65536]
[2016-08-02 09:45:02,442][INFO ][node ] [ansted-search-01]
initialized
[2016-08-02 09:45:02,442][INFO ][node ] [ansted-search-01]
starting ...
[2016-08-02 09:45:02,564][INFO ][transport ] [ansted-search-01]
publish_address {10.100.180.149:9300}, bound_addresses {10.100.180.149:9300}
[2016-08-02 09:45:02,569][INFO ][discovery ] [ansted-search-01]
graylog/ApmM7aokRfqW6Xz2Tf5sbg
[2016-08-02 09:45:05,673][INFO ][cluster.service ] [ansted-search-01]
detected_master
{ansted}{mhgQ99LVR_e1YgU7kQsQTg}{10.100.180.146}{10.100.180.146:9300}, added
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{meza650RQ7am6OhjpbKc6A}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false,
master=false},{ansted}{mhgQ99LVR_e1YgU7kQsQTg}{10.100.180.146}{10.100.180.146:9300},},
reason: zen-disco-receive(from master
[{ansted}{mhgQ99LVR_e1YgU7kQsQTg}{10.100.180.146}{10.100.180.146:9300}])
[2016-08-02 09:45:05,792][INFO ][http ] [ansted-search-01]
publish_address {10.100.180.149:9200}, bound_addresses {10.100.180.149:9200}
[2016-08-02 09:45:05,792][INFO ][node ] [ansted-search-01]
started
[2016-08-02 09:48:50,780][INFO ][cluster.service ] [ansted-search-01]
removed
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{meza650RQ7am6OhjpbKc6A}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-receive(from master
[{ansted}{mhgQ99LVR_e1YgU7kQsQTg}{10.100.180.146}{10.100.180.146:9300}])
[2016-08-02 09:49:04,890][INFO ][cluster.service ] [ansted-search-01]
added
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{4UvP3MGbSi-dACauYdsiJw}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-receive(from master
[{ansted}{mhgQ99LVR_e1YgU7kQsQTg}{10.100.180.146}{10.100.180.146:9300}])
[2016-08-02 10:24:23,531][INFO ][discovery.zen ] [ansted-search-01]
master_left
[{ansted}{mhgQ99LVR_e1YgU7kQsQTg}{10.100.180.146}{10.100.180.146:9300}], reason
[shut_down]
[2016-08-02 10:24:23,534][WARN ][discovery.zen ] [ansted-search-01]
master left (reason = shut_down), current nodes:
{{ansted-search-01}{ApmM7aokRfqW6Xz2Tf5sbg}{10.100.180.149}{10.100.180.149:9300},{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{4UvP3MGbSi-dACauYdsiJw}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}
[2016-08-02 10:24:23,535][INFO ][cluster.service ] [ansted-search-01]
removed
{{ansted}{mhgQ99LVR_e1YgU7kQsQTg}{10.100.180.146}{10.100.180.146:9300},},
reason: zen-disco-master_failed
({ansted}{mhgQ99LVR_e1YgU7kQsQTg}{10.100.180.146}{10.100.180.146:9300})
[2016-08-02 10:24:28,888][INFO ][node ] [ansted-search-01]
stopping ...
[2016-08-02 10:24:28,972][INFO ][node ] [ansted-search-01]
stopped
[2016-08-02 10:24:28,972][INFO ][node ] [ansted-search-01]
closing ...
[2016-08-02 10:24:28,982][INFO ][node ] [ansted-search-01]
closed
[2016-08-02 10:24:34,744][INFO ][node ] [ansted-search-01]
version[2.3.4], pid[5867], build[e455fd0/2016-06-30T11:24:31Z]
[2016-08-02 10:24:34,745][INFO ][node ] [ansted-search-01]
initializing ...
[2016-08-02 10:24:35,299][INFO ][plugins ] [ansted-search-01]
modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-08-02 10:24:35,324][INFO ][env ] [ansted-search-01]
using [1] data paths, mounts [[/ (rootfs)]], net usable_space [1013.8gb], net
total_space [1015.1gb], spins? [unknown], types [rootfs]
[2016-08-02 10:24:35,324][INFO ][env ] [ansted-search-01]
heap size [1007.3mb], compressed ordinary object pointers [true]
[2016-08-02 10:24:35,324][WARN ][env ] [ansted-search-01]
max file descriptors [65535] for elasticsearch process likely too low, consider
increasing to at least [65536]
[2016-08-02 10:24:37,252][INFO ][node ] [ansted-search-01]
initialized
[2016-08-02 10:24:37,252][INFO ][node ] [ansted-search-01]
starting ...
[2016-08-02 10:24:37,437][INFO ][transport ] [ansted-search-01]
publish_address {10.100.180.149:9300}, bound_addresses {10.100.180.149:9300}
[2016-08-02 10:24:37,442][INFO ][discovery ] [ansted-search-01]
graylog/fVxnXEzrTyCj-Lwm6yaOvQ
[2016-08-02 10:24:40,572][INFO ][cluster.service ] [ansted-search-01]
detected_master
{ansted}{EWLDMtHpTEKbcb-73PMDqg}{10.100.180.146}{10.100.180.146:9300}, added
{{ansted}{EWLDMtHpTEKbcb-73PMDqg}{10.100.180.146}{10.100.180.146:9300},{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{4UvP3MGbSi-dACauYdsiJw}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-receive(from master
[{ansted}{EWLDMtHpTEKbcb-73PMDqg}{10.100.180.146}{10.100.180.146:9300}])
[2016-08-02 10:24:40,600][INFO ][http ] [ansted-search-01]
publish_address {10.100.180.149:9200}, bound_addresses {10.100.180.149:9200}
[2016-08-02 10:24:40,601][INFO ][node ] [ansted-search-01]
started
[2016-08-02 10:25:23,699][INFO ][cluster.service ] [ansted-search-01]
removed
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{4UvP3MGbSi-dACauYdsiJw}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-receive(from master
[{ansted}{EWLDMtHpTEKbcb-73PMDqg}{10.100.180.146}{10.100.180.146:9300}])
[2016-08-02 10:25:38,436][INFO ][cluster.service ] [ansted-search-01]
added
{{graylog-591312dc-6d8c-4695-9ae2-877950b92131}{znxEOrq4QEiMVjctbytmNA}{10.100.180.146}{10.100.180.146:9350}{client=true,
data=false, master=false},}, reason: zen-disco-receive(from master
[{ansted}{EWLDMtHpTEKbcb-73PMDqg}{10.100.180.146}{10.100.180.146:9300}])
