All,
I am able to connect my server node to the database. This is successful,
but inevitably fails after about 10 seconds. The connections get dropped
reliably after about that time.
Can anyone give some pointers on troubleshooting this. I have graylog
clusters (EC2 ASG's) for each of graylog's roles. I am trying to connect
the server to the data nodes, but ubuntu@graylog:/opt/graylog/server$
./bin/graylogctl status shows this:
#=============================================================
graylog-server running with PID 6818
ubuntu@graylog:/opt/graylog/server$ ./bin/graylogctl status
graylog-server running with PID 6818
ubuntu@graylog:/opt/graylog/server$ ./bin/graylogctl status
graylog-server running with PID 6818
ubuntu@graylog:/opt/graylog/server$ ./bin/graylogctl status
graylog-server running with PID 6818
ubuntu@graylog:/opt/graylog/server$ ./bin/graylogctl status
graylog-server running with PID 6818
ubuntu@graylog:/opt/graylog/server$ ./bin/graylogctl status
graylog-server not running
On Mongo terminal(running on same box as server), I see this:
#=============================================================
55520 #3 (3 connections now open)
2015-07-09T19:52:28.040+0000 [initandlisten] connection accepted from
127.0.0.1:55521 #4 (4 connections now open)
2015-07-09T19:52:28.040+0000 [initandlisten] connection accepted from
127.0.0.1:55522 #5 (5 connections now open)
2015-07-09T19:52:29.069+0000 [conn3] build index on: graylog2.content_packs
properties: { v: 1, key: { _id: 1 }, name: "_id_", ns:
"graylog2.content_packs" }
2015-07-09T19:52:29.069+0000 [conn3] added index to empty collection
2015-07-09T19:52:32.610+0000 [clientcursormon] mem (MB) res:43 virt:507
2015-07-09T19:52:32.610+0000 [clientcursormon] mapped (incl journal
view):320
2015-07-09T19:52:32.610+0000 [clientcursormon] connections:5
2015-07-09T19:52:43.554+0000 [conn3] build index on:
graylog2.system_messages properties: { v: 1, key: { _id: 1 }, name: "_id_",
ns: "graylog2.system_messages" }
2015-07-09T19:52:43.554+0000 [conn3] added index to empty collection
2015-07-09T19:52:47.892+0000 [conn1] end connection 127.0.0.1:55518 (4
connections now open)
2015-07-09T19:52:47.892+0000 [conn2] end connection 127.0.0.1:55519 (3
connections now open)
2015-07-09T19:52:47.892+0000 [conn3] end connection 127.0.0.1:55520 (2
connections now open)
2015-07-09T19:52:47.892+0000 [conn4] end connection 127.0.0.1:55521 (1
connection now open)
2015-07-09T19:52:47.892+0000 [conn5] end connection 127.0.0.1:55522 (0
connections now open)
The graylog server log file
#=============================================================
WARN ][discovery.zen.ping.unicast] [Jamal Afari] [1] failed send ping to
[#zen_unicast_1#][graylog][inet[/172.31.198.14:9300]]
org.elasticsearch.ElasticsearchIllegalStateException: can't add nodes to a
stopped transport
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:719)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:699)
at
org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:220)
at
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:373)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2015-07-09 17:44:17,774][INFO ][node ] [Jamal Afari]
stopped
[2015-07-09 17:44:17,774][INFO ][node ] [Jamal Afari]
closing ...
[2015-07-09 17:44:17,779][INFO ][node ] [Jamal Afari]
closed
[2015-07-09 17:44:33,959][INFO ][node ] [Lunatica]
version[1.5.2], pid[2275], build[62ff986/2015-04-27T09:21:06Z]
[2015-07-09 17:44:33,959][INFO ][node ] [Lunatica]
initializing ...
[2015-07-09 17:44:33,963][INFO ][plugins ] [Lunatica]
loaded [], sites []
[2015-07-09 17:44:36,314][INFO ][node ] [Lunatica]
initialized
[2015-07-09 17:44:36,314][INFO ][node ] [Lunatica]
starting ...
[2015-07-09 17:44:36,369][INFO ][transport ] [Lunatica]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/172.30.198.14:9300]}
[2015-07-09 17:44:36,377][INFO ][discovery ] [Lunatica]
graylog/vseaJqvdTimJCB1Tjz5Jkw
[2015-07-09 17:45:06,377][WARN ][discovery ] [Lunatica]
waited for 30s and no initial state was set by the discovery
[2015-07-09 17:45:06,382][INFO ][http ] [Lunatica]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/172.30.198.14:9200]}
[2015-07-09 17:45:06,383][INFO ][node ] [Lunatica]
started
[2015-07-09 17:46:06,539][INFO ][cluster.service ] [Lunatica]
new_master
[Lunatica][vseaJqvdTimJCB1Tjz5Jkw][graylog][inet[/172.30.198.14:9300]],
reason: zen-disco-join (elected_as_master)
[2015-07-09 17:46:07,035][INFO ][gateway ] [Lunatica]
recovered [1] indices into cluster_state
[2015-07-09 17:52:25,143][INFO ][node ] [Lunatica]
stopping ...
[2015-07-09 17:52:25,166][INFO ][node ] [Lunatica]
stopped
[2015-07-09 17:52:25,166][INFO ][node ] [Lunatica]
closing ...
[2015-07-09 17:52:25,170][INFO ][node ] [Lunatica]
closed
[2015-07-09 18:01:44,217][INFO ][node ] [Spider
Doppelganger] version[1.5.2], pid[2407], build[62ff986/2015-04-27T09:21:06Z]
[2015-07-09 18:01:44,217][INFO ][node ] [Spider
Doppelganger] initializing ...
[2015-07-09 18:01:44,221][INFO ][plugins ] [Spider
Doppelganger] loaded [], sites []
[2015-07-09 18:01:46,542][INFO ][node ] [Spider
Doppelganger] initialized
[2015-07-09 18:01:46,543][INFO ][node ] [Spider
Doppelganger] starting ...
[2015-07-09 18:01:46,595][INFO ][transport ] [Spider
Doppelganger] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/172.30.198.14:9300]}
[2015-07-09 18:01:46,603][INFO ][discovery ] [Spider
Doppelganger] graylog/Sd2WCbx-Rw-k1HxDRGRGtg
[2015-07-09 18:02:16,603][WARN ][discovery ] [Spider
Doppelganger] waited for 30s and no initial state was set by the discovery
[2015-07-09 18:02:16,608][INFO ][http ] [Spider
Doppelganger] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/172.30.198.14:9200]}
[2015-07-09 18:02:16,608][INFO ][node ] [Spider
Doppelganger] started
[2015-07-09 18:03:16,766][INFO ][cluster.service ] [Spider
Doppelganger] new_master [Spider
Doppelganger][Sd2WCbx-Rw-k1HxDRGRGtg][graylog][inet[/172.30.198.14:9300]],
reason: zen-disco-join (elected_as_master)
[2015-07-09 18:03:17,256][INFO ][gateway ] [Spider
Doppelganger] recovered [1] indices into cluster_state
[2015-07-09 19:20:16,978][INFO ][node ] [Spider
Doppelganger] stopping ...
[2015-07-09 19:20:17,013][INFO ][node ] [Spider
Doppelganger] stopped
[2015-07-09 19:20:17,013][INFO ][node ] [Spider
Doppelganger] closing ...
[2015-07-09 19:20:17,016][INFO ][node ] [Spider
Doppelganger] closed
[2015-07-09 19:23:08,161][INFO ][node ] [M]
version[1.5.2], pid[2570], build[62ff986/2015-04-27T09:21:06Z]
[2015-07-09 19:23:08,162][INFO ][node ] [M]
initializing ...
[2015-07-09 19:23:08,166][INFO ][plugins ] [M] loaded [],
sites []
[2015-07-09 19:23:10,504][INFO ][node ] [M] initialized
[2015-07-09 19:23:10,505][INFO ][node ] [M] starting ...
[2015-07-09 19:23:10,559][INFO ][transport ] [M]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/172.30.198.14:9300]}
[2015-07-09 19:23:10,567][INFO ][discovery ] [M]
graylog/nTUnv4H8RvKLx1A0HbBcLg
[2015-07-09 19:23:40,567][WARN ][discovery ] [M] waited for
30s and no initial state was set by the discovery
[2015-07-09 19:23:40,571][INFO ][http ] [M]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/172.30.198.14:9200]}
[2015-07-09 19:23:40,572][INFO ][node ] [M] started
[2015-07-09 19:24:40,730][INFO ][cluster.service ] [M] new_master
[M][nTUnv4H8RvKLx1A0HbBcLg][graylog][inet[/172.30.198.14:9300]], reason:
zen-disco-join (elected_as_master)
[2015-07-09 19:24:41,223][INFO ][gateway ] [M] recovered
[1] indices into cluster_state
[2015-07-09 19:47:46,609][INFO ][node ] [M] stopping ...
[2015-07-09 19:47:46,645][INFO ][node ] [M] stopped
[2015-07-09 19:47:46,645][INFO ][node ] [M] closing ...
[2015-07-09 19:47:46,651][INFO ][node ] [M] closed
[2015-07-09 19:51:57,942][INFO ][node ] [Nuklo]
version[1.5.2], pid[2760], build[62ff986/2015-04-27T09:21:06Z]
[2015-07-09 19:51:57,942][INFO ][node ] [Nuklo]
initializing ...
[2015-07-09 19:51:57,946][INFO ][plugins ] [Nuklo] loaded
[], sites []
[2015-07-09 19:52:00,319][INFO ][node ] [Nuklo]
initialized
[2015-07-09 19:52:00,320][INFO ][node ] [Nuklo]
starting ...
[2015-07-09 19:52:00,372][INFO ][transport ] [Nuklo]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/172.30.198.14:9300]}
[2015-07-09 19:52:00,380][INFO ][discovery ] [Nuklo]
graylog/Bsr-NI-nTTCUdeFL2q05Bg
[2015-07-09 19:52:30,381][WARN ][discovery ] [Nuklo] waited
for 30s and no initial state was set by the discovery
[2015-07-09 19:52:30,385][INFO ][http ] [Nuklo]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/172.30.198.14:9200]}
[2015-07-09 19:52:30,386][INFO ][node ] [Nuklo] started
[2015-07-09 19:53:30,542][INFO ][cluster.service ] [Nuklo]
new_master
[Nuklo][Bsr-NI-nTTCUdeFL2q05Bg][graylog][inet[/172.30.198.14:9300]],
reason: zen-disco-join (elected_as_master)
[2015-07-09 19:53:31,056][INFO ][gateway ] [Nuklo]
recovered [1] indices into cluster_state
[2015-07-09 19:58:31,867][INFO ][node ] [Nuklo]
stopping ...
[2015-07-09 19:58:31,897][INFO ][node ] [Nuklo] stopped
[2015-07-09 19:58:31,897][INFO ][node ] [Nuklo] closing
...
[2015-07-09 19:58:31,901][INFO ][node
Thanks,
Tim
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
