OK, so I'm back to the original issue.
As soon as there are multiple nodes in the configuration, I start to see
the "Did not find meta info of this node. Re-Registering." in the logs. I
see this for all 3 nodes. I've checked the replica configuration in Mongo
(looks good), dropped down to a single Mongo node (no change).. the mongo
table has the hosts in there:
graylog:PRIMARY> db.nodes.find()
{ "_id" : ObjectId("573b620b89479f0ccf046c34"), "is_master" : false,
"hostname" : "gray02.somewhere.com", "last_seen" : 1463512536,
"transport_address" : "http://10.201.137.210:12900/";, "type" : "SERVER",
"node_id" : "8536ee95-b9c7-4553-9022-d997da315755" }
{ "_id" : ObjectId("573b6dbf05ee161654fa2122"), "is_master" : true,
"hostname" : "gray01.somewhere.com", "last_seen" : 1463512536,
"transport_address" : "http://10.201.137.209:12900/";, "type" : "SERVER",
"node_id" : "00ac0ad1-b96f-46c0-a2bc-bc9e7a90777f" }
{ "_id" : ObjectId("573b6dbfb2d64909218a56b8"), "is_master" : false,
"hostname" : "gray00.somewhere.com", "last_seen" : 1463512537,
"transport_address" : "http://10.201.137.208:12900/";, "type" : "SERVER",
"node_id" : "3116ac6b-604f-4436-955c-1458cb489415" }
last_seen is getting updated.. times are in-sync on all 3 nodes (as well as
the F5 and any web clients).. REST calls are configured to be sticky, as
are the webUI calls.. what am I missing?
On Tuesday, May 17, 2016 at 11:04:22 AM UTC-7, Jeff McCombs wrote:
>
> You can't tell, but I'm blushing right now.
>
> Thanks Jochen. :)
>
>
> On Tuesday, May 17, 2016 at 10:13:36 AM UTC-7, Jochen Schalanda wrote:
>>
>> Hi Jeff,
>>
>> you're probably looking for the web_endpoint_uri configuration setting
>> (see
>> http://docs.graylog.org/en/2.0/pages/configuring_webif.html#configuration-options).
>>
>> The rest_listen_uri (or rest_transport_uri) should always be an address
>> which the Graylog nodes in a given cluster can access.
>>
>> Cheers,
>> Jochen
>>
>> On Tuesday, 17 May 2016 17:49:54 UTC+2, Jeff McCombs wrote:
>>>
>>> Hi Jochen,
>>>
>>> Yes, that's actually intentional. Though it could just be that I'm
>>> misunderstanding the option..
>>>
>>> Consider the scenario below:
>>>
>>> +-----------------+
>>> | |
>>> | User |
>>> | (192.168.1.200)|
>>> +-----------------+
>>> |
>>> |
>>> +------------------------------------------------------------------+
>>> |
>>> |
>>> | graylog.somewhere.com
>>> | (192.168.1.100)
>>> +--------------+
>>> | |
>>> | F5 |
>>> | |
>>> | |
>>> +--------------+
>>> |
>>> |
>>> |
>>>
>>>
>>> +------------------------------------------------------------------------------------+
>>> | | |
>>> +-------------+ +-------------+ +-------------+
>>> | | | | | |
>>> | | | | | |
>>> | gray00 | | gray01 | | gray02 |
>>> | 10.201.5.1 | | 10.201.5.2 | | 10.201.5.3 |
>>> +-------------+ +-------------+ +-------------+
>>>
>>> (hope that shows up OK, if not, convert it to fixed width font)
>>>
>>> When the javscript running in the browser for a WebUI call needs to
>>> reach the individual nodes via the REST interface, the only way for that
>>> call to happen is to go through the F5 and be load balanced.. right? But if
>>> the nodes need to communicate with one another via the REST interface as
>>> well..then yeah I could see why the nodes would be complaining. They try
>>> and reach the 192 address, the request gets balanced, and winds up on the
>>> wrong node...
>>>
>>> So am I just misunderstanding the REST transport URI option? WITHOUT
>>> setting that configuration to the same address, the WebUI doesn't function
>>> properly because there's no direct communication between the individual
>>> graylog nodes, and the end user.
>>>
>>> Is there a WebUI/REST URI option somewhere I just don't know about?
>>>
>>>
>>> On Tue, May 17, 2016 at 5:07 AM, Jochen Schalanda wrote:
>>>
>>>> Hi Jeff,
>>>>
>>>> you're using the same transport address for the Graylog REST API on all
>>>> 3 Graylog nodes. Is this intentional? I'm asking because that won't work
>>>> in
>>>> the long run as Graylog nodes need to be able to communicate with each
>>>> other via the Graylog REST API and the announced transport address.
>>>>
>>>> Cheers,
>>>> Jochen
>>>>
>>>>
>>>> On Friday, 13 May 2016 22:53:07 UTC+2, Jeff McCombs wrote:
>>>>>
>>>>> So here's a question.. looking at the node output from tokred vs mine..
>>>>>
>>>>> When you have a cluster of Graylog servers behind a load balancer.. do
>>>>> you configure the API transport address to the cluster IP, or the
>>>>> individual nodes? Could this be the cause of the following errors I'm
>>>>> also
>>>>> seeing?
>>>>>
>>>>> 2016-05-13T13:43:27.749-07:00 WARN [ProxiedResource] Node
>>>>> <3116ac6b-604f-4436-955c-1458cb489415> not found while trying to call
>>>>> org.graylog2.shared.rest.resources.system.RemoteMetricsResource on it.
>>>>> 2016-05-13T13:46:58.766-07:00 WARN [ProxiedResource] Node
>>>>> <3116ac6b-604f-4436-955c-1458cb489415> not found while trying to call
>>>>> org.graylog2.shared.rest.resources.system.RemoteMetricsResource on it.
>>>>> 2016-05-13T13:49:14.735-07:00 WARN [ProxiedResource] Node
>>>>> <3116ac6b-604f-4436-955c-1458cb489415> not found while trying to call
>>>>> org.graylog2.shared.rest.resources.system.RemoteMetricsResource on it.
>>>>>
>>>>> On Friday, May 13, 2016 at 1:50:17 PM UTC-7, Jeff McCombs wrote:
>>>>>>
>>>>>> Hi Jochen,
>>>>>>
>>>>>> I see the records for the nodes:
>>>>>>
>>>>>> graylog:PRIMARY> db.nodes.find()
>>>>>> { "_id" : ObjectId("57363bab05ee16689e192953"), "is_master" : false,
>>>>>> "hostname" : "gray01somewhere.com", "last_seen" : 1463172221,
>>>>>> "transport_address" : "http://graylog.somewhere.com:12900/";, "type"
>>>>>> : "SERVER", "node_id" : "00ac0ad1-b96f-46c0-a2bc-bc9e7a90777f" }
>>>>>> { "_id" : ObjectId("57363c0889479f6906e17de9"), "is_master" : false,
>>>>>> "hostname" : "gray02.somewhere.com", "last_seen" : 1463172221,
>>>>>> "transport_address" : "http://graylog.somewhere.com:12900/";, "type"
>>>>>> : "SERVER", "node_id" : "8536ee95-b9c7-4553-9022-d997da315755" }
>>>>>> { "_id" : ObjectId("57363c79b2d6491223d87222"), "is_master" : true,
>>>>>> "hostname" : "gray00.somewhere.com", "last_seen" : 1463172220,
>>>>>> "transport_address" : "http://graylog.somewhere.com:12900/";, "type"
>>>>>> : "SERVER", "node_id" : "3116ac6b-604f-4436-955c-1458cb489415" }
>>>>>>
>>>>>> Interestingly, when I shut down all but the master, it continues to
>>>>>> spit errors:
>>>>>>
>>>>>> /var/log/graylog-server/server.log
>>>>>> 2016-05-13T13:47:06.662-07:00 WARN [NodePingThread] Did not find
>>>>>> meta info of this node. Re-registering.
>>>>>> 2016-05-13T13:47:32.639-07:00 WARN [NodePingThread] Did not find
>>>>>> meta info of this node. Re-registering.
>>>>>>
>>>>>> mongo nodes query:
>>>>>> graylog:PRIMARY> db.nodes.find()
>>>>>> { "_id" : ObjectId("57363d64b2d6491223d87339"), "is_master" : true,
>>>>>> "hostname" : "gray00.somewhere.com", "last_seen" : 1463172464,
>>>>>> "transport_address" : "http://graylog.somewhere.com:12900/";, "type"
>>>>>> : "SERVER", "node_id" : "3116ac6b-604f-4436-955c-1458cb489415" }
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> On Friday, May 13, 2016 at 2:16:17 AM UTC-7, Jochen Schalanda wrote:
>>>>>>>
>>>>>>> Hi Jeff,
>>>>>>>
>>>>>>> please check the "nodes" collection in MongoDB and that it contains
>>>>>>> valid node descriptors while Graylog is running.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Jochen
>>>>>>>
>>>>>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/c0285405-cb5a-41cd-98ec-3cec1ca46b14%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
