Thank you for your reply Jochen.
I found this part:
2015-08-19_15:46:20.95757 WARN [SearchResource] Unable to execute search:
Failed to execute phase [query], all shards failed; shardFailures
{[5eRP1nSkR5-ffQFtbFVobg][graylog_20][0]: RemoteTransportException[[Robert
Bruce
Banner][inet[/IP_ADDRESS_OF_MASTER:9300]][indices:data/read/search[phase/query]]];
nested:
ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData
cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData];
}{[5eRP1nSkR5-ffQFtbFVobg][graylog_20][1]: RemoteTransportException[[Robert
Bruce
Banner][inet[/IP_ADDRESS_OF_MASTER:9300]][indices:data/read/search[phase/query]]];
nested:
ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData
cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData];
}{[5eRP1nSkR5-ffQFtbFVobg][graylog_20][2]: RemoteTransportException[[Robert
Bruce
Banner][inet[/IP_ADDRESS_OF_MASTER:9300]][indices:data/read/search[phase/query]]];
nested:
ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData
cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData];
}{[5eRP1nSkR5-ffQFtbFVobg][graylog_20][3]: RemoteTransportException[[Robert
Bruce
Banner][inet[/IP_ADDRESS_OF_MASTER:9300]][indices:data/read/search[phase/query]]];
nested:
ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData
cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData];
}
Any idea what that might be ?
For now this is what I was able to test:
>From web server:
curl -i http://127.0.0.1:9000/
HTTP/1.1 303 See Other
Location: /login?destination=%2F
Content-Length: 0
from graylog server 1 :
lsof -i :12900
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 30323 graylog 289u IPv6 5147490 0t0 TCP *:12900 (LISTEN)
testing
Command:
curl -i --user admin:password -H 'Accept: application/json'
http://IP_ADDRESS_GRAYLOG1:12900/system/cluster/node
HTTP/1.1 200 OK
Content-Type: application/json
X-Graylog-Node-ID: 74296f78-e754-41eb-b892-18366043cb64
X-Runtime-Microseconds: 515
Transfer-Encoding: chunked
{"node_id":"74296f78-e754-41eb-b892-18366043cb64","type":"server","transport_address":"http://IP_ADDRESS_GRAYLOG1:12900","last_seen":"2015-08-19T14:04:23.000Z","short_node_id":"74296f78","is_master":true}
graylog server 2:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 16802 graylog 299u IPv6 673114 0t0 TCP *:12900 (LISTEN)
Command:
curl -i --user admin:password -H 'Accept: application/json'
http://IP_ADDRESS_GRAYLOG2:12900/system/cluster/node
HTTP/1.1 200 OK
Content-Type: application/json
X-Graylog-Node-ID: 74296f78-e754-41eb-b892-18366043cb64
X-Runtime-Microseconds: 515
Transfer-Encoding: chunked
{"node_id":"74296f78-e754-41eb-b892-18366043cb64","type":"server","transport_address":"http://IP_ADDRESS_GRAYLOG2:12900","last_seen":"2015-08-19T14:04:23.000Z","short_node_id":"74296f78","is_master":false}
Does not tell me anything specific unfortunately.
I have expanded the disks that right now the /var/opt/graylog/data right
now takes less than 10 % following a warning that there is less than 15 %
available space. Restarted services but issue persist.
On Wednesday, 19 August 2015 15:00:40 UTC+1, Jochen Schalanda wrote:
>
> Hi,
>
> are there anymore details about the SearchPhaseExecutionException in the
> Graylog server logs (like a complete stack trace or details on the shard
> failures) or general error messages in the Elasticsearch logs?
>
>
> Cheers,
> Jochen
>
> On Wednesday, 19 August 2015 15:42:17 UTC+2, Graylog2 wrote:
>>
>> Hi Guys,
>>
>> I have a cluster combined of 2 graylog servers with ElasticSearch on them
>> and one additional Vm acting as a web interface.
>>
>> All has been deployed using the graylog OVA (VMware).
>>
>> All was working fine untill today when I spotted that I cannot perform
>> any searches nor I can see any messages in defined streams. Error I am
>> getting is this :
>>
>> *org.elasticsearch.action.search.SearchPhaseExecutionException*
>> Unable to execute search
>>
>> 1. I have restarted the whole cluster, no joy.
>> 2. Checked log files but nothing specific was found in relation to this
>> error
>> 3. There is space on the boxes, server one capacity is roung 85 % used
>> and server 2 roung 81%. Webserver is ok alltogether.
>>
>> Could you stear me in the right direction on where to start to tackle
>> this issue please ?
>>
>> Cheers.
>>
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/f35991e6-6011-41f0-8be4-033de152d93a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
