Hi there,
We have two nodes running graylog-server (v 1.2.0) and ElasticSearch in both of
them and I'm trying to figure out what happened that forced us to manually
cycle the deflector in order to see our messages again. We are processing a few
hundred messages per second, running with 3.5GB heap in the ES JVMs & 1.5GB in
the graylog-server JVMs.
I can see the following (somewhat filtered) in the graylog logs, in the
ElasticSearch logs I only see JVM GC duration WARNings. On a side note, I'd be
great to have some ERROR log yelling about messages being lost (or at least,
nowehere to be found).
Node 1
======
2015-12-11_09:26:53.47853 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
2015-12-11_09:26:53.91663 INFO [AbstractValidatingSessionManager] Validating
all active sessions...
2015-12-11_09:26:54.83515 INFO [AbstractValidatingSessionManager] Finished
session validation. No sessions were stopped.
2015-12-11_09:26:54.99021 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
[snip many repetead WARNings]
2015-12-11_09:32:53.71532 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
2015-12-11_09:33:38.03343 INFO [SystemJobManager] SystemJob
<e0098452-9fe7-11e5-857c-005056b06219>
[org.graylog2.indexer.indices.jobs.OptimizeIndexJob] finished in 1023067ms.
2015-12-11_09:33:53.23876 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
2015-12-11_09:33:53.34890 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
[snip many repetead WARNings]
2015-12-11_09:42:06.48387 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
2015-12-11_09:42:06.49119 WARN [IndexHelper] Couldn't find latest deflector
target index
2015-12-11_09:42:06.49120 org.graylog2.database.NotFoundException: Couldn't
find index range for index graylog_204
2015-12-11_09:42:06.49121 at
org.graylog2.indexer.ranges.EsIndexRangeService.loadIndexRange(EsIndexRangeService.java:155)
2015-12-11_09:42:06.49121 at
org.graylog2.indexer.ranges.EsIndexRangeService.access$000(EsIndexRangeService.java:78)
2015-12-11_09:42:06.49121 at
org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:106)
2015-12-11_09:42:06.49122 at
org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:103)
2015-12-11_09:42:06.49122 at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
2015-12-11_09:42:06.49122 at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
2015-12-11_09:42:06.49122 at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
2015-12-11_09:42:06.49123 at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
2015-12-11_09:42:06.49123 at
com.google.common.cache.LocalCache.get(LocalCache.java:3937)
2015-12-11_09:42:06.49123 at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
2015-12-11_09:42:06.49123 at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
2015-12-11_09:42:06.49124 at
org.graylog2.indexer.ranges.EsIndexRangeService.get(EsIndexRangeService.java:130)
2015-12-11_09:42:06.49124 at
org.graylog2.indexer.IndexHelper.determineAffectedIndicesWithRanges(IndexHelper.java:119)
2015-12-11_09:42:06.49125 at
org.graylog2.indexer.searches.Searches.search(Searches.java:241)
2015-12-11_09:42:06.49125 at
org.graylog2.rest.resources.search.RelativeSearchResource.searchRelative(RelativeSearchResource.java:103)
[snip many repetead WARNings]
2015-12-11_09:42:53.88667 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
2015-12-11_09:43:05.75814 WARN [jvm] [graylog2-server]
[gc][young][331369][11365] duration [1s], collections [1]/[1.5s], total
[1s]/[2.1h], memory [1.2gb]->[777.9mb]/[1.3gb], all_pools {[young]
[583mb]->[12.5mb]/[600mb]}{[survivor] [57.6mb]->[68.6mb]/[75mb]}{[old]
[684.8mb]->[696.9mb]/[750mb]}
2015-12-11_09:43:19.87094 WARN [NodePingThread] Did not find meta info of this
node. Re-registering.
2015-12-11_09:43:53.24549 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
Node 2
======
2015-12-11_09:26:50.41251 INFO [AbstractValidatingSessionManager] Validating
all active sessions...
2015-12-11_09:26:51.07435 INFO [AbstractValidatingSessionManager] Finished
session validation. No sessions were stopped.
2015-12-11_09:28:17.94562 WARN [NodePingThread] Did not find meta info of this
node. Re-registering.
2015-12-11_09:28:18.14976 WARN [jvm] [graylog2-server]
[gc][young][345393][18320] duration [2.5s], collections [1]/[3.5s], total
[2.5s]/[12.2m], memory [1.1gb]->[685mb]/[1.3gb], all_pools {[young]
[539.7mb]->[14.4mb]/[600mb]}{[survivor] [47.3mb]->[66.2mb]/[75mb]}{[old]
[598.2mb]->[604.4mb]/[750mb]}
2015-12-11_09:29:03.96581 WARN [jvm] [graylog2-server]
[gc][young][345436][18326] duration [3.6s], collections [1]/[4s], total
[3.6s]/[12.3m], memory [1.2gb]->[711.2mb]/[1.3gb], all_pools {[young]
[570.4mb]->[14.3kb]/[600mb]}{[survivor] [53.6mb]->[67.3mb]/[75mb]}{[old]
[638mb]->[643.9mb]/[750mb]}
2015-12-11_09:29:03.97050 WARN [NodePingThread] Did not find meta info of this
node. Re-registering.
2015-12-11_09:31:29.53938 INFO [jvm] [graylog2-server]
[gc][young][345575][18351] duration [987ms], collections [1]/[1.4s], total
[987ms]/[12.4m], memory [1gb]->[573.2mb]/[1.3gb], all_pools {[young]
[538.2mb]->[4.1kb]/[600mb]}{[survivor] [69.6mb]->[72.4mb]/[75mb]}{[old]
[492.4mb]->[500.7mb]/[750mb]}
2015-12-11_09:33:04.55657 WARN [jvm] [graylog2-server]
[gc][young][345668][18367] duration [2.4s], collections [1]/[2.9s], total
[2.4s]/[12.4m], memory [1.1gb]->[694.3mb]/[1.3gb], all_pools {[young]
[536.6mb]->[612.6kb]/[600mb]}{[survivor] [44.2mb]->[73.7mb]/[75mb]}{[old]
[612.5mb]->[620mb]/[750mb]}
2015-12-11_09:33:04.69209 WARN [NodePingThread] Did not find meta info of this
node. Re-registering.
2015-12-11_09:38:33.23279 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
2015-12-11_09:38:33.25105 WARN [IndexHelper] Couldn't find latest deflector
target index
2015-12-11_09:38:33.25107 org.graylog2.database.NotFoundException: Couldn't
find index range for index graylog_204
2015-12-11_09:38:33.25107 at
org.graylog2.indexer.ranges.EsIndexRangeService.loadIndexRange(EsIndexRangeService.java:155)
2015-12-11_09:38:33.25107 at
org.graylog2.indexer.ranges.EsIndexRangeService.access$000(EsIndexRangeService.java:78)
2015-12-11_09:38:33.25108 at
org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:106)
2015-12-11_09:38:33.25108 at
org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:103)
2015-12-11_09:38:33.25108 at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
2015-12-11_09:38:33.25109 at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
2015-12-11_09:38:33.25109 at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
2015-12-11_09:38:33.25109 at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
2015-12-11_09:38:33.25110 at
com.google.common.cache.LocalCache.get(LocalCache.java:3937)
2015-12-11_09:38:33.25110 at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
2015-12-11_09:38:33.25111 at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
2015-12-11_09:38:33.25112 at
org.graylog2.indexer.ranges.EsIndexRangeService.get(EsIndexRangeService.java:130)
2015-12-11_09:38:33.25113 at
org.graylog2.indexer.IndexHelper.determineAffectedIndicesWithRanges(IndexHelper.java:119)
2015-12-11_09:38:33.25113 at
org.graylog2.indexer.searches.Searches.search(Searches.java:241)
2015-12-11_09:38:33.25113 at
org.graylog2.rest.resources.search.RelativeSearchResource.searchRelative(RelativeSearchResource.java:103)
2015-12-11_09:38:33.28821 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
2015-12-11_09:41:27.25182 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
2015-12-11_09:41:27.25322 WARN [IndexHelper] Couldn't find latest deflector
target index
2015-12-11_09:41:27.25323 org.graylog2.database.NotFoundException: Couldn't
find index range for index graylog_204
2015-12-11_09:41:27.25323 at
org.graylog2.indexer.ranges.EsIndexRangeService.loadIndexRange(EsIndexRangeService.java:155)
2015-12-11_09:41:27.25324 at
org.graylog2.indexer.ranges.EsIndexRangeService.access$000(EsIndexRangeService.java:78)
2015-12-11_09:41:27.25324 at
org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:106)
2015-12-11_09:41:27.25325 at
org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:103)
2015-12-11_09:41:27.25326 at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
2015-12-11_09:41:27.25326 at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
2015-12-11_09:41:27.25326 at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
2015-12-11_09:41:27.25326 at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
2015-12-11_09:41:27.25327 at
com.google.common.cache.LocalCache.get(LocalCache.java:3937)
2015-12-11_09:41:27.25327 at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
2015-12-11_09:41:27.25327 at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
2015-12-11_09:41:27.25328 at
org.graylog2.indexer.ranges.EsIndexRangeService.get(EsIndexRangeService.java:130)
2015-12-11_09:41:27.25328 at
org.graylog2.indexer.IndexHelper.determineAffectedIndicesWithRanges(IndexHelper.java:119)
2015-12-11_09:41:27.25329 at
org.graylog2.indexer.searches.Searches.search(Searches.java:241)
2015-12-11_09:41:27.25329 at
org.graylog2.rest.resources.search.RelativeSearchResource.searchRelative(RelativeSearchResource.java:103)
2015-12-11_09:41:35.42510 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
2015-12-11_09:41:35.42645 WARN [IndexHelper] Couldn't find latest deflector
target index
2015-12-11_09:41:35.78407 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
2015-12-11_09:42:23.12058 WARN [EsIndexRangeService] Couldn't load index range
for index graylog_204
2015-12-11_09:42:23.12507 WARN [IndexHelper] Couldn't find latest deflector
target index
2015-12-11_09:42:23.12509 org.graylog2.database.NotFoundException: Couldn't
find index range for index graylog_204
2015-12-11_09:42:23.12509 at
org.graylog2.indexer.ranges.EsIndexRangeService.loadIndexRange(EsIndexRangeService.java:155)
2015-12-11_10:00:44.90915 INFO [SystemJobManager] Submitted SystemJob
<0b637b50-9fee-11e5-bcf7-005056b04c73>
[org.graylog2.indexer.ranges.RebuildIndexRangesJob]
2015-12-11_10:00:44.91117 INFO [RebuildIndexRangesJob] Re-calculating index
ranges.
2015-12-11_10:00:45.83204 INFO [EsIndexRangeService] Calculated range of
[graylog_202] in [597ms].
2015-12-11_10:00:54.19679 INFO [EsIndexRangeService] Calculated range of
[graylog_203] in [763ms].
2015-12-11_10:00:55.81696 INFO [EsIndexRangeService] Calculated range of
[graylog_200] in [1431ms].
2015-12-11_10:00:56.68987 INFO [EsIndexRangeService] Calculated range of
[graylog_201] in [395ms].
2015-12-11_10:00:57.64916 INFO [EsIndexRangeService] Calculated range of
[graylog_198] in [303ms].
2015-12-11_10:00:58.77833 INFO [EsIndexRangeService] Calculated range of
[graylog_199] in [1064ms].
2015-12-11_10:00:59.14814 INFO [EsIndexRangeService] Calculated range of
[graylog_196] in [222ms].
2015-12-11_10:01:01.52932 INFO [EsIndexRangeService] Calculated range of
[graylog_197] in [193ms].
2015-12-11_10:01:01.94286 ERROR [EsIndexRangeService] Error while calculating
timestamp stats in index <graylog_204>
2015-12-11_10:01:01.94288
org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to
execute phase [query], all shards failed; shardFailures
{[CxNGmxp7SlGVyffVXlOGSw][graylog_204][0]: RemoteTransportException[[Captain
Barracuda][inet[/192.168.88.25:9300]][indices:data/read/search[phase/query]]];
nested:
ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData
cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData];
}{[CxNGmxp7SlGVyffVXlOGSw][graylog_204][1]: RemoteTransportException[[Captain
Barracuda][inet[/192.168.88.25:9300]][indices:data/read/search[phase/query]]];
nested:
ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData
cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData];
}{[RMZ7lw18Tv-HHYU5aHJIoQ][graylog_204][2]: RemoteTransportException[[Diamanda
Nero][inet[/192.168.88.24:9300]][indices:data/read/search[phase/query]]];
nested:
ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData
cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData];
}{[RMZ7lw18Tv-HHYU5aHJIoQ][graylog_204][3]: RemoteTransportException[[Diamanda
Nero][inet[/192.168.88.24:9300]][indices:data/read/search[phase/query]]];
nested:
ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData
cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData]; }
any ideas?
tks
--
pica
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/20151211104414.GA42961%40grummit.biaix.org.
For more options, visit https://groups.google.com/d/optout.
