Hi, Is there any more info I could have supplied to make this easier? To recap: graylog somehow managed to loose an index and needed manual intervention to avoid further message loss.
tks -- joan * Joan Picanyol i Puig <[email protected]> [20151211 11:44]: > Hi there, > > We have two nodes running graylog-server (v 1.2.0) and ElasticSearch in both > of > them and I'm trying to figure out what happened that forced us to manually > cycle the deflector in order to see our messages again. We are processing a > few > hundred messages per second, running with 3.5GB heap in the ES JVMs & 1.5GB in > the graylog-server JVMs. > > I can see the following (somewhat filtered) in the graylog logs, in the > ElasticSearch logs I only see JVM GC duration WARNings. On a side note, I'd be > great to have some ERROR log yelling about messages being lost (or at least, > nowehere to be found). > > Node 1 > ====== > > 2015-12-11_09:26:53.47853 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > 2015-12-11_09:26:53.91663 INFO [AbstractValidatingSessionManager] Validating > all active sessions... > 2015-12-11_09:26:54.83515 INFO [AbstractValidatingSessionManager] Finished > session validation. No sessions were stopped. > 2015-12-11_09:26:54.99021 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > > [snip many repetead WARNings] > > 2015-12-11_09:32:53.71532 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > 2015-12-11_09:33:38.03343 INFO [SystemJobManager] SystemJob > <e0098452-9fe7-11e5-857c-005056b06219> > [org.graylog2.indexer.indices.jobs.OptimizeIndexJob] finished in 1023067ms. > 2015-12-11_09:33:53.23876 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > 2015-12-11_09:33:53.34890 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > > [snip many repetead WARNings] > > 2015-12-11_09:42:06.48387 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > 2015-12-11_09:42:06.49119 WARN [IndexHelper] Couldn't find latest deflector > target index > 2015-12-11_09:42:06.49120 org.graylog2.database.NotFoundException: Couldn't > find index range for index graylog_204 > 2015-12-11_09:42:06.49121 at > org.graylog2.indexer.ranges.EsIndexRangeService.loadIndexRange(EsIndexRangeService.java:155) > 2015-12-11_09:42:06.49121 at > org.graylog2.indexer.ranges.EsIndexRangeService.access$000(EsIndexRangeService.java:78) > 2015-12-11_09:42:06.49121 at > org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:106) > 2015-12-11_09:42:06.49122 at > org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:103) > 2015-12-11_09:42:06.49122 at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) > 2015-12-11_09:42:06.49122 at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) > 2015-12-11_09:42:06.49122 at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > 2015-12-11_09:42:06.49123 at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > 2015-12-11_09:42:06.49123 at > com.google.common.cache.LocalCache.get(LocalCache.java:3937) > 2015-12-11_09:42:06.49123 at > com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > 2015-12-11_09:42:06.49123 at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > 2015-12-11_09:42:06.49124 at > org.graylog2.indexer.ranges.EsIndexRangeService.get(EsIndexRangeService.java:130) > 2015-12-11_09:42:06.49124 at > org.graylog2.indexer.IndexHelper.determineAffectedIndicesWithRanges(IndexHelper.java:119) > 2015-12-11_09:42:06.49125 at > org.graylog2.indexer.searches.Searches.search(Searches.java:241) > 2015-12-11_09:42:06.49125 at > org.graylog2.rest.resources.search.RelativeSearchResource.searchRelative(RelativeSearchResource.java:103) > > [snip many repetead WARNings] > > 2015-12-11_09:42:53.88667 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > 2015-12-11_09:43:05.75814 WARN [jvm] [graylog2-server] > [gc][young][331369][11365] duration [1s], collections [1]/[1.5s], total > [1s]/[2.1h], memory [1.2gb]->[777.9mb]/[1.3gb], all_pools {[young] > [583mb]->[12.5mb]/[600mb]}{[survivor] [57.6mb]->[68.6mb]/[75mb]}{[old] > [684.8mb]->[696.9mb]/[750mb]} > 2015-12-11_09:43:19.87094 WARN [NodePingThread] Did not find meta info of > this node. Re-registering. > 2015-12-11_09:43:53.24549 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > > > > Node 2 > ====== > > 2015-12-11_09:26:50.41251 INFO [AbstractValidatingSessionManager] Validating > all active sessions... > 2015-12-11_09:26:51.07435 INFO [AbstractValidatingSessionManager] Finished > session validation. No sessions were stopped. > 2015-12-11_09:28:17.94562 WARN [NodePingThread] Did not find meta info of > this node. Re-registering. > 2015-12-11_09:28:18.14976 WARN [jvm] [graylog2-server] > [gc][young][345393][18320] duration [2.5s], collections [1]/[3.5s], total > [2.5s]/[12.2m], memory [1.1gb]->[685mb]/[1.3gb], all_pools {[young] > [539.7mb]->[14.4mb]/[600mb]}{[survivor] [47.3mb]->[66.2mb]/[75mb]}{[old] > [598.2mb]->[604.4mb]/[750mb]} > 2015-12-11_09:29:03.96581 WARN [jvm] [graylog2-server] > [gc][young][345436][18326] duration [3.6s], collections [1]/[4s], total > [3.6s]/[12.3m], memory [1.2gb]->[711.2mb]/[1.3gb], all_pools {[young] > [570.4mb]->[14.3kb]/[600mb]}{[survivor] [53.6mb]->[67.3mb]/[75mb]}{[old] > [638mb]->[643.9mb]/[750mb]} > 2015-12-11_09:29:03.97050 WARN [NodePingThread] Did not find meta info of > this node. Re-registering. > 2015-12-11_09:31:29.53938 INFO [jvm] [graylog2-server] > [gc][young][345575][18351] duration [987ms], collections [1]/[1.4s], total > [987ms]/[12.4m], memory [1gb]->[573.2mb]/[1.3gb], all_pools {[young] > [538.2mb]->[4.1kb]/[600mb]}{[survivor] [69.6mb]->[72.4mb]/[75mb]}{[old] > [492.4mb]->[500.7mb]/[750mb]} > 2015-12-11_09:33:04.55657 WARN [jvm] [graylog2-server] > [gc][young][345668][18367] duration [2.4s], collections [1]/[2.9s], total > [2.4s]/[12.4m], memory [1.1gb]->[694.3mb]/[1.3gb], all_pools {[young] > [536.6mb]->[612.6kb]/[600mb]}{[survivor] [44.2mb]->[73.7mb]/[75mb]}{[old] > [612.5mb]->[620mb]/[750mb]} > 2015-12-11_09:33:04.69209 WARN [NodePingThread] Did not find meta info of > this node. Re-registering. > 2015-12-11_09:38:33.23279 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > 2015-12-11_09:38:33.25105 WARN [IndexHelper] Couldn't find latest deflector > target index > 2015-12-11_09:38:33.25107 org.graylog2.database.NotFoundException: Couldn't > find index range for index graylog_204 > 2015-12-11_09:38:33.25107 at > org.graylog2.indexer.ranges.EsIndexRangeService.loadIndexRange(EsIndexRangeService.java:155) > 2015-12-11_09:38:33.25107 at > org.graylog2.indexer.ranges.EsIndexRangeService.access$000(EsIndexRangeService.java:78) > 2015-12-11_09:38:33.25108 at > org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:106) > 2015-12-11_09:38:33.25108 at > org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:103) > 2015-12-11_09:38:33.25108 at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) > 2015-12-11_09:38:33.25109 at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) > 2015-12-11_09:38:33.25109 at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > 2015-12-11_09:38:33.25109 at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > 2015-12-11_09:38:33.25110 at > com.google.common.cache.LocalCache.get(LocalCache.java:3937) > 2015-12-11_09:38:33.25110 at > com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > 2015-12-11_09:38:33.25111 at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > 2015-12-11_09:38:33.25112 at > org.graylog2.indexer.ranges.EsIndexRangeService.get(EsIndexRangeService.java:130) > 2015-12-11_09:38:33.25113 at > org.graylog2.indexer.IndexHelper.determineAffectedIndicesWithRanges(IndexHelper.java:119) > 2015-12-11_09:38:33.25113 at > org.graylog2.indexer.searches.Searches.search(Searches.java:241) > 2015-12-11_09:38:33.25113 at > org.graylog2.rest.resources.search.RelativeSearchResource.searchRelative(RelativeSearchResource.java:103) > > 2015-12-11_09:38:33.28821 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > 2015-12-11_09:41:27.25182 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > 2015-12-11_09:41:27.25322 WARN [IndexHelper] Couldn't find latest deflector > target index > 2015-12-11_09:41:27.25323 org.graylog2.database.NotFoundException: Couldn't > find index range for index graylog_204 > 2015-12-11_09:41:27.25323 at > org.graylog2.indexer.ranges.EsIndexRangeService.loadIndexRange(EsIndexRangeService.java:155) > 2015-12-11_09:41:27.25324 at > org.graylog2.indexer.ranges.EsIndexRangeService.access$000(EsIndexRangeService.java:78) > 2015-12-11_09:41:27.25324 at > org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:106) > 2015-12-11_09:41:27.25325 at > org.graylog2.indexer.ranges.EsIndexRangeService$1.load(EsIndexRangeService.java:103) > 2015-12-11_09:41:27.25326 at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) > 2015-12-11_09:41:27.25326 at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) > 2015-12-11_09:41:27.25326 at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > 2015-12-11_09:41:27.25326 at > com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > 2015-12-11_09:41:27.25327 at > com.google.common.cache.LocalCache.get(LocalCache.java:3937) > 2015-12-11_09:41:27.25327 at > com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > 2015-12-11_09:41:27.25327 at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > 2015-12-11_09:41:27.25328 at > org.graylog2.indexer.ranges.EsIndexRangeService.get(EsIndexRangeService.java:130) > 2015-12-11_09:41:27.25328 at > org.graylog2.indexer.IndexHelper.determineAffectedIndicesWithRanges(IndexHelper.java:119) > 2015-12-11_09:41:27.25329 at > org.graylog2.indexer.searches.Searches.search(Searches.java:241) > 2015-12-11_09:41:27.25329 at > org.graylog2.rest.resources.search.RelativeSearchResource.searchRelative(RelativeSearchResource.java:103) > > 2015-12-11_09:41:35.42510 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > 2015-12-11_09:41:35.42645 WARN [IndexHelper] Couldn't find latest deflector > target index > > 2015-12-11_09:41:35.78407 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > 2015-12-11_09:42:23.12058 WARN [EsIndexRangeService] Couldn't load index > range for index graylog_204 > 2015-12-11_09:42:23.12507 WARN [IndexHelper] Couldn't find latest deflector > target index > 2015-12-11_09:42:23.12509 org.graylog2.database.NotFoundException: Couldn't > find index range for index graylog_204 > 2015-12-11_09:42:23.12509 at > org.graylog2.indexer.ranges.EsIndexRangeService.loadIndexRange(EsIndexRangeService.java:155) > > 2015-12-11_10:00:44.90915 INFO [SystemJobManager] Submitted SystemJob > <0b637b50-9fee-11e5-bcf7-005056b04c73> > [org.graylog2.indexer.ranges.RebuildIndexRangesJob] > 2015-12-11_10:00:44.91117 INFO [RebuildIndexRangesJob] Re-calculating index > ranges. > 2015-12-11_10:00:45.83204 INFO [EsIndexRangeService] Calculated range of > [graylog_202] in [597ms]. > 2015-12-11_10:00:54.19679 INFO [EsIndexRangeService] Calculated range of > [graylog_203] in [763ms]. > 2015-12-11_10:00:55.81696 INFO [EsIndexRangeService] Calculated range of > [graylog_200] in [1431ms]. > 2015-12-11_10:00:56.68987 INFO [EsIndexRangeService] Calculated range of > [graylog_201] in [395ms]. > 2015-12-11_10:00:57.64916 INFO [EsIndexRangeService] Calculated range of > [graylog_198] in [303ms]. > 2015-12-11_10:00:58.77833 INFO [EsIndexRangeService] Calculated range of > [graylog_199] in [1064ms]. > 2015-12-11_10:00:59.14814 INFO [EsIndexRangeService] Calculated range of > [graylog_196] in [222ms]. > 2015-12-11_10:01:01.52932 INFO [EsIndexRangeService] Calculated range of > [graylog_197] in [193ms]. > 2015-12-11_10:01:01.94286 ERROR [EsIndexRangeService] Error while calculating > timestamp stats in index <graylog_204> > 2015-12-11_10:01:01.94288 > org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to > execute phase [query], all shards failed; shardFailures > {[CxNGmxp7SlGVyffVXlOGSw][graylog_204][0]: RemoteTransportException[[Captain > Barracuda][inet[/192.168.88.25:9300]][indices:data/read/search[phase/query]]]; > nested: > ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData > cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData]; > }{[CxNGmxp7SlGVyffVXlOGSw][graylog_204][1]: RemoteTransportException[[Captain > Barracuda][inet[/192.168.88.25:9300]][indices:data/read/search[phase/query]]]; > nested: > ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData > cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData]; > }{[RMZ7lw18Tv-HHYU5aHJIoQ][graylog_204][2]: > RemoteTransportException[[Diamanda > Nero][inet[/192.168.88.24:9300]][indices:data/read/search[phase/query]]]; > nested: > ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData > cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData]; > }{[RMZ7lw18Tv-HHYU5aHJIoQ][graylog_204][3]: > RemoteTransportException[[Diamanda > Nero][inet[/192.168.88.24:9300]][indices:data/read/search[phase/query]]]; > nested: > ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData > cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData]; } > > any ideas? > > tks > -- > pica > > -- > You received this message because you are subscribed to the Google Groups > "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/20151211104414.GA42961%40grummit.biaix.org. > For more options, visit https://groups.google.com/d/optout. -- pica -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/20151215112935.GD23059%40grummit.biaix.org. For more options, visit https://groups.google.com/d/optout.
