Hi Jochen There wasn't any error that pointed at this explicitly. This is graylog-1.3.4 with ES-1.7 on CentOS7. There has been a general degradation as my (single) test box grew to 5TB in size - I think I'm simply tickling all the edge cases of performance issues and this is the consequence. I do think ES could do with some major improvements - I continually saw graylog reporting it couldn't push data into ES and yet ES reported no problems and showed "green". And yet if I restarted ES (ie stop/start) it immediately came up as "red" with 10,000+ UNASSIGNED shards - so there's a fairly serious bug in there IMHO.
In any case I ended up deleting all the indexes that wouldn't move off "UNASSIGNED" and basically ended up deleting most of my data, so I've thrown the entire thing away (hey, it is a test box :-) and restarted with the Beta graylog-2 with ES-2.3.1 and we'll see how that goes. This test box is almost at an end - new hardware has finally arrived - so I think the root cause of the issues will soon be resolved by actually having a cluster/etc. Thanks Jason On Wed, Apr 13, 2016 at 12:55 AM, Jochen Schalanda <[email protected]> wrote: > Hi Jason, > > what's your Graylog and Elasticsearch configuration? Are there any error > messages in the logs of either Graylog or Elasticsearch? > > Cheers, > Jochen > > > On Monday, 11 April 2016 07:15:17 UTC+2, Jason Haar wrote: >> >> Hi there >> >> Over the weekend I installed the ES marvel diagnostics package and the >> following day noticed that graylog was broken. Restarting ES showed 20,000 >> shards in an UNASSIGNED state. I disabled graylog-server (so there was no >> new data flowing in) and watched over the next couple of hours as that >> UNASSIGNED number dropped down to 0 (and GREEN). I then restarted ES and - >> bam - back to 20,000 UNASSIGNED shards again >> >> I've now done three iterations of that - looks like it's completely >> borked. There's over 5TB of data in there - how can I regain it? >> >> PS: I don't know if this has anything to do with marvel - it's just the >> last change I made. The reason I installed it was because I have had ES >> continually doing this kind of thing - but previously stopping graylog, >> restarting ES and waiting would lead to a happy ES - but no longer. >> >> -- >> Cheers >> >> Jason Haar >> Information Security Manager, Trimble Navigation Ltd. >> Phone: +1 408 481 8171 >> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 >> > -- > You received this message because you are subscribed to the Google Groups > "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/507dbfa3-0239-45ca-9799-2cae7db65418%40googlegroups.com > <https://groups.google.com/d/msgid/graylog2/507dbfa3-0239-45ca-9799-2cae7db65418%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrg%2BL5zOhKu%2BHLJVtQjpZyjSQTe8HtsmS_%3DDJyBU%2B44UpTg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
