Hi, you first have to fix the cluster health state of your Elasticsearch cluster before you should even think about deleting the Graylog disk journal.
Check the Elasticsearch logs for corresponding hints: http://docs.graylog.org/en/2.1/pages/configuration/file_location.html#omnibus-package Cheers, Jochen On Friday, 30 December 2016 08:01:20 UTC+1, cyph...@gmail.com wrote: > > Thank you again, we're almost there: > > df -m > Filesystem 1M-blocks Used Available Use% Mounted on > udev 1495 1 1495 1% /dev > tmpfs 300 1 300 1% /run > /dev/dm-0 15282 4902 9582 34% / > none 1 0 1 0% /sys/fs/cgroup > none 5 0 5 0% /run/lock > none 1500 0 1500 0% /run/shm > none 100 0 100 0% /run/user > /dev/sda1 236 121 103 55% /boot > /dev/sdb1 100664 8181 87347 9% /var/opt/graylog/data > > > As you predicted we're still getting errors: > > Elasticsearch cluster unhealthy (RED) > The Elasticsearch cluster state is RED which means shards are unassigned. > This usually indicates a crashed and corrupt cluster and needs to be > investigated. Graylog will write into the local disk journal. Read how to > fix this in the Elasticsearch setup documentation. > <http://docs.graylog.org/en/2.1/pages/configuration/elasticsearch.html#cluster-status-explained> > > I looked at the above provided link, but don't know how to delete the > journal, any help with this last step would be appreciated. > > > On Wednesday, December 28, 2016 at 4:59:35 PM UTC+1, Edmundo Alvarez wrote: > >> This documentation page covers how to extend the disk space in the OVA: >> http://docs.graylog.org/en/2.1/pages/configuration/graylog_ctl.html#extend-disk-space >> >> >> Please note that Graylog's journal is sometimes corrupted when it ran out >> of disk space. In that case you may need to delete the journal folder. >> >> Regards, >> Edmundo >> >> > On 28 Dec 2016, at 16:04, cyph...@gmail.com wrote: >> > >> > Thank you Edmundo. >> > >> > It appears we ran out of space. >> > >> > df -h >> > Filesystem Size Used Avail Use% Mounted on >> > udev 1.5G 4.0K 1.5G 1% /dev >> > tmpfs 300M 388K 300M 1% /run >> > /dev/dm-0 15G 15G 0 100% / >> > none 4.0K 0 4.0K 0% /sys/fs/cgroup >> > none 5.0M 0 5.0M 0% /run/lock >> > none 1.5G 0 1.5G 0% /run/shm >> > none 100M 0 100M 0% /run/user >> > /dev/sda1 236M 121M 103M 55% /boot >> > >> > We don't mind loosing all the history, we just want the server up and >> running. If the space available can be extended even better (keep in mind >> this is OVA). Any suggestions? >> > >> > On Wednesday, December 28, 2016 at 9:18:24 AM UTC+1, Edmundo Alvarez >> wrote: >> > Hello, >> > >> > I would start by looking into your logs in /var/log/graylog, specially >> those in the "server" folder, which may give you some errors to start >> debugging the issue. >> > >> > Hope that helps. >> > >> > Regards, >> > Edmundo >> > >> > > On 27 Dec 2016, at 20:55, cyph...@gmail.com wrote: >> > > >> > > We've been using Graylog OVA 2.1 for a while now, but it stopped >> working all of the sudden. >> > > >> > > We're getting: >> > > >> > > Server currently unavailable >> > > We are experiencing problems connecting to the Graylog server running >> on https://graylog:443/api. Please verify that the server is healthy and >> working correctly. >> > > You will be automatically redirected to the previous page once we can >> connect to the server. >> > > Do you need a hand? We can help you. >> > > Less details >> > > This is the last response we received from the server: >> > > Error message >> > > cannot GET https://graylog:443/api/system/cluster/node (500) >> > > >> > > >> > > ubuntu@graylog:~$ sudo graylog-ctl status >> > > run: elasticsearch: (pid 32780) 74s; run: log: (pid 951) 10764s >> > > down: etcd: 0s, normally up, want up; run: log: (pid 934) 10764s >> > > run: graylog-server: (pid 33146) 35s; run: log: (pid 916) 10764s >> > > down: mongodb: 0s, normally up, want up; run: log: (pid 924) 10764s >> > > run: nginx: (pid 32974) 57s; run: log: (pid 914) 10764s >> > > >> > > >> > > How can we begin to troubleshoot the issue, which logs to view...? >> > > >> > > -- >> > > You received this message because you are subscribed to the Google >> Groups "Graylog Users" group. >> > > To unsubscribe from this group and stop receiving emails from it, >> send an email to graylog2+u...@googlegroups.com. >> > > To view this discussion on the web visit >> https://groups.google.com/d/msgid/graylog2/4fb8da46-2e73-42c7-b67d-444c0b801484%40googlegroups.com. >> >> >> > > For more options, visit https://groups.google.com/d/optout. >> > >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Graylog Users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to graylog2+u...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/graylog2/9d79cf3a-b221-4419-b94f-f278ec598fe0%40googlegroups.com. >> >> >> > For more options, visit https://groups.google.com/d/optout. >> >> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/007aaf19-0299-4c1e-a993-519452f741f2%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
