On Mon, 24 Jul 2017 15:25:14 +0000 Andrew Williams <a...@andywilliams.me> said:
> It seems to be back now without a kick... Is there stuff going on at the > moment? i don't know. when i saw your mail i checked. all was up... i moved on. i have disabled the logrotate daily crons in e5v1 and www/git ... because because the going down suspiciously coincided with these. i've disabled auditing in libvirt too... i just want to get things working and then figure out which level i pulled did it. maybe one of the levers i pulled di it? e5v1, www, have been up over 1 whole day so not one rebooted them. 35 has now been up 2 days (last time there was an issue i restarted libvirt and its guests only and things worked again). > On Mon, 24 Jul 2017 at 15:30 Andrew Williams <a...@andywilliams.me> wrote: > > > Today it died between 1 and 2pm UTC... > > On Sat, 22 Jul 2017 at 16:26, Carsten Haitzler <ras...@rasterman.com> > > wrote: > > > >> On Sat, 22 Jul 2017 15:20:48 +0000 Andrew Williams <a...@andywilliams.me> > >> said: > >> > >> > Thanks to whoever kicked the server just now :) > >> > >> me :) > >> > >> > Are we any closer to what might be triggering this as soon as the > >> weekend > >> > starts? ;) > >> > >> no. i've experimented with disabling cron.daily things on e5... it does > >> seem > >> that now shutting down vm's and bringing them back up works now kvm is a > >> module... ? hmmm. > >> > >> > Andy > >> > > >> > On Wed, 19 Jul 2017 at 18:47 Andrew Williams <a...@andywilliams.me> > >> wrote: > >> > > >> > > It looks like the new key did work after all - but in the confiscation > >> > > about ensuring no proxies were set up I accidentally deleted the line > >> of > >> > > config that sets the username. > >> > > Either way I can now long in, thanks. > >> > > Now at least I can kick the server if/when it's not accessible this > >> > > weekend. > >> > > > >> > > Andy > >> > > > >> > > On Tue, 18 Jul 2017 at 17:14 Andrew Williams <a...@andywilliams.me> > >> wrote: > >> > > > >> > >> I have upgrade my key to a 4096 and using this new one I can use git > >> over > >> > >> ssh but cannot access e5 through ssh - same as with my old key. > >> > >> > >> > >> Andy > >> > >> > >> > >> On Tue, 18 Jul 2017 at 13:39 Andrew Williams <a...@andywilliams.me> > >> > >> wrote: > >> > >> > >> > >>> It's E5 that won't let me in (no proxy of course) > >> > >>> > >> > >>> On Tue, 18 Jul 2017 at 10:48, Carsten Haitzler < > >> ras...@rasterman.com> > >> > >>> wrote: > >> > >>> > >> > >>>> On Tue, 18 Jul 2017 08:03:04 +0000 Andrew Williams < > >> > >>>> a...@andywilliams.me> said: > >> > >>>> > >> > >>>> > As of Monday morning git was working well again (both https and > >> > >>>> git+ssh > >> > >>>> > working fine). > >> > >>>> > I still do not have ssh access back - one machine noted the > >> following: > >> > >>>> > > >> > >>>> > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > >> > >>>> > @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ > >> > >>>> > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > >> > >>>> > IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! > >> > >>>> > > >> > >>>> > If I accept the new identity I get "Permission denied > >> (publickey)." > >> > >>>> > Same key as I've been using for a year or 3 (2048 bits) > >> > >>>> > >> > >>>> mine's 4096 bits... and it works. i believe simotek moved to 4096 > >> too > >> > >>>> and then > >> > >>>> it worked. what machine is that? > >> > >>>> > >> > >>>> > Thanks, > >> > >>>> > Andy > >> > >>>> > > >> > >>>> > On Tue, 18 Jul 2017 at 08:30 Carsten Haitzler < > >> ras...@rasterman.com> > >> > >>>> wrote: > >> > >>>> > > >> > >>>> > > On Tue, 18 Jul 2017 07:16:30 +0000 Andrew Williams < > >> > >>>> a...@andywilliams.me> > >> > >>>> > > said: > >> > >>>> > > > >> > >>>> > > > These reports were related to the weekend when access was > >> very > >> > >>>> difficult > >> > >>>> > > > indeed. Whilst I'm thrilled that everything is up now I don't > >> > >>>> think that > >> > >>>> > > > should invalidate the fact we had a lot of issues on > >> Saturday and > >> > >>>> Sunday. > >> > >>>> > > > >> > >>>> > > ssh/git never stopped working for me... so i don't have any > >> idea > >> > >>>> why. is it > >> > >>>> > > working for you now? > >> > >>>> > > > >> > >>>> > > > Andy > >> > >>>> > > > On Tue, 18 Jul 2017 at 06:07, Carsten Haitzler < > >> > >>>> ras...@rasterman.com> > >> > >>>> > > wrote: > >> > >>>> > > > > >> > >>>> > > > > On Mon, 17 Jul 2017 12:27:05 +0000 Andrew Williams < > >> > >>>> > > a...@andywilliams.me> > >> > >>>> > > > > said: > >> > >>>> > > > > > >> > >>>> > > > > hmm well the server isn't down... it's up and working, www > >> is > >> > >>>> working. > >> > >>>> > > ssh > >> > >>>> > > > > works for me... i currently don't know why you are having > >> ssh > >> > >>>> > > issues... :/ > >> > >>>> > > > > > >> > >>>> > > > > > No, I have no proxy commands set up. I have no > >> peculiarity in > >> > >>>> > > networking. > >> > >>>> > > > > > The physical location and method of connecting were the > >> same > >> > >>>> as I > >> > >>>> > > have > >> > >>>> > > > > been > >> > >>>> > > > > > using for months. The connection was unreliable at best > >> and > >> > >>>> got > >> > >>>> > > worse as > >> > >>>> > > > > > Saturday wore on. > >> > >>>> > > > > > > >> > >>>> > > > > > Some times I could reach the server and connect. Some > >> times I > >> > >>>> got > >> > >>>> > > > > rejected. > >> > >>>> > > > > > Other times I got no response. > >> > >>>> > > > > > > >> > >>>> > > > > > Additionally I was not the only one. Those on IRC were > >> > >>>> agreeing that > >> > >>>> > > > > > connection was unreliable at best. > >> > >>>> > > > > > > >> > >>>> > > > > > Andy > >> > >>>> > > > > > > >> > >>>> > > > > > On Mon, 17 Jul 2017 at 08:33 Carsten Haitzler < > >> > >>>> ras...@rasterman.com> > >> > >>>> > > > > wrote: > >> > >>>> > > > > > > >> > >>>> > > > > > > On Sat, 15 Jul 2017 23:32:25 +0000 Andrew Williams < > >> > >>>> > > > > a...@andywilliams.me> > >> > >>>> > > > > > > said: > >> > >>>> > > > > > > > >> > >>>> > > > > > > > Hi, > >> > >>>> > > > > > > > > >> > >>>> > > > > > > > After an hour of trying to get through to our git > >> server > >> > >>>> I'm now > >> > >>>> > > > > giving > >> > >>>> > > > > > > up > >> > >>>> > > > > > > > for the night. > >> > >>>> > > > > > > > I understand that there is an unknown problem that > >> needs > >> > >>>> to be > >> > >>>> > > > > gotten to > >> > >>>> > > > > > > > the bottom of but I don't understand why those who > >> were > >> > >>>> able to > >> > >>>> > > help > >> > >>>> > > > > no > >> > >>>> > > > > > > > longer have access. > >> > >>>> > > > > > > > > >> > >>>> > > > > > > > Can someone in the know please communicate what is > >> going > >> > >>>> on? I > >> > >>>> > > know > >> > >>>> > > > > that > >> > >>>> > > > > > > we > >> > >>>> > > > > > > > are getting our hosting provided kindly for free but > >> > >>>> There are so > >> > >>>> > > > > many > >> > >>>> > > > > > > free > >> > >>>> > > > > > > > hosting solutions for open source projects I don't > >> get it > >> > >>>> - we > >> > >>>> > > were > >> > >>>> > > > > > > afraid > >> > >>>> > > > > > > > of the stability of "the new sourceforge" but GitHub, > >> > >>>> despite > >> > >>>> > > high > >> > >>>> > > > > > > > criticism for its downtime, has been unavailable for > >> > >>>> barely 2 > >> > >>>> > > days > >> > >>>> > > > > over > >> > >>>> > > > > > > the > >> > >>>> > > > > > > > last 2 years... > >> > >>>> > > > > > > > >> > >>>> > > > > > > do you have a proxy thing for git.e.org in your ssh > >> > >>>> config. i can > >> > >>>> > > get > >> > >>>> > > > > to > >> > >>>> > > > > > > git.e.org no problems. all weekend. i didn't have > >> issues. > >> > >>>> my > >> > >>>> > > email was > >> > >>>> > > > > > > down > >> > >>>> > > > > > > though (separate issue)... > >> > >>>> > > > > > > > >> > >>>> > > > > > > > Andy > >> > >>>> > > > > > > > On Sat, 15 Jul 2017 at 22:41, jaquilina < > >> > >>>> jaquil...@eagleeyet.net > >> > >>>> > > > > >> > >>>> > > > > wrote: > >> > >>>> > > > > > > > > >> > >>>> > > > > > > > > All I know is that beber adding my ssh key to the > >> > >>>> system and > >> > >>>> > > > > > > reencrypted > >> > >>>> > > > > > > > > the credentials store which could be the root > >> cause of > >> > >>>> these > >> > >>>> > > issues > >> > >>>> > > > > > > with > >> > >>>> > > > > > > > > git over ssh > >> > >>>> > > > > > > > > > >> > >>>> > > > > > > > > > >> > >>>> > > > > > > > > > >> > >>>> > > > > > > > > Sent from Samsung tablet. > >> > >>>> > > > > > > > > -------- Original message -------- > >> > >>>> > > > > > > > > From: Andrew Williams <a...@andywilliams.me> > >> > >>>> > > > > > > > > Date: 15/07/2017 22:59 (GMT+01:00) > >> > >>>> > > > > > > > > To: Jonathan Aquilina <jaquil...@eagleeyet.net>, > >> > >>>> Enlightenment > >> > >>>> > > > > > > developer > >> > >>>> > > > > > > > > list <enlightenment-devel@lists.sourceforge.net> > >> > >>>> > > > > > > > > Subject: Re: [E-devel] Server down > >> > >>>> > > > > > > > > > >> > >>>> > > > > > > > > Correction, for git I'm getting either connection > >> > >>>> closed or > >> > >>>> > > > > permission > >> > >>>> > > > > > > > > denied... every 5th or so connection makes it > >> through. > >> > >>>> > > > > > > > > On Sat, 15 Jul 2017 at 20:46, Andrew Williams < > >> > >>>> > > > > a...@andywilliams.me> > >> > >>>> > > > > > > > > wrote: > >> > >>>> > > > > > > > > > >> > >>>> > > > > > > > >> Hi, > >> > >>>> > > > > > > > >> > >> > >>>> > > > > > > > >> Thanks to whoever prodded it. Now git and web are > >> > >>>> working > >> > >>>> > > again. > >> > >>>> > > > > > > > >> I use git+ssh and it is working OK (on and off > >> > >>>> actually) but I > >> > >>>> > > > > can't > >> > >>>> > > > > > > get > >> > >>>> > > > > > > > >> in through ssh any more. Bu5hm4n and others have > >> noted > >> > >>>> the > >> > >>>> > > same > >> > >>>> > > > > > > thing. Have > >> > >>>> > > > > > > > >> permissions on SSH been altered? > >> > >>>> > > > > > > > >> > >> > >>>> > > > > > > > >> Thanks, > >> > >>>> > > > > > > > >> Andy > >> > >>>> > > > > > > > >> > >> > >>>> > > > > > > > >> On Sat, 15 Jul 2017 at 15:06 Jonathan Aquilina < > >> > >>>> > > > > > > jaquil...@eagleeyet.net> > >> > >>>> > > > > > > > >> wrote: > >> > >>>> > > > > > > > >> > >> > >>>> > > > > > > > >>> I think that it might be worth contacting the > >> hosting > >> > >>>> > > provider > >> > >>>> > > > > as I > >> > >>>> > > > > > > am > >> > >>>> > > > > > > > >>> wondering if they are doing something with their > >> > >>>> network > >> > >>>> > > > > hardware or > >> > >>>> > > > > > > maybe > >> > >>>> > > > > > > > >>> something that is causing our server to really > >> not be > >> > >>>> happy. > >> > >>>> > > > > > > > >>> --- > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> Regards, > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> Jonathan Aquilina > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> On 2017-07-15 15:59, Derek Foreman wrote: > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> Somewhat related, I've been monitoring the > >> server with > >> > >>>> > > > > "smokeping" > >> > >>>> > > > > > > for a > >> > >>>> > > > > > > > >>> few weeks now, and the majority of the blackouts > >> start > >> > >>>> > > between > >> > >>>> > > > > 5AM > >> > >>>> > > > > > > and > >> > >>>> > > > > > > > >>> 5:10AM north american central time (10ish UTC I > >> > >>>> guess?) > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> Sometimes they clear in under an hour (I don't > >> know > >> > >>>> if this > >> > >>>> > > is an > >> > >>>> > > > > > > > >>> automatic recovery or manual intervention). > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> During the blackouts generally the git server is > >> > >>>> partially > >> > >>>> > > > > responsive > >> > >>>> > > > > > > > >>> and the web server is completely borked. > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> Today's blackout started between 5:00AM and > >> 5:10AM > >> > >>>> CST. > >> > >>>> > > > > Currently > >> > >>>> > > > > > > the > >> > >>>> > > > > > > > >>> git server is semi responsive and the web server > >> is > >> > >>>> hosed. > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> I don't know if that gives any useful input into > >> > >>>> tracking > >> > >>>> > > down > >> > >>>> > > > > what's > >> > >>>> > > > > > > > >>> knocking over the house of cards (or if knowing > >> what's > >> > >>>> > > knocking > >> > >>>> > > > > it > >> > >>>> > > > > > > over is > >> > >>>> > > > > > > > >>> even useful), but there ya go. :D > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> Thanks, > >> > >>>> > > > > > > > >>> Derek > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> On 2017-07-15 05:26 AM, Andrew Williams wrote: > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> Hi, > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> That's the server keeled over again. I would fix > >> it > >> > >>>> but today > >> > >>>> > > > > it's > >> > >>>> > > > > > > > >>> rejecting my login. > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> Can someone please help, I was in the middle of a > >> > >>>> bunch of > >> > >>>> > > work > >> > >>>> > > > > on > >> > >>>> > > > > > > > >>> docuwiki that I would like to complete? > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> Thanks, > >> > >>>> > > > > > > > >>> Andy > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >> > >>>> > > > > > >> > >>>> > > > >> > >>>> > >> ------------------------------------------------------------------------------ > >> > >>>> > > > > > > > >>> Check out the vibrant tech community on one of > >> the > >> > >>>> world's > >> > >>>> > > most > >> > >>>> > > > > > > > >>> engaging tech sites, Slashdot.org! > >> > >>>> http://sdm.link/slashdot > >> > >>>> > > > > > > > >>> _______________________________________________ > >> > >>>> > > > > > > > >>> enlightenment-devel mailing list > >> > >>>> > > > > > > > >>> enlightenment-devel@lists.sourceforge.net > >> > >>>> > > > > > > > >>> > >> > >>>> > > > >> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > >> > >>>> > > > > > > > >>> > >> > >>>> > > > > > > > >>> -- > >> > >>>> > > > > > > > >> http://andywilliams.me > >> > >>>> > > > > > > > >> http://ajwillia.ms > >> > >>>> > > > > > > > >> > >> > >>>> > > > > > > > > -- > >> > >>>> > > > > > > > > http://andywilliams.me > >> > >>>> > > > > > > > > http://ajwillia.ms > >> > >>>> > > > > > > > > > >> > >>>> > > > > > > > -- > >> > >>>> > > > > > > > http://andywilliams.me > >> > >>>> > > > > > > > http://ajwillia.ms > >> > >>>> > > > > > > > > >> > >>>> > > > > > > > >> > >>>> > > > > > >> > >>>> > > > >> > >>>> > >> ------------------------------------------------------------------------------ > >> > >>>> > > > > > > > Check out the vibrant tech community on one of the > >> > >>>> world's most > >> > >>>> > > > > > > > engaging tech sites, Slashdot.org! > >> > >>>> http://sdm.link/slashdot > >> > >>>> > > > > > > > _______________________________________________ > >> > >>>> > > > > > > > enlightenment-devel mailing list > >> > >>>> > > > > > > > enlightenment-devel@lists.sourceforge.net > >> > >>>> > > > > > > > > >> > >>>> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > >> > >>>> > > > > > > > > >> > >>>> > > > > > > > >> > >>>> > > > > > > > >> > >>>> > > > > > > -- > >> > >>>> > > > > > > ------------- Codito, ergo sum - "I code, therefore I > >> am" > >> > >>>> > > > > -------------- > >> > >>>> > > > > > > The Rasterman (Carsten Haitzler) > >> ras...@rasterman.com > >> > >>>> > > > > > > > >> > >>>> > > > > > > -- > >> > >>>> > > > > > http://andywilliams.me > >> > >>>> > > > > > http://ajwillia.ms > >> > >>>> > > > > > > >> > >>>> > > > > > >> > >>>> > > > >> > >>>> > >> ------------------------------------------------------------------------------ > >> > >>>> > > > > > Check out the vibrant tech community on one of the > >> world's > >> > >>>> most > >> > >>>> > > > > > engaging tech sites, Slashdot.org! > >> http://sdm.link/slashdot > >> > >>>> > > > > > _______________________________________________ > >> > >>>> > > > > > enlightenment-devel mailing list > >> > >>>> > > > > > enlightenment-devel@lists.sourceforge.net > >> > >>>> > > > > > > >> > >>>> https://lists.sourceforge.net/lists/listinfo/enlightenment-devel > >> > >>>> > > > > > > >> > >>>> > > > > > >> > >>>> > > > > > >> > >>>> > > > > -- > >> > >>>> > > > > ------------- Codito, ergo sum - "I code, therefore I am" > >> > >>>> > > -------------- > >> > >>>> > > > > The Rasterman (Carsten Haitzler) ras...@rasterman.com > >> > >>>> > > > > > >> > >>>> > > > > -- > >> > >>>> > > > http://andywilliams.me > >> > >>>> > > > http://ajwillia.ms > >> > >>>> > > > >> > >>>> > > > >> > >>>> > > -- > >> > >>>> > > ------------- Codito, ergo sum - "I code, therefore I am" > >> > >>>> -------------- > >> > >>>> > > The Rasterman (Carsten Haitzler) ras...@rasterman.com > >> > >>>> > > > >> > >>>> > > -- > >> > >>>> > http://andywilliams.me > >> > >>>> > http://ajwillia.ms > >> > >>>> > >> > >>>> > >> > >>>> -- > >> > >>>> ------------- Codito, ergo sum - "I code, therefore I am" > >> -------------- > >> > >>>> The Rasterman (Carsten Haitzler) ras...@rasterman.com > >> > >>>> > >> > >>>> -- > >> > >>> http://andywilliams.me > >> > >>> http://ajwillia.ms > >> > >>> > >> > >> -- > >> > >> http://andywilliams.me > >> > >> http://ajwillia.ms > >> > >> > >> > > -- > >> > > http://andywilliams.me > >> > > http://ajwillia.ms > >> > > > >> > -- > >> > http://andywilliams.me > >> > http://ajwillia.ms > >> > >> > >> -- > >> ------------- Codito, ergo sum - "I code, therefore I am" -------------- > >> The Rasterman (Carsten Haitzler) ras...@rasterman.com > >> > >> -- > > http://andywilliams.me > > http://ajwillia.ms > > > -- > http://andywilliams.me > http://ajwillia.ms -- ------------- Codito, ergo sum - "I code, therefore I am" -------------- The Rasterman (Carsten Haitzler) ras...@rasterman.com ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ enlightenment-devel mailing list enlightenment-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
