I also experienced the same problem when running my mapathon today at the European Commission - JRC in Ispra (Italy): https://tasks.hotosm.org/ was down, but for the purposes of the mapathon (introducing humanitarian mapping and introducing participants to mapping) the instance at https://tasks.teachosm.org was totally fine.
This solution might be useful for many other people involved in mapathons during this week! Best, Marco Il giorno mer 13 nov 2019 alle ore 13:32 Felix Delattre < felix.delat...@hotosm.org> ha scritto: > Hi all, > > Yes, this is the worst time. Almost the whole tech team is sole focusing > on that emergency. > > We had earlier today some problems with a maintenance procedure that went > out of control. And at the moment we are facing the following issues on the > Tasking Manager that has been brought back: > > - Locking of tasks is not working and throws an error message. > - Login is failing for some, not for others > - Activities of late yesterday, such as projects that have been created, > might be lost in the database, as we had to use a backup from yesterday. > > We are working on solving it soon. And will report here and in our other > channels (slack, twitter.com/hotosm_tech and status.hotosm.org) more > information. > > In the meantime we can only offer alternative instances such as: > > * tm3.openstreetmap.id > * tasks-assisted.hotosm.org > * tasks.teachosm.org > > Please excuse the inconvenience and we will post any updates as soon as > possible. > > So far an update from the infrastructure experts on what happened earlier > today: > > > *Summary:* > > Tasking Manager is now back and working. Unfortunately, many email > addresses were lost and therefore some users who login will be asked for > their email addresses again. Projects are intact and unaffected > > > > *So what happened?* > > During the scheduled maintenance to upgrade Tasking Manager instance, > one of our automated scripts deleted our database. This is not unexpected. > But the backup we made before deletion was deleted along with the database. > In a nutshell, we lost the production database completely, including > automated backups. We however had a separate offline backup that was taken > manually yesterday. But we had stripped email addresses from this database > backup for privacy reasons. We had to restore email addresses from another > offline backup from 10 days ago to fix it. We are now closely monitoring > the DB and the application for any problems. > > > > *What can you expect in the future?* > > We learned many valuable lessons from this including our approach to > data backups, performance and stability. All our experiences from the > outages of recent past have made us think about how to overhaul our > approach to optimize our infrastructure. We will work to implement > processes to mitigate performance, deployment and stability problems over > the next few weeks. We are also working on a detailed RCA with technical > details of the outage. > We understand that this would have caused quite a bit of frustration and > we apologize for that. You can expect things to improve in the next few > weeks. As always, you can follow us on Twitter ( > https://twitter.com/hotosm_tech) or Slack for updates in addition to our > status page at (https://status.hotosm.org). > > > > On Wed, Nov 13, 2019 at 12:01 PM Jorieke Vyncke <jorieke.vyn...@gmail.com> > wrote: > >> Hi all, >> Yes, please HOT, can you do everything to keep the TM online and stable >> today / this week? >> It's GISday today and there are more than 30 mapathons worldwide and 100+ >> this week for OSMgeoweek. Mapathons are happening in Uganda, Nigeria, >> Amsterdam, Sydney, Rio, etc. >> At the moment I tell people to go to tasks.teachosm.org and MapSwipe, if >> the HOT TM is not working. >> Best regards, >> Jorieke >> >> Op wo 13 nov. 2019 om 11:51 schreef Pete Masters via HOT < >> hot@openstreetmap.org>: >> >>> I'm sure you are all aware, but just in case... this is geo week and >>> there are more than 70 mapathons planned for this week. Maybe updates from >>> the tech team need to be posted across channels on this as it is critical >>> infrastructure for all of these events and people might not be connected to >>> slack / list / facebook etc... >>> >>> Cheers, >>> >>> Pete >>> >>> On Wed, Nov 13, 2019 at 11:38 AM Micheal Yani <michealyan...@gmail.com> >>> wrote: >>> >>>> Hello Tech team, we just want to let you know that the tasking >>>> manager isn't stable yet, It has has worked for 2 hrs and now is on and >>>> off. >>>> _______________________________________________ >>>> HOT mailing list >>>> HOT@openstreetmap.org >>>> https://lists.openstreetmap.org/listinfo/hot >>>> >>> _______________________________________________ >>> HOT mailing list >>> HOT@openstreetmap.org >>> https://lists.openstreetmap.org/listinfo/hot >>> >> _______________________________________________ >> HOT mailing list >> HOT@openstreetmap.org >> https://lists.openstreetmap.org/listinfo/hot >> > > > -- > > *Felix Delattre* > Technical Project Manager > felix.delat...@hotosm.org > > -- > *Humanitarian OpenStreetMap Team* > *Using OpenStreetMap for Humanitarian Response & Economic Development* > web <http://hotosm.org/> | twitter <https://twitter.com/hotosm> | facebook > <https://www.facebook.com/hotosm> | donate <https://donate.hotosm.org/> > _______________________________________________ > HOT mailing list > HOT@openstreetmap.org > https://lists.openstreetmap.org/listinfo/hot >
_______________________________________________ HOT mailing list HOT@openstreetmap.org https://lists.openstreetmap.org/listinfo/hot