Re: [HOT] Tasking manager unstable

[Jorieke Vyncke]

Thanks for the follow up Felix. It seems to work fine now! I'm seeing a lot
of happy mapping people faces on social media. :-)

On Wed, 13 Nov 2019, 18:11 Felix Delattre, 
wrote:

> Hi all,
>
> we are back with the HOT Tasking Manager. We are continuing to monitor the
> application servers, database, and network, and are investigating further
> what went wrong. Again, we apologize for the delays and frustration this
> has caused, especially during GeoWeek.
>
> Felix
>
> On 11/13/19 3:52 PM, Jorieke Vyncke wrote:
>
> Hi all,
> We have just send out the following email to all mapathon organisers, but
> hopefully the HOT Tasking Manager is properly up and running again soon.
> Happy mapping everyone! :-)
> Cheers,
> Jorieke
>
>
>
> Hello Mappers,
>
>
>
> Happy GIS day! Unfortunately many of you have experienced some issues with
> HOT Tasking Manager during mapathons today. We want to apologize for this
> inconvenience, and to share some guidance on what to do if this happens
> during your mapathon.
>
>
>
> *HOT Tasking Manager is experiencing some issues - why?*
>
> HOT Tasking Manager is currently unstable due to a failed maintenance from
> last night. We don’t presently know when this problem will be fixed. The
> HOT tech team is working as fast as possible to fix the problem. To find
> out more about the status, please check:
>
>- Status.hotosm.org : this will tell you if
>the tool is down, not if it is experiencing delays (as it is presently)
>- follow @hotosm_tech on Twitter for updates
>- Join the HOT Slack (http://slack.hotosm.org/) and ask questions in
>the #tasking-manager channel
>
>
>
> *Contingency plans for your mapathon: what are your options if the HOT
> Tasking Manager is not working during your event?*
>
>1. Map directly in OSM! Check out an area you know well and contribute
>your own knowledge. If you’d like to host a map-walk, please see this guide
>on maps.me .
>2. Host a Mapswipeathon. MapSwipe  has several
>tasks up and running on the app. You can help swiping in Indonesia,
>Venezuela, Philippines and Colombia. Please note this is a mobile app only.
>3. There are alternative Tasking manager instances you can use like
>https://tm3.openstreetmap.id/ & tasks.teachosm.org. Please note that
>these will not have the same projects on them, so you’ll need to pick a new
>project.
>
> *Note:* Mapswipe, and the alternative Tasking Managers, are completely
> separate systems that let you contribute to OSM, so they should *not *be
> affected by the outage of the main HOT Tasking Manager.
>
>
>
> Thank you for your understanding, we appreciate your flexibility. We are
> working with the HOT Tech Team to get this fixed as soon as possible.  If
> you have any questions or need help course correcting please reach out.
>
>
>
> Op wo 13 nov. 2019 om 15:20 schreef Marco Minghini <
> marco.minghin...@gmail.com>:
>
>> I also experienced the same problem when running my mapathon today at the
>> European Commission - JRC in Ispra (Italy):
>> https://tasks.hotosm.org/ was down, but for the purposes of the mapathon
>> (introducing humanitarian mapping and introducing participants to mapping)
>> the instance at https://tasks.teachosm.org was totally fine.
>>
>> This solution might be useful for many other people involved in mapathons
>> during this week!
>>
>> Best,
>> Marco
>>
>>
>> Il giorno mer 13 nov 2019 alle ore 13:32 Felix Delattre <
>> felix.delat...@hotosm.org> ha scritto:
>>
>>> Hi all,
>>>
>>> Yes, this is the worst time. Almost the whole tech team is sole focusing
>>> on that emergency.
>>>
>>> We had earlier today some problems with a maintenance procedure that
>>> went out of control. And at the moment we are facing the following issues
>>> on the Tasking Manager that has been brought back:
>>>
>>> - Locking of tasks is not working and throws an error message.
>>> - Login is failing for some, not for others
>>> - Activities of late yesterday, such as projects that have been created,
>>> might be lost in the database, as we had to use a backup from yesterday.
>>>
>>> We are working on solving it soon. And will report here and in our other
>>> channels (slack, twitter.com/hotosm_tech and status.hotosm.org) more
>>> information.
>>>
>>> In the meantime we can only offer alternative instances such as:
>>>
>>> * tm3.openstreetmap.id
>>> * tasks-assisted.hotosm.org
>>> * tasks.teachosm.org
>>>
>>> Please excuse the inconvenience and we will post any updates as soon as
>>> possible.
>>>
>>> So far an update from the infrastructure experts on what happened
>>> earlier today:
>>>
>>> > *Summary:*
>>> > Tasking Manager is now back and working. Unfortunately, many email
>>> addresses were lost and therefore some users who login will be asked for
>>> their email addresses again. Projects are intact and unaffected
>>> >
>>> > *So what happened?*
>>> 

Re: [HOT] Tasking manager unstable

[Felix Delattre]

Hi all,

we are back with the HOT Tasking Manager. We are continuing to monitor
the application servers, database, and network, and are investigating
further what went wrong. Again, we apologize for the delays and
frustration this has caused, especially during GeoWeek.

Felix

On 11/13/19 3:52 PM, Jorieke Vyncke wrote:
> Hi all, 
> We have just send out the following email to all mapathon organisers,
> but hopefully the HOT Tasking Manager is properly up and running again
> soon. 
> Happy mapping everyone! :-) 
> Cheers, 
> Jorieke
>
>
>
> Hello Mappers,
>
>  
>
> Happy GIS day! Unfortunately many of you have experienced some issues
> with HOT Tasking Manager during mapathons today. We want to apologize
> for this inconvenience, and to share some guidance on what to do if
> this happens during your mapathon.
>
>  
>
> *HOT Tasking Manager is experiencing some issues - why?*
>
> HOT Tasking Manager is currently unstable due to a failed maintenance
> from last night. We don’t presently know when this problem will be
> fixed. The HOT tech team is working as fast as possible to fix the
> problem. To find out more about the status, please check:
>
>   * Status.hotosm.org : this will tell you
> if the tool is down, not if it is experiencing delays (as it is
> presently)
>   * follow @hotosm_tech on Twitter for updates
>   * Join the HOT Slack (http://slack.hotosm.org/) and ask questions in
> the #tasking-manager channel
>
>  
>
> *Contingency plans for your mapathon: what are your options if the HOT
> Tasking Manager is not working during your event?*
>
>  1. Map directly in OSM! Check out an area you know well and
> contribute your own knowledge. If you’d like to host a map-walk,
> please see this guide on maps.me
> .
>  2. Host a Mapswipeathon. MapSwipe has several
> tasks up and running on the app. You can help swiping in
> Indonesia, Venezuela, Philippines and Colombia. Please note this
> is a mobile app only.
>  3. There are alternative Tasking manager instances you can use like
> https://tm3.openstreetmap.id/
> . Please note that these will not have
> the same projects on them, so you’ll need to pick a new project.
>
> *Note:*Mapswipe, and the alternative Tasking Managers, are completely
> separate systems that let you contribute to OSM, so they should *not
> *be affected by the outage of the main HOT Tasking Manager.
>
>  
>
> Thank you for your understanding, we appreciate your flexibility. We
> are working with the HOT Tech Team to get this fixed as soon as
> possible.  If you have any questions or need help course correcting
> please reach out.
>
>  
>
>
> Op wo 13 nov. 2019 om 15:20 schreef Marco Minghini
> mailto:marco.minghin...@gmail.com>>:
>
> I also experienced the same problem when running my mapathon today
> at the European Commission - JRC in Ispra (Italy):
> https://tasks.hotosm.org/ was down, but for the purposes of the
> mapathon (introducing humanitarian mapping and introducing
> participants to mapping) the instance at
> https://tasks.teachosm.org was totally fine.
>
> This solution might be useful for many other people involved in
> mapathons during this week!
>
> Best,
> Marco
>
>
> Il giorno mer 13 nov 2019 alle ore 13:32 Felix Delattre
> mailto:felix.delat...@hotosm.org>> ha
> scritto:
>
> Hi all,
>
> Yes, this is the worst time. Almost the whole tech team is
> sole focusing on that emergency.
>
> We had earlier today some problems with a maintenance
> procedure that went out of control. And at the moment we are
> facing the following issues on the Tasking Manager that has
> been brought back:
>
> - Locking of tasks is not working and throws an error message. 
> - Login is failing for some, not for others
> - Activities of late yesterday, such as projects that have
> been created, might be lost in the database, as we had to use
> a backup from yesterday.
>
> We are working on solving it soon. And will report here and in
> our other channels (slack, twitter.com/hotosm_tech
>  and status.hotosm.org
> ) more information.
>
> In the meantime we can only offer alternative instances such as:
>
> * tm3.openstreetmap.id 
> * tasks-assisted.hotosm.org 
> * tasks.teachosm.org 
>
> Please excuse the inconvenience and we will post any
> updates as soon as possible.
>
> So far an update from the infrastructure experts on what
> happened earlier today:
>
> > *Summary:*
> > Tasking Manager is now back and working. 

Re: [HOT] Tasking manager unstable

[Jorieke Vyncke]

Hi all,
We have just send out the following email to all mapathon organisers, but
hopefully the HOT Tasking Manager is properly up and running again soon.
Happy mapping everyone! :-)
Cheers,
Jorieke



Hello Mappers,



Happy GIS day! Unfortunately many of you have experienced some issues with
HOT Tasking Manager during mapathons today. We want to apologize for this
inconvenience, and to share some guidance on what to do if this happens
during your mapathon.



*HOT Tasking Manager is experiencing some issues - why?*

HOT Tasking Manager is currently unstable due to a failed maintenance from
last night. We don’t presently know when this problem will be fixed. The
HOT tech team is working as fast as possible to fix the problem. To find
out more about the status, please check:

   - Status.hotosm.org : this will tell you if
   the tool is down, not if it is experiencing delays (as it is presently)
   - follow @hotosm_tech on Twitter for updates
   - Join the HOT Slack (http://slack.hotosm.org/) and ask questions in the
   #tasking-manager channel



*Contingency plans for your mapathon: what are your options if the HOT
Tasking Manager is not working during your event?*

   1. Map directly in OSM! Check out an area you know well and contribute
   your own knowledge. If you’d like to host a map-walk, please see this guide
   on maps.me .
   2. Host a Mapswipeathon. MapSwipe  has several
   tasks up and running on the app. You can help swiping in Indonesia,
   Venezuela, Philippines and Colombia. Please note this is a mobile app only.
   3. There are alternative Tasking manager instances you can use like
   https://tm3.openstreetmap.id/ & tasks.teachosm.org. Please note that
   these will not have the same projects on them, so you’ll need to pick a new
   project.

*Note:* Mapswipe, and the alternative Tasking Managers, are completely
separate systems that let you contribute to OSM, so they should *not *be
affected by the outage of the main HOT Tasking Manager.



Thank you for your understanding, we appreciate your flexibility. We are
working with the HOT Tech Team to get this fixed as soon as possible.  If
you have any questions or need help course correcting please reach out.



Op wo 13 nov. 2019 om 15:20 schreef Marco Minghini <
marco.minghin...@gmail.com>:

> I also experienced the same problem when running my mapathon today at the
> European Commission - JRC in Ispra (Italy):
> https://tasks.hotosm.org/ was down, but for the purposes of the mapathon
> (introducing humanitarian mapping and introducing participants to mapping)
> the instance at https://tasks.teachosm.org was totally fine.
>
> This solution might be useful for many other people involved in mapathons
> during this week!
>
> Best,
> Marco
>
>
> Il giorno mer 13 nov 2019 alle ore 13:32 Felix Delattre <
> felix.delat...@hotosm.org> ha scritto:
>
>> Hi all,
>>
>> Yes, this is the worst time. Almost the whole tech team is sole focusing
>> on that emergency.
>>
>> We had earlier today some problems with a maintenance procedure that went
>> out of control. And at the moment we are facing the following issues on the
>> Tasking Manager that has been brought back:
>>
>> - Locking of tasks is not working and throws an error message.
>> - Login is failing for some, not for others
>> - Activities of late yesterday, such as projects that have been created,
>> might be lost in the database, as we had to use a backup from yesterday.
>>
>> We are working on solving it soon. And will report here and in our other
>> channels (slack, twitter.com/hotosm_tech and status.hotosm.org) more
>> information.
>>
>> In the meantime we can only offer alternative instances such as:
>>
>> * tm3.openstreetmap.id
>> * tasks-assisted.hotosm.org
>> * tasks.teachosm.org
>>
>> Please excuse the inconvenience and we will post any updates as soon as
>> possible.
>>
>> So far an update from the infrastructure experts on what happened earlier
>> today:
>>
>> > *Summary:*
>> > Tasking Manager is now back and working. Unfortunately, many email
>> addresses were lost and therefore some users who login will be asked for
>> their email addresses again. Projects are intact and unaffected
>> >
>> > *So what happened?*
>> > During the scheduled maintenance to upgrade Tasking Manager instance,
>> one of our automated scripts deleted our database. This is not unexpected.
>> But the backup we made before deletion was deleted along with the database.
>> In a nutshell, we lost the production database completely, including
>> automated backups. We however had a separate offline backup that was taken
>> manually yesterday. But we had stripped email addresses from this database
>> backup for privacy reasons. We had to restore email addresses from another
>> offline backup from 10 days ago to fix it. We are now closely monitoring
>> the DB and the application for any problems.
>> >
>> > *What can 

Re: [HOT] Tasking manager unstable

[Marco Minghini]

I also experienced the same problem when running my mapathon today at the
European Commission - JRC in Ispra (Italy):
https://tasks.hotosm.org/ was down, but for the purposes of the mapathon
(introducing humanitarian mapping and introducing participants to mapping)
the instance at https://tasks.teachosm.org was totally fine.

This solution might be useful for many other people involved in mapathons
during this week!

Best,
Marco


Il giorno mer 13 nov 2019 alle ore 13:32 Felix Delattre <
felix.delat...@hotosm.org> ha scritto:

> Hi all,
>
> Yes, this is the worst time. Almost the whole tech team is sole focusing
> on that emergency.
>
> We had earlier today some problems with a maintenance procedure that went
> out of control. And at the moment we are facing the following issues on the
> Tasking Manager that has been brought back:
>
> - Locking of tasks is not working and throws an error message.
> - Login is failing for some, not for others
> - Activities of late yesterday, such as projects that have been created,
> might be lost in the database, as we had to use a backup from yesterday.
>
> We are working on solving it soon. And will report here and in our other
> channels (slack, twitter.com/hotosm_tech and status.hotosm.org) more
> information.
>
> In the meantime we can only offer alternative instances such as:
>
> * tm3.openstreetmap.id
> * tasks-assisted.hotosm.org
> * tasks.teachosm.org
>
> Please excuse the inconvenience and we will post any updates as soon as
> possible.
>
> So far an update from the infrastructure experts on what happened earlier
> today:
>
> > *Summary:*
> > Tasking Manager is now back and working. Unfortunately, many email
> addresses were lost and therefore some users who login will be asked for
> their email addresses again. Projects are intact and unaffected
> >
> > *So what happened?*
> > During the scheduled maintenance to upgrade Tasking Manager instance,
> one of our automated scripts deleted our database. This is not unexpected.
> But the backup we made before deletion was deleted along with the database.
> In a nutshell, we lost the production database completely, including
> automated backups. We however had a separate offline backup that was taken
> manually yesterday. But we had stripped email addresses from this database
> backup for privacy reasons. We had to restore email addresses from another
> offline backup from 10 days ago to fix it. We are now closely monitoring
> the DB and the application for any problems.
> >
> > *What can you expect in the future?*
> > We learned many valuable lessons from this including our approach to
> data backups, performance and stability. All our experiences from the
> outages of recent past have made us think about how to overhaul our
> approach to optimize our infrastructure. We will work to implement
> processes to mitigate performance, deployment and stability problems over
> the next few weeks.  We are also working on a detailed RCA with technical
> details of the outage.
> We understand that this would have caused quite a bit of frustration and
> we apologize for that. You can expect things to improve in the next few
> weeks. As always, you can follow us on Twitter  (
> https://twitter.com/hotosm_tech) or Slack for updates in addition to our
> status page at (https://status.hotosm.org).
>
>
>
> On Wed, Nov 13, 2019 at 12:01 PM Jorieke Vyncke 
> wrote:
>
>> Hi all,
>> Yes, please HOT, can you do everything to keep the TM online and stable
>> today / this week?
>> It's GISday today and there are more than 30 mapathons worldwide and 100+
>> this week for OSMgeoweek. Mapathons are happening in Uganda, Nigeria,
>> Amsterdam, Sydney, Rio, etc.
>> At the moment I tell people to go to tasks.teachosm.org and MapSwipe, if
>> the HOT TM is not working.
>> Best regards,
>> Jorieke
>>
>> Op wo 13 nov. 2019 om 11:51 schreef Pete Masters via HOT <
>> hot@openstreetmap.org>:
>>
>>> I'm sure you are all aware, but just in case... this is geo week and
>>> there are more than 70 mapathons planned for this week. Maybe updates from
>>> the tech team need to be posted across channels on this as it is critical
>>> infrastructure for all of these events and people might not be connected to
>>> slack / list / facebook etc...
>>>
>>> Cheers,
>>>
>>> Pete
>>>
>>> On Wed, Nov 13, 2019 at 11:38 AM Micheal Yani 
>>> wrote:
>>>
   Hello Tech team, we  just want to let you know that the tasking
 manager isn't stable yet, It has has worked for 2 hrs and now is on and
 off.
 ___
 HOT mailing list
 HOT@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/hot

>>> ___
>>> HOT mailing list
>>> HOT@openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/hot
>>>
>> ___
>> HOT mailing list
>> HOT@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/hot
>>
>
>
> --
>
> *Felix 

Re: [HOT] Tasking manager unstable

[Felix Delattre]

Hi all,

Yes, this is the worst time. Almost the whole tech team is sole focusing
on that emergency.

We had earlier today some problems with a maintenance procedure that went
out of control. And at the moment we are facing the following issues on the
Tasking Manager that has been brought back:

- Locking of tasks is not working and throws an error message.
- Login is failing for some, not for others
- Activities of late yesterday, such as projects that have been created,
might be lost in the database, as we had to use a backup from yesterday.

We are working on solving it soon. And will report here and in our other
channels (slack, twitter.com/hotosm_tech and status.hotosm.org) more
information.

In the meantime we can only offer alternative instances such as:

* tm3.openstreetmap.id
* tasks-assisted.hotosm.org
* tasks.teachosm.org

Please excuse the inconvenience and we will post any updates as soon as
possible.

So far an update from the infrastructure experts on what happened earlier
today:

> *Summary:*
> Tasking Manager is now back and working. Unfortunately, many email
addresses were lost and therefore some users who login will be asked for
their email addresses again. Projects are intact and unaffected
>
> *So what happened?*
> During the scheduled maintenance to upgrade Tasking Manager instance, one
of our automated scripts deleted our database. This is not unexpected. But
the backup we made before deletion was deleted along with the database. In
a nutshell, we lost the production database completely, including automated
backups. We however had a separate offline backup that was taken manually
yesterday. But we had stripped email addresses from this database backup
for privacy reasons. We had to restore email addresses from another offline
backup from 10 days ago to fix it. We are now closely monitoring the DB and
the application for any problems.
>
> *What can you expect in the future?*
> We learned many valuable lessons from this including our approach to data
backups, performance and stability. All our experiences from the outages of
recent past have made us think about how to overhaul our approach to
optimize our infrastructure. We will work to implement processes to
mitigate performance, deployment and stability problems over the next few
weeks.  We are also working on a detailed RCA with technical details of the
outage.
We understand that this would have caused quite a bit of frustration and we
apologize for that. You can expect things to improve in the next few weeks.
As always, you can follow us on Twitter  (https://twitter.com/hotosm_tech)
or Slack for updates in addition to our status page at (
https://status.hotosm.org).



On Wed, Nov 13, 2019 at 12:01 PM Jorieke Vyncke 
wrote:

> Hi all,
> Yes, please HOT, can you do everything to keep the TM online and stable
> today / this week?
> It's GISday today and there are more than 30 mapathons worldwide and 100+
> this week for OSMgeoweek. Mapathons are happening in Uganda, Nigeria,
> Amsterdam, Sydney, Rio, etc.
> At the moment I tell people to go to tasks.teachosm.org and MapSwipe, if
> the HOT TM is not working.
> Best regards,
> Jorieke
>
> Op wo 13 nov. 2019 om 11:51 schreef Pete Masters via HOT <
> hot@openstreetmap.org>:
>
>> I'm sure you are all aware, but just in case... this is geo week and
>> there are more than 70 mapathons planned for this week. Maybe updates from
>> the tech team need to be posted across channels on this as it is critical
>> infrastructure for all of these events and people might not be connected to
>> slack / list / facebook etc...
>>
>> Cheers,
>>
>> Pete
>>
>> On Wed, Nov 13, 2019 at 11:38 AM Micheal Yani 
>> wrote:
>>
>>>   Hello Tech team, we  just want to let you know that the tasking
>>> manager isn't stable yet, It has has worked for 2 hrs and now is on and
>>> off.
>>> ___
>>> HOT mailing list
>>> HOT@openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/hot
>>>
>> ___
>> HOT mailing list
>> HOT@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/hot
>>
> ___
> HOT mailing list
> HOT@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/hot
>


-- 

*Felix Delattre*
Technical Project Manager
felix.delat...@hotosm.org

--
*Humanitarian OpenStreetMap Team*
*Using OpenStreetMap for Humanitarian Response & Economic Development*
web  | twitter  | facebook
 | donate 
___
HOT mailing list
HOT@openstreetmap.org
https://lists.openstreetmap.org/listinfo/hot

Re: [HOT] Tasking manager unstable

[Pete Masters via HOT]

I'm sure you are all aware, but just in case... this is geo week and there
are more than 70 mapathons planned for this week. Maybe updates from the
tech team need to be posted across channels on this as it is critical
infrastructure for all of these events and people might not be connected to
slack / list / facebook etc...

Cheers,

Pete

On Wed, Nov 13, 2019 at 11:38 AM Micheal Yani 
wrote:

>   Hello Tech team, we  just want to let you know that the tasking manager
> isn't stable yet, It has has worked for 2 hrs and now is on and off.
> ___
> HOT mailing list
> HOT@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/hot
>
___
HOT mailing list
HOT@openstreetmap.org
https://lists.openstreetmap.org/listinfo/hot