Re: [OSM-talk] Upload slowness - what's going on?

[Gerd Petermann]

I did not see that improvement when Ben posted, but I got the impression that
my last
uploads where processed faster again, so if something changed during the
last 5 days it was
to the better.

Gerd 

Tom Hughes-3 wrote
> On 17/05/16 06:14, Ben Discoe wrote:
> 
>> API download and upload has gotten fast(er) tonight, for the first
>> time since the server move on May 9!
> 
> Well that makes no sense at all.
> 
> Are you saying it was still slow over the weekend and then got faster 
> last night?
> 
> Tom
> 
> -- 
> Tom Hughes (

> tom@

> )
> http://compton.nu/
> 
> ___
> talk mailing list

> talk@

> https://lists.openstreetmap.org/listinfo/talk





--
View this message in context: 
http://gis.19327.n5.nabble.com/Upload-slowness-what-s-going-on-tp5873456p5874049.html
Sent from the General Discussion mailing list archive at Nabble.com.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Upload slowness - what's going on?

[Tom Hughes]

On 17/05/16 06:14, Ben Discoe wrote:


API download and upload has gotten fast(er) tonight, for the first
time since the server move on May 9!


Well that makes no sense at all.

Are you saying it was still slow over the weekend and then got faster 
last night?


Tom

--
Tom Hughes (t...@compton.nu)
http://compton.nu/

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Upload slowness - what's going on?

[Ben Discoe]

API download and upload has gotten fast(er) tonight, for the first
time since the server move on May 9!
Whatever they've done today, it's working!
I am so happy. This really impacts my life.
Thanks,
Ben

On Fri, May 13, 2016 at 5:59 AM, Grant Slater
 wrote:
> Hi All,
>
> On Monday 9th May 2016 the master OSM database server was moved to
> York (Bytemark) from London (Imperial).
> This was to avoid multiple upcoming weekends of planned power testing
> & maintenance at the Imperial data centre. For the last few years
> Imperial has housed all our main critical systems including master &
> slave DB servers and frontend & backend web/api servers. We also added
> 4 new frontend/backend web/api server to York on Monday.
>
> We now have the master database server in York and the secondary
> database server in Imperial. We also have a warm standby slave db in
> AWS Ireland. A fourth SSD (NVMe) based DB server was delivered
> yesterday (Thursday), but it needs testing (burn-in, reliability,
> performance etc) before we can start using it. Slave DB servers can be
> promoted to master if required.
>
> The slave db servers serve Web/API read traffic and writes go to the
> master. When the frontend + backend servers were in the same data
> centre as the master db server the latency was <1ms. We now run a VPN
> to connect the servers up and the latency is ~8ms Imperial to
> Bytemark. Currently we are using the frontend & backends server at
> Imperial (closest to slave db read server) and sending writes over the
> VPN to Bytemark. The extra 8ms roundtrip is triggered multiple times
> based on the size of the upload changeset, this is the root cause for
> the slower uploads. The link between Imperial & Bytemark can handle
> gigabit speeds. Over the last few days we've been tweaking the VPN
> settings to get optimal latency & throughput over the links.
>
> Over today (for at least the weekend) we are switching to the new
> frontend & backend servers in York (Bytemark). London Imperial will be
> offline from approximately 5pm (GMT+1) for the first weekend of power
> maintenance.
>
> In summary: The slow uploads are a known issue and we'll fix as soon
> as practical. Our main concern has been setting up multiple data
> center redundancy to avoid extended downtime.
>
> Here is the list of all core hardware and hosting locations:
> https://hardware.openstreetmap.org/
>
> Hope that answers the questions. ;-)
>
> Photos or it didn't happen:
> * Syncing & powering down before we start London -> York DB move:
> https://twitter.com/OSM_Tech/status/729582996685213696
> * Staged photo of racking up the master DB server at Bytemark:
> https://twitter.com/OSM_Tech/status/729693392737832961
> * Testing the new Frontend / Backend servers a week ago:
> https://twitter.com/OSM_Tech/status/728286193696292865
>
> Bytemark are a fantastic hosting company and their ongoing support of
> the OpenStreetMap project is highly commendable. Please support them
> ;-) https://twitter.com/bytemark/status/729698435339853824
>
> Kind regards,
>
> Grant
> Part of the OSM Ops team.
>
>
> On 13 May 2016 at 11:44, Tim Waters  wrote:
>> I believe the Dev mailing list may have some of your technical answers
>> https://lists.openstreetmap.org/pipermail/dev/2016-May/thread.html
>>
>> It appears from that list that the database servers are now a few
>> hundreds of miles from where the web servers are, causing the increase
>> in latency. I do not know if this is a permanent change, the thread on
>> osm-dev does seem to indicate that things are still in flux.
>>
>> Tim
>>
>>
>>
>> On 13 May 2016 at 06:02, Ben Discoe  wrote:
>>> Several of us have noticed radically slowly upload speed for
>>> changesets, roughly since the server move on May 9.  Like, as
>>> painfully slow as it used to be, it's now several times slower.
>>>
>>> It's been discussed with @OSM_Tech on twitter, in this thread:
>>> https://twitter.com/OSM_Tech/status/730857486618664960
>>>
>>> Before I get too hysterical, can somebody tell me what happened, and
>>> can it be fixed?
>>>
>>> OSM_Tech's mysterious message:
>>>   "Large uploads will take around 3 times longer. Small uploads extra
>>> delay should be minimal."
>>>
>>> Does this mean that something did change?  It is database writes that
>>> are taking so much longer?  Changesets with as few as 400 object are
>>> taking several times longer, what constitutes "large" vs. "small"?
>>> Can it be fixed?  Can I donate large sums of money somewhere to help
>>> it get fixed?
>>>
>>> Thanks,
>>> Ben
>>>
>>> ___
>>> talk mailing list
>>> talk@openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/talk
>>
>> ___
>> talk mailing list
>> talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list

Re: [OSM-talk] Upload slowness - what's going on?

[nebulon42]

As said before: great explanation and great work. These are complicated
tasks, so thanks to the whole OWG for managing all this.

nebulon42



signature.asc
Description: OpenPGP digital signature
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Upload slowness - what's going on?

[Tom Hughes]

On 13/05/16 14:43, Maarten Deen wrote:

On 2016-05-13 14:59, Grant Slater wrote:

Kudos for the very clear explanation.


Over today (for at least the weekend) we are switching to the new
frontend & backend servers in York (Bytemark). London Imperial will be
offline from approximately 5pm (GMT+1) for the first weekend of power
maintenance.


Is my understanding correct that this will put all servers in York and
eliminate the latency and therefore the slowness?


As of now everything is running in York so that we are ready for the 
power to go off in London over the weekend.


On Monday we plan to move service back to London though I will probably 
try and keep the uploads going through the backend servers at York to 
see if that improves the speed issues seen this week.


In due course the new database server in London will likely become the 
master with the one in York as a slave for servicing reads but with the 
ability to flip the master over if we need to run from York for some reason.



Am I also correct to assume that the current situation is only because
of this move (and prompted by the maintenance at Imperial) and is not
planned to be a permanent setup?


Well long term we want to have things spread over multiple sites so that 
we can cope with a data centre going down and part of this is us 
learning the best way to split things up and arrange things so that we 
can achieve redundancy without impacting other things.


So there will likely be some tinkering and experimentation over the 
coming weeks as we try different things to see what works best.


Tom

--
Tom Hughes (t...@compton.nu)
http://compton.nu/

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Upload slowness - what's going on?

[Maarten Deen]

On 2016-05-13 14:59, Grant Slater wrote:

Kudos for the very clear explanation.


Over today (for at least the weekend) we are switching to the new
frontend & backend servers in York (Bytemark). London Imperial will be
offline from approximately 5pm (GMT+1) for the first weekend of power
maintenance.


Is my understanding correct that this will put all servers in York and 
eliminate the latency and therefore the slowness?
Am I also correct to assume that the current situation is only because 
of this move (and prompted by the maintenance at Imperial) and is not 
planned to be a permanent setup?


Regards,
Maarten

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Upload slowness - what's going on?

[Jo]

Hi Grant,

Thank you for the technical explanation and especially for all the efforts
improving the infrastructure! It's unfortunate uploads are so much slower
now, but if the whole setup is more reliable/redundant, that's a small
price to pay.

Polyglot

2016-05-13 14:59 GMT+02:00 Grant Slater :

> Hi All,
>
> On Monday 9th May 2016 the master OSM database server was moved to
> York (Bytemark) from London (Imperial).
> This was to avoid multiple upcoming weekends of planned power testing
> & maintenance at the Imperial data centre. For the last few years
> Imperial has housed all our main critical systems including master &
> slave DB servers and frontend & backend web/api servers. We also added
> 4 new frontend/backend web/api server to York on Monday.
>
> We now have the master database server in York and the secondary
> database server in Imperial. We also have a warm standby slave db in
> AWS Ireland. A fourth SSD (NVMe) based DB server was delivered
> yesterday (Thursday), but it needs testing (burn-in, reliability,
> performance etc) before we can start using it. Slave DB servers can be
> promoted to master if required.
>
> The slave db servers serve Web/API read traffic and writes go to the
> master. When the frontend + backend servers were in the same data
> centre as the master db server the latency was <1ms. We now run a VPN
> to connect the servers up and the latency is ~8ms Imperial to
> Bytemark. Currently we are using the frontend & backends server at
> Imperial (closest to slave db read server) and sending writes over the
> VPN to Bytemark. The extra 8ms roundtrip is triggered multiple times
> based on the size of the upload changeset, this is the root cause for
> the slower uploads. The link between Imperial & Bytemark can handle
> gigabit speeds. Over the last few days we've been tweaking the VPN
> settings to get optimal latency & throughput over the links.
>
> Over today (for at least the weekend) we are switching to the new
> frontend & backend servers in York (Bytemark). London Imperial will be
> offline from approximately 5pm (GMT+1) for the first weekend of power
> maintenance.
>
> In summary: The slow uploads are a known issue and we'll fix as soon
> as practical. Our main concern has been setting up multiple data
> center redundancy to avoid extended downtime.
>
> Here is the list of all core hardware and hosting locations:
> https://hardware.openstreetmap.org/
>
> Hope that answers the questions. ;-)
>
> Photos or it didn't happen:
> * Syncing & powering down before we start London -> York DB move:
> https://twitter.com/OSM_Tech/status/729582996685213696
> * Staged photo of racking up the master DB server at Bytemark:
> https://twitter.com/OSM_Tech/status/729693392737832961
> * Testing the new Frontend / Backend servers a week ago:
> https://twitter.com/OSM_Tech/status/728286193696292865
>
> Bytemark are a fantastic hosting company and their ongoing support of
> the OpenStreetMap project is highly commendable. Please support them
> ;-) https://twitter.com/bytemark/status/729698435339853824
>
> Kind regards,
>
> Grant
> Part of the OSM Ops team.
>
>
> On 13 May 2016 at 11:44, Tim Waters  wrote:
> > I believe the Dev mailing list may have some of your technical answers
> > https://lists.openstreetmap.org/pipermail/dev/2016-May/thread.html
> >
> > It appears from that list that the database servers are now a few
> > hundreds of miles from where the web servers are, causing the increase
> > in latency. I do not know if this is a permanent change, the thread on
> > osm-dev does seem to indicate that things are still in flux.
> >
> > Tim
> >
> >
> >
> > On 13 May 2016 at 06:02, Ben Discoe  wrote:
> >> Several of us have noticed radically slowly upload speed for
> >> changesets, roughly since the server move on May 9.  Like, as
> >> painfully slow as it used to be, it's now several times slower.
> >>
> >> It's been discussed with @OSM_Tech on twitter, in this thread:
> >> https://twitter.com/OSM_Tech/status/730857486618664960
> >>
> >> Before I get too hysterical, can somebody tell me what happened, and
> >> can it be fixed?
> >>
> >> OSM_Tech's mysterious message:
> >>   "Large uploads will take around 3 times longer. Small uploads extra
> >> delay should be minimal."
> >>
> >> Does this mean that something did change?  It is database writes that
> >> are taking so much longer?  Changesets with as few as 400 object are
> >> taking several times longer, what constitutes "large" vs. "small"?
> >> Can it be fixed?  Can I donate large sums of money somewhere to help
> >> it get fixed?
> >>
> >> Thanks,
> >> Ben
> >>
> >> ___
> >> talk mailing list
> >> talk@openstreetmap.org
> >> https://lists.openstreetmap.org/listinfo/talk
> >
> > ___
> > talk mailing list
> > talk@openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Upload slowness - what's going on?

[Grant Slater]

Hi All,

On Monday 9th May 2016 the master OSM database server was moved to
York (Bytemark) from London (Imperial).
This was to avoid multiple upcoming weekends of planned power testing
& maintenance at the Imperial data centre. For the last few years
Imperial has housed all our main critical systems including master &
slave DB servers and frontend & backend web/api servers. We also added
4 new frontend/backend web/api server to York on Monday.

We now have the master database server in York and the secondary
database server in Imperial. We also have a warm standby slave db in
AWS Ireland. A fourth SSD (NVMe) based DB server was delivered
yesterday (Thursday), but it needs testing (burn-in, reliability,
performance etc) before we can start using it. Slave DB servers can be
promoted to master if required.

The slave db servers serve Web/API read traffic and writes go to the
master. When the frontend + backend servers were in the same data
centre as the master db server the latency was <1ms. We now run a VPN
to connect the servers up and the latency is ~8ms Imperial to
Bytemark. Currently we are using the frontend & backends server at
Imperial (closest to slave db read server) and sending writes over the
VPN to Bytemark. The extra 8ms roundtrip is triggered multiple times
based on the size of the upload changeset, this is the root cause for
the slower uploads. The link between Imperial & Bytemark can handle
gigabit speeds. Over the last few days we've been tweaking the VPN
settings to get optimal latency & throughput over the links.

Over today (for at least the weekend) we are switching to the new
frontend & backend servers in York (Bytemark). London Imperial will be
offline from approximately 5pm (GMT+1) for the first weekend of power
maintenance.

In summary: The slow uploads are a known issue and we'll fix as soon
as practical. Our main concern has been setting up multiple data
center redundancy to avoid extended downtime.

Here is the list of all core hardware and hosting locations:
https://hardware.openstreetmap.org/

Hope that answers the questions. ;-)

Photos or it didn't happen:
* Syncing & powering down before we start London -> York DB move:
https://twitter.com/OSM_Tech/status/729582996685213696
* Staged photo of racking up the master DB server at Bytemark:
https://twitter.com/OSM_Tech/status/729693392737832961
* Testing the new Frontend / Backend servers a week ago:
https://twitter.com/OSM_Tech/status/728286193696292865

Bytemark are a fantastic hosting company and their ongoing support of
the OpenStreetMap project is highly commendable. Please support them
;-) https://twitter.com/bytemark/status/729698435339853824

Kind regards,

Grant
Part of the OSM Ops team.


On 13 May 2016 at 11:44, Tim Waters  wrote:
> I believe the Dev mailing list may have some of your technical answers
> https://lists.openstreetmap.org/pipermail/dev/2016-May/thread.html
>
> It appears from that list that the database servers are now a few
> hundreds of miles from where the web servers are, causing the increase
> in latency. I do not know if this is a permanent change, the thread on
> osm-dev does seem to indicate that things are still in flux.
>
> Tim
>
>
>
> On 13 May 2016 at 06:02, Ben Discoe  wrote:
>> Several of us have noticed radically slowly upload speed for
>> changesets, roughly since the server move on May 9.  Like, as
>> painfully slow as it used to be, it's now several times slower.
>>
>> It's been discussed with @OSM_Tech on twitter, in this thread:
>> https://twitter.com/OSM_Tech/status/730857486618664960
>>
>> Before I get too hysterical, can somebody tell me what happened, and
>> can it be fixed?
>>
>> OSM_Tech's mysterious message:
>>   "Large uploads will take around 3 times longer. Small uploads extra
>> delay should be minimal."
>>
>> Does this mean that something did change?  It is database writes that
>> are taking so much longer?  Changesets with as few as 400 object are
>> taking several times longer, what constitutes "large" vs. "small"?
>> Can it be fixed?  Can I donate large sums of money somewhere to help
>> it get fixed?
>>
>> Thanks,
>> Ben
>>
>> ___
>> talk mailing list
>> talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Upload slowness - what's going on?

[Tim Waters]

I believe the Dev mailing list may have some of your technical answers
https://lists.openstreetmap.org/pipermail/dev/2016-May/thread.html

It appears from that list that the database servers are now a few
hundreds of miles from where the web servers are, causing the increase
in latency. I do not know if this is a permanent change, the thread on
osm-dev does seem to indicate that things are still in flux.

Tim



On 13 May 2016 at 06:02, Ben Discoe  wrote:
> Several of us have noticed radically slowly upload speed for
> changesets, roughly since the server move on May 9.  Like, as
> painfully slow as it used to be, it's now several times slower.
>
> It's been discussed with @OSM_Tech on twitter, in this thread:
> https://twitter.com/OSM_Tech/status/730857486618664960
>
> Before I get too hysterical, can somebody tell me what happened, and
> can it be fixed?
>
> OSM_Tech's mysterious message:
>   "Large uploads will take around 3 times longer. Small uploads extra
> delay should be minimal."
>
> Does this mean that something did change?  It is database writes that
> are taking so much longer?  Changesets with as few as 400 object are
> taking several times longer, what constitutes "large" vs. "small"?
> Can it be fixed?  Can I donate large sums of money somewhere to help
> it get fixed?
>
> Thanks,
> Ben
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Upload slowness - what's going on?

[Michał Brzozowski]

Their response is indeed so vague that I suspect even they don't know
what's happening ;)

Certainly, when you use JOSM, you can choose how much objects to
upload at once. Small bundles (100...200) seem to work better. You can
experiment with it.

Michał

On Fri, May 13, 2016 at 7:02 AM, Ben Discoe  wrote:
> Several of us have noticed radically slowly upload speed for
> changesets, roughly since the server move on May 9.  Like, as
> painfully slow as it used to be, it's now several times slower.
>
> It's been discussed with @OSM_Tech on twitter, in this thread:
> https://twitter.com/OSM_Tech/status/730857486618664960
>
> Before I get too hysterical, can somebody tell me what happened, and
> can it be fixed?
>
> OSM_Tech's mysterious message:
>   "Large uploads will take around 3 times longer. Small uploads extra
> delay should be minimal."
>
> Does this mean that something did change?  It is database writes that
> are taking so much longer?  Changesets with as few as 400 object are
> taking several times longer, what constitutes "large" vs. "small"?
> Can it be fixed?  Can I donate large sums of money somewhere to help
> it get fixed?
>
> Thanks,
> Ben
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] Upload slowness - what's going on?

[Ben Discoe]

Several of us have noticed radically slowly upload speed for
changesets, roughly since the server move on May 9.  Like, as
painfully slow as it used to be, it's now several times slower.

It's been discussed with @OSM_Tech on twitter, in this thread:
https://twitter.com/OSM_Tech/status/730857486618664960

Before I get too hysterical, can somebody tell me what happened, and
can it be fixed?

OSM_Tech's mysterious message:
  "Large uploads will take around 3 times longer. Small uploads extra
delay should be minimal."

Does this mean that something did change?  It is database writes that
are taking so much longer?  Changesets with as few as 400 object are
taking several times longer, what constitutes "large" vs. "small"?
Can it be fixed?  Can I donate large sums of money somewhere to help
it get fixed?

Thanks,
Ben

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk