Re: [OSM-dev] osm2pgsql tile expiry performance

2019-05-29 Thread j
> To be fair, I am not sure this is even related to tile expiration. I
> have not tried 0.96 updates without tile expiration as a baseline.

After 3 hours, the same update has reached the same state of progress.
osm2pgsql (debian backport) 0.96 maintains < 1GB of memory usage without
expiring tiles. While expiring tiles, it ballooned to >22GB before
being killed.

Seems the issue is specific to tile-expiration overhaul. 

I note that initial testing of this code used an extract of Europe:
https://github.com/openstreetmap/osm2pgsql/pull/747

I did not note it earlier, but I am working with the planet. I doubt
these issues would present themselves on a modestly sized machine
working with Europe only.

j

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] osm2pgsql tile expiry performance

2019-05-29 Thread j
I'm beginning to implement a tile expiration solution and have run into
some issues with the new tile expiry expansion.

I'm afraid I do not have precise timings, but I'm seeing what appears
to be at least an order of magnitude performance penalty, probably due
to memory exhaustion.

Test machine is Debian stretch 4CPUs, 26GB RAM, SSD Array. Initial osm
import was performed with osm2pgsql 0.92 and finished in under 48 hours.

Under 0.92 I was running multiple render chains while importing
changesets. Tests w/ 0.96 have been run against an otherwise idle
machine.

Database & flatnodes file were restored to initial state between each
round of testing.

WEEKLY changeset using 0.92:
-=-=-=-
time $OSM2PGSQL --append -s -C 3000 -G --hstore -d gis -H $PGHOST -U \
$PGUSER -r xml changes.osc \
--flat-nodes /database/postgresql/OSM-FLATNODES --slim \
--number-processes 4 --style openstreetmap-carto.style \
--tag-transform-script openstreetmap-carto.lua -e19 -o \
$WORKDIR_OSM/expired-tiles

With -e19, I was able to import a weekly changeset in roughly 24 hours.


Using git repository (5/27? pull):
-=-=-=-
time $OSM2PGSQL --append -s -C 3000 -G --hstore -d gis -H $PGHOST -U
$PGUSER -r xml changes.osc \
--flat-nodes /database/postgresql/OSM-FLATNODES --slim \
--number-processes 4 --style openstreetmap-carto.style \
--tag-transform-script openstreetmap-carto.lua -e10-16 -o \
$WORKDIR_OSM/expired-tiles

Consistently crashed w/ a bad alloc(). I didn't note any unusual output
in the compile. Crashes even with a 24 hour changeset.


DAILY changeset using Debian backport to stretch (0.96):
-=-=-=-
As above, using -e10-16. 

22 hours spent processing a 24 hour changeset and still importing new
relations. It's 35GB into swap, with osm2pgsql claiming 89% of the
memory usage.

---

I realize there is a large difference between zoom level 19 and 10-16,
but I assume it should take significantly less RAM/CPU for 10-16.

Please feel free to point out any stupidity I've generated here and/or
recommend a better way to generate a list of dirty tiles at lower zoom
levels based on the 0.92 output.

To be fair, I am not sure this is even related to tile expiration. I
have not tried 0.96 updates without tile expiration as a baseline.

j

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] Workflow for bulk upload of GPX files

2019-05-29 Thread Ajay Singh1 via dev
Hi Joost,

Thanks for your help. I'm exploring HOTOSM's tasking-manager for this task.
Here also, we get to break down our task into smaller tasks and can be
better tracked. Also, I'm trying to get in touch with other mappers to get
some help here.

Is there anybody you know of who has prior experience with this process?


On Tue, May 28, 2019 at 4:13 PM joost schouppe 
wrote:

> You might have a decent shot of getting volunteers to map this if you
> create a Maproulette challenge. I imagine the roads are sparse, and spread
> out of a larger area. So turn them into individual objects, which can be
> mapped in little time. For example, from a few dozen to a few kilometer
> long sections of a single road. Or perhaps a few adjecent roads. These
> become individual microtasks in Maproulette. You might split them up per
> country and get in touch with several local communities. These tasks are
> rather fun to do (especially if your pre-processing is good) so you might
> actually get some work done.
>
> As for "how to recrute paid mappers", I'm not sure. I know Kaart and
> Telenav do similar work, so they might be interested. Here's some
> activities that follow the "organized editing guideliness", and hence have
> a record in "activities". Note that many of these are not commercial.
>
> https://wiki.openstreetmap.org/wiki/Organised_Editing/Activities
> https://wiki.osmfoundation.org/wiki/Organised_Editing_Guidelines
>
> Op ma 27 mei 2019 om 11:53 schreef Ajay Singh1 via dev <
> dev@openstreetmap.org>:
>
>> Hi Marc,
>>
>> Sorry for the late reply but I meant that are there any mappers whom I
>> can hire to get the mapping done as there are thousand of roads to be
>> mapped?
>>
>> On Mon, May 20, 2019 at 10:40 PM marc marc 
>> wrote:
>>
>>> Le 20.05.19 à 19:03, Ajay Singh1 a écrit :
>>> > I can just upload and someone else will make roads by looking at that
>>> trace.
>>>
>>> of course.
>>> but since there is no toolsosm.org api for a contributor to download
>>> the unprocessed traces, it is like hoping that someone else will do
>>> all the work again that you have done of detecting that these traces are
>>> missing ways in osm
>>> ___
>>> dev mailing list
>>> dev@openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/dev
>>>
>>
>>
>> --
>> Regards,
>> Ajay Singh
>> ___
>> dev mailing list
>> dev@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/dev
>>
>
>
> --
> Joost Schouppe
> OpenStreetMap  |
> Twitter  | LinkedIn
>  | Meetup
> 
>


-- 
Regards,
Ajay Singh
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] 403 forbidden

2019-05-29 Thread Mateusz Konieczny
Technical requirements are at 
https://operations.osmfoundation.org/policies/tiles/ 


To check what gmap.net is doing you need to either check its source code,
capture your network traffic and review it (with for example Wireshark)
or use program that is not maliciously violating OSM policies.

29 May 2019, 13:18 by vijaya21tiw...@gmail.com:

> I would appreciate if you can let me know the exact request header.
>

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] 403 forbidden

2019-05-29 Thread Mateusz Konieczny
29 May 2019, 11:57 by openstreet...@firefishy.com:

> Hi Vijaya,
>
> We have recent had to crack down on users who are violating our usage policy:
> https://operations.osmfoundation.org/policies/tiles/
>
> Please check what User-Agent your app uses or if you fake any browser headers.
>
> The tile.openstreetmap.org CDN is at breaking point.
> We now frequently push more than 3Tb per second of traffic, with over
> 22000 requests per second
>
Thanks for doing this, within last months map view during editing was really 
laggy
(and often still, hopefully more policy-violating users can be blocked).
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] 403 forbidden

2019-05-29 Thread Vijaya Nand
I would appreciate if you can let me know the exact request header.

Thanks,
Vijaya

On Wed, 29 May 2019, 16:35 Grant Slater, 
wrote:

> Hi Vijaya,
>
> If you have changed the User-Agent to comply with the usage policy and
> still being blocked then you are likely in violation for faking
> another request header.
> Please review the Technical Requirement section here:
> https://operations.osmfoundation.org/policies/tiles/
>
> Kind regards,
>
> Grant
>
> On Wed, 29 May 2019 at 11:43, Vijaya Nand 
> wrote:
> >
> > DeadSureMobile/1.0.0.1
> >
> > This is my user agent. Please do let this approved so that I can use it.
> I am using gmap.net.
> >
> > Thanks, please confirm so that I can test.
> >
> >
> >
> >
> > On Wed, 29 May 2019, 16:07 Grant Slater, 
> wrote:
> >>
> >> Hi Vijaya,
> >>
> >> I would suggest using a user-agent with name and version at least eg:
> >> org.github.app-name/1.0.0
> >> Do make sure you read and follow:
> >> https://operations.osmfoundation.org/policies/tiles/
> >>
> >> Here is why gmap.net got blocked and will remain blocked:
> >>
> https://github.com/radioman/greatmaps/blob/799f3a38a65350ff00c21acf9b88c96848bb3880/GMap.NET.Core/GMap.NET.MapProviders/GMapProvider.cs#L329-L336
> >>
> >> Kind regards,
> >>
> >> Grant
> >>
> >>
> >> On Wed, 29 May 2019 at 11:04, Vijaya Nand 
> wrote:
> >> >
> >> > Hi Grant.
> >> >
> >> >
> >> > Please suggest me what user agent should I pass. My application name
> is DeadsureMobile.
> >> >
> >> > Please suggest.
> >> >
> >> > It's Very urgent for me.
> >> >
> >> > Thanks,
> >> > Vijaya
> >> >
> >> > On Wed, 29 May 2019, 15:27 Grant Slater, 
> wrote:
> >> >>
> >> >> Hi Vijaya,
> >> >>
> >> >> We have recent had to crack down on users who are violating our
> usage policy:
> >> >> https://operations.osmfoundation.org/policies/tiles/
> >> >>
> >> >> Please check what User-Agent your app uses or if you fake any
> browser headers.
> >> >>
> >> >> The tile.openstreetmap.org CDN is at breaking point.
> >> >> We now frequently push more than 3Tb per second of traffic, with over
> >> >> 22000 requests per second/
> >> >> We are reaching the limit of what our current stack/config can
> handle.
> >> >>
> >> >> Kind regards,
> >> >>
> >> >> Grant
> >> >> Part of the OpenStreetMap sysadmin team
> >> >>
> >> >> On Wed, 29 May 2019 at 10:26, Vijaya Nand 
> wrote:
> >> >> >
> >> >> > Dear sir,
> >> >> >
> >> >> > I am using a gmap.net control to use OSM but suddent it started
> giving 403 error. Please suggest.
> >> >> >
> >> >> > Thanks,
> >> >> > Vijaya
> >> >> > ___
> >> >> > dev mailing list
> >> >> > dev@openstreetmap.org
> >> >> > https://lists.openstreetmap.org/listinfo/dev
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] 403 forbidden

2019-05-29 Thread Grant Slater
Hi Vijaya,

If you have changed the User-Agent to comply with the usage policy and
still being blocked then you are likely in violation for faking
another request header.
Please review the Technical Requirement section here:
https://operations.osmfoundation.org/policies/tiles/

Kind regards,

Grant

On Wed, 29 May 2019 at 11:43, Vijaya Nand  wrote:
>
> DeadSureMobile/1.0.0.1
>
> This is my user agent. Please do let this approved so that I can use it. I am 
> using gmap.net.
>
> Thanks, please confirm so that I can test.
>
>
>
>
> On Wed, 29 May 2019, 16:07 Grant Slater,  wrote:
>>
>> Hi Vijaya,
>>
>> I would suggest using a user-agent with name and version at least eg:
>> org.github.app-name/1.0.0
>> Do make sure you read and follow:
>> https://operations.osmfoundation.org/policies/tiles/
>>
>> Here is why gmap.net got blocked and will remain blocked:
>> https://github.com/radioman/greatmaps/blob/799f3a38a65350ff00c21acf9b88c96848bb3880/GMap.NET.Core/GMap.NET.MapProviders/GMapProvider.cs#L329-L336
>>
>> Kind regards,
>>
>> Grant
>>
>>
>> On Wed, 29 May 2019 at 11:04, Vijaya Nand  wrote:
>> >
>> > Hi Grant.
>> >
>> >
>> > Please suggest me what user agent should I pass. My application name is 
>> > DeadsureMobile.
>> >
>> > Please suggest.
>> >
>> > It's Very urgent for me.
>> >
>> > Thanks,
>> > Vijaya
>> >
>> > On Wed, 29 May 2019, 15:27 Grant Slater,  
>> > wrote:
>> >>
>> >> Hi Vijaya,
>> >>
>> >> We have recent had to crack down on users who are violating our usage 
>> >> policy:
>> >> https://operations.osmfoundation.org/policies/tiles/
>> >>
>> >> Please check what User-Agent your app uses or if you fake any browser 
>> >> headers.
>> >>
>> >> The tile.openstreetmap.org CDN is at breaking point.
>> >> We now frequently push more than 3Tb per second of traffic, with over
>> >> 22000 requests per second/
>> >> We are reaching the limit of what our current stack/config can handle.
>> >>
>> >> Kind regards,
>> >>
>> >> Grant
>> >> Part of the OpenStreetMap sysadmin team
>> >>
>> >> On Wed, 29 May 2019 at 10:26, Vijaya Nand  
>> >> wrote:
>> >> >
>> >> > Dear sir,
>> >> >
>> >> > I am using a gmap.net control to use OSM but suddent it started giving 
>> >> > 403 error. Please suggest.
>> >> >
>> >> > Thanks,
>> >> > Vijaya
>> >> > ___
>> >> > dev mailing list
>> >> > dev@openstreetmap.org
>> >> > https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] 403 forbidden

2019-05-29 Thread Vijaya Nand
DeadSureMobile/1.0.0.1

This is my user agent. Please do let this approved so that I can use it. I
am using gmap.net.

Thanks, please confirm so that I can test.




On Wed, 29 May 2019, 16:07 Grant Slater, 
wrote:

> Hi Vijaya,
>
> I would suggest using a user-agent with name and version at least eg:
> org.github.app-name/1.0.0
> Do make sure you read and follow:
> https://operations.osmfoundation.org/policies/tiles/
>
> Here is why gmap.net got blocked and will remain blocked:
>
> https://github.com/radioman/greatmaps/blob/799f3a38a65350ff00c21acf9b88c96848bb3880/GMap.NET.Core/GMap.NET.MapProviders/GMapProvider.cs#L329-L336
>
> Kind regards,
>
> Grant
>
>
> On Wed, 29 May 2019 at 11:04, Vijaya Nand 
> wrote:
> >
> > Hi Grant.
> >
> >
> > Please suggest me what user agent should I pass. My application name is
> DeadsureMobile.
> >
> > Please suggest.
> >
> > It's Very urgent for me.
> >
> > Thanks,
> > Vijaya
> >
> > On Wed, 29 May 2019, 15:27 Grant Slater, 
> wrote:
> >>
> >> Hi Vijaya,
> >>
> >> We have recent had to crack down on users who are violating our usage
> policy:
> >> https://operations.osmfoundation.org/policies/tiles/
> >>
> >> Please check what User-Agent your app uses or if you fake any browser
> headers.
> >>
> >> The tile.openstreetmap.org CDN is at breaking point.
> >> We now frequently push more than 3Tb per second of traffic, with over
> >> 22000 requests per second/
> >> We are reaching the limit of what our current stack/config can handle.
> >>
> >> Kind regards,
> >>
> >> Grant
> >> Part of the OpenStreetMap sysadmin team
> >>
> >> On Wed, 29 May 2019 at 10:26, Vijaya Nand 
> wrote:
> >> >
> >> > Dear sir,
> >> >
> >> > I am using a gmap.net control to use OSM but suddent it started
> giving 403 error. Please suggest.
> >> >
> >> > Thanks,
> >> > Vijaya
> >> > ___
> >> > dev mailing list
> >> > dev@openstreetmap.org
> >> > https://lists.openstreetmap.org/listinfo/dev
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] 403 forbidden

2019-05-29 Thread Grant Slater
Hi Vijaya,

I would suggest using a user-agent with name and version at least eg:
org.github.app-name/1.0.0
Do make sure you read and follow:
https://operations.osmfoundation.org/policies/tiles/

Here is why gmap.net got blocked and will remain blocked:
https://github.com/radioman/greatmaps/blob/799f3a38a65350ff00c21acf9b88c96848bb3880/GMap.NET.Core/GMap.NET.MapProviders/GMapProvider.cs#L329-L336

Kind regards,

Grant


On Wed, 29 May 2019 at 11:04, Vijaya Nand  wrote:
>
> Hi Grant.
>
>
> Please suggest me what user agent should I pass. My application name is 
> DeadsureMobile.
>
> Please suggest.
>
> It's Very urgent for me.
>
> Thanks,
> Vijaya
>
> On Wed, 29 May 2019, 15:27 Grant Slater,  wrote:
>>
>> Hi Vijaya,
>>
>> We have recent had to crack down on users who are violating our usage policy:
>> https://operations.osmfoundation.org/policies/tiles/
>>
>> Please check what User-Agent your app uses or if you fake any browser 
>> headers.
>>
>> The tile.openstreetmap.org CDN is at breaking point.
>> We now frequently push more than 3Tb per second of traffic, with over
>> 22000 requests per second/
>> We are reaching the limit of what our current stack/config can handle.
>>
>> Kind regards,
>>
>> Grant
>> Part of the OpenStreetMap sysadmin team
>>
>> On Wed, 29 May 2019 at 10:26, Vijaya Nand  wrote:
>> >
>> > Dear sir,
>> >
>> > I am using a gmap.net control to use OSM but suddent it started giving 403 
>> > error. Please suggest.
>> >
>> > Thanks,
>> > Vijaya
>> > ___
>> > dev mailing list
>> > dev@openstreetmap.org
>> > https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] 403 forbidden

2019-05-29 Thread Vijaya Nand
Hi,

Right now I want to consume tiles from tiles.openstreetmap.org. later I
will build my own server.

I want to use user agent DeadSureMobile. Please unblock this so that my
application can be used for now.


On Wed, 29 May 2019, 15:53 Andy Townsend,  wrote:

> On 29/05/2019 11:04, Vijaya Nand wrote:
> >
> > Please suggest me what user agent should I pass. My application name
> > is DeadsureMobile.
> >
> >
> For the avoidance of doubt, if you want to consume map tiles of the same
> format as the ones that you see at OpenStreetMap.org the safest way is
> to create your own.  See
> https://switch2osm.org/manually-building-a-tile-server-18-04-lts/ for
> how to do this.
>
> If you'd rather use someone else's tiles, see
> https://switch2osm.org/providers/ .
>
> Other tile technologies are available - there's lots of stuff in the OSM
> wiki and elsewhere.
>
> Best Regards,
>
> Andy
>
> (not affiliated with the OSM admins in any way; just trying to provide
> some helpful links)
>
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] 403 forbidden

2019-05-29 Thread Andy Townsend

On 29/05/2019 11:04, Vijaya Nand wrote:


Please suggest me what user agent should I pass. My application name 
is DeadsureMobile.



For the avoidance of doubt, if you want to consume map tiles of the same 
format as the ones that you see at OpenStreetMap.org the safest way is 
to create your own.  See 
https://switch2osm.org/manually-building-a-tile-server-18-04-lts/ for 
how to do this.


If you'd rather use someone else's tiles, see 
https://switch2osm.org/providers/ .


Other tile technologies are available - there's lots of stuff in the OSM 
wiki and elsewhere.


Best Regards,

Andy

(not affiliated with the OSM admins in any way; just trying to provide 
some helpful links)




___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] 403 forbidden

2019-05-29 Thread Vijaya Nand
Hi Grant.


Please suggest me what user agent should I pass. My application name is
DeadsureMobile.

Please suggest.

It's Very urgent for me.

Thanks,
Vijaya

On Wed, 29 May 2019, 15:27 Grant Slater, 
wrote:

> Hi Vijaya,
>
> We have recent had to crack down on users who are violating our usage
> policy:
> https://operations.osmfoundation.org/policies/tiles/
>
> Please check what User-Agent your app uses or if you fake any browser
> headers.
>
> The tile.openstreetmap.org CDN is at breaking point.
> We now frequently push more than 3Tb per second of traffic, with over
> 22000 requests per second/
> We are reaching the limit of what our current stack/config can handle.
>
> Kind regards,
>
> Grant
> Part of the OpenStreetMap sysadmin team
>
> On Wed, 29 May 2019 at 10:26, Vijaya Nand 
> wrote:
> >
> > Dear sir,
> >
> > I am using a gmap.net control to use OSM but suddent it started giving
> 403 error. Please suggest.
> >
> > Thanks,
> > Vijaya
> > ___
> > dev mailing list
> > dev@openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/dev
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] 403 forbidden

2019-05-29 Thread Maarten Deen

On 2019-05-29 11:56, Maarten Deen wrote:

On 2019-05-29 11:23, Vijaya Nand wrote:

Dear sir,

I am using a gmap.net [1] control to use OSM but suddent it started
giving 403 error. Please suggest.


Going to gmap.org


gmap.net of course.

Maarten

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] 403 forbidden

2019-05-29 Thread Grant Slater
Hi Vijaya,

We have recent had to crack down on users who are violating our usage policy:
https://operations.osmfoundation.org/policies/tiles/

Please check what User-Agent your app uses or if you fake any browser headers.

The tile.openstreetmap.org CDN is at breaking point.
We now frequently push more than 3Tb per second of traffic, with over
22000 requests per second/
We are reaching the limit of what our current stack/config can handle.

Kind regards,

Grant
Part of the OpenStreetMap sysadmin team

On Wed, 29 May 2019 at 10:26, Vijaya Nand  wrote:
>
> Dear sir,
>
> I am using a gmap.net control to use OSM but suddent it started giving 403 
> error. Please suggest.
>
> Thanks,
> Vijaya
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


Re: [OSM-dev] 403 forbidden

2019-05-29 Thread Maarten Deen

On 2019-05-29 11:23, Vijaya Nand wrote:

Dear sir,

I am using a gmap.net [1] control to use OSM but suddent it started
giving 403 error. Please suggest.


Going to gmap.org redirects me to https://www.cggh.org/
Is it related to https://github.com/radioman/greatmaps ? Maybe the 
domain was transferred?


Regards,
Maarten


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] Map not showing

2019-05-29 Thread Vijaya Nand
> Dear sir,
>
> I am using a gmap.net control to use OSM but suddent it started giving
> 403 error. Please suggest.
>
> Thanks,
> Vijaya
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


[OSM-dev] 403 forbidden

2019-05-29 Thread Vijaya Nand
Dear sir,

I am using a gmap.net control to use OSM but suddent it started giving 403
error. Please suggest.

Thanks,
Vijaya
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev