Re: [OSM-dev] status/prognosis of Merkaator?

[Ilya Zverev]

28.10.2019 18:59, marc marc пишет:

Le 28.10.19 à 15:29, Greg Troxel a écrit :

   Is Merkaator a reasonable choice for OSM editing, it terms of working
   with current APIs and preset notions?


I never use it myself but accord to the stat, it's still in use
in the top 10 of end-user editor.
https://wiki.openstreetmap.org/wiki/Editor_usage_stats


While it is in the top 10 by number of edits, it is far down by the 
number of users, below Level0 and GNOME Maps. Its usage has been very 
close to Potlatch 1 since 2014.


Ilya


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Fwd: Request for a Map as JPEG export

[Ilya Zverev]

The easiest way with no programming would be to install QGIS and use a shape 
file for your region from Geofabrik. You can import your kml file there, and it 
gives a high level of control over styling and layout.

https://qgis.org/ru/site/forusers/download.html 

https://download.geofabrik.de/  

Tutorials and such can be easily found by googling.

Good luck,
Ilya

> On 16 Apr 2019, at 01:23, Flaviu S  wrote:
> 
> Dear Dev Community,
> 
> I searched for a way or a tool which can do following:
> 
> I need to create a JPEG with the Size of A0 (841 x 1189 mm) at least or of 
> 2A0 (1189 x 1682 mm) if possible.
> 
> The corners of the map have following coordinates:
> 48.03731, 15.85933
> 47.88788, 15.85933
> 47.88742, 16.33449
> 48.03601, 16.33476
> 
> 
> The Zoom Level should be as high as possible, so that the names of the 
> streets are still readable.
> For example like this 
> 
> 
> And then I need also this polygons (a KMZ File) to be on the map
> 
> 
> Do you have any possibility to make a JPEG like this in that high Zoom Level?
> Because without programming skills, the only way I found is to make more 
> screenshots and then stitch it together to one big picture and then resize it 
> to A0, but this would mean, I can't change anything afterwards.
> 
> If you can help me, I would be very thankful ☺
> 
> P.s.:
> If something is unclear or you need more information, please feel free to 
> contact me.
> 
> Best regards
> Flavius Sarca
> 
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSMand Live can steal your money

[Ilya Zverev]

I’d like to remind everyone that OsmAnd is an open app, with both mobile and 
webside code available on GitHub. The author would be grateful if anybody here 
updated the php code to use OAuth instead of login and password:

https://github.com/osmandapp/osmandapp.github.io/tree/master/website 


Ilya

> 12 янв. 2018 г., в 16:15, Darafei Komяpa Praliaskouski  
> написал(а):
> 
> Hi,
> 
> https://osmand.net/osm_live  requests user's OSM 
> password and e-mail in exchange of promise of bitcoin payment.
> 
> There is no way to check that the password is not being collected, with or 
> without knowledge of service authors. At least 1100 accounts may be affected.
> 
> Simplest attack vector may be "if password matches on google drive of this 
> e-mail and there's a backup of wallet there and password matches there too, 
> get all the money from there".
> 
> What can be done on osm.org  side to mitigate it?
> Can password reset be forced for affected users, and for those who keep 
> coming to that form?
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Wrong node ids in osmupdated planet

[Ilya Zverev]

Hi,

I have downloaded a planet file around... March?.. and have been updating it 
using osmupdate 0.3H every two weeks or so. And now I've met a curious bug. 
Look at these way nodes:

$ ./osmconvert planet-171009.o5m --out-osm | grep -A 3 '



But OSM API gives: http://www.openstreetmap.org/api/0.6/way/419952872





How could this happen?

Thanks,
Ilya


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Planet file with preprocessed lines/polygons

[Ilya Zverev]

The answer hugely depends on your task. There is no such preprocessed geojson 
planet dump, because it would be huge and a pain to use.

Most people use libosmium or pyosmium to scan through the features in an OSM 
file. You will get geometries and even reconstructed multipolygons. See 
http://osmcode.org/

Then there are QA Tiles, a whole planet split into small tiles in the MVT 
format. People use them to build validation and QA services, like counting 
buildings and finding geometry errors. Links and a few examples are here: 
https://www.mapbox.com/blog/osm-qa-tiles/

To get billions of GeoJSON objects for every feature in the planet, you can use 
osm2pgsql to import the planet dump into a PostgreSQL+PostGIS database. It will 
make three tables for points, lines and polygons. After that, pgsql can give 
you geometries and tags in geojson format: you would have to make three queries 
and assemble the result into one big geojson file. Or process these on the 
spot. The drawback is that you would need half a terabyte for that.

Ilya

> 12 мая 2017 г., в 19:18, Christoph Lingg  написал(а):
> 
> Hello,
> 
> when it comes to read raw OSM dumps it's quite straightforward to parse 
> nodes: their geometry properties can be read alongside with their tags. When 
> it comes to linestrings and relations it is more complicated to access their 
> geometry: the geometry of referenced nodes needs to be combined into lines 
> and polygons. Also one needs to decide which linestrings/relations are 
> actually lines and which are areas.
> 
> I know how this can be done but I am wondering if there are preprocessed 
> datasets around that have geometries already precomputed. That would make 
> sense to me as a lot of people face the sample problem and this step is quite 
> resource intense.
> 
> A huge file containing all osm items as geojson would be my dreamcase. Does 
> this exist?
> 
> Thank you for your time and your help!
> Christoph Lingg
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] The EWG meeting is tomorrow

[Ilya Zverev]

Hi everyone,

As you might know, the new Engineering Working Group is meeting every two weeks 
on Mumble. Read the meeting minutes on the wiki: 
https://wiki.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2017-03-14

We are not discussing any software development, but trying to help current and 
future OSM developers. The two tasks we are working on now are:

1) Google Summer of Code 2017. The student application period opened a week ago 
and ends on 3rd of April, next Monday. Do not put off answering students' 
emails.
2) The Top Ten Tasks list. The previous one was compiled six years ago, and we 
are looking at refreshing it. For now we are collecting ideas on the etherpad: 
https://pads.ccc.de/k4rlFOGIHb

The next EWG meeting is tomorrow at 20:00 UTC:

https://www.timeanddate.com/worldclock/fixedtime.html?msg=EWG+Meeting&iso=20170328T20&ah=1

We would like everybody who considers themselves a EWG member, interested in 
organizing mentors or in compiling the Top Ten Tasks list to join Mumble and to 
discuss these topics. We use the usual HOT server, see instructions here: 
https://wiki.openstreetmap.org/wiki/Mumble

Ilya
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OpenStreetMap accepted for this year's GSoC

[Ilya Zverev]

Hi,

There was a misunderstanding concerning the new Engineering Working 
Group. We had that group for a while, it discussed a lot of engineering 
topics, became a group for programmers and fizzled out two years ago.


The new EWG is not the same group. Programmers will have nothing to do 
on it. The EWG now is for managing developers and development projects. 
Basically it connects new developers with OSM-related projects, helping 
them find a suitable task, and helping OSM-related projects with finding 
new developers. EWG members do not write code (as members) and do not 
tell projects maintainers what to do.


The first - and for a time, the only - task is managing the Google 
Summer of Code and its projects. We believe that it would be better done 
by a group than by a single person, like it was last year. If you want 
to join, known fully that it is a management, not development group, we 
would be glad to have your help. Be prepared that your ideas will be put 
on hold: we are starting slow.


The old EWG and the new one are completely different (though the stated 
goals overlap), so we start the member list from scratch. I told the 
"old EWG" members about the transition last Summer, and if any of them - 
and any of you - want to join, you are welcome to enroll and help the 
project.


Also, Peter forgot the link to the Mumble settings and instructions 
page: https://wiki.openstreetmap.org/wiki/Mumble


Timezone-aware meeting date: 
https://www.timeanddate.com/worldclock/fixedtime.html?msg=EWG+Meeting&iso=20170228T20&ah=1


Ilya

27.02.2017 22:45, Peter Barth пишет:

Hi all,

we got accepted as a mentoring organization for this year's GSoC!
https://summerofcode.withgoogle.com/organizations/5161142197420032/

Yay! So what's next? Students now have a chance to get to know their
favorite organization, talk to possible mentors, make them self known
to the project idea they like best and all that. Starting from March
20. the official application period for students opens.

Besides I'm excited to tell you that the EWG tries to resurrect :-)
They're a perfect match for our GSoC participation by finding mentors,
finding help for students and all that. As such the EWG will have their
first meeting tomorrow at 20:00 UTC on Mumble (the HOT server). If
you're a potential mentor or want to join/help the EWG feel free to
join! The first meeting is in particular to tidy up our ideas list and
try to assemble a list of mentors. Hope to see you.

Happy GSoC,
Peda




___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Querying for non-native characters in name field

[Ilya Zverev]

Try this one: http://overpass-turbo.eu/s/lAf

Though I'm not sure why it doesn't catch all the weird Icelandic 
characters. See this post for better option, although in development: 
http://www.openstreetmap.org/user/mmd/diary/40197


IZ

29.01.2017 20:43, Jóhannes Birgir Jensson пишет:

One side effect of Maps.me is the editing of name= field (this has been
mostly fixed in the latest versions though, most entries now go into
name:phone language).

I want to be able to do an overpass query for Iceland where name= field
contains non-Icelandic characters. These could be for example Chinese,
Cyrillic or even other European characters (such as âà for example). I'm
guessing it could be difficult for the latin characters but hopeful it
would be easier for non-latin alphabets.

Is there a magic formula for achieving this?

--Stalfur / Jói


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev




___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] A new reverter

[Ilya Zverev]

Hi everyone,

If you needed to revert a changeset and thought, all tools seem quite 
complex, I've got you another one. It is just a python script with no 
dependences on third-party libraries or frameworks.


https://github.com/Zverik/simple-revert

Just do a git clone and use it (responsibly!) Pass a user name to see 
their changesets, a series of changesets ids to revert them (the result 
will be better if you revert a bunch instead of one by one).


There are limitations: it won't revert old changesets that modify a lot 
of references. The script was made to revert mostly tag modifications 
and node moves, and it does it by creating a diff and applying it to the 
latest version. Also it skips tags that were changed after the given 
changeset.


The second script there can restore any old version of any object, 
undeleting missing references. The result might be messy for complex 
relations, so you should check it afterwards. To get an osmChange file, 
redirect scripts' output to it.


IZ

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Unsigned 32 bit node numbers in applications

[Ilya Zverev]

Thanks for the reminder, I've updated http://textual.ru/64/ with a new 
countdown.

As http://wiki.openstreetmap.org/wiki/64-bit_Identifiers states, most 
applications have switched to 64-bit ids. Among the exceptions seems to be OSRM 
(unsigned ints).

IZ

> 9 февр. 2016 г., в 10:05, Andrew Hain  
> написал(а):
> 
> Just a quick reminder: is there anyone still using unsigned 32 bit 
> node numbers? The database is approaching node 40.
> 
> --
> Andrew
> 
> 
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Using native social SDK for signing in to OSM on mobile

[Ilya Zverev]

First, Facebook SDK is open-source. So there are no proprietary solutions 
related to this discussion. Google login SDK is probably open too, though I 
didn't check that.

The discussion with Martijn is about a different things: while he proposes to 
change the whole registration process altogether, possibly removing some 
important steps, I just want to use already existing processes from a mobile 
application. That would involve adding some endpoints to osm.org, and granting 
access to some mobile apps.

JOSM already circumvents the permissions confirmation step, but it still works 
only with login and password. I want to employ social accounts, so a user has 
nothing to type at all.

IZ


> 28 дек. 2015 г., в 15:15, Simon Poole  написал(а):
> 
> 
> If I understand Ilya correctly what he wants to avoid is (the hassle of) the 
> authorisation step when using OAuth. During this process you need to login to 
> openstreetmap.org with your credentials and then confirm that the app is 
> allowed to access the API on your behalf.
> 
> To see what is involved in practical terms you can try to use the HOT task 
> manager, maproulette etc (or on a mobile device Vespucci, I assume Go Map! 
> uses OAuth too, as any current third party app for OSM should).
> 
> The authorisation is a one time process (per app) and as such I'm not quite 
> convinced that the whole discussion isn't a solution looking for a problem, 
> but Ilya is correct in that it does involve the hassle of people remembering 
> their google/FB/whatever password. Naturally on a mobile device you want to 
> minimize typing in any case so I'm mildly in support of at least 
> investigating what this would entail (it is unlikely that we would use a 
> proprietary solution in Vepsucci though, on other devices and with other apps 
> the trade-offs might be different).
> 
> The above is a separate but related issue to making the signup process 
> "mobile friendly"  see 
> https://github.com/openstreetmap/openstreetmap-website/issues/894 for a 
> longish discussion.
> 
> Simon
> 
> Am 24.12.2015 um 15:58 schrieb Greg Troxel:
>> Ilya Zverev 
>>  writes:
>> 
>> 
>>> This can be made a part of a policy for allowing apps to use OSM
>>> official social accounts.
>>> 
>> Can you explain what you mean by "OSM official social accounts"?
>> Perhaps it is just me that doesn't get it, but I am not following what
>> you really mean.
>> 
>> 
>> 
>> ___
>> dev mailing list
>> 
>> dev@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/dev
> 
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Using native social SDK for signing in to OSM on mobile

[Ilya Zverev]

> 25 дек. 2015 г., в 8:36, Ineiev  написал(а):
> 
> On Thu, Dec 24, 2015 at 01:55:06PM +0300, Ilya Zverev wrote:
>> My answer is, absolutely. If a random person walking with their phone adds
>> opening hours or marks a shop closed, I don't care if she/he is a part of
>> community or not.
> 
> What if they add copyright-encumbered data as well?

What if a person with a desktop editor and access to thousands of mapping 
portals and paper maps starts adding copyright-encumbered data? It is much 
easier from a desktop than from a mobile editor. Should we ban JOSM, since it 
is used for most questionable imports and copyright violations?

> 
>> I think it's time to stop taking OSM for a playground of a few geo-geeks.
>> There are thousands of people who fall for "free and open wiki map", and
>> millions who don't care, but who use our map and want to have the freshest
>> data on it.
> 
> I'm used to think that OSM is _the_ project for those who care; for those
> who don't, there is plenty of other possibilities, isn't it?

It was. It still is, but the definition of "care" is much wider now. HOT 
attracts people who care not about the map, but about refugees and crisis 
victims. With a wider mobile reach we can attract people caring about their 
neighbourhood accessible in a mapping application, but who don't care which map 
it is. For now, other mapping projects are luring these people, so why can't we?

IZ
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Using native social SDK for signing in to OSM on mobile

[Ilya Zverev]

Hi Greg,

Signing in on mobile devices is hard. You click "sign in" and then have to 
remember your login and password. There is an alternative method: facebook and 
google buttons on a login form. You can try them yourself: after linking your 
account with a social account, signing in is just a matter of pressing one 
button.

Registering a new user is much harder on mobile, since you have to type 
everything twice, and then leave for another app to confirm a e-mail. Again, 
with a facebook/google button, you don't have to type anything: e-mail and 
display name are pre-filled, and password is optional. E-mail confirmation step 
is also omitted, since social networks have already validated your e-mail. So, 
registering with facebook or google is a straightforward process of 3-4 button 
clicks and no typing.

But that's on a desktop browser. You most likely have logged in to some social 
accounts on your phone. For example, most android phones are unusable without a 
google account. And millions of iPhone users use facebook. So it's safe to 
assume most mobile users have a permanent login to a social account.

I haven't used Vespucci in a while, but I can assume that when you try to sign 
in with it, it opens an inline browser with a login form. You can remember and 
type your login/password, or click on any social button. Alas, after that you 
would have to remember your google/facebook login and password, and type them 
in. Because even if you are already signed in to these in your system, browser 
panes do not share the account. So there is no simplification of a signing in 
process with social buttons in a mobile OSM editing app.

I want to use a system-wide social account on mobile devices. So that a user 
could sign in in a mobile OSM app with a single button click (provided he/she 
is logged in to facebook/google on a device). To do that, I need to change 
osm.org source code a bit, adding an endpoint, and to allow third-party apps to 
create "tokens", identification strings, that osm.org would be able to use.

The latter requires that the mobile app developer either knows two keys, which 
are currently stored only on osm.org server, or that any app that wants to 
allow social login to be registered by an OWG member who has access to 
facebook/google system account. When you click "facebook" button on osm.org 
login form, you are taken to an "OpenStreetMap" facebook app and are asked to 
allow it your name and e-mail. That is the social app/account I'm talking 
about. You cannot make a working social login in a mobile app without having 
some kind of access to that "OpenStreetMap" facebook app.

IZ

> 24 дек. 2015 г., в 17:58, Greg Troxel  написал(а):
> 
> 
> Ilya Zverev  writes:
> 
>> This can be made a part of a policy for allowing apps to use OSM
>> official social accounts.
> 
> Can you explain what you mean by "OSM official social accounts"?
> Perhaps it is just me that doesn't get it, but I am not following what
> you really mean.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Using native social SDK for signing in to OSM on mobile

[Ilya Zverev]

My answer is, absolutely. If a random person walking with their phone adds 
opening hours or marks a shop closed, I don't care if she/he is a part of 
community or not. Having them logged in with an OSM account makes them 
contactable, and that's all we need.

I think it's time to stop taking OSM for a playground of a few geo-geeks. There 
are thousands of people who fall for "free and open wiki map", and millions who 
don't care, but who use our map and want to have the freshest data on it. Can 
we allow people with a slightly less responsibility here? Again, they are 
signing with their social accounts, that give much more private information 
about them, than an e-mail address. They are surely as contactable as regular 
OSM users.

As for 5 seconds, I am not proposing to drop CTs and descriptions of the 
project on sign up. Note that the original post is about signing in, not 
signing up. Of course, signing up would also be made easier, but I agree that 
new users, no matter how casual, should agree to CTs and learn a couple things 
about OpenStreetMap. This can be made a part of a policy for allowing apps to 
use OSM official social accounts.

IZ

> My question is: Do we want to encourage casual editing?
> 
> And my answer is "not 100% sure but perhaps rather not".

> There are some benefits to casual editing; if people could just fire off a 
> quick edit to something without even signing in then surely we could get more 
> people to do just that - upload a quick OCR'd photo of shop opening times or 
> whatever. Point, shoot, upload, bam! - OSM improved in 5 seconds. I see that 
> benefit and I would like to have it.
> 
> But I also view us OSM contributors as a community. We share something. We 
> care for this project together. We participate in various communication 
> channels. We watch our backyard. We chat up new users and invite them to 
> meetings.
> 
> I think it is important for people to make a decision to join this community. 
> This decision is not just a quick "I agree" screen where you put your work 
> under a certain license; it also means you should know that you're signing up 
> for something here; that you take responsibility; that you have to be 
> contactable, and will be contacted, about your contributions.
> 
> Making it too easy to breeze through the signup process, on a mobile device, 
> using your stored credentials from elsewhere - how can we expect anyone who 
> signs up this way to understand what this project is about, what he's signing 
> up to?
> 
> "Making signup easier" is certainly a good goal to have, but signup includes 
> getting people to understand OSM. A workflow that lets people sign up in 5 
> seconds but lands us with users who don't even know what the consequences of 
> their actions are is not a step forward, it is detrimental to the project in 
> my opinion.
> 
> This is not saying you shouldn't write an easy to use mobile editor, or you 
> shouldn't attempt to reduce the mobile signup workflow to a few clicks, but 
> from anyone who ships an app that does quick signups I'd like to see a 
> concept of how they intend to make sure that users understand what they are 
> signing up for (legally and socially).

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Using native social SDK for signing in to OSM on mobile

[Ilya Zverev]

Hi everyone,

Suppose I'm working on a mobile editor for iOS and Android. You can't 
edit anything without signing in, so first a user has to enter their 
login and password. If they are not registered in OSM, too bad: the 
registration workflow is overly complex for mobile. One has to enter 
login/password/e-mail, wait for confirmation, switch to mailing app, 
click a link, close a browser, return to the editing app, enter login 
and password again.


But osm.org for some time has Facebook and Google login. They don't 
require password and e-mail confirmation, and most mobile phone users 
has at least one. For example, most Android phone users have a Google 
account. There are native SDKs for both of these networks, so one 
doesn't even have to enter their login and password: these are stored 
somewhere on the phone.


I mention "native", because the common way of authentication on osm.org, 
with a WebView, won't work: you would have to enter your Facebook or 
Google credentials from scratch. Which is not simpler than registering 
on osm.org.


I would like osm.org to support authentication via native social SDKs. 
It would benefit current and future mobile editing apps, and would 
drastically increase the number of OSM editors (that is, users). I'm 
writing all this, so authors of other editing apps could show their support.


To do that, we need two things. First, authentication on osm.org with 
social login tokens bypassing omniauth web flow. For Facebook, this pull 
request https://github.com/openstreetmap/openstreetmap-website/pull/1114 
is a way, although not perfect. I assume there is something like that 
for Google.


Second, there are social accounts, to which official OSM social logins 
are linked. E.g. a facebook app. To allow signing in with a native SDK, 
an app id should be registered with the social account. That is, we need 
a policy for including mobile editing apps there, and a person 
responsible for that. I've sent a draft to OWG, which has these items:


- The application code must be published and accessible by members of OWG.
- There must be a downloadable version of the app with the usual login 
via a password.
- The application should have some data editing capabilities, for which 
signing in is required, and must use proper changeset tags.
- The application must already have some releases, and must be used, 
with at least 100 users / downloads. It must have a wiki page and at 
least one related blog entry.
- Secret keys provided by social apps must not be included anywhere in 
the application: not in code, not in resources, etc. They can be used 
only on a server.


Well, it's only a draft, intended mostly to start a conversation. There 
is a way to circumvent this, but it includes making osm.org accept any 
tokens from any social accounts/apps, which is not good.


The third step would be adding an API for signing in and registering new 
members, but that is too complex and not a topic of this discussion.


What do you think of this?

My goal is to have "Facebook"/"Google" buttons, a single press on which 
is all that's needed for editing OpenStreetMap (well, there are extra 
steps for registering, but the idea is the same).


IZ

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Registering on osm.org with a social account

[Ilya Zverev]

Hi everyone,

We at maps.me (well, me alone for now) are looking into simplifying OSM sign up 
process on mobile devices. As you know, the current process, which involves 
opening a browser, then opening a mail app, then opening a browser again, then 
returning to an app and entering credentials once again, is quite disruptive. 
But since we already support linking accounts to social login services (like 
Facebook and MS Live), there are few things that could simplify registering 
users on mobile. I wrote some thoughts on hackpad:

https://hackpad.com/Facebook-Registation-for-OSM.org-4MI6M5EeeRh

My point is to create a set of API methods that allow for user registration 
without at any point leaving the app, but not removing any steps from the 
registration process (except e-mail verification, which is done by a third 
party). This would greatly increase number of OSM editors, not only because web 
registration will take less clicks, but from a great number of mobile users, 
who don't want to bother with a website.

I can probably learn RoR and make these changes myself, but for now I'd like to 
learn if the community is good with this proposal.

IZ
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] load-osm.sh

[Ilya Zverev]

Paul Norman wrote:

On 10/12/2014 2:49 PM, Ilya Zverev wrote:

Hi! Yesterday I had a simple task: there is a rendering server, which
is not to be minutely updated. So its PostgreSQL database doesn't need
temporary "slim" tables. But I cannot import an extract without
creating those, so basically I can import 3 times less data than
possible.

It's worth noting that the --drop mode in osm2pgsql will delete the slim
tables when done, and also not create the indexes on those tables. As
those indexes are large, this is a substantial space savings in itself.
--drop also will create more compact indexes for the rendering tables.


Oh. I forgot to check osm2pgsql options and missed --drop. Now it's been 
added as an extra mode "loadc". It is ~7% faster than "load"+"clean", 
and produces an sql dump of nearly the same size. Thanks.


IZ

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] load-osm.sh

[Ilya Zverev]

Hi! Yesterday I had a simple task: there is a rendering server, which is 
not to be minutely updated. So its PostgreSQL database doesn't need 
temporary "slim" tables. But I cannot import an extract without creating 
those, so basically I can import 3 times less data than possible.


The solution is obvious: I rented a huge hourly-priced droplet, 
installed postgresql and osm2pgsql there, imported a big OSM extract, 
deleted "slim" tables and transferred database contents to said 
rendering server.


Well, actually, being a programmer, I first spend half a day on 
automating this task. The script load-osm.sh has just been uploaded to 
https://github.com/Zverik/regional , along with instructions. Basically 
you would need to run it 4 times, once as root, on a clean droplet. 
Tested on DO's ubuntu 12 & 14, fedora 19 & 20, debian 7.


Not sure if anyone needs it, and that it hasn't been done before, but still.

IZ

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] MapBox imagery in the offset database

[Ilya Zverev]

Hi! Recently someone changed MapBox Imagery URL in JOSM, which resulted 
in different keys for the same imagery in imagery offset database. This 
prompted me to create a distinct id for mapbox imagery. The regular 
expression for matching URL is 
"tiles\.mapbox\.com/v[3-9]/openstreetmap\.map". All mapbox imagery 
offsets in the database were converted to use this id, JOSM plugin from 
this moment also uses it. If you actively use offsets plugin, please 
update it now to access your old stored offsets.


I really hope this is the same layer, with same offsets. If not, please 
tell me earlier, so I can undo the database change.


If you maintain a different database of imagery, please update it 
accordingly.



IZ

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Munitely updated regional osm2pgsql database

[Ilya Zverev]

sly (sylvain letuffe) wrote:
>> and since diffs are only available for the whole planet
>
> FYI, that isn't true anymore :
> http://wiki.openstreetmap.org/wiki/Planet.osm/diffs#Regionally_limited_diffs
> Geofabrik is providing daily diffs for "any subregion you can imagine"
...daily diffs. And, e.g. just for a part of Russia, not the whole.

> and french association provides minute diffs for several countries, and the
> list can be increased on case by cases along with a real need.
Which need is real and which is imaginary? What do I do if I have a
polygon, then increase it as space allows, then change it once more
when I have more money to spare — every time I would ask french
association (which doesn't have any reason to help me) to alter a
cutting polygon?

I prefer to rely only on official services when possible, thus no
augmented diff processing, for example. If, with comments above, this
script is still not the best way to have regional minutely updates,
please point me to an alternative.


IZ


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Munitely updated regional osm2pgsql database

[Ilya Zverev]

Hi! I think it's unfair that the entry level to producing OSM tiles is
prohibitively high: you cannot just install postgis/mapnik/mod_tile on
a $10 vps and have minutely updated country tiles. There certainly
won't be enough memory to run osm2pgsql, and since diffs are only
available for the whole planet, your database would grow 600 MB a day.
I ended up reloading it from a local machine every 10 days, and
quickly grew tired of this.

So, https://github.com/zverik/regional — some scripts that I wrote
for my server, including trim_osc.py, which processes osmChange files,
removing nodes and ways outside a given polygon. It obviously isn't
very effective (it cannot touch ways with no node information, i.e.
those which have only their tags modified), but still, after I
installed it, daily database size increase dropped from 600 to 100
megabytes (which is around 10 times more than the actual extract
growth).

I hope it will be useful to somebody, who's rendering tiles for fun
and not wanting to spare more than 5-10 dollars a month on it.

IZ


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Area data type -- again!

[Ilya Zverev]

Tobias Knerr wrote:

>/  Malcolm Herring wrote:
/>>/  My other wish is that disjunct outers be dis-allowed. Where multiple
/>>/  complete areas need to be associated, then this is the proper use of
/>>/  relations.
/
+1

>/  Why? How would you draw an administrative district with an exclave?
/
A relation for the administrative area with two area elements to
represent the territory, plus possibly a member for the capital and
whatever other members mappers come up with.

As we have relations for representing the relationship between elements,
I believe that we should design the area data type as a polygon (with
holes), not as a multipolygon. After all, it would be odd to have a
relationship between multiple areas as the one special case that works
without relations, while every other combination of elements uses relations.


So... a park with two parts separated by a railway? A city with exclave 
(not administrative boundary, but place multipolygon now)? I live in 
such city now, and I have that kind of park nearby. One can come up with 
more examples, where multiple outer rings are needed.


Why should we restrict ourselves (adding extra tests to API by the way), 
just to have to circumvent this restriction later with relations? Why 
cannot administrative boundary relations be redefined to having an 
entire area as a member, instead of working as a crippled multipolygon 
for areas that were meant to replace multipolygon relations?


My opinion is that there is no much difference in having multiple versus 
one outer rings: the processing is still the same, but there would be 
less checks. And without a good reason to do that, I'm against 
restricting areas to simple polygons with a single outer ring.


IZ

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Area data type -- again!

[Ilya Zverev]

Malcolm Herring wrote:
> I don't like this proposal because the structure is made up of nodes
> rather than ways. This will make for extremely tedious construction as
> well as create much more data. Also it is too different from the 
> existing multipolygon method of creating generic area geometries. This
> proposed structure would mean that all the tools & consumer apps would
> face big re-writes.

Sadly, there is no way to not rewrite tools and apps after area type
introduction, even if we just rename multipolygons to areas and be
done with it. For example, how do you tell closed way from an area in
an editor? On the other hand, tools like osm2pgsql and osmium will not
only become quicker and simplier, because they won't have to make
assuptions on what is area and what isn't, but they will screen most
data consumers (e.g. mapnik) from this change.

As for creating much more data, it is simply untrue. The proposal
would replace ways+relation for most cases with a simple area object
on same nodes, and for complex cases it will function similar to
a multipolygon relation. Construction would be easier, since we would
have knowledge of inner and outer sides, even when we have only some of
nodes or ways.

> There is nothing wrong with multipolygons except for the fact that they
> are relations. This leads to chaos when multiple area geometries need to
> be associated but do not form a logical single geometric entity. If an
> area data type were defined that incorporates the existing multipolygon
> structure, then all existing code that parse and create multipolygon 
> relations would not need re-writing.

That was the basis for my thoughts: I started with an idea to just
make areas a special kind of relation, basically rename multipolygons.
But it would imply converting all regular polygons (there are 120
million just building outlines, compared to 7 million multipolygons in
total) to 2 objects: way + area. That would be bad. So I solved the
problem for other two types of areas: small polygons (just added node
references) and coastlines (reversed way references). Multipolygon
code would still be used for assembling areas, but it would be
unnecessary in most cases, since in most cases of "polygon with a
hole" a simple, node-based area would be enough.

> My other wish is that disjunct outers be dis-allowed. Where multiple
> complete areas need to be associated, then this is the proper use of 
> relations.

Why? How would you draw an administrative district with an exclave?

IZ


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Area data type -- again!

[Ilya Zverev]

steggink at steggink.org wrote:

> Can you please explain how ways referencing areas "work"? Preferably
> with some illustrations. Either I don't get it, or it doesn't make  
> sense. The way as I see it now you'll get some closed loops  
> along/inside ways.

I've made a visual example:
http://wiki.openstreetmap.org/wiki/User:Zverik/Areas#Visual_example

Please read the proposal, you'll find some of your ideas already
introduced there. Also check
http://wiki.openstreetmap.org/wiki/The_Future_of_Areas page, which
outlines problems with the current model and links to different
proposals for area datatype that were made in 2011-2012.

IZ


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Area data type -- again!

[Ilya Zverev]

Hi all! There has been no discussion of areas since November 2012, and
the last EWG meeting made me think of them for some reason. I believe
a proper area type can be done, and it would nicely represent every
kind of areas — from simple buildings to the coastline.

Please see http://wiki.openstreetmap.org/wiki/User:Zverik/Areas

It suggests having an area as both an atomic object - much like ways:
list of lists of nodes, - and as a virtual object to which ways refer.
Like multipolygon relations, but reversed: it does not reference ways
(there could be too many: e.g. the coastline), but ways reference it.
It allows for better consistency checks and smarter partial editing.

IZ


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] List parent objects in OSM XML produced by API

[Ilya Zverev]

Hi! I've just opened this ticket, in which I propose to include /> tags to objects returned by OSM API which were requested individually 
(not with map call):


https://github.com/openstreetmap/openstreetmap-website/issues/768

The tag would allow editors to warn users when they delete or modify an 
object which is a part of other objects, not loaded in an editing 
session. Consequently, it would minimize number of failed preconditions, 
conflicts and broken relations.


I'm sure it won't be much trouble to add support for this tag to popular 
editors — I, of course, would do this for Level0.


Are there any reasons not to do this? Can anyone with Rails Port 
programming experience code this? Parent objects are already listed on 
osm.org's browse pages, so this shouldn't be too hard, or too heavy on 
the database.


IZ

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Tiles provider

[Ilya Zverev]

Hi.

Of course tile providers do not want you to bulk download tiles. Tiles
are not static images, but generated on every request, which places a
load on CPU and disk drives. Simply displaying a map in web browsers
is bearable, because number of tiles requested is low. What you want
is around 7-10 million of tiles. It is a lot. Too much for any server.
You want to overload some server for days just for providing free
tiles. Please do not do this. It's sometimes acceptable to download
~1-3 thousand tiles at maximum, but no more. For your case there are
two options:

* you build tile server yourself and generate tiles needed (be warned,
it's tens of gigabytes);
* you pay some company (e.g. MapBox or Geofabrik) to do that for you.

I did not hear of anyone providing tiles in archives for bulk downloading,
if that's what you had in mind.

IZ

> Thank you for the response. I do have a offline tiles server on
> windows, I was just wondering if there was an easier way to download
> all the tiles of a big area for someone who does not want to install a tiles 
> server.

> I didn't find any tiles provider in the list that allow you to
> download all the tiles from a large area. Everything seems to be online.


>> Date: Wed, 26 Mar 2014 16:58:01 +0400
>> From: zve...@textual.ru
>> To: dev@openstreetmap.org
>> CC: scream...@hotmail.com
>> Subject: Re: Tiles provider
>> 
>> Vince Berubey:
>> > Hi,To download the tiles from level 0 to 17 from a US state, is the
>> > only option to have an offline tiles server? Is there a company
>> > allowing you to do bulk downloading of tiles?
>> 
>> The best option in my opinion is installing postgresql and mapnik on a
>> local machine, and then generating tiles locally. That's what I have
>> done last weekend instead of downloading tiles for zooms 14-15, and it
>> turned out to be much easier that it looks. I've documented steps


IZ


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Tiles provider

[Ilya Zverev]

Vince Berubey:
> Hi,To download the tiles from level 0 to 17 from a US state, is the
> only option to have an offline tiles server? Is there a company
> allowing you to do bulk downloading of tiles?

The best option in my opinion is installing postgresql and mapnik on a
local machine, and then generating tiles locally. That's what I have
done last weekend instead of downloading tiles for zooms 14-15, and it
turned out to be much easier that it looks. I've documented steps
here:

http://wiki.openstreetmap.org/wiki/User:Zverik/Tile_Server_on_Fedora_20

(Someday that page will have something on setting up a tile server,
but for now I stopped at generating static tiles)

All you need is a linux machine and an extract of OSM data for your
state. Try here: http://download.geofabrik.de/north-america.html — you
need an [.osm.pbf] file.

IZ


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] BigMap 2

[Ilya Zverev]

Hi! For a long time there was only one reasonable tool to download and 
stitch map tiles into an image: Frederik's BigMap script. Sadly it 
suffered from some little design problems, which meant I had to fix 
generated perl scripts by hand every time.


Today I have released BigMap 2: http://bigmap.osmz.ru

It shares with the predecessor the spartan user interface and perl 
scripts as the main mode of operation for experienced mappers. But the 
improvements will allow even non-programmers to get images of, for 
example, a mapping party region. I'll just copy the list from github:


* Better landing page with Leaflet and URL parsing.
* More than fifteen popular tile layers to choose from.
* KML, WLD and OziExplorer MAP meta files generation.
* Map downloading script can be produced in Python language.
* Fixed user agent and small pauses while downloading.
* Attribution on generated images.
* Server-side stitching with a queue.

Source code is published on https://github.com/Zverik/bigmap2 under 
WTFPL, as always. Quotas for server-side stitching may be lowered if the 
service eats too much resources (it's up to 100 tiles per image at the 
moment).


IZ

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] GPX Planet extracts

[Ilya Zverev]

Hi! GPX Planet extracts are nearly ready: 
http://zverik.osm.rambler.ru/gps/files/extracts/index.html


Extracts for Germany regions are still being processed, I hope to 
upload them by tomorrow morning (Thursday in the worst case). All other 
files are complete. You can use those extracts of manageable size or cut 
them even more (e.g. just your city) with filter-gpx.pl from 
https://github.com/Zverik/gpxplanet-tools


If you are going to calculate statistics on those files, be warned that 
when a user had uploaded their traces in an gz/bz2 archive, it was made 
into a single trace in the database, and therefore in this dump. So a 
single gpx in a GPX planet doesn't mean a single source gpx.


There are a lot of uses for this trace array, including generation of a 
tile layer for use in editors and preparing printable city maps of those 
traces, like Steve did for his kickstarter (though not with OSM data). 
Or maybe one can visualize a fleet of mappers collecting traces through 
the last 7 years. Not to mention academic possibilites. There is 
probably no larger open dataset of collected GPS points, and it would be 
quite unfortunate to leave it unprocessed.


Thanks,
IZ

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] GPX Planet Dump to PostgreSQL

[Ilya Zverev]

Hi! In case someone has resources for rendering GPX tiles, I've 
finished a tool that loads all tracks from the upcoming GPX planet dump 
into PostgreSQL tables (with PostGIS extension): 
https://github.com/Zverik/gpx2pgsql


I've succesfully tested it on the data Firefishy provided a couple days 
ago: http://not.textual.ru/zverik/2/3/test-25k-gpx.png


All metadata (except tags) is stored in a separate table, so tracks 
could be colored by age or visibility, for example. Though I prefer 
plain black traces -- which I would do, like the last time, with my perl 
scripts (also on github). I plan to update them shortly.



IZ

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Imagery Offset Database beta-testing

[Ilya Zverev]

Hi! I have been working on the imagery offset database JOSM plugin, and 
now it seems to be finished. The same could be said for the server. The 
documentation is here: 
http://wiki.openstreetmap.org/wiki/Imagery_Offset_Database (my English 
seems to have degraded, so please fix any grammar mistakes you find).


Until Sunday, 31th of March the project is in beta-testing. I invite 
you to check out the plugin and populate the database with imagery 
offsets and calibration objects for your regions. I have just begun 
working on a web-interface, but I plan to finish it by the end of the 
week: http://offsets.textual.ru/


The database is published hourly under PDDL license in XML and JSON 
formats. API also supports those two formats, and I hope to see IODB 
support in more editors.


Please don't announce IODB before the release, there are only 6 days to 
go :)


I'm ready to answer any questions and fix any bugs and inconveniences 
you may encounter.



IZ

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Reminder: Node 32-bit exhaustion

[Ilya Zverev]

Andrew M. Bishop wrote:


If using the "standard" toolchain of osm2psql, postgresql and mapnik
what is the minimum software versions that are needed to continue
creating maps after ids reach 2^31-1?


I doubt that mapnik's handling of 32bit ids would in any way affect OSM 
rendering. So you can still use older versions of it without worrying 
that something will break.



The wiki page says that osm2psql version 0.81.1 is required but the
version that I have reports itself as "osm2pgsql SVN version 0.80.0
(32bit id space)".  This suggests to me that it will work up until id
2^32-1 appears but this may be wishful thinking on my part.


Well, you've got less than 24 hours to update your osm2pgsql. 
http://textual.ru/64/ shows that there are 1.3 million nodes left, and 
for the last two days nearly 1.5 million nodes were added daily.


Though I fail to understand what would be broken if you continue to use 
32-bit osm2pgsql. Indices, maybe.



IZ

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Reminder: Node 32-bit exhaustion

[Ilya Zverev]

Stephan Knauss wrote:

could you give details about a use case where mapnik needs the 
osm_id?

The "official" styles do not contain a reference to osm_id, it's an
internal thing in the database.


Or is it going into using osm directly as a datasource instead of 
postrgres?


While it is possible to feed osm files to mapnik directly, as far as I 
understand, osm_id can be fetched even from postgresql database as a 
"feature id". Judging by a relevant issue 
https://github.com/mapnik/mapnik/issues/1662 before December it was 
32-bit. Latest commits dealing with this problem were uploaded in 
January, including an option to turn off BIGINT support (it will be on 
by default) and some tests.



IZ

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Reminder: Node 32-bit exhaustion

[Ilya Zverev]

Andrew M. Bishop wrote:


Since 2^31 nodes will make older versions of some software unusable
and 2^32 nodes will make other versions unusable it would be helpful
to have a wiki page to record this information.  I think that most
useful would be a table of the minimum version of each piece of
software to support true 32-bit (> 2^31) and 64-bit node numbers.


I've started it at 
http://wiki.openstreetmap.org/wiki/64-bit_Identifiers

Please update with any software I've missed.


IZ

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] WHODIDIT

[Ilya Zverev]

I tweaked Zverik's WHODIDIT script to speed-up the queries. These
performance improvements are mainly due to the usage of a spatial 
index

in the database.


Did you benchmark it against old version? Is the performance worth 
doubling the index size and using spatial extensions? To my shame, I 
didn't ever assume there were spatial functions in MySQL, and wrote 
around them. To me it looks the speed of queries remained the same 
(considering your server a bit faster than the one it's currently 
deployed on).



IZ

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev