Re: [OSM-dev] GSOC Project: "Make the website use the API"

[Andy Allan]

On 2 March 2018 at 04:43, Paul Norman  wrote:
> I wrote a blog post on how to get started with the API projects:
> http://paulnorman.ca/blog/2018/02/make-the-website-use-the-api-gsoc-project/
>
> I recommend steps 2, 4, and 5 for anyone applying for a project which
> interacts with the API.

It would be remiss of me to point out that I don't think this project
is a good idea. Although I said similar things last year (
https://lists.openstreetmap.org/pipermail/dev/2017-February/029702.html
) I'll say it again now.

The idea of having a single-page javascript app to replace the browse
pages is overcomplicating a problem that barely exists. We would then
have a browser asking the website for some javascript code, which then
runs and asks the website for some API request. The website will fetch
the data from the database, construct a response in XML, then the
javascript decodes that, and finally constructs some html for the user
to view. Much simpler, in my view, to have the browser make a request,
and the server to respond with the html required.

Now what would be useful is some refactoring of the code. Currently we
have app/controllers/browse_controller.rb#way which returns some html
representing a way, and also completely separate code at
app/controllers/way_controller.rb#read which returns some XML, also
representing the same way. I don't think we need different controllers
just to return responses in different formats. So I'd like to see the
browse pages refactored to use the same code as the XML api, but
that's quite different from what is being proposed in this GSoC
project.

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSMand Live can steal your money

[Andy Allan]

In general, I'd like to disable HTTP Basic Auth to our API, and only
use OAuth. This removes any need to share your OSM password with third
parties. However, developers often find it easier to build
integrations using basic auth, so I can imagine some opposition to
this.

Thanks,
Andy

On 12 January 2018 at 13:15, Darafei "Komяpa" Praliaskouski
 wrote:
> Hi,
>
> https://osmand.net/osm_live requests user's OSM password and e-mail in
> exchange of promise of bitcoin payment.
>
> There is no way to check that the password is not being collected, with or
> without knowledge of service authors. At least 1100 accounts may be
> affected.
>
> Simplest attack vector may be "if password matches on google drive of this
> e-mail and there's a backup of wallet there and password matches there too,
> get all the money from there".
>
> What can be done on osm.org side to mitigate it?
> Can password reset be forced for affected users, and for those who keep
> coming to that form?
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Andy Allan joining web site maintainers

[Andy Allan]

On 10 July 2017 at 13:25, Matthijs Melissen <i...@matthijsmelissen.nl> wrote:
> On 10 July 2017 at 11:14, Andy Allan <gravityst...@gmail.com> wrote:
>> Thanks Tom. My intentions for the next few months are to continue to
>> do whatever I can to encourage new contributors. I've done a lot of
>> refactoring recently which will help whoever tries to make code
>> changes, but there's other things I think I can do too.
>
> Currently, the openstreetmap-website is both responsible for the
> openstreetmap.org web application, and for the API.
>
> How hard would it be to separate these two functions, perhaps even
> into two projects?

It's not hard, but I personally think it's not worthwhile. But as I
said in my response to Frederik, opinions vary and other core OSM
developers will disagree with me. I'll give some counterpoints to your
statements:

> Would there be any advantages in doing so? Intuitively both functions
> seem to be separate projects.

I don't see them as being different at all. If you want to "view a
way" or "view a diary entry", then there's a choice of output formats
- XML, html or json (in the future, perhaps). So one app can easily
have three different views, rather than having an entirely different
application just to change the display format.

Our API currently covers only a small part of the function of the
website. In future we could allow you to post diary entries via the
API, or add new friends, or get a list of comments on your diary
entries. Again, it would seem strange to me to have a rails app that
lets you view your list of friends as a webpage, but then need a
different app running to see that same list in json or XML.

> I could imagine having two smaller
> applications might make maintenance easier.

Or it would spread the developers thinner, and lead to a lot of
duplicated effort.

> It would also mean we
> could, for example phase Ruby out for the app but not for the api or
> vice versa.

I'd rather phase out low-level code like C++! Even better, we can also
phase out home-grown ruby code and use third-party libraries much more
often. There's very little about what we do that needs anything
special, beyond the map call. Most of OpenStreetMap is completely
routine things like signing up, resetting passwords, CRUD on simple
objects and RESTful API calls. And we don't even do much of that -
we're not dealing with millions of requests per second or anything
like that. So the less code we have of our own to maintain, the
better, and the higher-level language it can be written in the better,
as far as I'm concerned.

> Lastly, making the website make only use of well-defined
> api endpoints instead of interface with the database directly might
> also makes things clearer.

I've said similar things in the past, but I realise different
developers have said similar meaning slightly different things. For
me, that means that we should unify the internals of the rails port
(e.g. not having a separate "BrowseController" for viewing a way, and
a different "WayController" for viewing a way in XML). For other
developers it means changing the rails port to make API calls (e.g.
BrowseController would fire off HTTP calls). For other other
developers it means turning the browse html pages into a javascript
SPA that fire off http calls to the API for the data it needs. I'm not
in favour of 2 and 3.

> Have there been any thoughts in this direction?

There's been many thoughts over the years, but nowhere near enough
implementation!

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Andy Allan joining web site maintainers

[Andy Allan]

On 10 July 2017 at 10:48, Frederik Ramm <frede...@remote.org> wrote:
> Hi,
>
> On 10.07.2017 11:14, Andy Allan wrote:
>> My intentions for the next few months are to continue to
>> do whatever I can to encourage new contributors.
>
> I think it would be helpful for new contributors if the following two
> points could be explained:
>
> * What kinds of changes to the API are acceptable while still being "API
> 0.6"? Is anything that adds new API calls automatically on hold until we
> do "0.7", or should we just refrain from breaking existing API calls?

As far as I'm concerned, adding new API calls to the non-core parts of
the API are fine. So e.g. DiaryEntries or Notes but avoiding
nodes/ways/relations/changesets etc.

For those, I'm also personally fine with adding new APIs, but I know
that others prefer for them to go into cgimap at the same time. But
see also below.

For 0.6 there should be no breakage of existing API calls, of course.
I know people expect big things from 0.7, but I'd be happy to see a
0.7 release that doesn't change any data models, but fixes the API.
The API for notes is pretty bad, for example[1]. We could have 0.7 as
an interim API that runs in parallel with 0.6 for a year or two, and
save the big changes for 0.8.

> Maybe this is even on some mental roadmap somewhere - I heard people
> talking about making the web site actually *use* API calls, rather than
> accessing the database directly.

I have many mental roadmaps, but there's a line to be drawn between
long-term personal roadmaps and astronautism :-)

> * What is the relationship between "cgimap" and the web site, and in how
> far are contributions that touch an area handled by "cgimap" required to
> cover both the C++ and the Rails implementation?

Currently they work in parallel. So everything that is in cgimap has a
corresponding ruby version that's not used in production for OSMF (but
is probably used elsewhere in production). More calls are gradually
being added to cgimap - originally it only handled the map call, hence
the name. You can get a feel for the current state by looking at which
calls are passed to cgimap[2] by apache, the rest are handled by
openstreetmap-website.

I would like to change this situation, by using the cgimap-ruby
gem[3]. This would allow us to use cgimap within
openstreetmap-website, and get rid of the corresponding ruby code.
This would be useful for development and self-deployments, even if
those paths are still bypassed in production and still handled by a
standalone cgimap process. It would also mean anything added to cgimap
wouldn't need a corresponding ruby version. It would also mean that
there is less pressure to add more and more API endpoints to cgimap,
and instead only add those that are necessary from a performance point
of view.

There is a slight difference of opinion on whether we should use
cgimap for every API call at some point in the future. I don't think
it's worthwhile and I think the C++ code should only be for
performance-critical things, but opinions vary.

> One reason why this is of interest to me is that I'm still very much
> interested in being able to access deleted objects through the API and
> through the web site. I've made a few half-baked attemtps at
> implementing that in the past
> (https://github.com/woodpeck/openstreetmap-cgimap/tree/deleted_call,
> https://github.com/woodpeck/openstreetmap-website/tree/browse-deleted-objects)
> and would appreciate guidance of how to "do this right", if at all
> possible within the constraints of "0.6".

So there's already the AMF API, as I know you're aware but others
might not be. For the deleted items map call, it's apparently very
inefficient, and there have been plans to "do it properly" for as long
as I can remember. However, I don't see why porting the deleted items
map call to to the XML API should have to wait indefinitely, so I'd
definitely support that. Not least because we could then consider
turning off the AMF api, which is awesome but gnarly code to maintain.

Thanks,
Andy

[1] "POST /api/0.6/notes/addPOIexec" for example, see
https://github.com/openstreetmap/openstreetmap-website/blob/39c5d8caa71b29d70bb136d6e0a560426e089112/config/routes.rb#L102-L106
[2] 
https://github.com/openstreetmap/chef/blob/7720e35c56ebf155de90facca5a421e43974774b/cookbooks/web/templates/default/apache.backend.erb#L66-L75
[3] https://github.com/zerebubuth/cgimap-ruby

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Andy Allan joining web site maintainers

[Andy Allan]

On 22 June 2017 at 21:15, Tom Hughes <t...@compton.nu> wrote:
> I'm pleased to be able to invite you all to join me in welcoming Andy Allan
> as the new co-maintainer on the code base for the OpenStreetMap web site.
>
> Going forward we intend to look to add additional maintainers to further
> diversify and spread the load, so we will be actively looking for candidates
> among those working on the code base.
>
> Please continue to open issues and pull requests on GitHub as before.

Thanks Tom. My intentions for the next few months are to continue to
do whatever I can to encourage new contributors. I've done a lot of
refactoring recently which will help whoever tries to make code
changes, but there's other things I think I can do too. So it will
likely be a combination of:

* My own refactoring priorities, which will pay off for developers in
the long term but aren't very visible to most people.
* Working through the PR and issue backlogs, to try and help some
straightforward things get "over the line" - hopefully motivating for
developers.
* Work on a few end-user facing tasks, so that we're making visible progress.

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] GSoC API mentoring help needed

[Andy Allan]

On 20 February 2017 at 01:04, Paul Norman  wrote:

> 1. cgimap-ruby
>
> I don't yet have a student interested in this, but I'd like to see if one of
> the ones who has contacted me is. This could use a mentor who has dealt with
> ruby gems before, which I haven't. I have a feeling this part of the work
> isn't enough for a full project, so it could pull in something from a
> different API-related proposal.

I think this would be more than enough for a full project. I have
experience with the ruby side of things, but not the C++ side, so I
don't think I would be a good mentor. But the integration of
cgimap-ruby into the rails code would be very valuable, but perhaps
hard to get right. For example, while the basic integration would be
to get cgimap-ruby to create the document and let rails send that to
the client, the more advanced solution is to allow cgimap-ruby to
stream the response to the socket directly, without rails buffering
the whole thing.

> 2. website from API
>
> This is a project to change parts of the website to call the API instead of
> the database for object information. Good candidate pages would be the
> object browse ones (/node/, etc). There are two different technical
> approaches to this, and depending on which route a student takes, the mentor
> might need different skills. The first of these is to do the processing of
> API results on the server and return HTML to the client, like
> http://osm.mapki.com/history/ does. The second is to do the processing of
> API results on the client with Javascript, like
> http://osmlab.github.io/osm-deep-history/ does. For the first, the student
> would be reproducing existing HTML output so the mentor would need knowledge
> of ruby and API calls. For the second, client-side javascript would be
> needed, but less ruby.

I can see the purpose of this, but I've never seen it as being as high
a priority as other developers do. For example, I can see why the
browse pages shouldn't have access to the data in a manner that's not
exposed in the API, but that to me suggests improving the API, rather
than forcing a lowest-common-denominator approach of forcing the
browse pages to use the API.

I would avoid the pure-javascript approach, as it's just rewriting for
the sake of rewriting. My suggestion would be to just change the
existing browse pages code slightly - the controllers should make
(internal) calls to the API endpoints, to replace their direct db
access. Even better if those internal API calls are processed by
cgimap-ruby!

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Karlsruhe Hack Weekend 18/19 Feb

[Andy Allan]

On 4 January 2017 at 14:25, Frederik Ramm  wrote:
> Hi,
>
>a new year - a new hack weekend! Everyone's welcome in Karlsruhe on
> the 18th/19th February if they prefer hacking on something together, or
> at least not totally on their own ;)
>
> Details on the Wiki
> https://wiki.openstreetmap.org/wiki/Karlsruhe_Hack_Weekend_February_2017

For anyone considering travelling to Karlsruhe from London or nearby,
I can recommend going via train which is what I did back in October.
The LGV Est line from Paris to Strasbourg is now open, so it's quicker
to go via Paris than the previous alternative of going via Brussels.
It's quite charming to have an hour in Paris for lunch each way - Gare
du Nord (Eurostar) and Gare de l'Est (TGV direct to Karlsruhe) are
only a few minutes walk apart. https://www.trainline.eu/ was the best
site I found for combining the Eurostar + SNCF / DB tickets for online
purchase.

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Main database wiki page

[Andy Allan]

On 12 December 2016 at 11:54, Darafei "Komяpa" Praliaskouski
 wrote:

> I've been googling for current human-readable document that describes what
> OSM database currently is.

As far as I'm aware, there's no up-to-date human-readable description
of the primary OSM database. The database contains 40+ tables, and a
bunch of foreign keys, sequences, and of course, many columns, so
maintaining any documentation is not a trivial task. That's likely why
any documentation you find is unlikely to have been kept up to date
over the years.

> Can someone with deeper knowledge update this page, and/or make it point to
> a better place to read about current OSMF database cluster setup?

There are two aspects to the database - A) the internal structure
(tables, columns etc) and B) the production cluster setup
(replication, backups etc)

A) The internal structure is governed by the openstreetmap-website
code base, specifically the db/structure.sql definition.

https://github.com/openstreetmap/openstreetmap-website/blob/master/db/structure.sql

This is not very human-friendly, but it is accurate. I never use this
myself, since I find it easier to read the definitions in app/models/
to find how the different tables relate to each other, and I use the
postgresql command line to find out the details of the structure, if
necessary.

B) The cluster configuration is managed via the Chef configuration, found at:

https://github.com/openstreetmap/chef

What happens on each machine is defined in the roles, and the set-up
of database-related machines is controlled by the "db" cookbook.

https://github.com/openstreetmap/chef/tree/master/roles
https://github.com/openstreetmap/chef/tree/master/cookbooks/db

Again, these are not particularly human-readable, but since they are
the real code that controls the configuration, they are always
accurate. Again, I don't usually read the role files directly, but I
would use https://hardware.openstreetmap.org/ , which is generated
from the Chef configuration automatically, as a starting point and a
more human-friendly approach.

Please let me know if you have any questions and I'll do what I can to help.

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Planet change tile expiry list service

[Andy Allan]

On 1 September 2016 at 09:25, Yves  wrote:
> Along with minutely diffs, I wonder if expired tiles lists would be
> something to be shared: this is something each tile server is doing by
> itself by now, and seems a waste in ressources.

It's not really something that should be relied on, since there's no
"one true list" of expired tiles. For example, the osm.org tileservers
ignore relation expiry entirely. Other servers might expire relations
only when certain tags are changed (little point in expiring all of
germany when an unrendered tag on an admin boundary changes). Some
tileservers use the osm2pgsql output. With large polygons, only the
exterior is expired, but  different operators will have different
ideas as to how large a "large" polygon is, or they may want to expire
the interior too.

Or to put it a different way - which list would you like? The one
generated by osm2pgsql, or the one generated by
https://github.com/openstreetmap/chef/blob/6db231309db4c58f4264beab8a72fd46fdd0d01d/cookbooks/tile/files/default/ruby/expire.rb
? Which is "right"?

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Tile usage without proper identification

[Andy Allan]

On 12 August 2016 at 16:32, Komяpa  wrote:
> What are the plans to scale the tile provision infrastructure?

We should be clear here - we have more than enough capacity to handle
all the traffic generated by our mappers, editing software and every
website run by the OSMF, local chapters and local mapping groups.
Several times over.

We've always allowed other people to use our spare capacity on the
tileservers, but recently it's got completely out of hand. Most of the
use of our tileservers has become developers looking for free maps,
nothing to do with the rest of the project. Often these are commercial
companies who are using our tileservers and selling their apps.
Subsidising commercial companies isn't the best use of community
donations and volunteer sysadmin time, when there are many alternative
services (such as those run by CartoDB, Stamen, etc) that provide
zero-cost map layers based on OSM data anyway.

We do have plans to scale the tile infrastructure later in the year
(cascading an old database server), in addition to the current process
making sure that our OSMF tileservers are being mainly used for
OpenStreetMap related projects.

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Tile usage without proper identification

[Andy Allan]

On 12 August 2016 at 12:34, Maarten Deen  wrote:
> On 2016-08-12 13:14, Paul Norman wrote:
>>
>> The usage policy for tile.openstreetmap.org requires that users send a
>> valid user-agent[1], or, in the case of a web browser, a HTTP Referer.
>> Ops are looking into automatically rate limiting clients violating this
>> part of the policy.[2]
>
>
> I have wondered about this. Does rate limiting now not occur? I have seen a
> lot of times that the map in the JOSM downloadscreen gets slow if you pan
> around a lot or use it a lot to download lots of small areas. Also the
> mapnik overlay has this issue.

There is already throttling based on other aspects of the requests to
the tileservers (specifically, IP address). So you might be seeing
that.

> Of course this can also be due to other issues.

Yep, the tileservers have been overloaded recently too. You're most
likely to notice when you get slow or missing requests at high zoom
levels.

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Numeric OSM tag keys

[Andy Allan]

On 5 August 2016 at 12:51, Stadin, Benjamin
 wrote:

> What is the allowed syntax for OSM tag keys? Is there actually a
> restriction, like OSM tag keys must follow common variable name syntax?

Tags keys and tag values can be 256 UTF-8 characters long, with a few
invalid characters based on the XML specification. But otherwise
there's no limits to what can be found in them.

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Tile server and Nominatim setup @ Openstreetmap.org

[Andy Allan]

On 21 October 2015 at 15:39, Stadin, Benjamin
 wrote:

> Do you have some details how to accomplish installing the complete stack,
> close to how it works at openstreetmap.org?

The production servers are configured using Chef, and all the chef
cookbooks used are available at

https://github.com/openstreetmap/chef

So this will let you find out the actual configurations used! Of
course, the cookbooks are customised for the OSM production
environment so they aren't designed for easy re-use in other
situations, and in many cases it's tricky to figure out what they are
actually doing. But I just wanted to make sure everyone is aware that
the information is available if you are curious!

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Outdated web pages / default pages on some OSM addresses

[Andy Allan]

On 19 April 2015 at 11:33, Tom Hughes t...@compton.nu wrote:

 The protocol being used is not the point. The point is that neither site had
 been configured, because neither is a URL that we have ever told anybody to
 use!

Don't be so harsh. Given that

www.osm.org - www.openstreetmap.org,
wiki.osm.org - wiki.openstreetmap.org
forum.osm.org - forum.openstreetmap.org
planet.osm.org - planet.openstreetmap.org

then it's *not* unreasonable to suspect that the foundation.osm.org
should work, whether or not it has ever been used before.

Since we don't actually publish a list of these are all the domains
that work, please don't use any other ones, there's also no point in
jumping on someone for not having read said non-existent list.

 You might as well ask why fraggle.osm.org doesn't redirect to
 osmfoundation.org.

That's hardly the same, and you know it.

 If you want to write chef recipes to handle automatically setting up all the
 different domains then please be my guest.

If it's something that would have value, but that you don't want to do
yourself, we can open a ticket on the chef issue tracker and point
people towards it.

https://github.com/openstreetmap/chef/issues

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] License for Javascript code [repeat]

[Andy Allan]

On 27 January 2015 at 03:44, Richard Stallman r...@gnu.org wrote:

 The GPL says that this decision is made and stated with notices in the
 source files, but these source files have no such notices, meaning no
 decision was ever stated.

Thanks for bringing this to our attention, and you are correct that we
haven't put license information in each of our files. I for one was
unaware that there was this ambiguity, although I'm not a maintainer
of the software, only an occasional committer.

Since this list is used for a wide range of openstreetmap-related
development projects, not just the code for the website, here are some
more targetted links:

http://git.openstreetmap.org/rails.git/
https://trac.openstreetmap.org/query?status=newstatus=assignedstatus=reopenedcomponent=websiteorder=priority
https://github.com/openstreetmap/openstreetmap-website

 Can I ask the developers of those files to make a choice, so that it
 can be listed in the source files and finish specifying how they are
 licensed?

There are approximately 112 developers involved in the whole codebase,
so I hope we won't need to contact each of them! For a number of years
the README has stated:

This software is licensed under the GNU General Public License 2.0.
See included LICENSE file for details.

... so I expect the answer is that it's v2 only, not v2 and later. In
saying that, we could certainly be more clear in each of the source
files in addition to the README

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Release openstreetmap-carto v2.25.0

[Andy Allan]

On 11 December 2014 at 15:34, Sven Geggus li...@fuchsschwanzdomain.de wrote:

 BTW, I'm also still looking for a less annoying way to maintain the german
 style which is basically a fork of your style starting from various versions.

Please do let me know if there's anything we can do to
openstreetmap-carto to make this easier for you, even if it's just a
small change.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Release openstreetmap-carto v2.25.0

[Andy Allan]

On 11 December 2014 at 12:10, Christoph Hormann chris_horm...@gmx.de wrote:
 I am somewhat reluctant to bring up this problem since i think the more
 active development is a good thing in total and i don't want to
 badmouth this.

I'm glad that you bring this up, please don't be reluctant! I know
that the development process for openstreetmap-carto could do with
some improvements.

First subject: The merging of changes. My normal process it to go to
the list of pull requests assigned to me, and then work down them from
the top down, i.e. dealing with the newest things first.

https://github.com/gravitystorm/openstreetmap-carto/pulls/assigned/gravitystorm

I first try to merge anything that's complicated, like a refactoring.
Small changes like changing zoom levels of an icon are easier to redo
later on if there are merge conflicts.
Secondly I merge any small 'no brainers' like fixing text-dy. There's
no need for them to wait.
Then I do other things, starting with stuff where I'm happy with the
fix and they can be merged with no conflicts.
Then I start working through any of the above that now needs manual
conflict resolution.
Then I revisit things that I've passed over - things that take more
thought, or where I'm considering rejecting the pull requests or
asking for modifications.

It seems elaborate perhaps, but it's basically just me trying to get
as many things merged in the limited time that I have available.

From that it would be a reasonable conclusion to think that I'm being
a bottleneck on the development - well, perhaps I am. But what is
frustrating me most is that I end up spending all my time working on
pull requests that I simply don't think are the most important tasks,
but there are so many of them (and so many calls for me to hurry up
with the reviews) that I never actually work on anything else. The
time I'm not spending reviewing and merging PRs I'm instead spending
wading through all the comments on issues, which are still running at
over 100 a week.

But these are just implementation details. The bigger problem is that
after 2 years of work we're not making much progress on the most
important things.

The style we have now is pretty much the same as it was two years ago.
We've made hundreds of changes and I'm a fan of iterative development,
but I'm not sure that we're iterating in any particular direction. I'd
love to sort out the mishmash of colours, but I'm disheartened by
how long it is taking us to sort out just the buildings, and solving
the 'trunk roads in forests' and the hundred other problems like this
seems a long way off. Even when we're changing small things we get
bogged down in endless debates and scrutiny, and unfortunately all the
debating doesn't actually lead to significantly better results.

I don't like reviewing the pull requests. It's not fun. Almost every
one I merge I think that I'd have done something slightly differently
- an extra zoom here, a different colour there, a changed icon etc.
I've deliberately tried not to bikeshed every single pull request and
so ended up merging a lot of stuff that's not quite to my taste or
changes that aren't entirely cohesive. Am I getting this wrong? Should
I be delaying more PRs until I get time to tweak them? Or should I
stop reviewing the PRs entirely and just merge any of them that don't
cause syntax errors?

One thing that I'm still sure of is that we need a *team* of
cartographers. There are too many things in the map, and too many
ideas and discussions for the style for it to be run by just one
person. And the work that we're doing now is 10 times more than I
could do without the help of everyone who is working on it. But I
don't know how best to organize ourselves. And I'm still unsure if
it's possible to have a consistent, coherent, nice and yet detailed
map with dozens of people making iterative changes. I hope it is!

So I pose a question that's most pressing on my mind - should the
other maintainers be merging PRs without me reviewing them first? Will
this lead to a better result?

If anyone has any advice, I'm all ears.

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Chinese spam diaries, an analysis

[Andy Allan]

On 3 December 2014 at 15:46, Andrew Hain andrewhain...@hotmail.co.uk wrote:
 A spammer is periodically posting messages in Chinese to the User Diaries.

Thanks for the analysis, I hope it provides developers with ideas for
combatting it via the automated spam filters that we already have[1].

However, spam is an arms race, and I think we might need a different
long-term approach. I know in the past using 3rd-party spam filtering
services was too expensive (and not really very OSM-ish either).
Perhaps we need a new set of human content moderators on the site, say
40-80 people with a variety of languages between them. We can consider
grey-listing all accounts - i.e. the first few posts of every account
is held for review automatically by default, and enable direct posting
after we're more certain they aren't a spammer.

Of course, this would all need coding, but I'm interested in other
people's ideas. The current situation where our spam filters can be
overwhelmed, and all the removal of spam depends on full-blown
system-administrator[2] accounts, isn't perfect!

Thanks,
Andy

[1] 
https://github.com/openstreetmap/openstreetmap-website/blob/master/app/models/user.rb#L211
[2] 
https://github.com/openstreetmap/openstreetmap-website/blob/master/app/controllers/diary_entry_controller.rb#L10

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Chinese spam diaries, an analysis

[Andy Allan]

On 3 December 2014 at 16:25, Martin Koppenhoefer dieterdre...@gmail.com wrote:

 maybe we could have a crowd-sourced approach and introduce a spam-flag
 that logged-in users could set, i.e. another button in the comment,
 reply line which says something like flag as spam, with a counter, and
 if more than x people have clicked on it we would automatically or manually
 hide/delete the post.

Good idea.

I'd also like to do something to prevent (as well as react to) spam,
since reaction-only processes still fill RSS/atom feeds and relayers
like https://twitter.com/osmblogs with spam posts.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Chinese spam diaries, an analysis

[Andy Allan]

On 3 December 2014 at 16:33, Serge Wroclawski emac...@gmail.com wrote:

 First, right now there's only a single person who can remove spam from
 diary entries or profiles.

Not strictly true - any user with site administrator priviledges can
remove spam - see my previous link to the code. There are multiple
people who have those privileges. Of course, in reality it's mainly
one person (Tom) who does the work.

 Second of all, we need a flagging mechanism. I know that Tom wants a
 complete solution that includes a work queue, etc. I think that's a
 very laudable goal, but something that just sends an email would be
 great for now.

The flagging mechanism is still reactive - I'd like to look at ideas
for blocking spam before it hits the site.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Mapnik-Glitch Schwarzwald

[Andy Allan]

On 17 November 2014 12:16, Florian Schäfer flor...@schaeferban.de wrote:

 What version of osm2pgsql is running on the main osm.org-tileserver?

Tom has already answered (0.81.0), but here's a longer explanation for
anyone who is interested. This is how I went about finding the answer
:-)

The server configurations are managed by chef, so the place to start
is with the chef configuration repository at
http://git.openstreetmap.org/chef.git/ , which is mirrored at
https://github.com/openstreetmap/chef

Each component is normally managed by a cookbook, so I had a look in
the cookbooks directory for one for osm2pgsql. Since there isn't, I
searched for osm2pgsql and found in the tile cookbook the line
'package osm2pgsql', which shows that osm2pgsql is being installed
from packages. I doubt that we're using the stock Ubuntu package, so I
had a look in the apt cookbook recipe/default.rb for any PPA
definitions, and found http://ppa.launchpad.net/osmadmins/ppa/ubuntu/.
In the 'pool/main/o/osm2pgsql/' there's packages there with 0.81.0 in
the name, so it's a reasonable bet these are the ones being used.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Adding Slippy Map to Taginfo's Projects Tab?

[Andy Allan]

On 8 October 2014 15:19, Stefan Keller sfkel...@gmail.com wrote:
 Hi,

 On Mon, 15 Sep 2014 Jochen Topf wrote about Upcoming new feature:
 Projects in [Taginfo-dev] which you recently released [1]. That's a
 cool feature and many projects are already integrated [2]!

 Would'nt it be very helpful to have also Slippy Map
 (openstreetmap-carto [3]) listed there?

 Did anybody already thought about a tool to convert CartoCSS (or
 Mapnik) style to the json project file required by Taginfo [4]?

We had a discussion on this within the openstreetmap-carto project:

https://github.com/gravitystorm/openstreetmap-carto/issues/961

The outcome is that it's almost impossible[1] to do this automatically
since we do a lot of tag processing in our SQL statements.

Thanks,
Andy

[1] Nothing's ever impossible with software though :-)

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Cache settings for OSM Mailing List monthly index pages

[Andy Allan]

On 17 September 2014 13:09, Robert Whittaker (OSM lists)
robert.whittaker+...@gmail.com wrote:
 I'm not sure if this is the right list to post this to, but I've
 noticed recently that I have to refresh the index pages of OSM mailing
 list posts e.g.
 https://lists.openstreetmap.org/pipermail/talk/2014-September/date.html
 in order to get new posts to show up. This seems to happen to me with
 multiple browsers on different computers.

Yeah, I've noticed this too, but I haven't looked into it. Until now :-)

The configuration of the servers is controlled via chef, and the chef
repository is at
http://git.openstreetmap.org/chef.git

There's a mailman cookbook, which has only one change made recently:
http://git.openstreetmap.org/chef.git/history/e9aa039648867546ebda79d0ceae2f6aa6f9621d:/cookbooks/mailman

The relevant commit contains the bug:

http://git.openstreetmap.org/chef.git/blobdiff/e517f49bcdd9b1afac9809154c8d727f2bc96847..8604bd193412c0bf604d8b8e38b76d0e350f107c:/cookbooks/mailman/templates/default/apache.erb

( http://bit.ly/1o9fn36 if the link above is borked)

```
-   Location ~ /pipermail/([^/]+)/(2004|2005|2006|2007|2008|2009)
+   Location ~
/pipermail/([^/]+)/(2004|2005|2006|2007|2008|2009|2010|2011|2012|2013|2014)
ExpiresActive On
ExpiresDefault access plus 180 days
/Location
```

So a possible fix is to remove 2014 from the end of that list until
the end of the year.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] User Testing OSM.org

[Andy Allan]

On 7 December 2013 20:30, SomeoneElse li...@mail.atownsend.org.uk wrote:
  Didn't someone have a go at this a few years ago
 (either at a London meetup or driven by some academic research) - it'd be
 interesting to see if you can try and measure some of the same things and
 see how the results compare.

You could be thinking of one of these:

http://blog.gravitystorm.co.uk/2012/03/01/make-the-hard-things-simple-and-the-simple-things-occasionally-surprisingly-hard/
http://blog.gravitystorm.co.uk/2011/10/25/openstreetmap-usability-revisited/
http://blog.gravitystorm.co.uk/2010/10/04/quick-and-dirty-usability-testing-of-osm/

or academically, perhaps you're thinking of

http://2011.sotm-eu.org/t...@31.html

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] mapnik-render-image 0.3 (was: generate_image.py)

[Andy Allan]

On 25 November 2013 22:03, Stephan Bösch-Plepelits
sk...@xover.htu.tuwien.ac.at wrote:

 Now I renamed the script to 'mapnik-render-image' and renamed the project
 on Github accordingly:
 https://github.com/plepe/mapnik-render-image

How does your new version of the script compare to the nik2img.py utility?

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Server configuration: (Urgent problem) pink tiles while requesting lvl 16, 17 18 tiles

[Andy Allan]

On 14 October 2013 17:20, Álvaro Enríquez de Luna Muñoz
aenriquezdel...@many-worlds.es wrote:

 This co-worker isn't working with us anymore, and I am currently facing an 
 urgent problem. Requested tiles from levels 16, 17 and 18 from zones that 
 have never been visited (the app was being used in very restricted areas and 
 now different zones have been enabled) are showing as pink squares.

Hi there,

It sounds to me like your rendering daemon has died - either renderd,
or tirex, depending on your original setup. Without the rendering
daemon mod_tile will continue to serve tiles that it has rendered
already, but cannot request rendering of any new tiles. The default
appearance of missing tiles on OpenLayers is a (nasty) pink
background, which again sounds like it fits your situation.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Standard for Map Key

[Andy Allan]

On 16 September 2013 12:14, Peter Wendorff wendo...@uni-paderborn.de wrote:

 as far as I know all map keys are created by hand.

I think that's true, and therefore somewhat tedious!

 2) it would be really great to have some automatic map key generation
 out of the style files. For Mapnik that's incredible ugly as mapnik does
 not have any semantic connection between e.g. highway casing, fill and
 label. For CartoCSS (which is used since a while for the default style)
 this might be possible in parts.

I don't think it's particularly feasible - the definitions for all the
symbolizers that make up a zoom level 17 trunk bridge are spread
around a fair amount, and so you'd need a CartoCSS-complete parser
to figure out what symbolizers apply to a feature, and then a
mapnik-complete rendering engine to ensure that the symbolizer
attributes are accurately reflected in the legend images. Which is
why...

 In any case it would require additional information like which label to
 use for a particular feature in the map key, which items should NOT be
 displayed (as Tom mentioned) or in which order items should be displayed.

... I've been developing something that does this. It takes a
legend.json file which describes what features should be drawn (and in
what order, what description to use, which geometries etc) and spits
out a series of mapnik renderings for each feature. It's still very,
very alpha quality but it's available at

https://github.com/gravitystorm/mapnik-legendary

I aim to make it configurable enough to recreate what we have at the
moment, and to make it easy/feasible for me to create legends for my
other styles, and to add more functionality like sprite images. The
ultimate goal is to be able to create legends like the Ordnance survey
have - for those not familiar with them, have a look at
https://github.com/gravitystorm/mapnik-legendary and check out the way
that the railway crossings in the 1:25,000 or water features in the
1:50,000 maps are shown for some inspiration.

Overall, I have no intention of manually updating the legend that we
have at the moment. If anyone wants to take it on then feel free, but
my efforts are going into mapnik-legendary

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Default map style on osm.org

[Andy Allan]

On 28 August 2013 12:02, Peter K peat...@yahoo.de wrote:

 Or at least: will this default style be
 further developed and improved?

See the following video for some background regarding the current
state of the main map style
http://vimeopro.com/openstreetmapus/state-of-the-map-us-2013/video/68093876

Since the conference, the new openstreetmap-carto stylesheets are now
live, and many small improvements are starting to show up.

Thanks,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Default map style on osm.org

[Andy Allan]

On 28 August 2013 16:05, Peter K peat...@yahoo.de wrote:
  But I've always heard/read that the style has to stay
 the same.

And where did you hear that? It's not something I've heard before.

 Who can decide and develop this in the future?

That would be me, and whoever else contributes to the style. Of
course, we all listen to feedback too.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Default map style on osm.org

[Andy Allan]

On 28 August 2013 16:26, Holger Jeromin mailgm...@katur.de wrote:

 I think the rewrite in carto wanted to maintain the visual result to be
 sure to be able to switch the main rendering. Otherwise the switch could
 be stopped by some for visual reasons.

Good point - that's probably the source of the confusion.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Default map style on osm.org

[Andy Allan]

On 28 August 2013 16:48, Eugene Alvin Villar sea...@gmail.com wrote:

 I think that being pretty is not the goal of the default map style at all.
 The primary goal of the default style is to expose as much of the OSM data
 as possible.

That's widely held opinion, but there's an equally sized opinion that
would heartily disagree.

The cartography work that I do is not intended as a simplistic
multicoloured visualisation of the database. It's a map, and
cartographic decisions about what to show, and what to leave out, are
key to its success. Expose as much of the OSM data as possible is
only a goal with a sufficiently loose definition of 'as possible'.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [OSM-talk] Is there some lag in the backend data?

[Andy Allan]

On 24 July 2013 04:22, Ian Dees ian.d...@gmail.com wrote:

 iD clears out its internal representation of OSM data when a user's Save is
 successful and then re-requests the area in the viewport immediately. The
 replication delay for a read-only API needs to be less than 2 or 3 seconds
 in this scenario.

The physics of this starts to make it impossible.

Perhaps it's better to explore a sort of inverse if-modified-since
header. Clients would be able to request an initial bounding box
without any particular constraint, and assume that it's within a few
minutes of being up-to-date. But after making an upload they could
request a bounding box with only if replicated since: timestamp.
The server can then indicate that it can't fulfill the request and the
client retries later, or the server could proxy the request to the
main API, or something else.

Cheers,
Andy

P.S. This is why I'm focussed on db-level replication for scaling API
read requests, but we should work this through and be able to choose
between alternatives!

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Expiring Tiled OSM Data

[Andy Allan]

On 19 June 2013 20:58, Ian Dees ian.d...@gmail.com wrote:

 So we're talking potentially several billion (entity type)+(entity id) -
 [(tile x)+(tile y), ...] rows in a database of some sort along with the
 forward and reverse indexes.

Indeed. Although there's probably a much clever approach than the one
I'm suggesting.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Expiring Tiled OSM Data

[Andy Allan]

On 19 June 2013 16:25, Ian Dees ian.d...@gmail.com wrote:

 The last bit of work before I can call it complete is to correctly expire
 tiles based on the minutely diffs. The naive and incorrect approach would be
 to expire based on node changes. This ignores changes to way and relation
 tags, for example.

 Any thoughts on how to do this correctly and efficiently?

I think the correct way is, for each tile, to store the list of nodes,
ways and relations returned within that tile. For example, the map
call walks up the tree from nodes - ways, but then also back down the
three from ways - nodes in order to complete the way. So if a
minutely diff marks an entity as expired, I can't see any other way to
determine which tiles that request is mentioned in, beyond re-creating
every tile from scratch, or doing some complex graph walking. Even for
a simple node tag change, that node could appear in any (or all)
tiles, and doing an exhaustive search of all the ways and relations
that mention it, and relation members in turn, to find every possible
bounding box seems the wrong way.

So basically, when generating a tile, store an entity - tile lookup
table too. When regenerating a tile, flush the entity - tile store
for that tile and repopulate.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] mod_tile causes segfault on debian 7.0

[Andy Allan]

On 22 May 2013 12:06, Sven Geggus li...@fuchsschwanzdomain.de wrote:

 I get segmentation faults in mod_tile on a more or less regular basis which
 will leave apache in a 100% CPU on all cores state!

Ah, that's interesting, I'm seeing something very similar. I'm using
1341e129e5 (22 April) with apache2-mpm-event 2.2.22 (on Ubuntu 12.04)

I'm not sure that commit is necessarily involved, since it doesn't
change anything for apache 2.2, but perhaps the bug was introduced
elsewhere and that commit is just a symptom? I'm just using renderd
with a normal disk backend, if that helps.

I'm not sure what's triggering it, but it looks like it might be
happening every few days. If someone can figure out the trigger, I'm
happy to git-bisect.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] New database server

[Andy Allan]

On 21 May 2013 02:32, Jason Remillard remillard.ja...@gmail.com wrote:

 The server that we are planning on purchasing is monster. Very
 complicated and expensive. I am concerned that this might not be the
 best way to go.

Indeed, it might not be the best way to go, and any thoughts and
brainpower applied to the problem are always very welcome.

Of course, at OWG we *do* think it's the best approach, given all the
trade-offs involved, and it's a decision that hasn't been undertaken
lightly.

 We have a google summer of code proposal to write an edge proxy server
 for the OSM API. I don't know if the project will be accepted, but it

It's worth bearing in mind that we don't have any places on this
year's GSoC, and even if OSGeo decided to go for this project on our
behalf, it could be months before we even have any idea whether or not
the implementation is feasible. I don't mean to be negative, but
weighing up the here and now requirements against a hypothetical
alternative at some point in the future is one of these trade-offs
that we routinely have to make at OWG.

 For the money we planning on spending on the big
 server, we could get could get several of these smaller edge servers
 with flash disks and a less expensive redundant write/history database
 server.

Well, we could certainly spend the money on small edge servers, but
it's not clear to me why you think that would make the central server
less expensive. I think this proposal may be worthwhile but it's
somewhat orthogonal to the goals of the new server.

At the moment we have two osm-database-class machines, the older of
which (smaug) is no longer capable of handling the full load on its
own, but is still useful as a database-level read slave. The newer
machine (ramoth) can handle the load entirely on its own, but is
approaching the limits of dealing with the full read+write load.

When it comes to the master database, we need certain characteristics:
A) To be able to handle the write load (and the associated reads
involved in checking constraints etc)
B) To be able to store the entire database
C) To be more than one of these machines, for failover

Smaug most likely doesn't fulfil A, and so currently we don't really
fulfil C. So we need a new machine that can do A+B, and these are
unfortunately expensive. In order to last more than 6 months, the new
machine also needs plenty of space (B) on fast disks (A) which is
where most of the money goes.

Having map-call-proxies, as you discuss, doesn't solve any of A, B or
C for the master database. Sharing out the read-only load is a good
idea, but it's not clear to me whether it is better done with
postgres-level database replication (as we have been doing),
proxy-level replication (as per this GSoC idea), or even just
examining logs and ban-hammering people scraping the map call (my
personal favourite!).

 As we need to scale

It's best in these conversations to be precise about what we mean by
to scale. Scaling read-requests is only one aspect, and we have a
variety of feasible (and active) options. Long-term, we may[1] need to
work around the need for all these machines to store the entire
database (B), and that's Hard. We may[2] also need to figure out how
to solve A, and that's Hard too.

Like I said at the start, thoughts and brainpower are always welcome!

Cheers,
Andy

[1] If we grow osm faster than Moore's law, otherwise: happy days
[2] If db-write activity outpaces disk-io and/or network bandwidth
increases, otherwise: happy days

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] New database server

[Andy Allan]

On 21 May 2013 18:50, Paweł Paprota ppa...@fastmail.fm wrote:

 And what I can tell you is that people in OSM (I mean admins here) are very
 supportive and open to changes.

I'm glad to hear that! Many people say they have a different
experience, but we try to be helpful.

 Sadly, I
 simply don't have enough time to finish it to the extent that it is
 acceptable (to me, not to mention others/admins) to consider it for
 production use.

I think many of our software projects have similar issues, and one of
the key aspects is to try to get more than one volunteer to develop
each one.

I'm currently working on improving the documentation for the rails
port (with a view to making it easier for new developers to get
started). When I finish with that, do you think that your OWL project
could benefit from something similar? Or are there other things that
you think are holding back other developers from getting started?

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Add an iD link to the Potlatch no-Flash screen

[Andy Allan]

On 22 April 2013 15:50, Tom MacWright t...@macwright.org wrote:

 it seems as if doesn't expose way membership to the user currently. In any
 area with multipolygons or turn restrictions or, likely more difficult to
 fix, other types or relations, mappers will really break a lot without even
 being able to notice.

 It doesn't expose relations in the UI, but does not break them and makes the
 same relatively smart choices as P2 when users make operations on ways and
 nodes in relations.

I think the concern is that if there is no indication that a way is a
member of a relation, then mappers might believe that it's not part of
any relation. For example, an untagged way in the middle of nowhere
might seem ripe for removal, but unbeknown to the mapper, it's an
inner ring of a boundary multipolygon relation. Or two stretches of
road have identical tagging, and a mapper may be tempted to merge them
without realising they belong to separate bus routes.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osm2pgsql and mod_tile have moved to git

[Andy Allan]

On 3 April 2013 16:10, Kai Krueger kakrue...@gmail.com wrote:
 Hello everyone,

 as many people seem to prefer using git over svn, we have decided to convert
 the master repository of osm2pgsql and mod_tile over to git.

That's fantastic news!

 Previously anyone who had an osm-svn account had commit access to the
 repositories and could submit patches directly to the master repository, and
 indeed people did take advantage of this (were forced to take advantage of
 this). After the move to git, this will no longer be the case. For the
 moment it is pretty limited who has push rights to the repository (Jon and
 myself for mod_tile, and Sahra, Brian and myself for osm2pgsql), but that is
 something that can be discussed and changed. Personally I am in favour of a
 fairly open policy and would be happy to see others have push rights as
 well.

This is great too!

 With that change I guess the bug tracking will also likely mainly move over
 to the github issue trackers, although track tickets on
 trac.openstreetmap.org now also get forwarded to the tile-serving mailing
 list. Therefore they should be noticed and dealt with as well, if you prefer
 to use trac.

I hope to see the issues moving over to github, in order to remove any
barriers to discovery for people working on the code. I find finding
osm2pgsql-related tickets on trac a lot harder than finding osm2pgsql
issues on github, to the point that in all the projects I work on, I
very rarely bother trying to find tickets on trac.

 As the new repositories live in the same location as previously the svn
 mirrors you will need to re-check out those repositories, as the history has
 been re-written to be a cleaner SVN conversion than the mirrors previously
 were. I hope that doesn't cause any trouble or inconvenience for people who
 have local forks of the previous mirror.

That's a small price to pay :-)

 Any comments or suggestions on the development process of these two projects
 are welcome,

For now, just my thanks for undertaking the work - it's really
massively appreciated.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] mod_tile stable version ?

[Andy Allan]

On 28 March 2013 15:20, Sarah Hoffmann lon...@denofr.de wrote:
 On Thu, Mar 28, 2013 at 07:36:56AM -0700, Kai Krueger wrote:
 Despite possibly being the person who has recently been most active in
 committing code to mod_tile / renderd and osm2pgsql, I have been reluctant
 to claim the official maintainer title, as I didn't want to be the gate
 keeper or bottleneck. But perhaps I should just take that responsibility, or
 at least make sure to try and be more responsive to patches.

 I'd be very happy to see you step up as maintainer and move the stuff
 to github. I'd feel much more comfortable to submit pull requests to
 github instead of doing direct SVN commits that nobody even knows
 about unless the code fails in some annoying way.

Me too. There's no way I'll ever commit my ham-fisted coding directly to SVN!

But Kai, if you really want to it's possible to recreate our current
mess in git. Simply give commit rights to openstreetmap/osm2pgsql.git
to whoever asks, then we're back to the same situation as with our SVN
repo.

But it would be better to have a maintainer (or two), even if all they
do is move the project to git, and retrospectively tag non-broken
releases with a version number every now and then.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] mod_tile stable version ?

[Andy Allan]

On 27 March 2013 14:09, Pieren pier...@gmail.com wrote:

 OSM data  tools are mostly developed by volunteers on their free
 time.

And I'm sure we all strive to make high-quality tools. If someone
encounters bugs, like crashes and broken documentation, then even just
being told by email is valuable, or logging tickets, or mentioning it
on IRC. There's no need to round on them for not having fixed all the
problems single-handedly.

Of course, some people have unreasonable expectations, and we all tire
of their demands. But things like being able to compile the software
is a perfectly reasonable expectation.

 Feel free to improve the wiki documentation if you find issues
 or incompleteness. Creating an account for wiki edition takes less
 time than writing your last 5 messages. Do something positive for the
 project instead of complaining. Thus, the next one installing a tile
 server on F18 will be grateful for your help.

And this is one of the reasons our documentation is such a mess. With
the best will in the world, someone who doesn't understand the
components and has just managed to get them working - for the very
first time - isn't in the best position to write clear documentation.

The osm2pgsql page on the wiki, like many others, is degrading over
the years as people add to it and nobody edits it. Such pages are not
converging towards great documentation, they start off as useful
documentation and slowly degenerate, while accumulating increasing
amounts of cruft, inconsistencies and even inline-patches that you are
encouraged to apply. Sheesh. And that doesn't even count the
increasing numbers of other pages you can stumble across which
duplicate one another and confuse anyone who reads them.

So think twice before telling a new user that they should start
editing wiki pages.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Coastline changes Antarctica

[Andy Allan]

On 12 March 2013 09:12, Tom Hughes t...@compton.nu wrote:

 I am not sure it is a good idea to switch to water polygons. Those
 polygons
 are much more complicated because they contain lots of holes, so they are
 slower to render. I'd only do that if really necessary (for instance when
 you want to mask something below the water polygon.


 Interesting - that change was Andy's idea and I think the thought was to
 reduce the damage done by any breakage and to ensure that we're not matching
 any massive polygons in the busy (land) areas.

So my line of thinking is as follows:

1) Jochen's proposed changes are going to break all coastcheck-created
shapefiles, so openstreetmap-carto should instead use ones from
OSMCoastline
2) Jochen does some form of QA for OSMCoastline so they are good to use directly
3) If you're not using hillshading, inverse vs normal makes no visual
difference. But as soon as you use hillshading, you want inverse. So
all else being equal, inverse should be the default
4) It's also confusing to have the map background as blue, when most
people editing stylesheets think the background colour would be the
land colour.
5) The vast majority of rendered tiles are all land. So it seems
strange to draw a blue background, to immediately colour it with a
land polygon, in almost every case.

I take the point about the complexity of water polygons vs land
polygons though, it's not something I'd thought of. I wonder if it
invalidates the performance advantages described in 5) ?

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Leaflet.EditInOSM

[Andy Allan]

On 28 January 2013 14:20, Yohan Boniface yohanbonif...@free.fr wrote:
 Hi all,

 A short mail to introduce the little plugin Leaflet.EditInOSM [1], which add
 a control in Leaflet with links to edit the current view in JOSM, Potlatch
 or iD.
 Demo here: http://yohanboniface.github.com/Leaflet.EditInOSM/

Cool!

In general I prefer directing people towards openstreetmap.org/edit ,
rather than having a bunch of options or linking to instances/remote
control directly. The advantage of openstreetmap.org/edit is that the
user will have already chosen their favourite editor, and so you only
need one button. Additionally it's a better user experience when a new
user hasn't yet signed up to openstreetmap - it reduces confusion with
the JOSM / Merkartor remote control link not working, for example. And
it lets those who still prefer p1 to stick with it! :-)

Obviously many other websites continue to have multiple links, so
you're not alone in making that decision. But as the number of editors
grows, and as there are plenty of people who don't use the most
popular (n) editors, I think that standardising on the
openstreetmap.org/edit approach is the better one.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Leaflet.EditInOSM

[Andy Allan]

On 28 January 2013 16:39, Jochen Topf joc...@remote.org wrote:

 That approach might work for in-browser editors, but I don't want to go to
 a different web page first before my JOSM is opened at the right place. I
 want to stay in the context of the web page where I am opening the editor
 from.

Fair enough.

 Maybe we could make the editor setting for a user available through an API on
 osm.org, then every site can fetch this setting and directly do the right
 thing.

That's a good idea - but how would that work for Yohan's example?
You'd need to give his website your username or something similar,
right?

 Well, actually it is not that easy. I am almost never logged in to the main 
 OSM
 site. About the only reason I log onto the site is to use the editor. But not
 my actual preferred editor (which is JOSM). I have set my preferred editor to
 Potlatch 1, because that is the easiest way to still access Potlatch 1, which
 I don't need that often, but I still need it for one thing, namely reviving
 deleted objects. So I don't actually want this preferred editor at all most
 of the time...

Well, I can think of a few bugs that need solving, going by your
explanation! Staying logged in for longer, accessing p1 more easily
when duty calls, implemented deleted map calls in the API, then
implemented the undo in other editors, etc.

I'd say the only showstopper that you describe above is that you want
to stay on the same page when triggering the remote control plugin.
Would it work if openstreetmap.org/edit responds with a 30x code
(307?) pointing to the remote control url? Or would your browser still
show an empty page?

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Snapshot Server

[Andy Allan]

On 20 January 2013 00:45, Rob Nickerson rob.j.nicker...@gmail.com wrote:

 I reinstalled snapshot-server following your updated instructions, which was
 a lot simpler than last time :-) and gave the web load a test. It worked a
 charm :-)

That's great news. If you, or anyone else on the list, have problems
or suggestions for making it easier to install a copy of
snapshot-server, please let me know!

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Snapshot Server

[Andy Allan]

On 8 January 2013 17:59, Rob Nickerson rob.j.nicker...@gmail.com wrote:

 A web based loading screen would be a big plus for me.
 I had a quick look and found a 5mb shapefile. I would imagine 10mb is
 probably the largest I have. Not sure how that converts to .osm filesize.

On Tuesday I coded up an upload form, and it's in the most recent
version. It works at about 30s for a 1Mb file on my laptop, which is
fine for the data that I've been handling but not great for others who
need larger uploads.

I've stuck in a few tickets on github for things that would be good to
do in future - if anyone fancies having a stab at them, be my guest!
There's nothing complex in the snapshot-server code.

https://github.com/gravitystorm/snapshot-server

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Snapshot Server

[Andy Allan]

On 28 December 2012 05:52, Paul Norman penor...@mac.com wrote:
 Readme.md for snapshot-server is not quite accurate. I’ve got an updated
 readme at https://github.com/pnorman/snapshot-server/blob/patch-3/README.md
 that covers enabling PostGIS and hstore but Andy hasn’t had time to update
 upstream.

I've updated the readme, and there's actually a chance now that you
could get it installed. I've tested the instructions on a VM, but of
course feedback is welcome. I've gone for the route that involves the
least typing, so rake db:create will sort out all the postgis-related
things for you.

Rob, did you manage to get your snapshot-server instance up and
running in the end?

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Reverts from the woodpeck_repair account

[Andy Allan]

On 2 January 2013 13:31, Paweł Paprota ppa...@fastmail.fm wrote:

 For me such reverts just introduce unneeded clutter in the history database
 - thousands of new versions in the history. And the end result will be that
 this revert will then again be slowly reverted by individual mappers
 replacing deprecated tags...

That's only one possibility. There are deprecated tags that are more
popular than their voted on replacement, that have better support in
editors, are used by more renderings and are still preferred by
mappers. The assumption that deprecated tags will be replaced by other
ones is a false assumption, and hence why we don't automatically
change tags.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Carto-based Mapnik OSM Rendering

[Andy Allan]

On 12 December 2012 21:35, Alex Barth a...@mapbox.com wrote:
 Again, this is awesome to have this up.

 If you find issues, please file here 
 https://github.com/gravitystorm/openstreetmap-carto/issues

 Or fork, fix and issue a pull request :-)

 And really, on a first glance there is very little difference. Just filed one 
 for leisure=garden

 https://github.com/gravitystorm/openstreetmap-carto/issues/6

Thanks Ian (Dees) and everyone else, this is awesome! I'll work
through all the issues reported so far, and also be merging in Ian
(Villeda)'s work on making the setup easier.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Announcing openstreetmap-carto v1.0 (and v2.0!)

[Andy Allan]

On 6 December 2012 15:54, Tom Hughes t...@compton.nu wrote:

 One thing Andy did say was that he'd like other people to review what he's
 done to make sure he hasn't introduced regressions - he is probably best
 placed to explain how he envisions that working.

There's two main things I'd like other people to review:

1) Performance. I'm interested to know how the generated XML compares
with the original in terms of tiles per second. Yevaud has plenty of
overhead, so long as there's not a substantial performance
degradation. Anyone who can test performance of the two stylesheets,
even only approximately using nik2img.py, would be doing a great
service.

2) Visual changes. I've spotted a few typos in the conversion as I
went along, but only for the places that I looked at (primarily London
and Spaghetti Junction in Birmingham). If we can set up the style and
get other people to check and compare against the current style, that
would be great. I expect in some of the less-common features (e.g.
highway construction, abandoned canals in tunnels) there could well be
one or two typos in the stylesheet, e.g. a casing is 1 pixel too wide
or a zoom level is off by one.

Of course, neither of them are blockers for deploying on the
tileserver, it's just more professional if we can fix them first,
rather than dealing with things later!

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Announcing openstreetmap-carto v1.0 (and v2.0!)

[Andy Allan]

Hi all,

openstreetmap-carto is a project to re-implement the standard
OpenStreetMap mapnik style, in CartoCSS[1]. At the OpenStreetMap Hack
Weekend in London last Sunday I released v1.0 of project, and went to
the pub before telling anyone!

Version 1.0 is the complete re-implementation of the entire style in
CartoCSS, including all the features of the original layer. It is
pretty much pixel-for-pixel identical to the original style, with only
a couple of noticeable differences around vertical text spacing and
stacking of overlapping landcover polygons. I've spent a long time
comparing the output with that of the current mapnik stylesheets and
tracking down mistakes in the conversion, which you can see in the
run-up to the release. There might be some more mistakes that are
found later, in which case I'll release more versions in the v1.x
series.

It's been a huge amount of effort, but it's entirely worth it to see
it all working in TileMill! Playing around with the style is now much
easier, and much more fun.

As per the roadmap, the next stage is to simplify the stylesheets to
make them easier to wrangle. I've already started this, with
converting a couple of key colours into variables (so now you can
change the colour of all the water features with a one-liner) and
refactoring the style rules to make compilation times much faster (and
the output XML around 23% of the size of v1.0). So today marks the
release of v2.0, the first release of the refactoring work.

I intend to continue work on the refactoring, and I've started make
notes (in the form of github issues) to give you an idea of what the
upcoming work is likely to involve. I want to be cautious about making
the stylesheet too clever, so if you've got an ideas for major
refactoring, lets discuss them!

If you have any questions, or if you have any successes or problems
trying to use the stylesheets, let me know!

Cheers,
Andy

[1] https://github.com/gravitystorm/openstreetmap-carto

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osm2pgsql segmentation fault

[Andy Allan]

On 17 November 2012 11:44, NopMap ekkeh...@gmx.de wrote:
 It identifies itself as


 The crash message is


 The last output of osm2pgsql is



Hi Nop,

None of your quotes came through!

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Porting the mapnik stylesheet to carto

[Andy Allan]

Hi All,

Matt has mentioned it in the other thread regarding the stylesheets,
but I thought I'd expand a bit on what I'm up to.

For a long time (coming up for a year now) I've been contemplating
creating a carto version of the main stylesheets. I ported all my own
stylesheets from xml to carto and found a renewed joy in developing
them, both through the change in syntax and from the improved
toolchain (i.e. using TileMill). Also, I much prefer using git than
SVN, since it makes it much easier to try things out on a branch and
seek feedback, rather than passing patch files around or just stomping
over the main branch - or even having patches slip by because people
(like me) are uneasy just committing them and never get round to it.
Finally, I've seen so many thinly-customised versions of the OSM
stylesheets that I know people are trying and struggling to work with
them, and I have painful experience of working with multiple different
XML stylesheets (OpenCycleMap and MapQuest, to name but two).

I had no idea how few commits there had been to the stylesheets over
the last six months (pretty much none, really), so there's a good
opportunity for a conversion given the reasonably static target.
Yesterday I decided that rather than emailing the list, I would dive
in and see how much work it would be to convert them to carto, and
whether it's feasible

https://github.com/gravitystorm/openstreetmap-carto

By a rough metric of number of mapnik rules, I'm 48% of the way
through the conversion. Currently the map looks like this:

http://thunderflames.org/temp/openstreetmap-carto-wip.png

So far so good!

I strongly believe that this is the way forward. Ideally I'd like to
finish the conversion and hit a 1.0.0 release of the new stylesheets -
one that we can swap yevaud over to without causing much kerfuffle.
Then I've already identified a bunch of improvements to the stylesheet
that will make life easier, and which I hope will attract new
cartographers (the many who are familiar with Tilemill) and/or
rekindle interest from our battle-hardened veterans.

At the very least, we'll soon have a carto version of the current
stylesheet. Regardless of what we decide wrt to old XML stylesheets -
if, say, someone else decides to take them on and develop them in the
traditional manner - I'm sure the carto version will be of interest to
many people.

Please, ask away if you have any questions. I'll keep the list posted
with how I get on.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Porting the mapnik stylesheet to carto

[Andy Allan]

On 16 November 2012 17:04, Matt Williams li...@milliams.com wrote:

 Great work. Do you have any idea how/if the performance changes when
 using Carto rather than the old XML?

In my experience it makes no appreciable difference. Carto compiles
down to XML, and usually most of the speed is in the sql queries and
the time taken by mapnik to draw things, rather than anything else.

However, it's reasonably easy in Carto to write some CSS that compiles
into pathological XML stylesheets, and they of course take ages to
parse and have a runtime impact too. I can make 20 lines of carto
compile into 20,000 lines of XML[1], but I also know how to avoid
doing that too :-)

However, my view is always that servers are cheap, cartographers are
thin on the ground, and (with reason) it's not worth worrying too much
when they are within a few percent of one another. I'd be interested
to see benchmarks if anyone fancies doing it, but my finger-in-the-air
would be the carto-generated XML will be within 10% of the
hand-crafted stuff since the bulk of it is the same.

Cheers,
Andy

[1] https://github.com/mapbox/carto/issues/20

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [Potlatch-dev] P2, snapshot-server, imports, vector layers and more

[Andy Allan]

On 15 November 2012 08:37, Richard Fairhurst rich...@systemed.net wrote:

 For 1; we could continue with vectors.xml, and also provide a way to provide
 a URL in the embedding page/query string (much as you can with background
 imagery).

Recently I've been looking at the tilejson spec, which the Mapbox guys
developed for map layers. It's a way of describing what tiles are
available, the extent of the map, the attribution and so on in a json
document.

https://github.com/mapbox/tilejson-spec
http://c.tiles.mapbox.com/v3/examples.map-4l7djmvo.json

I think we could make imagery.xml more lightweight if we encourage
sources (e.g. our ooc maps) to provide a tilejson document - then our
imagery list can be just a list of URLs to the tilejson documents. It
will also then simplify the experience for a user adding their own
imagery, since all the extents, zoom levels etc are figured out
automatically.

I'm therefore also wondering if we can come up with a similar json doc
for vector background layers, that applications (p2/josm) can use to
figure out all the information they need. At the moment
snapshot-server displays the xml for you to copy and paste into
vectors.xml, but Paul has highlighted the need for users to add their
own snapshot layers to whichever instance of p2 they are running.
Pasting a json-endpoint URL is perhaps much simpler than building a
dialog to paste in xml snippets.

Thoughts?

 (It might be possible to do what JOSM does by putting the sources on a wiki
 page, rather than relying on a commit to the P2 repository, but I'm always a
 bit wary of people adding Google imagery etc. without us noticing. Any
 thoughts?)

Eugh, no. Let's stick to keeping things specified in the repository -
with github's web-based file editing, it's as easy to contribute to as
a wiki, but with enforced review during the pull requests.

 For 2; we could add a new menu like the 'Background' one showing data layers
 in the current area.

I'd like to see something like this.

Cheers,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [OSM-dev] Available Languages on wiki. and real estate

[Andy Allan]

On 8 November 2012 17:25, Jeff Meyer j...@gwhat.org wrote:

 Here's the ticket I was going to file (but cannot figure out (a) where to
 file it; and (b) where the code is underlying the wiki code below...):

Tickets can be filed in trac.openstreetmap.org (component is 'wiki')

 Any alternative that reduces the amount of vertical screen space for this
 information would be great. For example, why not collapse the list by
 default, the same way Other Languages are currently collapsed by default.

I like the way that wikipedia does it (by using the sidebar) but I'm
not sure if there's any technical reasons why we're doing it
differently.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Notes from today's iD huddle on #osm-dev

[Andy Allan]

On 5 November 2012 18:11, Alex Barth a...@mapbox.com wrote:

 Notes here:

 https://github.com/systemed/iD/wiki/Check-in-Nov-05-2012

From the notes:

@tmcw: I would love anyone else to look at this and try fixing /
proofing some of the work (RichardF, Allan, JOSM team)

Can you / Tom elaborate on this?

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ODbL full history planet

[Andy Allan]

On 23 October 2012 09:15, kimaidou kimai...@gmail.com wrote:
 Hi Paul,

 Thanks for pointing this out. I would say this file is the one you pointed :
 https://github.com/gravitystorm/openstreetmap-license-change/blob/master/check_history.rb

It's actually a different file for loading history extracts into an
apidb database:

https://github.com/gravitystorm/openstreetmap-license-change/blob/master/extract_loader.rb

There's a lot of stuff in there about license acceptance, which of
course was more relevant to the original purpose of the script. Don't
worry though, all the licensing stuff is only for calculating the
agreed_time column in the users table - it doesn't affect the main
output of nodes/ways/relations.

As far as I'm aware, it's still the only code that can load .osh files
into an apidb database.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] ODbL full history planet

[Andy Allan]

On 25 October 2012 14:43, kimaidou kimai...@gmail.com wrote:

 I am not a ruby guy, so I will surely need some time to understand how to
 install and run this file, but I will give it a try.

There is a readme, but in short use ruby 1.9 and run bundle install
to install the dependencies.

 What about performance
 ? Does it handle a big history file ?

It was used to load a whole-country extract (Ireland) before, but it's
no where near as fast as other tools. I'd suggest creating a very
small history extract first, and see how you get on.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM Wishlist (community wishlist built experimentation)

[Andy Allan]

On 18 October 2012 16:27, Kai Krueger kakrue...@gmail.com wrote:
 I would suggest putting it on help.openstreetmap.org rather than on the
 wiki. I know that is a bit of an abuse of help, but the builtin voting
 system and reordering according to votes can be useful to help filter the
 list of wishes and suggestions. It should make the whole thing cleaner and
 more manageable.

I'd suggest not putting it on help, since it's an abuse of the help system.

If you really want to have lots of ideas and let them vote on them,
then just re-activate the uservoice thing.

http://osm.uservoice.com/forums/41047-general

It's already full of suggestions. I suspect this new effort will
achieve similar results.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Why are so many changeset so large?

[Andy Allan]

On 17 October 2012 13:53, Matt Amos zerebub...@gmail.com wrote:

 getting to the point: this might to some extent mitigate the large
 changesets issue, as it would allow bboxes to be collected at a smaller
 granularity. however, it wouldn't be a full solution and we'd probably
 still need something like OWL to break down the geographic footprint of
 changesets further.

Further to this, I find this changeset extent problem is often
caused by looking at things the wrong way around. If you want to find
out what area the changeset covers, then we supply a bounding box to
help. However, if you want to know which changesets affect a given
area, this reverse question is much less easily answered. Hence OWL,
etc.

Beyond that, the extent is more of a promise that there are no edits
on the outside, rather than any guide to what's within. No changeset
completely fills, nor even evenly fills, its extent. There is a
widespread and very shakey assumption that smaller changesets are
somehow more likely to be rectangular or have a more even distribution
across themselves, but this won't hold in the real world in pretty
much any circumstances[1].

Basically, I see no need to worry about the extent of bounding boxes,
and no need to move to having bboxes on uploads instead of changesets
or other complications. No matter what we do, if your interest in a
changeset extends beyond the details of its extent, you need a
mechanism (again, e.g. OWL) to detail the actual locations of the
edits to the entities, and different interests (and different
entities) will have even have different buffers of interest around
them. Lets focus on things like that.

Cheers,
Andy

[1] Unless we all live in cities with north/south street grids and map
each city block in individual changesets :-)

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Notes - OSM improvements BoF at SOTM PDX

[Andy Allan]

On 17 October 2012 05:26, Roland Olbricht roland.olbri...@gmx.de wrote:

 It has roughly the same importance than the main API

The OSMF Operations Working Group classify the planet.osm.org feeds at
the same (highest) level of importance to the project already. It's
absolutely core.

 On the other hand, for somebody who is not yet familiar with the OSM data
 format (XML, PBF or both), the planet.osm.org is unlikely to be useful, so
 both are services with quite distinct audiences.

I find it hard to argue that giving someone a bbox-extract from the
website, a regional extract from geofabrik or the whole planet file
are distinct from one another in any meaningful way beyond size. An
integrated approach seems more logical.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Notes - OSM improvements BoF at SOTM PDX

[Andy Allan]

On 16 October 2012 11:02, Jochen Topf joc...@remote.org wrote:

 I don't think we can bring all of this into a single export tab. In fact, if
 we reduce all of this into a single export tab we risk alienating people who
 try out whats offered, find it lacking and then go away not understanding that
 they tried the wrong approach.

We can take the existing situation concerns osm formatted extracts and
make it more sensible.

If you choose a small area, you (might) get it from the export tab.
If you choose a medium sized area, you should be directed to a list of
appropriate third-party extracts
If you choose a giant area, you should be directed to the planet file.

Given that's exactly what happens when you either know it all already,
spend hours reading wikis, ask on help.openstreetmap.org or phone your
nearest OSM consultant, I really don't see what stops us from just
doing it on the export tab in the first place. Other than programming,
of course.

We already have policies and procedures for choosing third-party
layers on the front page, so linking off to third-party extracts
shouldn't be a showstopper, especially given planet.osm will always be
there as a fallback. Distributing our data in a non-confusing fashion
is pretty important.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Notes - OSM improvements BoF at SOTM PDX

[Andy Allan]

On 16 October 2012 00:17, Tom Hughes t...@compton.nu wrote:
 On 15/10/12 23:40, Alex Barth wrote:

 - Translations are a frequent bottleneck for copy changes, unclear how to
 solve this.


 I'm not sure why you think this, but I can only think it is because you are
 overthinking the issue and aiming for a level of perfection that we are
 never likely to achieve.

 The way it works is this - you don't worry about translations as such at
 all. You just make sure strings are translatable and we commit that and then
 the translators get to work. Yes, when something new launches it will take a
 while to get translated, but that is something we live with as a volunteer
 project and it had worked perfectly well up to now.

It hasn't worked perfectly well up to now - as you've just described.
We should be able to make changes on a staging branch, get
translations sorted, and push live with as little untranslated
material as possible. The current situation, where it's impossible to
get translations ahead of merging into master, along with translation
updates being now-and-then rather than automatically fed back withing
a few tens of minutes, is less than ideal.

I'm not sure why you don't see this as a problem?

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM Wishlist

[Andy Allan]

On 12 October 2012 00:58, Tom MacWright t...@macwright.org wrote:
 Hey all (or, well, those subscribed to dev@ - my flamewar shields are at 50%
 so I'm not risking an email to talk),

Bear in mind that technical (generally software) stuff is often better
on dev@ anyway - and there's lots of developers on this list who
aren't subscribed to the general flamewar^Wtalk@ list.

 The three of these were closed by Tom Hughes, which is fine - they might be
 all bad ideas, and if we're treating the issue tracker as Stuff we all
 agree is definitely good, then they don't belong there: they're relatively
 undiscussed ideas, since I had just posted them.

We have rails-dev@ was the place to discuss matters relating to the
api/website, and here for matters with a wider impact. Does discussing
new ideas on github bring advantages over mailing lists that I'm
missing? It's not something that I'm used to.

 What do you want to see happen with OSM's software this year? What are the
 tasks which everyone agrees on, but nobody has had the time to tackle?

Ah, good stuff.

There's a lot of stuff on Potlatch2 that I'd like to see, but that I
don't have time to work on. There's a junction editor that needs
writing, an in-editor tutorial framework that needs finishing off, and
a lot of half-done stuff in translations, UI and unit testing. I
haven't seen anything specifically rulling it out, but would I be
right in saying that mapbox don't have plans to contribute to P2?

Beyond that, I'd like to see the issues around the
deleted-item-map-call sorted out, and OWL (or equivalent) integrated
into the rails port to power the history tab. As for all-new things,
I'd like to explore how to integrate all the various QA feeds into a
combined overview, rather than having to hunt around different sites.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Will the real OpenStreetBugs stand up?

[Andy Allan]

On 11 October 2012 14:49, Alex Barth a...@mapbox.com wrote:

 Thanks so much for the history lesson :) It provided exactly the context I 
 was looking for.

I think this is an important point - with a surge of interest in OSM
development (not just from the Mapbox guys) it's important that we all
work together at bringing each other up to speed - even if that feels
to some people like re-hashing old discussions and going over old
ground. The chances of the wiki having all the details is almost as
remote as the chances of the docs being up-to-date.

If anyone has questions on any of the why does X do Y and not Z
format, please do ask! There's usually some sort of a  reason,
although often it's just (to quote TomH) hysterical raisins[1].

Cheers,
Andy

[1] That's a play on historical reasons for those who don't know,
and generally means that either the reasoning has been lost in the
midsts of time, or the underlying reasoning no longer applies.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Weird branches on github mirrors

[Andy Allan]

Hi all,

When I clone various projects from github (e.g. mod_tile) I get weird
cascading branches

From https://github.com/openstreetmap/mod_tile
 * [new branch]  git-svn- upstream/git-svn
   96e542f..1739c6b  master - upstream/master
 * [new branch]  mirror/git-svn - upstream/mirror/git-svn
 * [new branch]  mirror/master - upstream/mirror/master
 * [new branch]  mirror/mirror/git-svn - upstream/mirror/mirror/git-svn
 * [new branch]  mirror/mirror/master - upstream/mirror/mirror/master

[...]

 * [new branch]
mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/master
- 
upstream/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/mirror/master

I realise this is an artifact/bug of the mirroring scripts, but who
needs to do what to fix them?

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Non-ASCII characters in XML generated from PostGIS

[Andy Allan]

On 17 May 2012 20:51, Nick Whitelegg nick.whitel...@solent.ac.uk wrote:
 Hi,

 I'm having some problems with generating XML from a postgis database from
 PHP on the Freemap server:

 http://www.free-map.org.uk/0.6/ws//bsvr.php?bbox=44.0,11.0,445000.0,115000.0poi=place,amenity,naturalannotation=1inProj=27700outProj=epsg:4326

 It's basically falling over on the French acute 'e' accent on one of the
 points of interest. This is the first time I've had this problem.
 I specify the encoding as 'iso-8859-1' in the ?xml? prolog which I thought
 was the way to deal with this, but no luck. I'm guessing therefore it's an
 issue with the way that PHP and/or Postgres are set
 up.

 On the client side I get an XML parsing error with either Firefox or the XML
 parser in Android.

 In the database Cafe (acute e) is encoded as CafC3E9. It's postgis
 1.5.1, postgres 8.4 and PHP 5.3.3. I'm guessing several other people have
 had this issue... is anyone able to offer any pointers?

Hi Nick,

There's a few issues here that are colliding.

First of all, you're returning HTML entities in an XML document.
That's going to throw an error. In your example, you have Atilde; ,
which isn't defined for XML, only for HTML.

http://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references

Secondly, you're getting your character encodings in a muddle. The
database is storing your e-acute in UTF8. It's unicode character
U+0039, and represented in the UTF-8 encoding as two bytes (C3 and
E9).

http://www.fileformat.info/info/unicode/char/e9/index.htm - look at
the UTF-8 (hex) section

I don't understand why you're trying to force the UTF-8 characters
into an ISO-8859-1 encoding - which has barely enough code points to
cover Western European languages, never mind Greek, Russian or any
other OSM data. Stick to UTF-8 encoding in your XML, and you should be
fine. Provided, of course, you can persuade PHP that's it's an XML
rather than an HTML document, to sort out the entity encoding as I
explained above.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [Potlatch-dev] Flash Player minimum version

[Andy Allan]

On 21 March 2012 16:00, Tom Hughes t...@compton.nu wrote:

 By earlier on do you mean earlier today?

 Only we started requiring 10.2 about 9am today and that was only reverted
 back to 10.1 just before 11am when Richard's fixed version of Potlatch was
 deployed.

Yeah, I just read the commits. We were having trouble About 5 hours
ago according to help.openstreetmap.org, which is, roughly, 11am.
I've therefore no idea whether we were testing the current or previous
build of p2!

Bad timing really.

Cheers,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [Potlatch-dev] [OSM-talk] Way with only one single node

[Andy Allan]

On 21 March 2012 12:56, Steve Bennett stevag...@gmail.com wrote:

 1) We can fix this, but it's hard to prevent it coming back when new
 features are built. I've probably written such bugs.

Mumble grumble unit testing.

 3) I still think we should do this. We have a responsibility to not
 mess up the database. Saying well, we should avoid writing bugs
 isn't addressing that: we should both implement this failsafe
 mechanism, *and* avoid writing such bugs.

Well, I'm not sure I agree on this one. If it was a truly awful thing
to do, the API would prevent it. Given that it's not truly awful, it's
not something to worry about too much. It's not messing up referential
integrity, it's just something unexpected and, at the moment,
seemingly useless.

Furthermore, handling it at save time is, to me, the wrong thing to
do. What do we do when we find our data has an issue? Alert the user?
Silently swallow it? Neither is a good thing. Should that way be
deleted? Should a node be re-inserted? What did the mapper mean to
happen? We can't tell.

Finally, in other projects I follow the convention of don't code
defensively, as described at
http://www.erlang.se/doc/programming_rules.shtml#HDR11 and elsewhere.
The save code should trust that the data is good. If the data isn't
good, whatever made it bad should be fixed. Otherwise it's turtles all
the way down - should the save code check the ids are actually
numerical, just in case, should it do x, y or z, just in case? So the
answer is that the input should be checked, somewhere in the
controller code, and then it should be trusted.

Anyway, mumble grumble unit testing. When we find out what's the
trigger, for the love of god someone should help me write the unit
test so that when it's fixed, it stays fixed.

Thanks,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [Potlatch-dev] [OSM-talk] Way with only one single node

[Andy Allan]

On 21 March 2012 16:34, Richard Fairhurst rich...@systemed.net wrote:

 I've corresponded with Thomas B and found some, ahem, fairly easy steps to
 reproduce:

 1. Click on map (_once_) to start new way
 2. With elastic band still engaged, click 'Save'
 3. Whatever the opposite of PROFIT is

 Will look at it this weekend I guess.

My initial thought would be to not create the way until the second
node was drawn. I think that would also remove a lot of the gnarlyness
around POI creation, where we are footling with the
way-that-we-never-actually-wanted as we finish handling the double
click.

Also, if anyone smarter than I am (Dave?) wants to point me in the
right direction, I'd love to bind the controller state and the data
state together a bit better, so that when you're doing things like
undo you not only change the data, but put the controller into the
appropriate state at the same time. That's only tangentially related
to the above problem, however.

Cheers,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [Potlatch-dev] Testing new versions

[Andy Allan]

On 7 March 2012 20:12, SomeoneElse li...@mail.atownsend.org.uk wrote:
 This may be a silly question - asking for something that exists already, but
 just in case not:

 How hard would it be have a version of Potlatch2 available after merging for
 final testing before becoming the default P2 version on the site?  To be
 useful it'd need to be able to be invoked with a GPX file (not necessarily
 as another edit dropdown, though that'd be nice).  With the best will in
 the world, sometimes when fixing stuff other stuff breaks, and it'd be nice
 to be able to catch any issues before they go live for everyone.

There's two approaches here.

First, for the when fixing stuff other stuff breaks - the correct
solution there is automated tests. We use them extensively on other
projects, such as the rails port. For most of the projects I work on I
have a simple rule of thumb - writing exhaustive tests is exhausting,
but any time there's a real-world breakage, write a test for that.
Breakages are inevitable, but breaking the same thing twice is
embarrassing!

But there's plenty of worthwhile acceptance testing beyond just
automated tests. The issue at the moment is that we run a very tight
loop between Richard committing to potlatch2 master and it being
deployed on osm.org. For final testing to work we would need to
deliberately add a delay, and we need to work out whether the delay is
fundamentally worthwhile.

If it is, then one easy way to control the delay is using numbered
releases. We could commit to master (which means it appears on e.g.
http://random.dev.openstreetmap.org/potlatch2/ ), and then give it a
day or so before tagging that with a release number. OSM.org then
would only deploy versions with an explicit release tagged.

But at the moment, I'm not sure whether it's worth adding in the
artificial delay between fixing a problem and deploying the new
version. I suspect it's more worthwhile to work on the automated
tests, so that when we fix something, we're confident that we haven't
broken anything else.

Cheers,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [Potlatch-dev] State of play

[Andy Allan]

On 5 March 2012 04:01, Steve Bennett stevag...@gmail.com wrote:

 undo on creating way fix,

 Definitely worthy of review - it's totally possible that I broke
 something else. Anyone else want to have a look at it? I guess we
 don't have any tests?

We don't have any tests that cover starting drawing a new way.

I'd absolutely love to have tests that cover all the Actions though -
it's so important not to mess up the data in some subtle way while
refactoring. After that, we can test whether pressing the buttons
has the desired effects.

There are some very basic Action tests at
https://github.com/systemed/potlatch2/tree/master/tests/src/net/systemeD/halcyon/connection/actions
already, but to say they are very basic is slightly overselling
them! We're also lacking tests on most of the Entity functionality
too.

Cheers,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [OSM-dev] Nested relations and generic implementation of geometry constructs - Was :osm2pgsql patch for nested relations

[Andy Allan]

On 2 March 2012 12:44, Peter Körner osm-li...@mazdermind.de wrote:

 type=route        -- build-linestring

Remember that there's more than one way to process a route relation -
you can either duplicate the underlying geometries, one per relation,
or push information from the relation down onto the existing geometry,
and perhaps concatenate values.

But otherwise, your idea of building a list of algorithms is a good
one, and lends itself well to configuration files. There's so much
stuff baked into osm2pgsql that should really be configurable.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[Potlatch-dev] Flex 4.6 - should we upgrade?

[Andy Allan]

Hi all,

Flex 4.6 was released in November. Do we want to upgrade (from 4.5)?

I'm not sure if there's anything in particular that we're after in
4.6, but some of the new controls look nice. Most of the work seems to
be targeting mobile apps.

http://www.adobe.com/devnet/flex/articles/introducing-flex46sdk.html

The biggest drawback would be having to bump the minimum target player
to 11.1 , from 10.2. Anyone got any opinions on that?

http://help.adobe.com/en_US/flex/using/WS2db454920e96a9e51e63e3d11c0bf69084-7ee0.html

Cheers,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [Potlatch-dev] Designation tag on most map features

[Andy Allan]

On 19 February 2012 07:16, Steve Bennett stevag...@gmail.com wrote:

 Obviously 2) is easy to fix. But what about 1? Is it time we
 implemented locale-specific map_features?

I refer to them as region-specific map_features, not locale-specific
map features - locale and language are too easily confused!

This has been a long-term goal of mine, stretching back many years. I
want to tailor things like road signs (and hints on what format the
references should be) on a per-country basis - so that when you're
mapping in Switzerland there are green Autoroute signs and when you're
in France it knows that highway=secondary is a Departmental road with
refs starting with a D.

However, to say that adds complexity would be an understatement. I'm
not going to start on that until we have a much better idea how to
manage it - we aren't even at the stage of widespread UI translations
yet, and we don't have a translatable global map_features yet, so I'm
not going to start making life even more complex by adding in regions!

The end goal, for me, is definitely a cross matrix of language and
regions, so that no matter what language you speak, and what country
you happen to be in today, the interface looks as much like what you
have in the photo as possible.

Cheers,
Andy

P.S. http://wiki.openstreetmap.org/wiki/Highway:International_equivalence
shows the tip of the iceberg

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [OSM-dev] Deleted objects

[Andy Allan]

On 19 February 2012 20:46, Frederik Ramm frede...@remote.org wrote:

 This requires some sort of show me all deleted objects in this area API
 call, and once that exists, editors could support it in some way. Potlatch 1
 already has an undelete function but uses a custom API that can not easily
 be re-used by other editors including P2.

Yes. I've been asking around for an undelete API for a couple of years
- it's got nothing to do with the license change though! It's the
major feature in Potlatch 1 that has not, and cannot[1], be
implemented in other editors right now.

 Has work already been done in that direction, or what kind of person would
 be required to do such work?

If talked to Matt Amos (zere) about it a few times, but he has been
planning some other changes to the way that history is stored that
would conflict with a naive implementation of this. I believe that the
naive implementation uses the fact that deleted items are still in the
current tables, marked as deleted, whereas there are plans to remove
deleted objects from the current tables and enforce things with
foreign keys. Obviously if this is done, then the undelete API would
need a major rethink to be based on the entity history tables.

However, there's not been much progress (that I'm aware of) on the
foreign-keys-related work, and we still need the undelete XML API!

Cheers,
Andy

[1] using the XML api. Technically if you want to do some AMF parsing,
the world is your oyster, but you'll have a bad (salty) taste left in
your mouth.

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [Potlatch-dev] Github, and tram/road issues

[Andy Allan]

 On 16 February 2012 06:03, Steve Bennett stevag...@gmail.com wrote:
 Hi all,
  I just did a pull request to the Github repository (which is fairly

You just made a pull request to a read-only mirror.

 up to date) but it looks like no one is watching. Is there somewhere
 else these pull requests should go? It's here anyway:

http://wiki.openstreetmap.org/wiki/Potlatch_2/Developer_Documentation#Sending_patches

We like patches in any way, but if you're sending a pull request,
please send it to systemed on github ([1]) aka Richard, rather than
git.openstreetmap.org or the openstreetmap  account on github.
Thanks.

Cheers,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [Potlatch-dev] Github, and tram/road issues

[Andy Allan]

On 16 February 2012 13:11, Steve Bennett stevag...@gmail.com wrote:

 Are there any policies/practice around using feature branches? I'm not
 sure if it's a Github limitation (or standard Git), but it seems one
 doesn't actually send individual pull requests - at any one time,
 you can have a grand total of 1 pull request outstanding. If I'm
 working on a bug fix for bug #1234, is it easier if I make a branch
 (eg, bug-1234) first? Does that help anything?

It's good practise, regardless of whether you're making pull requests,
to use feature branches. I'm not quite sure what your situation is,
but I've never come across any limitation on the number of pull
requests that I can have open at the same time.

Cheers,
Andy

___
Potlatch-dev mailing list
potlatch-...@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

[OSM-dev] New release of snapshot server

[Andy Allan]

Hi All,

Snapshot Server is a rails app that can host .osm files and serve them
out via map calls. The main use for this is to serve Vector Background
Layers to clients like Potlatch 2 to help with mapping.

http://wiki.openstreetmap.org/wiki/Snapshot_Server

The latest release allows one installed version to host multiple
projects, which is a huge gain and the single most asked-for feature.
In addition, I've given the user interface an overhaul and there's a
per-project breakdown of completion stats. You can see an example
installation, being used for a UK-based cycling project currently in
the demo stages, at
http://gravitystorm.dev.openstreetmap.org/cnxc-snapshot/ .

Code is available on github, and I'm happy to provide support and
answer any questions. Thanks go to Cyclestreets for funding further
development work last week.

https://github.com/gravitystorm/snapshot-server

I hope this will encourage more people to think background layer
rather than bulk import next time they have an interesting .osm file
to use!

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [Potlatch-dev] Potlatch doesn't work after recompile

[Andy Allan]

On 30 December 2011 20:09, Eric Wolf ebw...@gmail.com wrote:
 I had to stick this in my blog because the listserv choked on the original
 email - I was being too detailed.

 In a nutshell, if I make a trivial code change and recompile Potlatch with
 ant, the resulting code doesn't work. The problem is explained in detail
 here:

 http://www.asifanyonecares.com/2011/12/potlatch-dev-problems.html

Hi Eric,

I tried investigating this last week, but since it takes sooo long to
compile potlatch on my laptop I had to give up! I spent an hour or so
compiling over and over this evening instead.

First off, don't worry about ant test not working. The unit tests
have been broken for a few months since various internal APIs were
changed without also updating the tests[1].

Secondly, I see the same problem as you're seeing with the builds
running successfully but resulting in unusable .swf files. The error
messages are slightly perplexing so I've no idea what's the cause.
I'll note simply that I can make debug builds using the fcsh command
line compiler, and also using ant release to build non-debug builds
that work fine too.

If anyone has any clues as to what's up with the build system, please shout!

Cheers,
Andy

[1] It's a shame the unit tests are so unloved - I find these prove
remarkably useful in so many other projects.

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [OSM-dev] [API] What characters to escape in tag keys/values?

[Andy Allan]

On 30 December 2011 13:47, Chris Browet c...@semperpax.com wrote:
 Hi All, long time no see.

 I have an issue with a tag value ending with a % (percent), e.g. 15%,
 returning a 500 error from the api, while 15%20 does not.
 I assume there is some escaping logic behind but it is not clear to me which
 characters should be escaped and how.
 AFAIK, no escaping should be necessary in a xml attribute coded in UTF8, but
 I might be wrong.

 Could someone enlight me, please.

There is some escaping necessary, but not for percentage symbols. See
http://www.w3.org/TR/xml/#syntax for the definition and also
http://stackoverflow.com/a/1091953/105451 which is slightly easier to
read. Of course, there are also a handful of unicode characters that
are not valid in XML (and hence not valid in OSM) but again, that's
not what you're asking.

Sorry not to be able to help directly! Maybe if you can provide a
payload that the devs can replay against a local copy of the rails
port, that would help. And feel free to experiment against
http://api06.dev.openstreetmap.org

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?

[Andy Allan]

On 15 December 2011 12:32, sly (sylvain letuffe) li...@letuffe.org wrote:

 Will the maintenance and stress on this server (therefore on it's sys admins)
 whould be painless if all read api calls were directed to other servers ?

Let me reply to this in slightly hand-wavy terms.

Most of the load on our core infrastructure comes from read requests.
Anyone who can use another service, which gets its updates via the
minutely replication service, helps make the core go faster. This
counts especially for all those things that make requests against the
main API for non-editing purposes.

Also, we (the Operations Working Group) are aiming to spread core
postgres read requests over multiple replicated servers. We're working
hard on getting the relevant hardware in place to start trialling this
out, but it's a big project.

The original topic was about replication for rendering, so a comment
on that - that's not something we're tackling centrally. We have the
minutely diffs to allow robust, low resource replication out of the
core infrastructure, and we don't have plans to change this. As
Frederik has already said, if you want to replicate rendering
databases, jxapi databases or anything else, set one machine up to
read the diffs, and replicate from that using whatever system works.
It can be done today, if anyone wants to do so.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [Potlatch-dev] Potlatch 2.3

[Andy Allan]

On 26 November 2011 12:32, NopMap ekkeh...@gmx.de wrote:

 I played at migrating to P2.3 again and ran into a few problems.

Hi Nop,

First off, two apologies. P2.3 was released with a major upgrade to
the flex libraries that pretty much broke i18n completely, although
due to the seemingly never-ending saga of me being unable to reproduce
problems meant that it wasn't explained clearly in the release
announcement.

http://lists.openstreetmap.org/pipermail/potlatch-dev/2011-September/001220.html

Second apology is in not getting back to you at the weekend, since I
knew I was going to be doing a lot of work around i18n and hoping to
fix things. See the other thread for more on this.

Now, to your points:

 Is there a full set or browsable instance of the locales somewhere?

See http://random.dev.openstreetmap.org/potlatch2/ which is a
continually updated build of potlatch2 with directory browsing
enabled. I've also added a trac ticket
http://trac.openstreetmap.org/ticket/4112 to provide bundled builds in
order to cut down on these hassles.

 2. setting the locale. How do you do that in P2.3? The parameter format has
 changed, it used to be fo.addVariable(locale, de_DE); but what is it
 now?

Try again with 2.3-112 or later, it should be working, thanks to work
by Hiroshi.

 3. Config files. There was a hint earlier in this thread that the config
 file format has slightly changed. At first glance, my unchanged files appear
 to work. Can you give me more information where adjustments are necessary?

I'm not sure which config files you are referring to, but perhaps
Richard can describe more the subcategory panels work.

 4. Authorization. P2.3 asked for a fresh authorization key. What changes
 cause it to do that? That request seems to pop up from time to time, I found
 that I had about 10 authrizations in my osm account. The question is also
 asked occasionally by users.

That will happen when either the flash cookies are cleared, or the
.swf is served from a different URL. Perhaps your users have browser
privacy plugins that are clearing saved flash cookies?

 When i executed the authorization, I had firebug running as I used it to spy
 on P2. This caused Firefox to always crash after the authorization window
 opened. After disabling firebug it worked. Is this a known problem?

Haven't heard of that before.

Cheers,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [Potlatch-dev] Alternate look and feel

[Andy Allan]

On 26 November 2011 23:48, Richard Fairhurst rich...@systemed.net wrote:

 I'd like to offer two bundled stylesheets for the app: one that works with
 the osm.org site design, and another, more generic one suitable for
 third-party deployments. (You'll also be able to supply your own stylesheet
 if you compile just the one stylesheet file with Flex SDK.)

That's a good spur for me to continue porting the visual components
over to their spark equivalents - not only is it good for consistency,
but it'll make the stylesheets easier! There's plenty of easy upgrades
that can be made, but I'm surprised to find there's still a few
components that don't have spark equivalents yet. I guess the key is
to upgrade all the ones we can.

Cheers,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [OSM-dev] New transport render (and bugs on ÁÉÍÓÚÑ)

[Andy Allan]

On 25 November 2011 13:02, Iván Sánchez Ortega i...@sanchezortega.es wrote:

 So, question: What typography is the Transport map using?

 Yeah, yeah, I know I'm raining in Andy's parade. Don't make me feel bad about
 that :-(

No worries at all - great to get feedback. There's clearly an issue
with the font file that I'm using for the stations, but I'll need to
investigate exactly what's going on - I'm not an expert on this. I
guess the font is claiming to have glyphs that it doesn't really,
which is preventing it from falling back to unifont. I might try and
roll my sleeves up and add some more characters!

I'll try and get it fixed ASAP.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osm2pgsql update

[Andy Allan]

On 24 November 2011 09:53, Frederik Ramm frede...@remote.org wrote:

 Is anyone aware of multipolygon handling not working right when not using
 --slim? We might have to (re)introduce the primary key for osm_id at least
 on the polygon table to allow this deletion of duplicate areas.

I've always heard it stated the multipolygon handling is unlikely to
work properly in non-slim mode. But I honestly have no idea if anyone
seriously uses non-slim mode for any production services.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OAuth down

[Andy Allan]

On 17 November 2011 06:59, Pierre GIRAUD pierre.gir...@gmail.com wrote:
 Can you point me to urls to show me what changed recently ? (commits, diffs)

https://github.com/openstreetmap/openstreetmap-website/network

The what changed recently was that we upgraded from rails 2.x to 3.1
- so many things needed changing. You can see the commits that were
rebased onto master, there's plenty of them, and lots of plugins have
changed too. It's unlikely that the diffs will help you figure out
what changed, since so much has! I guess if you can reproduce the
issue then a binary search might help, but it's likely to be a plugin
upgrade or some other large change.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [Potlatch-dev] [Talk-us] Address improvement through imports?

[Andy Allan]

On 5 November 2011 18:11, Michal Migurski m...@stamen.com wrote:

 You probably don't even need to fork it. I suspect you could get most of the
 way there with a custom P2 style, a custom map_features.xml, and Andy's
 awesome new snapshot stuff (which is expressly designed for manually
 bringing in data from other sources). potlatch-dev is happy to help. :)


 Hi, this sounds cool! I don't closely follow potlatch-dev but I'm curious 
 about this feature.

Hi Mike,

It's a prototype feature, and poorly documented! The basic scenario is
that you have a .osm file representing some kind of external
datasource, that you want to combine into OSM - i.e. not blindly
upload directly to the main server. So you load the data up into a
small rails app (snapshotserver)[1] which serves it out as map
requests, and you use potlatch 2 to allow people to go through each
feature and reconcile them with what's already in OSM. As a bonus, you
can mark features as complete, thereby collaboratively reviewing the
external datasource.

There's some information about this on the wiki[2] and a blog by
Cyclestreets, who commissioned the work[3]

I'd love any feedback you have on it.

Cheers,
Andy

[1] https://github.com/gravitystorm/snapshot-server
[2] http://wiki.openstreetmap.org/wiki/Snapshot_Server
[3] http://www.cyclestreets.net/blog/2011/10/24/osm-merging-tool/

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [OSM-dev] osm2pgsql, direction of a virtual way (based on a route)

[Andy Allan]

On Wed, Oct 26, 2011 at 5:47 PM, Frederik Ramm frede...@remote.org wrote:

 this way has to be reversed to form a linestring with the others,

more like the direction of the route is with/against the direction of
the underlying way

 while
 others seem to use backward as the bus only uses this section of road on
 the return journey.

Which is, and always has been, completely stupid.

 But I have not systematically looked into this. Someone should perhaps do
 that. - Most route relations are probably not directional (cycle routes come
 to mind).

Bear in mind that cycle routes are one of the earliest and heaviest
users of forward/backward roles. It is easy to find them on
opencyclemap, e.g. at

http://osm.org/go/euu4OUOi?layers=C

I'm disappointed to see that whoever came up with the new PT schema
didn't understand what these roles are useful for, which was nothing
to do with the overall route characteristics (towards or away from one
particular place), instead they are for expressing the relation
between the direction of travel and the direction of the way. The
separation of every bus routes into two bus routes, one for each
direction, even when they are simply duplications of one another, is
surely a waste of time and effort.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [Potlatch-dev] Potlatch 2.3

[Andy Allan]

On Thu, Oct 20, 2011 at 11:07 PM, NopMap ekkeh...@gmx.de wrote:

 Unfortunately, this did not work. There is no counterpart of activity viewer
 for firefox.

Firebug's Net console gives you the equivalent, if I understand
Richard correctly.

 Trying to execute the above line with git produced the error message:
 Cloning into c:/Data/Maps/potlatch...
 Permission denied (publickey).
 fatal: The remote end hung up unexpectedly

Richard gave you his private link to github (since he was logged in at
the time) so that's why you didn't have permission. The public
equivalents are

https://github.com/systemed/potlatch2.git or
git://github.com/systemed/potlatch2.git

You can also grab it from the osmf git mirror, which reflects the
deployed version, at

http://git.openstreetmap.org/potlatch2.git

I've also updated the Developer Documentation to include the upgrade
to flex 4.5.1

Finally, if you want the contents of the source code directory without
messing around with git, then there's a Zip download button on the
github pages which will simply give you a tarball to download.

If there's anything else, give me a shout.

Cheers,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [OSM-dev] complying with google/bing/yahoo geocoding terms of service

[Andy Allan]

On Tue, Oct 4, 2011 at 5:43 AM, Paul Norman penor...@mac.com wrote:
 Nominatim is the standard way to do reverse geocoding.

Nominatim is good for both geocoding and reverse geocoding. It powers
both the search and the Where am I functions on the main site.


 If you want to do
 geocoding (telling you the name of where you are)

That's actually reverse geocoding. Geocoding is the process of
creating a geocode - i.e. lat/lon - for a given input string. Think
of it as encoding. Reverse geocoding - aka decoding the geocode,
means you start with a lat/lon and you want to decode that into a
string.

If find neither term particularly self-explanatory, nor easy to
remember which is which!

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[OSM-dev] osm2pgsql and only-named multipolygons

[Andy Allan]

Hi All,

I stumbled across an issue this week and I don't have a great solution
to it, so I thought I'd share it here.

In osm2pgsql we handle two types of multipolygons - those with the
useful tags on the relation, and those with no useful tags on the
relation where we use the outer ways instead. Note that we also
explicitly allow the relations to have a name. See
http://trac.openstreetmap.org/browser/applications/utils/export/osm2pgsql/output-pgsql.c?rev=26664#L1001

I've found this very common, especially with buildings. In fact, I
used it as an example in my talk at SOTM-EU, where another application
was showing buildings incorrectly - in some cases filling in the whole
thing (tags-on-outer-ways) and missing the building entirely
(tags-on-buildings). So both situations are established and we need to
deal with them.

However, one relation has caused me lots of issues, namely
http://www.openstreetmap.org/api/0.6/relation/1715069 . It's a giant
relation with a name, type, and nothing else, and so osm2pgsql creates
a polygon using one of the outer ways to find the useful tags.
Unfortunately it picked up waterway=stream from one of them, and I
supported drawing them as polygons, so I ended up with several hundred
square km of stream. Argh! But since no other map supports this
particular level of micro-mapping, it appears that it's my map that's
broken, so I've changed my styles to explicitly not support
streams-as-polygons.

But that's not fixing the general issue - people have a (strange)
fascination with creating relations for all kinds of (strange)
reasons, and that will only continue. Can anyone see a better way of
handling this situation? Should I have drawn a line in the sand and
demanded the relation in question was changed? Or is it an obligation
on map makers to tread only in the footsteps of the main layer? I
guess if osm2pgsql had picked a different outer way for the useful
tags - say one with highway=pedestrian, and it showed up on the main
style, then the relation would have been changed pronto.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [Potlatch-dev] Potlatch 2.3

[Andy Allan]

On Fri, Sep 30, 2011 at 6:29 PM, Richard Fairhurst rich...@systemed.net wrote:

 Thanks to Andy for huge amounts of work on this.

You're welcome - it's great to see it all finally coming together!
Preparations for this release started way back in May with the fp10
branch.

There's a lot of work to be done to take advantage of various new
things - Flash Player 10 means that we can use Vectors instead of
Arrays in order to improve type safety and opens the door to local
filesystem access - think loading gpx traces - but the main thing I'm
looking forward to is using the new features of Flex 4.5 user
interface components. At the moment we have a bit of a mixed bag of
components between the old mx ones and some of the new spark ones, but
when the migration is completed we'll have great possibilities. It's
also a significant help to any other flex developers or designers who
want to jump in and help us out!

I'm hoping that some of our developers can help get to the bottom of
all of our translation issues - it's been a real bugbear for some time
now and I'd like to get it well and truly fixed.

Cheers,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [Potlatch-dev] [Talk-GB] 'Can't load map'

[Andy Allan]

 So now if you get a simple Couldn't load the map it means a 500 server
 error genuinely came back from the API. This is the sort of thing that
 basically shouldn't happen, but it can occur, AIUI, if the daemon restarts
 halfway through the response; I've had that very occasionally.

Taking this to potlatch-dev

I'm really, really hoping that this has fixed everything. However, I
still have a lingering suspicion that there's a client-side issue
where the URL is never actually requested. I've seen this happen with
the linux debug player against a local rails install using wireshark,
and I suspect it happens on the release flash plugin too against the
main server, but I haven't captured with wireshark conclusively. And
so I can't pin it down and it takes friggin ages to try and trigger
it.

I've found it happening when panning around lots, and I wonder if
there's a max-concurrent-requests issue somewhere between flash and
the browser. I've seen enough reports about this, and experienced it
often enough myself,  to believe that it's not actually the site
that's returning that many errors.

Maybe we'll need to trap map call errors in potlatch and ping a
logging url to let us reconcile times-potlatch-thinks-it-got-an-error
with times-the-rails-serves-logged-sending-an-error and see how they
correlate!

Cheers,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [OSM-dev] Best way to render tiles with apidb changes?

[Andy Allan]

On Sat, Sep 24, 2011 at 5:21 PM, Евгений Фарфель je...@jenfa.od.ua wrote:
 Hello!

 I am currently setting up local openstreetmap server (Rails port, tile
 renderer and Postgres API DB). So far I have a working rails port
 installation and able to submit changesets into apidb, they are
 working as expected. However, if I understand correctly, those
 changesets will not be visible on tiles rendered by Mapnik, because
 Mapnik uses separate database schema, am I right? So, what is the
 correct way to present apidb changes on Mapnik tiles? Or, maybe, I
 should not use Mapnik at all?

The best thing to do is to set up a minutely mapnik server. This uses
all the standard software so no coding is needed. When you have the
mapnik+postgis+osm2pgsql+mod_tile server set up, you can then export
the minutely diff files from the apidb using osmosis, and import the
diff files into the mapnik database. Instructions are at
http://wiki.openstreetmap.org/wiki/Minutely_Mapnik

I've set up both the apidb and the mapnik db on the same machine many
times, it works very well. Although it can be tempting to try setting
up something to read the apidb directly, having two databases and the
standard api replication system is much more robust and well tested.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Speeding up Osm2pgsql through parallelization?

[Andy Allan]

On Tue, Sep 13, 2011 at 1:07 AM, Kai Krueger kakrue...@gmail.com wrote:
 Hi,

 I was thinking about ways to try and speed up osm2pgsql. Currently a good
 fraction of time, both in full imports and during diff-processing, is spent
 in the going over pending ways / relations section. Therefore speeding up
 that section should bring the overall time down quite a bit. One thought to
 try and speed up the going over pending ways / relations is to try and
 parallelize it.

That's funny, I'd been looking last week at the next step in the
processing and wondering if I could get a speed increase by
un-parrallel-ising it. I don't have the time or the skills to make
much headway, so I'll happily confuse this thread by talking about
other things.

http://trac.openstreetmap.org/browser/applications/utils/export/osm2pgsql/output-pgsql.c?rev=26651#L1365

At the moment, the creation of the temporary tables is done in
parallel, and so you need up to the sum of the sizes of the geometry
tables in free space (albeit some tables run faster than others, so -
depending on timing - you need less space). My concern is that doing
this serially will lead to improved IO (instead of thrashing between
threads) and less free space required since you'll only need up to
max(sizeoftables) instead of potentially sum(sizeoftables).

As for the create tmp - sort - overwrite, is there anything to be
gained by using the built-in CLUSTER instead? I'm not sure how well
our method will actually arrange things on-disk, but again I've done
nothing to investigate any hunches.

Just some thoughts from having stared at the output for too many hours :-)

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] osm2pgsql, -e option might affect too many tiles

[Andy Allan]

I'm not sure that osm2pgsql is massively sophisticated with this
situation, since it certainly doesn't know anything about the
stylesheets.

If, for example, the node was moved slightly, and all those lines were
dashes, then the full length of every line would need to be redrawn.
Alternatively, if it was just a tag change but now became e.g. a pub,
then any blocking symbolizer along those lines (e.g. road names) could
have moved and again everything needs redrawing.

That said, I'm sure there's plenty of optimizations that can be made
in this area.

Cheers,
Andy

On Thu, Sep 1, 2011 at 7:10 AM,  mar...@gmx.eu wrote:
 Hello,

 I'm using osm2pgsql with the excellent -e option which allows me to generate 
 dirty-tiles lists. It's my observation that the output can be somewhat 
 extensive if you just change ONE node. For example:

 I change the name of a single node. This causes osm2pgsql to put not only the 
 node's tile on the list but also every tile which is touched by every way 
 which uses this single node. I guess there's a reason for this behaviour. 
 Does anybody know situations in which it is really necessary to rerender all 
 these ways?

 Markus

 ___
 dev mailing list
 dev@openstreetmap.org
 http://lists.openstreetmap.org/listinfo/dev


___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

Re: [Potlatch-dev] Git starter?

[Andy Allan]

On Wed, Aug 31, 2011 at 4:13 PM, Steve Bennett stevag...@gmail.com wrote:

Yes, but this was causing lots of issues, as I'm sure you remember.

 Actually not - apart from my commiting changes in insufficient
 granularity. Happy to take your word for it, though.

A few other things spring instantly to mind - an ill-advised redesign
of the Undo system, a few problems with the Magic Roundabout system
(e.g. having a 50% chance of nuking a way when trying to shorten it)
and, more than anything else, lots of cases of commits generating
compiler warnings. That these commits were to trunk kept necessitating
other developers stepping up and fixing the build before they could
continue with their own work. That was getting quite disruptive.

 You're probably aware of the long-running debates about pre-commit
 review versus post-commit review. Quick summary: pre-commit review
 reduces developer activity level but improves quality.

I don't know much about the different approaches from a theoretical
viewpoint, except that the way were were doing it before caused us
lots of issues, and was making refactoring nigh on impossible.

As Richard has said, his master branch is the canonical 'this is
Potlatch2', so he's in charge

 Cool. This is the definitive statement lacking from the wiki page,
 which confused me.

I'd written that before the situation was clear. But there's also the
distinction that it's not necessarily what OSMF is deploying, and I
try to discourage anyone from getting worried about which repo to
clone from. If you clone from mine, and then want to pull changes from
someone else, it all comes out the same. Any notion of One True Repo
just causes more confusion later on!

 1) There is still a definitive repository

Make sure you realise that there's nothing that makes it the
definitive repository other than social factors. Unlike svn there's no
central repository. It's only definitive in that Richard is the
current maintainer, and the only difference he has over the rest of us
is that TomH generally doesn't disagree with him (on p2 matters at
least!). But you could also view the OSMF repo as definitive if you
care about the version that's actually deployed on osm.org

There are circumstances (we've done it two or three times already that
I can recall) where TomH pulls in bugfixes from my repo straight into
the OSMF one and deploys that, when RichardF wasn't around to update
his own repository. That's fine, git is totally decentralised like
that so it works.

Cheers,
Andy

___
Potlatch-dev mailing list
Potlatch-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/potlatch-dev

Re: [OSM-dev] How to check for database updates after minutely Mapnik Installation?

[Andy Allan]

On Fri, Aug 19, 2011 at 1:10 PM, Parveen Arora m...@parveenarora.in wrote:

 You can instruct osm2pgsql to create a so-called dirty-tile-list which then
 What is the meaning of dirty tiles this term always confuses me.

A dirty tile is a map image that has been marked as being out-of-date,
and therefore needs to be redrawn when the system gets the chance.

Mod_tile does this by having a file
(/var/lib/mod_tile/planet-import-complete) whose modification
timestamp is the last time the planet database was updated. When it
serves a tile, it checks the timestamp on the tile and makes sure the
tile is newer than the timestamp on planet-import-complete. If it is
older, it asks renderd or tirex to re-render the tiles.

When you are running minutely updates, it's common to use the expiry
lists generated by osm2pgsql to expire the tiles that need changing.
Either the tiles are rendered directly, or they can be marked dirty.
A tile can be marked as dirty by changing the timestamp to be really
old (amusingly, in OSM terms this means somewhere in 2000, which may
as well be the start of all time for us) so that it appears to be
older than the database import. That way it'll get re-rendered
automatically next time it's viewed.

When you make your own system not using tirex or renderd, you will
want to only re-render the tiles that need to be (rather than them
all). You'll need to take the list the osm2pgsql outputs, and then
either re-render them all in a script, or keep track of which ones are
needing re-rendering and do it later. If you update every minute then
sometimes you won't be able to keep up, hence the more complicated
approaches involving timestamps.

Cheers,
Andy

___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev