Re: [OSM-dev] [GSoC] Data tile example site
Ander, You'll have to store the coordinates yourself. Because you needed the coordinates to request the tile in the first place, you already have them -- you just need some way of associating the coordinates with the tile. When caching I don't think you need to store the coordinates explicitly -- you would probably use the coordinates as a key and the tile's JSON contents as a value. One thing you can do if you want to cache tiles on disk is store them as individual files in directories. Conceptually, you would store the tile with coordinates z=16 x=32744 y=21792 in "/path/to/cache/16/32744/21792.js". This won't work well as you cache more and more tiles, because there will be a lot of files in these directories. mod_tile uses a hashing scheme to avoid this problem: https://github.com/openstreetmap/mod_tile/blob/a39905d1957fd37cba0aa3a4ec7302ba4865b34c/dir_utils.c#L68 Another approach is to use a hash (like SHA1) of the tile's contents as a cache key, and maintain a mapping between coordinates and hashes. This avoids storing tiles twice which have the same content. There's a brief description of this approach here: http://mapbox.com/developers/mbtiles/#using_views_to_reference_redundant_images -- Michael On 2012-08-20, at 3:09 AM, Ander Pijoan wrote: > Hi! I have a big question: > > Imagine I download one of the tiles and keep it in my hard drive or in > a cache or send it to a client to open it in a program. The tiles > bounding box or X,Y,Zoom values are not in it so it won't be able to > know where to place that tile. > > I know that when asking for the tile I have those X,Y and Zoom values > so I can calculate its position but if it is not written anywhere in > the tile. I made some tests with Kothic's tiles allocated at > Osmonimski.ru and those tiles at least have their bounding box and > also their X,Y and Zoom values explicit in them. > > > > 2012/8/17, Michael Daines : >> Hi everyone, >> >> An example data tile server covering the British Isles is now online: >> >> http://data-tiles.mdaines.com/ >> >> The library I have been working on is rendering tiles as a Tirex backend >> using OpenStreetMap data. The map is drawn using Kothic JS over top of image >> tiles rendered with Mapnik which show land polygons (basically the contents >> of the layer-shapefiles.xml.inc file in OSM's Mapnik stylesheet). >> >> Here is an example data tile: >> >> http://data-tiles.mdaines.com/render/16/32744/21792.js >> >> The data in the tileset is defined by this configuration file which uses a >> DSL that works sort of like CSS selectors: >> >> https://github.com/mdaines/cover/blob/master/config/render.rb >> >> >> -- Michael >> >> >> ___ >> dev mailing list >> dev@openstreetmap.org >> http://lists.openstreetmap.org/listinfo/dev >> > > > -- > Ander Pijoan Lamas > Ingeniero Técnico en Informática de Gestión > Universidad de Deusto > > Contacto: > Email: ander.pij...@deusto.es > Móvil: +34 664471228 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] [GSoC] Data tile example site
Hi everyone, An example data tile server covering the British Isles is now online: http://data-tiles.mdaines.com/ The library I have been working on is rendering tiles as a Tirex backend using OpenStreetMap data. The map is drawn using Kothic JS over top of image tiles rendered with Mapnik which show land polygons (basically the contents of the layer-shapefiles.xml.inc file in OSM's Mapnik stylesheet). Here is an example data tile: http://data-tiles.mdaines.com/render/16/32744/21792.js The data in the tileset is defined by this configuration file which uses a DSL that works sort of like CSS selectors: https://github.com/mdaines/cover/blob/master/config/render.rb -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [GSoC] Data tiles - demo
> https://github.com/mdaines/render-data-tiles The link to the repository I gave is incorrect. The correct link is: https://github.com/mdaines/render-data-tile -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] [GSoC] Data tiles - demo
Hello, I now have a small demo of my data tiles project online here: http://data-tiles.mdaines.com/ This shows data tiles for the SF bay area at various zoom levels rendered as an overlay over image tiles. The tiles can also be used to render map images with Kothic JS -- choose the "Kothic" base layer and deselect the "Debug" layer. The tiles were rendered using the library I am working on here: https://github.com/mdaines/render-data-tiles The tiles were produced as follows: - The SF bay area metro extract was obtained from http://metro.teczno.com/ and imported into a PostGIS database using osm2pgsql (with the --store option) - Land polygons were obtained from http://openstreetmapdata.com/ and imported. - The script/build command from my project was then used to render data tiles for the area imported at zoom levels 9-13 using the "basic.rb" configuration file included in my project's repository. This produces an MBTiles database with zlib compressed JSON tiles. The viewer.rb application from my project is used to serve the tiles from the MBTiles database. Since tiles are stored compressed, and their sha1 hash is used to avoid duplication in the database, the application can serve the tiles compressed and set etag headers with little added cost. At zoom levels 9-12, the tiles include a subset of the features available from the osm2pgsql import, and the features' geometry is simplified. At zoom level 13, the tiles include all features without simplification. At all zoom levels, all tags from the import are available. This configuration file which determines this is here: https://github.com/mdaines/render-data-tile/blob/master/basic.rb I am interested in feedback on what subset of OSM data would be useful to include in an "official" set of tiles, particular at lower zoom levels. However, the idea is that it should also be possible to build whatever kind of tileset you want for your purposes. -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] [GSoC] Data tile service project updates
Some updates on my project: Last week I worked on feature simplification and filtering, and on building a tile set for a somewhat larger area: California. I have a demo data tile set posted in mbtiles format. The individual tiles are GeoJSON feature sets as specified on my project page, each compressed using gzip. http://mdaines-tiles.s3.amazonaws.com/california-z10.mbtiles This can be viewed using the script/debug command in my project's repository -- it provides a web interface which draws the data using a Leaflet canvas layer. https://github.com/mdaines/render-data-tile The tileset was produced from the GeoFabrik extract of California. The data was imported into a PostGIS database with osm2pgsql and then the tiles were produced with this script and config file: https://github.com/mdaines/render-data-tile/blob/master/build_california_tileset.rb https://github.com/mdaines/render-data-tile/blob/master/basic.rb I created a more complete tile set with zoom levels 12 to 14, but it's a bigger file (450 MB) -- let me know if you'd like to have a look. Zoom level 14 includes all features from the extract. -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [GSoC] Data Tile Service
One more thing: screenshots from the web-based debugging tool. http://osm-data-tiles.tumblr.com/ On 2012-06-18, at 1:03 PM, Michael Daines wrote: > Hello, > > Here are some updates for the GSoC Data Tile Service project. > > The repository for the tile rendering script is available here: > > https://github.com/mdaines/render-data-tile > > The script will render JSON tiles using data imported to PostgreSQL/PostGIS > with osm2pgsql. It can render individual tiles or a list of indices. There is > also a web-based debugging tool in the debug directory which renders tiles on > demand and displays them using HTML5 canvas and Leaflet. > > I have been running the render and debug scripts with the geofabrik extract > of the state of California. I have also been experimenting with rendering > tiles for that entire area. Here is an example tile: > > http://temporary.mdaines.com/gsoc-2012/15-5238-12699.js > > A note on file sizes: this particular tile is about 12K, and 3.2K gzipped. > The corresponding Mapnik image is 14K (about the same size gzipped). > > I have also created a page on the wiki for documenting the data tile format > produced by the script: > > > http://wiki.openstreetmap.org/wiki/Google_Summer_of_Code/2012/Data_Tile_Service/Tile_Format > > The format is basically a subset of GeoJSON with the addition of the Kothic > data tile format's "granularity" property. This allows coordinates to be > expressed as integers relative to the tile's position. > > This coming week I will be working on documentation, geometry simplification > and filtering, and on making a set of tiles for the data I have been working > with. > > > -- Michael > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] [GSoC] Data Tile Service
Hello, Here are some updates for the GSoC Data Tile Service project. The repository for the tile rendering script is available here: https://github.com/mdaines/render-data-tile The script will render JSON tiles using data imported to PostgreSQL/PostGIS with osm2pgsql. It can render individual tiles or a list of indices. There is also a web-based debugging tool in the debug directory which renders tiles on demand and displays them using HTML5 canvas and Leaflet. I have been running the render and debug scripts with the geofabrik extract of the state of California. I have also been experimenting with rendering tiles for that entire area. Here is an example tile: http://temporary.mdaines.com/gsoc-2012/15-5238-12699.js A note on file sizes: this particular tile is about 12K, and 3.2K gzipped. The corresponding Mapnik image is 14K (about the same size gzipped). I have also created a page on the wiki for documenting the data tile format produced by the script: http://wiki.openstreetmap.org/wiki/Google_Summer_of_Code/2012/Data_Tile_Service/Tile_Format The format is basically a subset of GeoJSON with the addition of the Kothic data tile format's "granularity" property. This allows coordinates to be expressed as integers relative to the tile's position. This coming week I will be working on documentation, geometry simplification and filtering, and on making a set of tiles for the data I have been working with. -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Socket problem with osm2pgsql on Mac/Homebrew
> I have confirmed that /usr/local/bin/osm2pgsql and /usr/local/bin/psql are > the versions being used by my system (with which psql, which osm2pgsql). > When I try to run osm2pgsql I get a strange connection error: > > osm2pgsql us-south.osm.pbf -r pbf > osm2pgsql SVN version 0.80.0 (32bit id space) > > Error: Connection to database failed: could not connect to server: Permission > denied > Is the server running locally and accepting > connections on Unix domain socket "/var/pgsql_socket/.s.PGSQL.5432"? > > Any suggestions? I can connect locally to my databases fine with Navicat and > via psql. Is it possible that osm2pgsql is trying to access the wrong psql? When using homebrew, it appears that you need to include the following option: --host=/tmp >From this guide: http://bdon.org/2012/03/23/osm2pgsql-homebrew-and-postgis-on-lion/ -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] GSoC - Data Tile Service
Hello, Here's a quick update on my GSoC project so far: I have been documenting the data tile format from the Kothic project (including the details of how map features are cropped, etc.) which is at present what the data tile format I am working on will be based on. I am also working on describing other schemes for cropping OSM data, and other JSON representations of OSM data. Second, I am working on actually producing data tiles. Currently, I am using a city-sized subset (the extract of San Francisco from http://metro.teczno.com) which is processed with osm2pgsql. I have set up a simple web service which produces tiles on demand and a Leaflet canvas layer which I am using to inspect the results of producing the tiles, so I can see renderings of the tiles in context. I will post this work (the documentation and the tile production script and web testbed) on the wiki and in a GitHub repository in the coming week. Finally, I have been working on a script for producing augmented diffs (as suggested a little while ago on the list by my mentor) by querying the Overpass API, meant to make keeping a replica of the planet up to date with minute diffs more tenable in the face of data processing speed and the possibility of missing data and broken geometry. An initial version is here: https://github.com/mdaines/augment-osc Thanks to everyone who has offered suggestions and advice! -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC - Data Tile Service
> You already matched my interests. > Client side (interactive) rendering, with json data format. > For low zoom levels with reduced data (only coastline, borders, rivers, > motorways) - nearly the data you see in current map tiles. > For higher zoom levels you may divide the data into different data sets like > train lines, motorways/streets, coastline/rivers, buildings, public > transport, others. Stefan, Good to hear that you're interested. Are you working on a client-side renderer? -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC - Data Tile Service
> * Problem: Since we are speaking about "vector tiling" it's > unavoidable that polygons are clipped, which makes me wonder on how > the client is informed and how he should deal with this? I'm not sure if I have a complete answer for that yet. Do you have any suggestions? One possibility is to do what the API does for bbox queries, where complete geometry for ways with a node intersecting the tile is included, and also the relations which refer to the stuff in the tile, but that's it. (So if you have a tile intersecting with the border of a country, you don't get the entire border, just one part of it.) This may be a bit unsatisfying for client-side rendering, since it's possible that a tile could be sitting entirely inside some kind of polygon, and the client wouldn't know until it downloaded one (or more) of the tiles around that. Another idea: if a tile is contained entirely within a polygon, include the polygon's tags (or some subset) "along with" the tile, rather than as a feature of the tile. > @Michael: > * In the website you refer to [1] there's mentioned "Similar to kothic > js tile format": Why only "similar"? Why don't you not base/fork on > Kothic/KothicJS? One trivial reason: I'm not sure yet if it makes sense for tiles to be in JSONP (rather than JSON). Also, I may want to abbreviate keys in JSON, and I was considering the possibility of some properties being represented by JSON arrays (for example, a list of element ids). -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC - Data Tile Service
> So during this stage we'll also develop a small prototype server that serves > tiled and .osm files that are tiled and filtered per zoom level. The > assumptions there look similar to yours. > We'll identify the most needed changes (to our client code or the temporary > .osm file format adjustments e.g. for handling cross-tile-polygons). And then > we'll continue development depending on that. Hi Torsten, The project sounds interesting. I'm curious to hear more of your thoughts on dealing with cross-tile polygons. How does drawing .osm files work currently? -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] GSoC - Data Tile Service
Hi everyone, My name is Michael Daines, and for this year's GSoC I proposed to create a service to provide fast, up-to-date map data accessible by tile. Some example uses are client-side rendering, editors, and client-side interactivity. My plan is for the project to be available as both a publicly-accessible service and as an application that's easy to set up and customize for yourself (even on a small home server). The wiki page for my project is here: http://wiki.openstreetmap.org/wiki/Google_Summer_of_Code/2012/Data_Tile_Service I'm looking for feedback about a couple things, so if this seems like it might be helpful for a project you're working on or want to work on, I'd like to hear from you about what I've written about so far on the wiki page. - How would you use a data tile service? I've added some notes about the requirements for a few use cases so far on the wiki page, but I'm hoping to collect more ideas. - What do you think about the tile format? - What do you think about the "customizations" for data tiles that I've mentioned on the wiki? These are mainly optimizations to reduce the size of tiles, particularly at lower zoom levels. - Would you like to set up a data tile service on your own server? Since this is an important part of the project, I'm interested in finding testers. Another issue is what kind of tile sets will be made publicly available as part of this project. I imagine the answer will become clearer as the project progresses, but one possibility is a tile set with complete data at a single high zoom level. A smaller goal my timeline mentions as a milestone is making available a tile set with a single zoom level which includes all data for a particular geographical area (possibly the SF bay area). Thanks! ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Possible GSoC project: tag/area monitoring service
> I was thinking about very long routes, and country borders. If I want > to monitor changes to my state and interstate routes within the state, > I don't have any good options at the moment to do that. I don't think > it should be terribly difficult to implement, I'm just not sure how > well it will scale. To make it more interesting you could allow for > watching all changes to objects with certain tags that are within a > certain distance of a route relation, or located inside a > multipolygon. This is exactly the sort of thing that I'd want to support -- to be able to say "show me updates that are both in this area and match this tag expression". Serge suggested that areas might be implemented more efficiently by splitting the map up into tiles, which would allow for an approximation of any kind of area: just figure out which tiles intersect with the area you're interested in. -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Possible GSoC project: tag/area monitoring service
> I would expand 6 to be documentation for use as well as the ability to > replicate the server environment using OSM planet data update feeds. I > personally expect the restrictions on the tile servers to be extended to the > API servers when enough application coders implement a way to use the API > directly from thousands or millions of clients at which point they'll be > instructed to fire up their own server and need more than just use-based > documentation. Definitely. Customization would also be important, since I imagine that users would like to avoid storing a bunch of data they're not going to use in their project. (Depends on how things are implemented, though.) If your project only dealt with a single city, it might make things easier to set up if you knew the storage or computation requirements were less onerous than dealing with the entire world. -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Possible GSoC project: tag/area monitoring service
> Write some code to query jaxpi for bounding boxes in Python based on tile > name. > Use this and write "Data tile" support in TileStache. I'd store cached > tiles in Redis (for reasons that become apparent in a few sentences). > I'd use the parsing/storing bits of Changepipe to tell me which tiles > are effected by a changeset (even though I believe it uses the > changeset's bbox, which is oftentimes wrong). > Since Changepipe is already using Redis, using Redis for the tiles makes > sense. > > And then the issue would be how to hack in some code for the > websocket/stream/whatever. This seems like it'd be relatively simple > using Redis pubsub and something like gevent, but I haven't looked > into it. Do I have this right: the server in this implementation would act as sort of a fast, tile-addressed cache for data available through XAPI or similar? > The right answer would be to keep a local copy of the database and > then update it as necessary. I believe Ian Dees has a copy of some > MongoDB code that uses quadtile to index OSM objects (I'm very fuzzy > on the details). (Update, Ian sent me this url, but I haven't taken a > look: > https://github.com/iandees/mongosm/commit/c46c2081edde0b3b2b0446dd06d5ef02b292631c > ) > > Then as objects would change, you'd be able to update the tiles. It looks like mongosm includes an implementation of a data tile server? The quadtile indexing is interesting in that you use only a single parameter to refer to tiles, rather than the z/x/y triple commonly used with image tiles. Keeping a local copy seems simpler and more reliable, but you have to store all the data... This is where I see some kind of customization as being useful -- if you were running your own server, and were only interested in a single city, or only interested in roads and building shapes, you could store just that data. >> Perhaps some kind of quadtree-type approach could be used, where tiles are >> split if they have high density? > > That'd certainly work. I'd started with a naive approach of "If I only > have one zoom level, things are easy", and then you just accept that > some areas are dense, and others not. At the same time, there won't be > as much demand for low density areas. > > There's certainly value in cleverness and not transmitting too much > data, but there's also value in simplicity for clients. Particularly for GSoC I think I'd want to err on the side of simplicity. The zoom level could always be adjusted? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Possible GSoC project: tag/area monitoring service
> First, a longstanding wishlist item for OSM has been "data tiles", > that is the API data, split into preset sized areas (eg z14), which a > client could call. This may not seem reelvant to your project but > you'll see why it is soon. This was actually part of my original motivation for proposing this project -- in my 2010 GSoC project, I used bbox queries to load data in tile-like sections, but as I mentioned this turned out to be very slow. Data tiles seem like they could speed things up for that sort of use. Ideally, the work involved in accessing a data tile would be comparable to accessing an image tile. Also, it seems easier to cache data addressed by tile than it is to cache the results of arbitrary bbox queries. I'd also be interested in working on data tiles -- is that in itself a reasonable project idea? My hope is that if either of these ideas are things people have been wanting for a while, they'll want to use them, and that if a project has people using it, it would be more likely to be around after the summer. One thing I was wondering about -- how do you choose a tile size to minimize both the number of accesses (larger tiles) and the byte size of tiles (smaller tiles)? Some areas have a much higher density of data than others. Perhaps some kind of quadtree-type approach could be used, where tiles are split if they have high density? The ideas you suggest for streaming-type updates on data tiles are very interesting. If you were writing an editor, you could be more certain that you were displaying the most recent data without having to reload all of it. > While you could use Changepipe to make arbitrary polygons and then > stream the changes, IMHO this is not as generally useful as one might > imagine. Network hiccups alone can mean that it's possible to miss an > event. And arbitrary polygons become "complicated" as the number of > queues can be large. I hadn't thought about using arbitrary polygons to specify areas as it seemed too complex -- would there be much call for that? I assume the use cases would be things like keeping track of updates to a city (the area of which isn't always conveniently specified as a bounding box). > By splitting the areas up, you can now take a changeset and know which > areas (tile) it effects. And then each client can simply subscribe to > an area (tile). You've greatly simplified the problem, whether you > allow for arbitrary shapes (one shape -> many tiles) or 1:1 tiles to > connections. > > Now, to your original question... Another advantage of "tiling" the > data is you can easily do both. Each tile can have a list of changes > associated with it. If you tried to do this on arbitrary polygons, > it'd get difficult very quickly. This makes sense, as I guess it means there are fewer bins to put things in when an update needs to be sent out to clients. (You only have to do the work once if several clients are looking at a particular tile.) And, if a client really did want to look at an arbitrary polygon, maybe it could "rasterize" the polygon into a list of tiles. For people who are interested in updates to tags, a similar approach could be used, perhaps -- in that case I guess a tile would be analogous to a particular value or set of values for a tag. -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Possible GSoC project: tag/area monitoring service
> I wouldn't worry about monitoring area changes, as we have OWL[0] > (supposedly being integrated with the "Rails port"), Changepipe[1], > and possibly others that do this already. I'd suggest you consider > focusing on the idea of monitoring for changes based on tags and > object IDs. I've been interested in changes to some large relations, > and other widely dispersed objects, which isn't addressed by any of > the current tools. Integration with Rails would be great, so we can > "Watch" any object directly from the website. Of course performance > would have to be considered before implementing such a service went > live, but I don't think that's terribly important for a GSoC project. When you mention changes to large relations and widely dispersed objects, I was wondering if you had any specific use cases in mind? I'd also be interested in hearing what kind of expressions you might expect to be able to use. For example, I was thinking you could say something like "give me updates for things with the tag highway=residential and is_in=Canada". It looks like Changepipe is similar to what I'm proposing, and OWL is sort of the opposite. My idea is that clients would tell the API they're interested in hearing about something (a tag, an id, some expression involving multiple tags, an area) and then updates would be sent to them as they happen, instead of polling for what's happened in the past. This scheme would reduce the number of incoming requests at the cost of the client being responsible for receiving the information and doing something with it. I believe this approach would reduce the complexity of processing new information since it would be known in advance what updates are required. Another approach would be that clients tell the API they're interested in getting information about something and can then request an RSS feed which has recent updates. But instead of requesting an RSS feed for an arbitrary watch expression, it's this sort of bin which stuff is thrown into as the map is updated. This seems easier to translate into something you can do manually with a web browser than what's described above. I'm curious to hear which of these approaches would be useful to people interested in this sort of thing. It seems like being able to ask for RSS feeds would be more immediately useful, but having data "pushed" to clients would allow for more flexible applications. -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Possible GSoC project: tag/area monitoring service
Hi everyone, I'm writing to seek opinions about a possible Google Summer of Code project. I did GSoC in 2010, and I'd like to apply again this year. My project in 2010 was a simplified, web-based map editor. Since the wiki page for project ideas mentions that proposals for the development of existing OSM infrastructure would be preferred, I was having a look at the API v0.7 page, and noticed some interest in a monitoring feature. My proposal is to build a monitoring service to augment the existing API, similar to the Twitter streaming API [1]. Users would request to receive map updates matching tags or which involve elements in some area, and updates would be sent either over a persistent connection (as Twitter does) or possibly by making requests to an endpoint specified by the user. My general idea for the architecture is basically to grab diffs and then send the relevant parts to clients depending on what they've asked to receive. Clients of such a monitoring service could do things like send daily email updates on map activity to users interested in a specific area or tag, invalidate tiles in custom-rendered maps, or assemble a subset of available OSM data for fast, up-to-date querying within that subset (a single city, for example) without worrying about making lots of requests to the OSM API. That third application would be useful for solving one of the problems I ran into with my 2010 project -- I was optimistically loading map data with bbox queries as the user panned the map, which was too slow on the production API to be practical (and probably isn't what that part of the API is really meant for). Another project idea might be to work directly on a service which would provide fast querying on tag or area subsets. However, the project as I've proposed it above seems to me to be sort of a generalization of that, and also seems like it would require less bandwidth and disk space. Thanks for taking a look, and please let me know what you think! -- Michael Daines [1] https://dev.twitter.com/docs/streaming-api ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Announcing: Simple Map Editor (GSoC 2010)
> Ah, something we need to clear up here: there's a difference between > "simple editor/application" and "simple UI". There is no such thing as > a simple editor. It's just not possible to be both "simple" and > "working". The OSM data model is complex and sophisticated, and any > attempts at a 'simple' editor will simply mess up other peoples work. > > But even the simple "rename a pub" editor is more complex than most > people realise. Naive implementations of a point-of-interest editor > break down as soon as they realise that all POIs can also be > represented by a closed way, and that these ways - both closed and > unclosed - can be part of multipolygons. It's complicated. POI != node > and Road != way. When I first started working on the editor, I had generality in mind, and I decided that designing with a few simple use cases in mind was the best way to go. My thinking was that if these could be done reasonably well, they could be extended to include most of the node and way editing I might eventually want. Editing the names of roads was one of these simple cases. Another case was adding supplementary information to amenities. Of course, anybody who looks at even these simple cases is bound to get lost clicking around the map feature documentation on the wiki. Part of my goal is to not be a tag editor, and in fact to never to use the words "node", "way", "tag", or "relation". I want to provide an interface that would seem more or less unsurprising to a beginner interested in editing the map. It works like this: you find something, you click on it, you see that you can change some things about it, and then you save. But the map is very complicated, and you've got to have some idea of what you're doing when editing. I absolutely agree that this complexity can't simply be forgotten about. But let a simple editor have a few corners of the map! Here's what I mean: At some point during development, Graham brought up the idea of custom editors. He suggested an editor geared toward wheelchair accessibility, and pointed me to an application that would display smoking and non-smoking establishments (related to a proposed smoking ban). When deciding where grab a bite to eat late one night, the possibility occurred to me of an editor which would parse the opening_hours tag and display restaurants which were open right then and allow me to fill in this information when it was missing. While you can't say every way is a road, or every pub is represented by one node, I think there are lot of cases where this sort of thing actually is okay. (It would be more okay if the editor would be able to present ways as areas. Then a lot of parks and buildings would suddenly become editable.) Of course, it follows that there are situations where this kind of thing doesn't work, and my response to this is: the editor doesn't support that! Use Potlatch or JOSM, because they are more powerful. The question is what to support. Without a lot of editorial work, by which I mean deciding how to translate the existing map data and editorial guidelines into some ideal editor configuration, I don't think my editor is fit for general editing quite yet, but I think that is a reasonable goal. It's just not really easy to pick what things to support. (You can be as descriptive or prescriptive as you want.) Another way to apply the work I've done is by making it easier (for technical people, I guess) to create small custom editors that are good at doing a specific thing, like any of the situations I mentioned. This is possible already by forking the project and editing the configuration files. But if you could load your own configuration file into the editor somehow, then you could avoid forking the code and setting things up on your own server, etc. -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Announcing: Simple Map Editor (GSoC 2010)
> It's awesome that it uses OAuth, I tip my hat at you for getting that > working! I hope to see many people building on your work. Could you > explain more what the ruby proxy layer does? Is it just handling the > session stuff or does it do translations between OSM API and the > javascript code? Sessions are handled by keeping the OAuth access token in the user's localStorage. (I believe this is incompatible with several versions of Internet Explorer, but it has an equivalent method I can implement later.) I don't know that this is the best solution, but I think it's the closest to what a desktop application might do when dealing with OAuth, and it allows me to not have to keep state in the proxy, so no database. The proxy layer is split into two parts: - A proxy for doing bbox queries on the map, which requires no authentication. I use the rack-streaming-proxy gem for this. - A proxy for OAuth and the API methods requiring authentication. This is in lib/osm_auth.rb. The /auth/request and /auth/access URLs handle getting request and access tokens, and the /proxy/* handlers use the OAuth gem to make requests to the API. I send the access token and secret from the JS part using special headers. > I'll also put out a plea to any Ruby/OAuth experts that we're still > looking for someone to tackle > http://wiki.openstreetmap.org/wiki/OAuth/10a I believe there's also some newer version of OAuth which specifically takes into account browser applications (although the last time I checked, that stuff wasn't really "standardized"). Another thing that would be nice would be being able to tell the API I want to revoke tokens. Right now, if the user authorizes and deauthorizes several times, old tokens can pile up in their OAuth settings. I "log out" by simply clearing the token from localStorage. -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Announcing: Simple Map Editor (GSoC 2010)
> osmdoc.com gives 12092 uses for "phone=" and 891 uses for "addr:phone=" so > I'd change what's in the wiki and use "phone=" and "fax=" and "website=". > > For any Reasons "email=" is not as popular as "addr:email=". This is the reason I have prioritized "phone" in the telephone field. However, I should mention that the form is smart enough to see other possibilities for the same tag, so if you edit something that uses "contact:phone", the editor will modify the "contact:phone" tag instead of adding its own "phone" tag. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Announcing: Simple Map Editor (GSoC 2010)
> - I can only add predefined tags - I think there should be a more common way > to add tags. The best way is to simplify adding relevant tags by using data > mining (Which tags are often used together with the tags already defined?) to > propose new tags instead of manually predefining tags! Or do you have > implemented it already this way but the test data set lacks some data? > Anyway, I (not a beginner) want to be able to add any tag I want. The idea is that as a beginner you wouldn't yet be sure what tags add anyway. (Or wouldn't even yet care how OSM uses "tags" or "nodes", etc.) This is more of a design decision than a technical limitation. > - I want to be able to zoom out to see the whole geometry. Right now the editor loads map data through the standard /map bbox query. Loading a lot of map data when zoomed out would be slow, (it is already slow on the production API) but perhaps in the future, XAPI could be used to only load the kinds of things the editor can actually do things with? > - I agree with Sebastian Klein: Do not force the user to cancel or save, but > save the whole as one changeset. How do you see this working? Just opening a changeset at the beginning of the editing session and letting it close automatically? Or would it be reflected in the interface? I want to avoid presenting the concept of the "changeset" and also make it clear that some action is taking place when editing. I don't want the user's edits to be silently transmitted to the servers without them realizing it. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Announcing: Simple Map Editor (GSoC 2010)
Sebastian, > Let me comment on the user interface: > > * It would be good to cancel the current operation if you click on the map or > another object. This is a possibility, however I was worried about lost data -- if the user clicks on the map or something else unintentionally, what is done about the element they were currently editing? > * If you click an object far right, it is hidden by the dialog, which is kind > of strange. I've put in a fix for this, but it probably needs to be smarter -- now the size of the map is adjusted when the editing pane appears, and if the element is out of view, the map pans to that element. Instead, the map should move as little as possible. One alternative might be to use popups. It's possible for me to make this change, but I went with just a simple "editing pane" for the time being to make things easier for me. > * A highlighting on mouse-over would be very fancy. I can try this -- I assumed it would be too expensive for hovering, though. (For hit-testing I use the distance of the click from the node or way's geometry to allow for inaccurate clicks, but maybe OpenLayers already has an optimized routine for what I'm doing and I didn't notice?) > * Allow to click POIs that are part of a way. This is already possible -- it's just that these nodes currently aren't recognized as "visible" by the editor's configuration. > Another thing I noticed: It creates a new changeset for every operation. It > should remember the changeset and add subsequent requests to it. Good point -- I see reviewing the API docs that changesets close automatically. I did things this way because I thought it was "cleaner", and because I didn't want the user to have to decide when to close a changeset. > I recommend you have a look at the JOSM presets. They are quite comprehensive > have been actively maintained for a long time. I'll have a look. My original intent was to have the editor interface be "curated" -- this is why the editing forms are all in HTML. I didn't quite get as far as writing help for each tag, though, as in this mockup: http://www.mdaines.com/simple-map-editor/add-category.html -- Michael ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Announcing: Simple Map Editor (GSoC 2010)
Hello OSM, I'd like to announce my GSoC 2010 project which is nearing completion, a simplified editor for OSM aimed at beginner users. It's also meant to be generally useful for quick edits to the map, and to be customizable to allow creation of "site-specific" editors. Some of the use cases I considered were adding names to roads and entering contact information for businesses and amenities, and developing a custom editor for keeping track of restaurants that are open late. The project consists of: - a proxy layer in Ruby, for communicating with the OSM API, - a collection of JavaScript classes and routines for connecting to the OSM API, working with map data, and authentication with OAuth, which can hopefully be used in other OpenLayers-based project that require map editing functionality, - an HTML/JavaScript application capable of modifying tags or nodes and ways and creating new nodes, using the above components. The application is available to try out here: http://simple-map-editor.heroku.com/ Please note: - I haven't done much testing in Internet Explorer. Please use Firefox, Chrome, or Safari for now. (I will deal with browser compatibility issues soon.) - Right now, the app connects to the development API, so it may look like there is no data anywhere. You can add nodes yourself, or search for "South Bank, London", where I happened to find some test data at the beginning of the project. - Also, you'll need an account on the dev app server. If you don't have one, you'll be prompted to create one when the app tries to authorize. - The app only knows about a few kinds of map features. More will be added. Source is available here: http://github.com/mdaines/simple-map-editor Questions, comments, and bug reports are much appreciated. Also, I plan to update the wiki page with more detailed and current information soon. I believe the editor can be production-ready with a bit more testing and polish. Let me know what you think! Thanks, -- Michael Daines ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
