Re: [OSM-dev] Fixing the history tab
Paweł Paprota wrote: Currently I am on a two-month work trip in Germany with not much free time but I really miss OSM and OWL development so I plan to get back to it some time in June when I'm back at home. \o/ When you look at it, there is really not a lot of stuff to be done before OWL and New History Tab can be rolled out to production. Mainly there are UI tweaks to be done so it looks nicer - that's why I'm a bit sad that no one volunteered to help out with JS/CSS - it's the easy part of the project... You're not by any chance coming to SOTM-US, are you? It would be great to attack this on the Monday hackday. (If not, anyone feel like sponsoring Paweł?) cheers Richard -- View this message in context: http://gis.19327.n5.nabble.com/Fixing-the-history-tab-tp5759340p5759387.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
Pawel, i do not understand how the New History Tab will scale and how it will be fast. Right now it's slow. The old owl fell over b/c of scale problems and my understanding is that the underlying technical challenge of how to make massive changesets browsable in rev chronological order on a map is unsolved. I must be missing something, I don't think you have explained this anywhere. I'd realy like to understand, where can I read up? On Friday, May 3, 2013, Paweł Paprota wrote: IMHO people really should put their minds to fixing *the* problem of Querying all changesets that actually modified data in an arbitrary bounding box of the world and displaying them in reverse chronological order is computationally expensive instead of coming up with yet another half way solution. It looks nice as such tools often do but does not solve the problem properly - why not use the time to help out with OWL / New History Tab instead of starting another project from scratch? Anyway, let me use this opportunity to give people a quick update about OWL and the History Tab Beta. As is painfully obvious from my Contributions graphic on Github[1], I've not been active at all in OSM since late February. Coincidentally, I started at my new job at that time... Currently I am on a two-month work trip in Germany with not much free time but I really miss OSM and OWL development so I plan to get back to it some time in June when I'm back at home. When you look at it, there is really not a lot of stuff to be done before OWL and New History Tab can be rolled out to production. Mainly there are UI tweaks to be done so it looks nicer - that's why I'm a bit sad that no one volunteered to help out with JS/CSS - it's the easy part of the project... On the backend side (OWL), there is support for relation to be added which is a gigantic challenge but I think this could be scheduled for 2.0... In any case, please consider helping out with the project, especially the frontend side, as it is not that far from being in potentially-mergable-state. [1] https://github.com/ppawel Paweł On 05/02/2013 05:18 PM, Alex Barth wrote: At this weekend's Chicago hack weekend Tom and I worked on a prototype that could be a viable solution for our currently broken history tab. It is taking a very different approach in comparison to Pawel's history tab [1] by not showing the entire history up front, but only latest changes to visible elements. I wrote up the details in a diary entry, would love to hear peoples thoughts on this. I think from a user story perspective this would work and it would be much cheaper to implement than a fast historic changeset browser. http://www.openstreetmap.org/**user/lxbarth/diary/19185http://www.openstreetmap.org/user/lxbarth/diary/19185 Alex [1] http://lists.openstreetmap.**org/pipermail/talk/2013-** January/065556.htmlhttp://lists.openstreetmap.org/pipermail/talk/2013-January/065556.html __**_ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.**org/listinfo/devhttp://lists.openstreetmap.org/listinfo/dev __**_ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.**org/listinfo/devhttp://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
What exactly do you mean by slow? OWL API response is slow or client-side Javascript is slowing down the browser (likely because of the number of GeoJSON features displayed at once)? The OWL API performance issues is indeed the most challenging problem to solve here. There are couple of things: 1. Database size - as the number of changesets grows, the number of vector tiles grows as well. So far there is several millions of changesets in the database and scale factor is not too bad actually - for high zoom levels it is performing pretty well. For low zoom levels and especially on frequently modified areas there need to be optimizations. All in all, I'm not concerned with the amount of data - even the basic solution as it is right now (using btree index in Postgres) is acceptable for most use cases - or at least it is not blocking for the production deployment. 2. Selecting last X changesets on low zoom levels where there is a lot of changesets present - this is indeed a very big problem because btree index helps only to narrow down the geographical aspect but the temporal aspect cannot be indexed and there is a sequential sort/scan - this is slow when you want to display last 20 changesets at zoom level 8 for London for instance. There is a couple of things here to research/try: * PostgreSQL now supports k nearest neighbors indexing[1][2] - the distance function could be defined * Matt suggested looking into buffer trees - haven't spent too much time on it myself yet but it looks interesting and tackles exactly the problem in OWL. Of course there may be other ways to optimize this and it is exactly my point - that people could try to solve this problem - I don't have a ready answer for it because it *is* a hard problem. Once solved, there would be a great new (and performant) way to look at OWL data. There are more things in OWL API that I introduced before I went inactive - now there are generic vector tiles, way subversions so basically you can already build on top of that and create Kothic browser for OSM data. It is definitely not easy and more work is needed on the OWL API part to serve proper Kothic tiles and also only the data that makes sense for given zoom levels etc. but it is all doable - all it needs is some work = time. [1] http://wiki.postgresql.org/wiki/What%27s_new_in_PostgreSQL_9.1#K-Nearest-Neighbor_Indexing [2] excoventures.com/talks/knn.pdf Paweł On Fri, May 3, 2013, at 14:21, Alex Barth wrote: Pawel, i do not understand how the New History Tab will scale and how it will be fast. Right now it's slow. The old owl fell over b/c of scale problems and my understanding is that the underlying technical challenge of how to make massive changesets browsable in rev chronological order on a map is unsolved. I must be missing something, I don't think you have explained this anywhere. I'd realy like to understand, where can I read up? On Friday, May 3, 2013, Paweł Paprota wrote: IMHO people really should put their minds to fixing *the* problem of Querying all changesets that actually modified data in an arbitrary bounding box of the world and displaying them in reverse chronological order is computationally expensive instead of coming up with yet another half way solution. It looks nice as such tools often do but does not solve the problem properly - why not use the time to help out with OWL / New History Tab instead of starting another project from scratch? Anyway, let me use this opportunity to give people a quick update about OWL and the History Tab Beta. As is painfully obvious from my Contributions graphic on Github[1], I've not been active at all in OSM since late February. Coincidentally, I started at my new job at that time... Currently I am on a two-month work trip in Germany with not much free time but I really miss OSM and OWL development so I plan to get back to it some time in June when I'm back at home. When you look at it, there is really not a lot of stuff to be done before OWL and New History Tab can be rolled out to production. Mainly there are UI tweaks to be done so it looks nicer - that's why I'm a bit sad that no one volunteered to help out with JS/CSS - it's the easy part of the project... On the backend side (OWL), there is support for relation to be added which is a gigantic challenge but I think this could be scheduled for 2.0... In any case, please consider helping out with the project, especially the frontend side, as it is not that far from being in potentially-mergable-state. [1] https://github.com/ppawel Paweł On 05/02/2013 05:18 PM, Alex Barth wrote: At this weekend's Chicago hack weekend Tom and I worked on a prototype that could be a viable solution for our currently broken history tab. It is taking a very different approach in comparison to Pawel's history tab [1] by not showing the entire history up front, but only latest changes to visible
Re: [OSM-dev] Fixing the history tab
Sorry, I did not finish one of the sentences properly - about knn in Postgres: * PostgreSQL now supports k nearest neighbors indexing[1][2] - the distance function/operator could be defined for OSM changesets so that a query for given tile (or tile range) uses the knn index and at the same time returns only k nearest neighbors (where nearest would be latest). Also - to add to my previous comments about the solution - I am 100% sure that there are people who know how to solve this kind of problem, even using the tools we have (Postgres) - so part of the task is reaching out to them and consulting about the solution. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
On 05/03/2013 02:21 PM, Alex Barth wrote: Pawel, i do not understand how the New History Tab will scale and how it will be fast. Right now it's slow. Since the solution you made doesn't scale either. (Try browsing any city/town in The Netherlands.) How is it an improvement? -- --- m.v.g., Cartinus ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
Since the solution you made doesn't scale either. (Try browsing any city/town in The Netherlands.) How is it an improvement? Well while http://osmlab.github.io/latest-changes is not fast right now there's a clear and scalable path to make it fast. Speeding up /latest-changes is much simpler as you'll never have to include history. Essentially what we'd need is a tiled data rendition of the latest status of OSM data. No history. Clearly, creating a tiled data source is probably not just a weekend project either but it's conceptually a solved problem in OSM. On Fri, May 3, 2013 at 10:05 AM, Cartinus carti...@xs4all.nl wrote: On 05/03/2013 02:21 PM, Alex Barth wrote: Pawel, i do not understand how the New History Tab will scale and how it will be fast. Right now it's slow. Since the solution you made doesn't scale either. (Try browsing any city/town in The Netherlands.) How is it an improvement? -- --- m.v.g., Cartinus ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
Btw, I forgot to mention that we've got a very similar browsing interface on OSM.org - we just don't break it down by changesets http://cl.ly/image/09421k3A0G0b On Thu, May 2, 2013 at 11:18 AM, Alex Barth a...@mapbox.com wrote: At this weekend's Chicago hack weekend Tom and I worked on a prototype that could be a viable solution for our currently broken history tab. It is taking a very different approach in comparison to Pawel's history tab [1] by not showing the entire history up front, but only latest changes to visible elements. I wrote up the details in a diary entry, would love to hear peoples thoughts on this. I think from a user story perspective this would work and it would be much cheaper to implement than a fast historic changeset browser. http://www.openstreetmap.org/user/lxbarth/diary/19185 Alex [1] http://lists.openstreetmap.org/pipermail/talk/2013-January/065556.html ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
On Thu, May 2, 2013 at 11:18 PM, Alex Barth a...@mapbox.com wrote: At this weekend's Chicago hack weekend Tom and I worked on a prototype that could be a viable solution for our currently broken history tab. It is taking a very different approach in comparison to Pawel's history tab [1] by not showing the entire history up front, but only latest changes to visible elements. I wrote up the details in a diary entry, would love to hear peoples thoughts on this. I think from a user story perspective this would work and it would be much cheaper to implement than a fast historic changeset browser. The biggest limitation of this solution is that it will never show changesets for objects that were deleted in the bounding box. If the changeset happens to include still-existing objects in the bounding box, then the changeset will be listed. But if a changeset only contains deletions (most likely a vandal, but legitimately can be someone just removing a few outdated POIs), then there's no way that this solution can help you find out what happened. (I swear I there was a restaurant here, where did it go?) At least the current system will show you the deletion changeset. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
On Fri, May 3, 2013 at 1:03 PM, Eugene Alvin Villar sea...@gmail.comwrote: On Thu, May 2, 2013 at 11:18 PM, Alex Barth a...@mapbox.com wrote: At this weekend's Chicago hack weekend Tom and I worked on a prototype that could be a viable solution for our currently broken history tab. It is taking a very different approach in comparison to Pawel's history tab [1] by not showing the entire history up front, but only latest changes to visible elements. I wrote up the details in a diary entry, would love to hear peoples thoughts on this. I think from a user story perspective this would work and it would be much cheaper to implement than a fast historic changeset browser. The biggest limitation of this solution is that it will never show changesets for objects that were deleted in the bounding box. If the changeset happens to include still-existing objects in the bounding box, then the changeset will be listed. But if a changeset only contains deletions (most likely a vandal, but legitimately can be someone just removing a few outdated POIs), then there's no way that this solution can help you find out what happened. (I swear I there was a restaurant here, where did it go?) At least the current system will show you the deletion changeset. ... after paging through dozens of big changesets. This is by no means perfect, but at least it gives you changeset details for the area you're looking at, not the changesets that overlap your area. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
On May 3, 2013, at 11:23 AM, Ian Dees wrote: On Fri, May 3, 2013 at 1:03 PM, Eugene Alvin Villar sea...@gmail.com wrote: On Thu, May 2, 2013 at 11:18 PM, Alex Barth a...@mapbox.com wrote: At this weekend's Chicago hack weekend Tom and I worked on a prototype that could be a viable solution for our currently broken history tab. It is taking a very different approach in comparison to Pawel's history tab [1] by not showing the entire history up front, but only latest changes to visible elements. I wrote up the details in a diary entry, would love to hear peoples thoughts on this. I think from a user story perspective this would work and it would be much cheaper to implement than a fast historic changeset browser. The biggest limitation of this solution is that it will never show changesets for objects that were deleted in the bounding box. If the changeset happens to include still-existing objects in the bounding box, then the changeset will be listed. But if a changeset only contains deletions (most likely a vandal, but legitimately can be someone just removing a few outdated POIs), then there's no way that this solution can help you find out what happened. (I swear I there was a restaurant here, where did it go?) At least the current system will show you the deletion changeset. ... after paging through dozens of big changesets. This is by no means perfect, but at least it gives you changeset details for the area you're looking at, not the changesets that overlap your area. Would it be silly to suggest that changesets get their own geometries in PostGIS and an associated spatial index, consisting of every way and node deleted, moved, etc.? -mike. michal migurski- contact info and pgp key: sf/cahttp://mike.teczno.com/contact.html ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
On Sat, May 4, 2013 at 2:27 AM, Michal Migurski m...@teczno.com wrote: Would it be silly to suggest that changesets get their own geometries in PostGIS and an associated spatial index, consisting of every way and node deleted, moved, etc.? Isn't this similar to what Paweł's New History Tab is doing behind the scenes? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
On Sat, May 4, 2013 at 2:23 AM, Ian Dees ian.d...@gmail.com wrote: On Fri, May 3, 2013 at 1:03 PM, Eugene Alvin Villar sea...@gmail.comwrote: At least the current system will show you the deletion changeset. ... after paging through dozens of big changesets. This is by no means perfect, but at least it gives you changeset details for the area you're looking at, not the changesets that overlap your area. I personally have not found the big changesets that much of a bother. But I think that is entirely due to the fact that I monitor changes in my patch in Southeast Asia which lies at the periphery of the world coordinate system. So big changesets don't often intersect my area of interest and I get a higher signal-to-noise ratio. I think I can imagine how useless the History tab would be for people in central locations like Europe. But for my needs in monitoring changes in my area, the History tab is currently acceptable. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
On May 3, 2013, at 12:22 PM, Eugene Alvin Villar wrote: On Sat, May 4, 2013 at 2:27 AM, Michal Migurski m...@teczno.com wrote: Would it be silly to suggest that changesets get their own geometries in PostGIS and an associated spatial index, consisting of every way and node deleted, moved, etc.? Isn't this similar to what Paweł's New History Tab is doing behind the scenes? Maybe—where would I look to find out? https://github.com/ppawel/openstreetmap-website is obviously the starting point, but where then? -mike. michal migurski- contact info and pgp key: sf/cahttp://mike.teczno.com/contact.html ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
On Fri, May 3, 2013 at 2:31 PM, Michal Migurski m...@teczno.com wrote: On May 3, 2013, at 12:22 PM, Eugene Alvin Villar wrote: On Sat, May 4, 2013 at 2:27 AM, Michal Migurski m...@teczno.com wrote: Would it be silly to suggest that changesets get their own geometries in PostGIS and an associated spatial index, consisting of every way and node deleted, moved, etc.? Isn't this similar to what Paweł's New History Tab is doing behind the scenes? Maybe—where would I look to find out? https://github.com/ppawel/openstreetmap-website is obviously the starting point, but where then? http://owl.apis.dev.openstreetmap.org/ then click the history tab. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
Hi, On 03.05.2013 20:27, Michal Migurski wrote: Would it be silly to suggest that changesets get their own geometries in PostGIS and an associated spatial index, consisting of every way and node deleted, moved, etc.? The current OSM API doesn't even use PostGIS and therefore no spatial index either.[*] Of course if you set up a separate history browsing database then that could make sense. Bye Frederik [*] A selfmade quad-tile implementation maps two coordinate dimensions into one integer for plain one-dimensional indexing. -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
Actually, my fork of the openstreetmap-website repository only contains the UI part of the new history tab. Heavy lifting (including creating and indexing geometry) is done by OWL which is here: https://github.com/ppawel/openstreetmap-watch-list Paweł On Fri, May 3, 2013, at 22:23, Eugene Alvin Villar wrote: On Sat, May 4, 2013 at 3:31 AM, Michal Migurski m...@teczno.com wrote: On May 3, 2013, at 12:22 PM, Eugene Alvin Villar wrote: On Sat, May 4, 2013 at 2:27 AM, Michal Migurski m...@teczno.com wrote: Would it be silly to suggest that changesets get their own geometries in PostGIS and an associated spatial index, consisting of every way and node deleted, moved, etc.? Isn't this similar to what Paweł's New History Tab is doing behind the scenes? Maybe—where would I look to find out? https://github.com/ppawel/openstreetmap-website is obviously the starting point, but where then? The source code is definitely the place where one should look. I think this is the more direct location: https://github.com/ppawel/openstreetmap-website/tree/owl-history-tab Anyway, basing on Paweł's numerous previous emails on the subject since January, I gather that his code processes each changeset and creates vector tiles which record the geometry changes. Vector tiles are obviously spatially indexable. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Fixing the history tab
At this weekend's Chicago hack weekend Tom and I worked on a prototype that could be a viable solution for our currently broken history tab. It is taking a very different approach in comparison to Pawel's history tab [1] by not showing the entire history up front, but only latest changes to visible elements. I wrote up the details in a diary entry, would love to hear peoples thoughts on this. I think from a user story perspective this would work and it would be much cheaper to implement than a fast historic changeset browser. http://www.openstreetmap.org/user/lxbarth/diary/19185 Alex [1] http://lists.openstreetmap.org/pipermail/talk/2013-January/065556.html ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fixing the history tab
IMHO people really should put their minds to fixing *the* problem of Querying all changesets that actually modified data in an arbitrary bounding box of the world and displaying them in reverse chronological order is computationally expensive instead of coming up with yet another half way solution. It looks nice as such tools often do but does not solve the problem properly - why not use the time to help out with OWL / New History Tab instead of starting another project from scratch? Anyway, let me use this opportunity to give people a quick update about OWL and the History Tab Beta. As is painfully obvious from my Contributions graphic on Github[1], I've not been active at all in OSM since late February. Coincidentally, I started at my new job at that time... Currently I am on a two-month work trip in Germany with not much free time but I really miss OSM and OWL development so I plan to get back to it some time in June when I'm back at home. When you look at it, there is really not a lot of stuff to be done before OWL and New History Tab can be rolled out to production. Mainly there are UI tweaks to be done so it looks nicer - that's why I'm a bit sad that no one volunteered to help out with JS/CSS - it's the easy part of the project... On the backend side (OWL), there is support for relation to be added which is a gigantic challenge but I think this could be scheduled for 2.0... In any case, please consider helping out with the project, especially the frontend side, as it is not that far from being in potentially-mergable-state. [1] https://github.com/ppawel Paweł On 05/02/2013 05:18 PM, Alex Barth wrote: At this weekend's Chicago hack weekend Tom and I worked on a prototype that could be a viable solution for our currently broken history tab. It is taking a very different approach in comparison to Pawel's history tab [1] by not showing the entire history up front, but only latest changes to visible elements. I wrote up the details in a diary entry, would love to hear peoples thoughts on this. I think from a user story perspective this would work and it would be much cheaper to implement than a fast historic changeset browser. http://www.openstreetmap.org/user/lxbarth/diary/19185 Alex [1] http://lists.openstreetmap.org/pipermail/talk/2013-January/065556.html ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
