Re: [osmosis-dev] Reading OSM History dumps
On Wed, Aug 25, 2010 at 11:14 PM, Peter Körner osm-li...@mazdermind.dewrote: Brett, the pgsql tasks currently write (in COPY mode) all data to temp files first. The process seems to be PlanetFile - NodeStoreTempFile - CopyFormatTempFile - PgsqlCopyImport in osm2pgsql the copy data is pushed to pgsql via unix pipes (5 or 6 COPY transactions running at the same time in different connections). This approach skips the CopyFormatTempFile stage. Is there any special reason this approach isn't used in the pgsnapshot package? Not too sure now :-) I think it was the simplest way to share code between both the --write-pgsql-dump task and what was then the --fast-write-pgsql (now simply --write-pgsql) task. In practice the COPY file creation and loading is fairly fast. The biggest downside is the extra disk space. The slowest parts of the whole process are the way geometry creation, index building, and the CLUSTER statements (in the newest schema). On relatively low-end hardware it takes many days to import an entire planet, only a small part of which is the COPY processing. In most cases I create the COPY files using --write-pgsql-dump and load them via the provided load script so that I can better monitor progress and resume if processing is interrupted. In short it just hasn't been a high priority to change it. While I'm on the topic, I've mostly completed the changes to the schema now. Performance is drastically improved over the old version for bounding box query processing. The --read-pgsql --dataset-bounding-box task combination would previously take approximately an hour to retrieve a 1x1 degree box in a populated area, now it is down to around 5 minutes due to far better disk layout. The biggest downside is that the table clustering takes a long time during initial database creation. ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Reading OSM History dumps
Am 25.08.2010 15:16, schrieb Marco Lechner - FOSSGIS e.V.: Hi Peter, I'm very intersted in your history-extension and I'm going to test as soon as a first snapshot is available. Will it be possible to eat an --bound-polygon stream from osmosis? Or will it just import the whole history-plane? You will be able to add a bbox or bound-polygon task before pushing things into the database. But without having special tasks to handle bonding boxes in regard to history dumps, you will get problems with nodes moving in- and out of your bounding box. The plugin will, for the time being, also not be able to handle change streams, so it will not be possible to keep the database updated. This is still work in progress in its earliest stage, so please don't expect it solving any real problems. Peter ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Reading OSM History dumps
Am 25.08.2010 15:26, schrieb Brett Henderson: In short it just hasn't been a high priority to change it. I was planning to share on FileInputStream/FileOutputStream level. You can feed a FileInputStream into the CopyManager as well as into a file, can't you? Maybe want to can copy the relevant bits later to pgsnapshot. Peter ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Reading OSM History dumps
Hi Marco The first snapshot is out. Unfortunately the hstore migration progress Brett is still in let the pgsnapshot tests fail, which is why hudson is not providing nightly builds anymore. Because of that you'll need to compile osmosis yourself. I attached instructions to this mail that also include the concrete plugin usage. The following tasks are available:--write-pgsql-history and --write-pgsql-history-dump. They correlate closely to --write-pgsql and --write-pgsql-dump. All features that are marked as experimental may work or they may not and of course they will be painfully memory intensive on larger datasets because of the lack of a good store implementation. Peter Am 25.08.2010 15:16, schrieb Marco Lechner - FOSSGIS e.V.: Hi Peter, I'm very intersted in your history-extension and I'm going to test as soon as a first snapshot is available. Will it be possible to eat an --bound-polygon stream from osmosis? Or will it just import the whole history-plane? Marco Am 25.08.2010 15:14, schrieb Peter Körner: Hi all After a little playing around I now got an idea of how I'm going to implement everything. I'll keep as close as possible at the regular simple schema and at the way the pgsql tasks work. Just as with the optional linestring/bbox builder, the history import tasks will serve more then one scheme. I'm leaving relations out, again. the regular simple scheme - its the basis of all but not capable of holding history data + history columns - create and populate an extra column in way_nodes to store the way version. - change the PKs of way_nodes to allow more then one version of an element + way_nodes version builder - create and populate an extra column in way_nodes that holds the node version that corresponds to the way's timestamp + minor version builder - create and populate an extra column in ways and way_nodes to store the ways minor versions, which are generated by changes to the nodes of the way between version changes of the way self. + from-to-timestamp builder - create and populate an extra column in the nodes and ways table that specifies the date until which a version of an item was the current one. After that time, the next version of the same item was current (or the item was deleted). the tstamp field in contrast contains the starting date from which an item was current. + linestring / bbox builder - just the same as with the regular simple scheme, works for all version and minor-version rows Until the end of the week I'll get a pre snapshot out that can populate the history table with version columns and changed PKs. The database created from this can be used to test Scotts SQL-Only solution [1]. It will also contain a first implementation of the way_nodes version builder but only with an example implementation of the NodeStore, that performs bad on bigger files. Brett, the pgsql tasks currently write (in COPY mode) all data to temp files first. The process seems to be PlanetFile - NodeStoreTempFile - CopyFormatTempFile - PgsqlCopyImport in osm2pgsql the copy data is pushed to pgsql via unix pipes (5 or 6 COPY transactions running at the same time in different connections). This approach skips the CopyFormatTempFile stage. Is there any special reason this approach isn't used in the pgsnapshot package? Peter [1] http://lists.openstreetmap.org/pipermail/dev/2010-August/020308.html ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev # download osmosis svn export http://svn.openstreetmap.org/applications/utils/osmosis/trunk/ osmosis-trunk # enter the source directory cd osmosis-trunk # download the history plugin svn export http://svn.toolserver.org/svnroot/mazder/osmhist/osmosis-plugin/history/ # enable the history plugin patch -p0 history/script/source-activation.patch # compile ant clean publish # reate a postgis user, if not already done sudo -u postgres createuser osmosis # create an empty database with hstore and postgis capabilities, if not already done sudo -u postgres createdb -E UTF8 -O osmosis osmosis-history sudo -u postgres createlang plpgsql osmosis-history # create the simple schema database psql -U osmosis osmosis-history package/script/pgsql_simple_schema_0.6.sql # add the history extension to the database psql -U osmosis osmosis-history history/script/pgsql_simple_schema_0.6_history.sql # the following lines add extra features to the database # execute before them before the import # they are experimental and very memory intensive # use only with small data sets # enable the node version builder #psql -U osmosis osmosis-history history/script/pgsql_simple_schema_0.6_history_way_nodes_version.sql # enable
Re: [osmosis-dev] Reading OSM History dumps
On Wed, Aug 25, 2010 at 11:33 PM, Peter Körner osm-li...@mazdermind.dewrote: Am 25.08.2010 15:26, schrieb Brett Henderson: In short it just hasn't been a high priority to change it. I was planning to share on FileInputStream/FileOutputStream level. You can feed a FileInputStream into the CopyManager as well as into a file, can't you? Sorry, I'm not sure what you mean. I think the only way to feed data into the CopyManager is via an InputStream. That InputStream can be a FileInputStream or a piped input stream or whatever you wish. But there are also classes like PGCopyOutputStream so perhaps you can use those directly to avoid using multiple threads. It's been a while since I looked at it. Maybe want to can copy the relevant bits later to pgsnapshot. Yep, sure. ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Reading OSM History dumps
On Thu, Aug 26, 2010 at 8:19 AM, Peter Körner osm-li...@mazdermind.dewrote: Hi Marco The first snapshot is out. Unfortunately the hstore migration progress Brett is still in let the pgsnapshot tests fail, which is why hudson is not providing nightly builds anymore. I hope to have this fixed over the next few days. I'm working with the server admins to get hstore support added to the database. Because of that you'll need to compile osmosis yourself. I attached instructions to this mail that also include the concrete plugin usage. If you wish to avoid compiling yourself you can also get nightly builds from: http://bretth.dev.openstreetmap.org/osmosis-build/ The 0.37.SNAPSHOT version in the above location is built via a cron job. No tests are run during the build. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [OSM-dev] new contributor terms ID list
John, John Smith wrote: That's debatable, I'd say anything with more than 0.5% contribution is most likely to be a bot or an import and should be excluded for the purposes of visualising support. The raw data can be interpreted in many ways. What we have seen is one interpretation. Some may be interested in how many *humans* have already agreed because they feel that the community is more important than the data. Others may only be looking at the *data* because the thing they fear most is that their local area of interest is adversely affected, no matter if the data came from imports or humans. A possible problem for those who are interested only in humans is that we don't have a bot flag in the database so what's a bot and what not is anybody's guess. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] new contributor terms ID list
On 25 August 2010 17:59, Frederik Ramm frede...@remote.org wrote: Some may be interested in how many *humans* have already agreed because they feel that the community is more important than the data. Others may only be looking at the *data* because the thing they fear most is that their local area of interest is adversely affected, no matter if the data came from imports or humans. In both cases a bot cleaning up tiger tags shouldn't be given any weight, it didn't contribute anything other than reducing excessive amounts of useless tags that shouldn't have been imported in the first place. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Mapping during license change / safe to edit
Hi, understandably, the license change process creates some unease amongst mappers. Most of them are pragmatic - they have agreed to the change, and now they want to continue what they do best, which is mapping stuff. They know that mapping new stuff (without reference to incompatible sources) is safe, and anything they create will definitely be in OSM to stay. But some of them are wary of modifying existing objects. How can they know that they are not building on sand - might the very object they are about to edit one day have to be removed because its author has not agreed to the license change? How could we make life easier for them? What technological measures could realistically be employed to determine what is safe to edit and how could this be signalled to the mapper? Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapping during license change / safe to edit
On Wed, Aug 25, 2010 at 10:08:18AM +0200, Frederik Ramm wrote: Hi, How could we make life easier for them? What technological measures could realistically be employed to determine what is safe to edit and how could this be signalled to the mapper? Get the Authors from the history of the node or way and match them against a list of users who have signed for the new license already? display like the validator layer in josm -- Celso González (PerroVerd) http://mitago.net ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapping during license change / safe to edit
Am 25.08.2010 10:08, schrieb Frederik Ramm: Hi Thank you for stepping forward. How could we make life easier for them? What technological measures could realistically be employed to determine what is safe to edit and how could this be signalled to the mapper? As a first base to build tools to do this calculation upon, I'm writing an osmosis plugin that is able to read the history dump into a postgres db in osmosis simple schema. There is still a lot of work to be done, but I may offer a snapshot until the end of this week. With the history dump in a database and (soon arriving) list of contributors, we can start creating reports in numeric or graphic form. I'll announce the availability of the snapshot on osm-dev and osmosis-dev. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapping during license change / safe to edit
Probably not realistic, but an idea nevertheless a new Web map layer showing all nodes, ways (relations?) in simple wireframe mode. Those that are not safe shown as red and those that are in black. But I'm a bit skeptical whether this all would do any good. I'd say go ahead and map. And save your local copies of OSM files for the stuff you edited, just in case. Igor On Wed, Aug 25, 2010 at 10:08 AM, Frederik Ramm frede...@remote.org wrote: Hi, understandably, the license change process creates some unease amongst mappers. Most of them are pragmatic - they have agreed to the change, and now they want to continue what they do best, which is mapping stuff. They know that mapping new stuff (without reference to incompatible sources) is safe, and anything they create will definitely be in OSM to stay. But some of them are wary of modifying existing objects. How can they know that they are not building on sand - might the very object they are about to edit one day have to be removed because its author has not agreed to the license change? How could we make life easier for them? What technological measures could realistically be employed to determine what is safe to edit and how could this be signalled to the mapper? Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapping during license change / safe to edit
2010/8/25 Celso González ce...@mitago.net Get the Authors from the history of the node or way and match them against a list of users who have signed for the new license already? display like the validator layer in josm If we want to be accurate, it may be more complex. A node can keep the same ID while set to a new (hopefully more accurate) set of coordinates. The same applies to tags linked to nodes, ways and relations. -- Christian ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapping during license change / safe to edit
At 2010-08-25 01:08, Frederik Ramm wrote: But some of them are wary of modifying existing objects. How can they know that they are not building on sand - might the very object they are about to edit one day have to be removed because its author has not agreed to the license change? To understand what needs to be done, I ask do we know what author means, exactly? Is it the original creator of a feature? What if others have edited it afterwards, and there is a mix of license acceptance among these users? What about nodes and ways with mixed license status? Is this stuff described (in a few paragraphs at most) somewhere? There's no way I'm reading through the license debate. I'm sure I'm not alone. (These are literal questions for which I do not know the answers - I'm implying nothing at all) -- Alan Mintz alan_mintz+...@earthlink.net ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapping during license change / safe to edit
At 2010-08-25 01:27, Igor Brejc wrote: But I'm a bit skeptical whether this all would do any good. I'd say go ahead and map. And save your local copies of OSM files for the stuff you edited, just in case. Is this going to be necessary, or will there be a point at which the license change occurs, with the state as of that change available for some period of time? It's impractical for me to manage local files with my edits, and I don't want to burden the servers unnecessarily by running an automated daily download of the (wide) area in which I'm editing. -- Alan Mintz alan_mintz+...@earthlink.net ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Wishlist API/db: bot-flag WAS new contributor terms ID list
2010/8/25 Frederik Ramm frede...@remote.org: A possible problem for those who are interested only in humans is that we don't have a bot flag in the database so what's a bot and what not is anybody's guess. Wouldn't that be generally a nice to have for the future? If you see an edit that doesn't seem right and know that it was executed by a bot you can exclude local knowledge when judging. It would also be nice to see the changeset-list with the option bots excluded to see who is really active in a certain area. cheers, Martin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] new contributor terms ID list
On 25 August 2010 04:33, Anthony o...@inbox.org wrote: On Tue, Aug 24, 2010 at 3:28 PM, Grant Slater openstreet...@firefishy.com wrote: On 24 August 2010 20:17, Stephan Knauss o...@stephans-server.de wrote: Is it possible to provide a list with userIDs that have agreed to the new terms? Yes this is planned to be made available shortly. It will be updated on a regular schedule. Last week's LWG minutes also have a nice graphic showing agreed contributors: https://docs.google.com/View?id=dd9g3qjp_77rbr8fgfw Yeah the list of user ids would be nice. I'd much rather see a picture of what the map will look like than raw number of edits. As John has alluded to (though only to a small extent), the graphic above is very misleading. Maybe the DWG could work on a slippy map based solely on the data which will be ODbL. It'd give them a way to test the algorithms for excluding/including data in addition to providing a visual. It is planned. User Ids will be released and we will provide planet to test the rules that are currently being discussed. Building a rendering could be something that we could do once we have the data up and running. Emilie Laffray ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] new contributor terms ID list
On Wed, Aug 25, 2010 at 11:36 AM, Emilie Laffray emilie.laff...@gmail.comwrote: On 25 August 2010 04:33, Anthony o...@inbox.org wrote: On Tue, Aug 24, 2010 at 3:28 PM, Grant Slater openstreet...@firefishy.com wrote: On 24 August 2010 20:17, Stephan Knauss o...@stephans-server.de wrote: Is it possible to provide a list with userIDs that have agreed to the new terms? Yes this is planned to be made available shortly. It will be updated on a regular schedule. Last week's LWG minutes also have a nice graphic showing agreed contributors: https://docs.google.com/View?id=dd9g3qjp_77rbr8fgfw Yeah the list of user ids would be nice. I'd much rather see a picture of what the map will look like than raw number of edits. As John has alluded to (though only to a small extent), the graphic above is very misleading. Maybe the DWG could work on a slippy map based solely on the data which will be ODbL. It'd give them a way to test the algorithms for excluding/including data in addition to providing a visual. It is planned. User Ids will be released and we will provide planet to test the rules that are currently being discussed. Building a rendering could be something that we could do once we have the data up and running. Presumably that information was already released to Matt in order for him to make the graphic. Why just to him? What's the delay in making it publically available? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapping during license change / safe to edit
For me, it's more a legal question than a technical question: Either we consider that all elements altered by a changeset are a kind of re-survey in which case the editor could replace altered existing elements by new ones at upload. By 'altered', I mean changes in positions or tags. By 'elements', I mean nodes, ways and relations. But I would not consider 'altered' an existing way with a new node inserted if it's not changing the current tags or positions of that way (e.g. creating a new intersection). For relations, it's more complicated because many are very localized and created by a single author in which case it should not be replaced by a second author. But some are large and complexe like boundaries where many contributors affected the relation. I don't see why in this case we should delete these relations with hundreds members only because the creator started it (probably also very incomplete) and refuses the license ... Or we don't consider that the last contributor is an original author, in which case it is impossible to do something automagically. Pieren ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapping during license change / safe to edit
On Wed, Aug 25, 2010 at 12:18 PM, Alan Mintz alan_mintz+...@earthlink.netalan_mintz%2b...@earthlink.net wrote: At 2010-08-25 01:27, Igor Brejc wrote: But I'm a bit skeptical whether this all would do any good. I'd say go ahead and map. And save your local copies of OSM files for the stuff you edited, just in case. Is this going to be necessary, or will there be a point at which the license change occurs, with the state as of that change available for some period of time? It's impractical for me to manage local files with my edits, and I don't want to burden the servers unnecessarily by running an automated daily download of the (wide) area in which I'm editing. I'm not involved in this license change in any way, so I'm not the one to answer. But being a developer myself and understanding at least a little about complexities of such a migration, my advice is to keep some kind of a backup. As I'm using JOSM, keeping OSM files used for uploading to server is not a big issue. And you probably don't need a daily backup, even if you loose a day or two of your changes, I guess it's not a total disaster (compared to the vast amount of data already in the system). Igor -- Alan Mintz alan_mintz+...@earthlink.net ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] new contributor terms ID list
Frederik Ramm wrote: The raw data can be interpreted in many ways. What we have seen is one interpretation. Until the mail eventually goes out saying please have a look at this and consider relicensing your data, it's entirely irrelevant anyway. At present, any stats measuring voluntary ODbL sign-ups against total OSM users are simply a measure of how effective (or otherwise) the communications have been in reaching users. Nothing more. You might as well use any of the zillions of polls that have been conducted, or http://wiki.openstreetmap.org/wiki/Category:ODbL_Supporter vs http://wiki.openstreetmap.org/wiki/Category:Users_Rejecting_ODbL, or http://wiki.openstreetmap.org/wiki/Category:Users_whose_contributions_are_in_the_public_domain vs http://wiki.openstreetmap.org/wiki/Category:Users_whose_contributions_are_not_in_the_public_domain, or stick a divining rod over Kent and say omm for all the benefit it will give you. [follow-ups to legal-talk, probably] cheers Richard -- View this message in context: http://gis.638310.n2.nabble.com/new-contributor-terms-ID-list-tp5458329p5460816.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapping during license change / safe to edit
Hi, Pieren wrote: For me, it's more a legal question than a technical question: [...] There's certainly several sides to this. There will be objects which are 100% safe under the new license (clearly, those which have never been touched by anyone who has not agreed; and which do not depend on other objects that are unsafe). There will also be objects which are 100% unsafe - only touched by people who have not yet signed up to the new license. Then there will be all the in-between cases, where some contributors have agreed and some not, or where an object depends on a not-agreed object. In these cases, we will have to establish whether the contribution by those who have not agreed is (a) non-trivial and (b) still present in the existing object, and if both conditions are met, the contribution must be removed. (A trivial contribution would e.g. be an automated removal of 100.000 created_by tags; and a contribution that is not present and longer, e.g. if someone added a tag and someone else removed it again, certainly does not require action either.) Removing a contribution, then, does not automaticall mean that the object has to be deleted; it may be possible to remove someone's contribution to an object by rolling it back to an earlier state, or even filtering out that contribution. This is what I think will have to be done when we decide what data to keep and what to remove. It may well be partially crowdsourced - I think it will be hard to find an algorithm that, for example, decides whether a contribution is trivial or not. It would be better to let real people do this. But I was thinking more about the immediate short term; at least the 100% cases could be easily made visible to users. This object is definitely safe, this object is not yet safe, this object may be partially safe, click here to inspect the history would already help a lot. Of course the work Peter is doing will already be a big step forward in driving analyses. However, having the whole thing somehow integrated into the editing process would also be cool. I'm unsure how to handle this technically since I guess that it is too expensive to calculate the current relicensing state of each element o the fly whenever it is loaded. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapping during license change / safe to edit
Am 25.08.2010 12:15, schrieb Alan Mintz: To understand what needs to be done, I ask do we know what author means, exactly? Please read this discussion about possible rules: http://lists.openstreetmap.org/pipermail/dev/2010-August/020124.html Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapping during license change / safe to edit
On Wed, Aug 25, 2010 at 1:42 PM, Frederik Ramm frede...@remote.org wrote: But I was thinking more about the immediate short term; at least the 100% cases could be easily made visible to users. This object is definitely safe, this object is not yet safe, this object may be partially safe, click here to inspect the history would already help a lot. Ideally, the server/API is the best place to deliver this information where e.g a flag could be attached to each downloaded element and set to safe when all versions have authors accepting the new license (and missing or unsafe in other cases). Either the flag is calculated at run-time on each request or it is regularly re-assessed by going through the list of contributors accepting the new licence. Then it would be the responsability of the editor to display this information (different colors or layers or whatever you like). Pieren ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mapping during license change / safe to edit
Am 25.08.2010 13:20, schrieb Igor Brejc: I'm not involved in this license change in any way, so I'm not the one to answer. But being a developer myself and understanding at least a little about complexities of such a migration, my advice is to keep some kind of a backup. All old planet dumps will stay online just as they're now and they will stay under CC-BY-SA. It's okay to keep a local backup of your changes, but it is not necessary in my eyes. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] OSM code sprint @FOSS4G??
Hi all, I'll be spreading OSM goodness at the upcoming FOSS4G conference (with lots of help from Dane Springmeyer A.K.A. The Mapnik Wizard). Besides the workshops and the conference, there is going to be a Code Sprint, which is kinda like a hack-a-thon. See: http://wiki.osgeo.org/wiki/FOSS4G_2010_Code_Sprint If other OSMers are going to be at the FOSS4G, this could be a good opportunity to spend some time working on OSM tools. I, for one, would like to spend some time reworking ogr2osm in order to adapt it for converting huge amounts of data. Are any other OSM devs interested in spending some time coding in Barcelona? Best, -- Iván Sánchez Ortega i...@sanchezortega.es Un ordenador no es una televisión ni un microondas: es una herramienta compleja. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [osmosis-dev] Reading OSM History dumps
Hi all After a little playing around I now got an idea of how I'm going to implement everything. I'll keep as close as possible at the regular simple schema and at the way the pgsql tasks work. Just as with the optional linestring/bbox builder, the history import tasks will serve more then one scheme. I'm leaving relations out, again. the regular simple scheme - its the basis of all but not capable of holding history data + history columns - create and populate an extra column in way_nodes to store the way version. - change the PKs of way_nodes to allow more then one version of an element + way_nodes version builder - create and populate an extra column in way_nodes that holds the node version that corresponds to the way's timestamp + minor version builder - create and populate an extra column in ways and way_nodes to store the ways minor versions, which are generated by changes to the nodes of the way between version changes of the way self. + from-to-timestamp builder - create and populate an extra column in the nodes and ways table that specifies the date until which a version of an item was the current one. After that time, the next version of the same item was current (or the item was deleted). the tstamp field in contrast contains the starting date from which an item was current. + linestring / bbox builder - just the same as with the regular simple scheme, works for all version and minor-version rows Until the end of the week I'll get a pre snapshot out that can populate the history table with version columns and changed PKs. The database created from this can be used to test Scotts SQL-Only solution [1]. It will also contain a first implementation of the way_nodes version builder but only with an example implementation of the NodeStore, that performs bad on bigger files. Brett, the pgsql tasks currently write (in COPY mode) all data to temp files first. The process seems to be PlanetFile - NodeStoreTempFile - CopyFormatTempFile - PgsqlCopyImport in osm2pgsql the copy data is pushed to pgsql via unix pipes (5 or 6 COPY transactions running at the same time in different connections). This approach skips the CopyFormatTempFile stage. Is there any special reason this approach isn't used in the pgsnapshot package? Peter [1] http://lists.openstreetmap.org/pipermail/dev/2010-August/020308.html ___ osmosis-dev mailing list osmosis-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Reading OSM History dumps
Hi Peter, I'm very intersted in your history-extension and I'm going to test as soon as a first snapshot is available. Will it be possible to eat an --bound-polygon stream from osmosis? Or will it just import the whole history-plane? Marco Am 25.08.2010 15:14, schrieb Peter Körner: Hi all After a little playing around I now got an idea of how I'm going to implement everything. I'll keep as close as possible at the regular simple schema and at the way the pgsql tasks work. Just as with the optional linestring/bbox builder, the history import tasks will serve more then one scheme. I'm leaving relations out, again. the regular simple scheme - its the basis of all but not capable of holding history data + history columns - create and populate an extra column in way_nodes to store the way version. - change the PKs of way_nodes to allow more then one version of an element + way_nodes version builder - create and populate an extra column in way_nodes that holds the node version that corresponds to the way's timestamp + minor version builder - create and populate an extra column in ways and way_nodes to store the ways minor versions, which are generated by changes to the nodes of the way between version changes of the way self. + from-to-timestamp builder - create and populate an extra column in the nodes and ways table that specifies the date until which a version of an item was the current one. After that time, the next version of the same item was current (or the item was deleted). the tstamp field in contrast contains the starting date from which an item was current. + linestring / bbox builder - just the same as with the regular simple scheme, works for all version and minor-version rows Until the end of the week I'll get a pre snapshot out that can populate the history table with version columns and changed PKs. The database created from this can be used to test Scotts SQL-Only solution [1]. It will also contain a first implementation of the way_nodes version builder but only with an example implementation of the NodeStore, that performs bad on bigger files. Brett, the pgsql tasks currently write (in COPY mode) all data to temp files first. The process seems to be PlanetFile - NodeStoreTempFile - CopyFormatTempFile - PgsqlCopyImport in osm2pgsql the copy data is pushed to pgsql via unix pipes (5 or 6 COPY transactions running at the same time in different connections). This approach skips the CopyFormatTempFile stage. Is there any special reason this approach isn't used in the pgsnapshot package? Peter [1] http://lists.openstreetmap.org/pipermail/dev/2010-August/020308.html ___ osmosis-dev mailing list osmosis-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [OSM-dev] Mapping during license change / safe to edit
is there already an announcement when the list of people that agreed will be made available online? cheers, Martin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Wishlist API/db: bot-flag WAS new contributor terms ID list
2010/8/25 André Riedel riedel.an...@gmail.com: At least xybot adds bot=yes to the changeset. It would be nice to exclude them (checkbox) from the history: http://www.openstreetmap.org/history?bbox=12.5679%2C42.8836%2C12.8501%2C43.0213 cheers, Martin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] XAPI problems and available queries
Hi all, This is my first time asking questions on this list and hopefully some of you can help me out here. I've been using XAPI for my application to get nodes and relations from openstreetmap and I've experienced some problems that I don't have an answer yet: 1) Sometimes, XAPI server responds pretty quick for my query. However, for some other time, it's very slow like 5 or 10 minutes to get all the nodes ( http://www.informationfreeway.org/api/0.6/node[highway=bus_stop][bbox=-82.738031,27.70168,-82.153672,28.252141]retrieves approximately 3900 nodes. I know this is a large dataset but the xml download file is not that big, under 2MB). 2) Since data needs to be updated by osmosis to planetdump before XAPI can have the new data, I understand that there's a mismatch between real openstreetmap database and xapi data. However, I've read on the wiki page that planetdump is updated minutely (??). It seems to me it takes more time to update the new data. I had to wait 3,4 hours before my new data/change took place in XAPI server. * Is there anything I can do in my application to stay away from the above problems? I don't intend to have my own database to handle the same job of XAPI server. * Is there a way to access OpenStreetMap Postgresql database to GET data only besides using the API? I'd love to use something like XAPI but more stable. * The last question seems irrelevant to this thread but I can't find an answer myself. If a node is deleted, how to get that node WITHOUT the node id (e.g. I'd like to use something like this http://www.informationfreeway.org/api/0.6/node[highway=bus_stop][bbox=-82.738031,27.70168,-82.153672,28.252141]but with all the deleted nodes] Thanks. -- Khoa Tran ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] New Tested
On Wed, Aug 25, 2010 at 8:39 PM, M∡rtin Koppenhoefer dieterdre...@gmail.com wrote: Some of the shortcut-issues might come from plugins. Please do not allow to plugins to redefine already in use major shortcuts like a, s, d, x, j, p, c, m, etc. Its horrible to hit ctrl+shift+m instead of m to simply merge 2 nodes. Sure, it's better that 'others' will have to hit ctrl+shift+m instead of your etc even if there shortcuts are in use since a long time. If you don't like plugins shortcut, then don't install them or redefine their shortcuts in the preferences. Pieren ___ josm-dev mailing list josm-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Mapping during license change / safe to edit
Frederik Ramm wrote: How could we make life easier for them? What technological measures could realistically be employed to determine what is safe to edit and how could this be signalled to the mapper? The osm format could be extended. Each node an tag gets an additional attribute safe=yes if the last position change is safe or the last tag change is safe. That calculation would need to be done on the server on every object change. Sounds costly for the server. This might be provided as api 0.6.1 so existing software still works. Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] New Tested
M∡rtin Koppenhoefer wrote: I'm talking about plugins that redefine common standard keys. That is IMHO bad for the usability. Is this possible? I always thought that once a shortcut is defined it can only be changed by the user. First one wins... Stephan ___ josm-dev mailing list josm-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] WMS and remote-control
Sebastian Klein wrote: Stephan Knauss wrote: Sebastian Klein wrote: How about adding the basic functionality into core? Providing the listener and accepting commands. By default it will have no commands included. Comments? I consider this is a viable solution. It fixes the problem of version dependency between wmsplugin and remotecontrol. The core part of remotecontrol would go into JOSM main, but it would not even listen to some network port, if no commands are registered. (This should satisfy the paranoid people.) Is this being worked on? Maybe for the next version after tested is released... Has to be done by a core developer, patch might require editing a bunch of files... Stephan ___ josm-dev mailing list josm-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Mapping during license change / safe to edit
At 2010-08-25 04:42, Frederik Ramm wrote: I'm unsure how to handle this technically since I guess that it is too expensive to calculate the current relicensing state of each element o the fly whenever it is loaded. That may well be true, but AFAICT, we can't even know that since there is no clear definition of what to do with the middle cases. It also makes me think that, instead of preserving existing features, when I survey a road or POI, I should delete it and then add it back again to make sure it falls in one of the 100% cases. That can't be right. Has there really been no consensus on implementation before foisting this whole thing on us? -- Alan Mintz alan_mintz+...@earthlink.net ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[josm-dev] New Tested
Hello, as we forgot the tested release in August, we now can schedule a new release in time for September. So please prepare everything for a new release in first week of September. Ciao -- http://www.dstoecker.eu/ (PGP key available) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] New Tested
2010/8/25 Pieren pier...@gmail.com: On Wed, Aug 25, 2010 at 8:39 PM, M∡rtin Koppenhoefer dieterdre...@gmail.com wrote: Some of the shortcut-issues might come from plugins. Please do not allow to plugins to redefine already in use major shortcuts like a, s, d, x, j, p, c, m, etc. Its horrible to hit ctrl+shift+m instead of m to simply merge 2 nodes. Sure, it's better that 'others' will have to hit ctrl+shift+m instead of your etc even if there shortcuts are in use since a long time. If you don't like plugins shortcut, then don't install them or redefine their shortcuts in the preferences. I'm talking about plugins that redefine common standard keys. That is IMHO bad for the usability. m to merge nodes was there for ages. Now it converts the geometry to a multipolygon. Which one do you use more often? If you hit ctrl+z to undo, the tags that were deleted from the geometry and put into the relation don't get copied back (the geometry has no tags any more). Cheers, Martin ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] New Tested
Stephan Knauss wrote: M∡rtin Koppenhoefer wrote: I'm talking about plugins that redefine common standard keys. That is IMHO bad for the usability. Is this possible? I always thought that once a shortcut is defined it can only be changed by the user. First one wins... The problem is, plugins are loaded first, so they win. :) It is not hard to fix the plugins that try to grab a well known shortcut, but we have to know about it! Please create a ticket on trac, if you find such a case. Sebastian ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] New Tested
2010/8/26 Sebastian Klein basti...@googlemail.com: It is not hard to fix the plugins that try to grab a well known shortcut, but we have to know about it! Please create a ticket on trac, if you find such a case. done cheers, Martin ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
