Re: [osmosis-dev] How to use EntityBuffer
marcus.wolsc...@googlemail.com wrote: Hello, I am trying to use the EntityBuffer -class but seem to be doing something wrong here. I did: XMReader task = new ... EntityBuffer buffer = new EntityBuffer(BUFFERCAPACITY); task.setSink(buffer); buffer.setSink(sink); buffer.run(); task.run(); It seems that this hangs indefinately waiting for entities to come in from the buffer. Do I need to call some special method or use a special manager-class to make the buffer start it's Executor or Thread? The key reason for using EntityBuffer is to separating processing into multiple threads. It is a key class implementing the --buffer task whose main reason for existing is to spread the pipeline execution across multiple threads and therefore CPU cores. So, the reason you're having a problem is because the buffer.run() method must be run in a separate thread to the task feeding data into it. The input thread calling the process method will block when the buffer becomes full. Likewise, the buffer.run thread will block until data becomes available. If you call both from the same thread the first one will never complete. Internally there are actually three buffers. The first is unsynchronised and allows the input thread to accumulate a number of entities before gaining the main lock. The second is synchronised and is shared between the two threads. The third is unsynchronised and allows the output thread to retrieve a number of records from the second buffer reducing the number of times the lock has to be obtained. If all you want is a way of buffering data within a single thread then EntityBuffer isn't what you're looking for but I've never found a case where buffering without an additional thread improves performance, I've always had more like using file buffers, database transactions, multi-row inserts, etc to achieve that. There is a class called TaskRunner which is a Thread class that allows you to detect when errors occur within the thread, it might be useful for you. If you want an example of its use check out the ActiveTaskManager which is used to run all tasks requiring their own thread (ie. all reading tasks and some others), and the ChangeDownloader task which is used for concurrently reading and merging many files into a single change stream. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Dropping 0.5 tasks
Frederik Ramm wrote: Hi, Brett Henderson wrote: I'm in the process of releasing 0.31 so that people have a known stable version to download that supports 0.6. It still includes all of the existing 0.5 tasks. So, would there be any objections to dropping 0.5 support? It's not urgent, but I'd like to get rid of them at some stage. There are some minor tools out there which don't yet support 0.6 (some simply do a version compare and balk if it isn't 0.5 - easy for a programmer to fix but maybe not so easy for every user). Only recently someone complained that the Ruby osmlib does not support 0.6. I told them to simply run the 0.6 file through osmosis and --write-xml-0.5... but of course 0.6-to-0.5-translation could also be achieved by an XSLT one-liner I guess. It's not a big deal, I'll keep them around for a bit longer then. I don't want to spend much effort looking after them though. If they need to be fixed to keep up with osmosis changes I'd rather drop them and avoid the effort. The other point is that so long as a copy of osmosis-0.31 is available somewhere they'll have that as a fallback option. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Dropping 0.5 tasks
Karl Newman wrote: When you do eventually drop it, it would be a good idea to create an SVN tag for the last Osmosis version with 0.5 support so it can be easily found later (I seem to recall having to sort through commit messages to find the last 0.4-supporting version for someone a few months back). Yep, that makes sense. I'll try not to forget. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] SQL schemas not available
Paul Balomiri wrote: Hi, the server gweb.bretth.com refuses to answer due to high traffic. Could you please send me the file hosted at http://gweb.bretth.com/apidb06-pgsql-latest.sql alternatively you could check it into svn at openstreetmap, or post it to this list? I'm really stuck on this one, Can you try again now? It's working here. It's a google server. Perhaps they had an outage when you tried ... I'm hesitant to check it in because I only provide it as a convenience and it's not always up to date. The proper way to create the schema is using the rails_port schema creation scripts (ie. the rake db:migrate command). At least the last four schema migrations are not included in the script, although they probably won't affect you. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problem using Osmosis (PluginLifecycleException)
On Wed, Nov 11, 2009 at 4:03 PM, Marcus Wolschon
marcus.wolsc...@googlemail.com wrote:
On Tue, Nov 10, 2009 at 11:03 PM, Brett Henderson br...@bretth.com
wrote:
Changing osmosis.bat is not the problem but how can I do an entry
Does
anybody know how to build a classpath list based on the jars contained in
a
directory within ant?
http://ant.apache.org/manual/CoreTasks/jar.html
That would be the attribute sub-element.
Unless I'm misunderstanding the attribute element that's not quite what I
meant.
I already have the following snippet in build.xml.
manifest file=build/binary/jar.txt
attribute name=Main-Class
value=org.openstreetmap.osmosis.core.Osmosis/
attribute name=Built-By value=${user.name}/
attribute name=Implementation-Title value=Osmosis Library/
attribute name=Implementation-Version
value=${project.version} (${TODAY})/
attribute name=Implementation-Vendor value=Brett
Henderson/
/manifest
I could add a classpath element, but I don't know how to add it dynamically
so that I don't have to hard code the list of jars.
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] WayKeyFilter
On Thu, Nov 19, 2009 at 11:27 PM, Andrew Byrd and...@fastmail.net wrote: On 19 Nov 2009, at 07:30, Brett Henderson wrote: Can you please update the wiki detailed usage page to reflect the new task? http://wiki.openstreetmap.org/wiki/Osmosis/DetailedUsage Done. I noted that it is currently only in svn. Great, thanks. On this note, if anybody is keen Osmosis could really do with somebody to take on creating a consistent set of filtering tasks for all entity types (at a minimum it appears that relations are missing from the existing set of tasks). I'm not sure what the requirements would be and I don't do any filtering myself so don't have much experience with it. I think all existing tasks have a bug at the moment where they don't handle ',' characters in the key names which probably should also be fixed. It should be rather simple to extend the existing filter set to relations. I have little experience with OSM/Osmosis myself, yet the task architecture was easy to understand. One thing needs to be clarified, though: should entities belonging to entity types not being filtered be passed through or thrown away? Currently, the node filters pass only nodes through, while the way filters pass non-way entities through unchanged. I see the practical reason for this, but it's not consistent. Maybe there should be an option to choose between behaviors. I think filters should behave consistently rather than having each entity type treated differently for a particular use case convenience. Usually I try to avoid changing existing behaviour, but only if there's a reasonable alternative. If the current filters are inconsistent I think we should change them. If we are going to break existing behaviour though I think we should attempt to make all the breaking changes at once rather than dribbling them in over a longer period. But I didn't write the original tasks and haven't used them so I'm not well placed to decide how they should work. According to the wiki, a tag is a Key-Value pair of Unicode strings of up to 255 characters (full Unicode characters, not bytes). There is no mention of excluded list separator characters, so if it was changed from comma to something else, there would still be a bug and it would cause havoc with people used to using commas. One reasonable solution is some kind of escape character (backslash?) for non- splitting commas that are in the key name. Yep, I think we'd have to escape the separating character if it is included in the key. Given the rarity of something like a ',' in a tag name, I don't think there's an issue with requiring users to escape it when it does occur. It would also be helpful to have a switch to invert filter logic from accept to remove, e.g. so you can accept everything with a highway tag then remove everything that has the highway=motorway value. This would make for a cleaner operation (and command line) than trying to make an exhaustive list of every key-value you want to keep. It would be nice to have comments from users and developers about how the filter set should be designed, and once that's clear I think the new tasks would show up, since they are useful to a lot of people, but not major undertakings. I'm gonna take a back seat on this one :-) If you have some time to gather some requirements, tidy up the existing tasks, and add new features then go for it. Don't feel the need to get full consensus before you do something. Obviously it's a good idea to gather ideas, but feel free to come up with what you think works well, implement it, and then get feedback. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Replication Changes
Hi All, I thought I should give an update on recent Osmosis replication changes. The existing minute diffs are irreparably broken. The timestamp approach for extracting changes only works if a long delay is used. 1 hour is probably a minimum and unacceptable for supposedly minute diffs. The existing minute-slow diffs were created as a means of auditing the minute diffs. The number of differences between using a 5 minute delay and 30 minute delay is quite scary. Both minute and minute-slow will get disabled soon. The new minute-replicate diffs are a new approach for replication that uses PostgreSQL transaction ids to identify changed data in the database. Bugs aside they should never miss data. I have recently fixed a one off error which was causing the last transaction id in each interval to be missed in some but not all cases. I believe this mechanism is now reliable but only time will tell for sure. Please post to the dev list if any missed data is detected. The very new hour-replicate diffs are a new experimental process (totally untested) that takes minute-replicate diffs and rolls them up into hourly changes. This is preferable to existing hourly diffs due to the elimination of a delay, and the elimination of the need to query the main database. Eventually I will disable the existing hourly diffs in favour of the new hour-replicate diffs but this is a bit further off. Note that unlike existing hourly diffs, the new hourly diffs are not exactly hour aligned. One file will get produced per hour, but the changes they contain won't necessarily be hour aligned. This is largely invisible to clients though, the key thing is still to apply them in order and starting from a point prior to your initial data import timestamp. The history diffs are in the process of being generated and are well through 2008 as we speak. These are effectively daily diffs but aren't getting deleted on a rolling window basis. This is effectively creating a full history dump of the database. This has been in the wings for a while, but only possible now that there is some more disk space available. These are still timestamp based extracts due to transaction id queries being useless for historical queries. As a result of the use of timestamps, these will be run with a large delay to avoid missing data. I'll probably set this delay to 1 day to be safe, but perhaps a couple of hours would be enough. The existing daily diffs will probably get disabled at some point in the near future as a result of these new diffs which perform much the same purpose. One big point to note is that all new extracts are *full* history diffs which means that they may contain multiple changes for a single entity. There is a new (as yet undocumented) task in osmosis called --simplify-change which allows a full history diff to be collapsed into an older style delta diff. This new task may be needed to make some existing Osmosis tasks work correctly. This is all fairly new so there will be a few bugs in Osmosis until this is worked through. Once this all settles down, all changesets on the planet server will be full history diffs. Full history for the life of the db will be available. Minute diffs will be available that allow an offline database to be kept within 1-2 minutes of the main API. That summarises the changes currently happening. Some of it is a bit experimental, but that doesn't mean it shouldn't work. Please yell if you see any problems, it's the only way they'll get investigated and fixed. On a related note, the good thing about the new transaction id based mechanism is that it allows zero delay to be used which means at least theoretically that we could get much lower than a 1-2 minute delay. But achieving less than 1-2 minutes can't be done easily with the existing file-based distribution approach. Moving away from a file-based distribution approach has serious implications for reliability in the face of server and network outages, cacheability, bandwidth consumption, and server resource usage. As a result, the existing approach is likely to represent the state of the art in the near to medium future. We need to stabilise the existing features before attempting new ones :-) Osmosis itself is in a fairly complete state, but the implications of full history changes need to be examined. Once this is sorted out I'll create an overdue release, in the meantime the nightly builds will have to suffice. I think that was all, feel free to ask any questions. But be patient for responses from me because my connectivity is limited at the moment, I'm not online every day and it's difficult for me to investigate problems. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] duplication of nodes on apply-change
In an attempt to fix this I have done the following: * Rolled back EntityByTypeThenIdComparator to the previous behaviour of only checking type then id. * Created a new EntityByTypeThenIdThenVersion comparator which (obviously :-) includes version as part of the checking. * Updated ChangeMerger (--merge-change) to use the new version inclusive comparison. * Updated SortedChangePipeValidator used by ChangeMerger (--merge-change) and ChangeSimplifier (--simplify-change) to use the new version inclusive comparison. * Created a new --simplify-change task that reduces full history diffs to delta diffs to allow old tasks relying on single changes per entity to continue to work. At some point I need to go through all change processing tasks and check to see if they need updates to support full history diffs but I'm not sure when I'll get around to it. If anybody sees any problems with the changes I've made please let me know. Brett 2009/11/18 Maxim Dubinin s...@gis-lab.info Thanks for explanation, Frederik. I confirm that it was indeed the problem and rolling back to rev18121 fixed the issue. I also note that 0.31.3 still has this bug. Maxim Вы писали 16 ноября 2009 г., 15:11:06: FR Hi, FR Maxim Dubinin wrote: MD But after I do: MD osmosis --read-xml-change file=data/updates/20091112-20091113.osc.gz MD --read-xml file=data/rus.osm --apply-change --write-xml file=data/rus-new.osm MD I'm getting two versions of the same node in the output: FR Is it possible that I am resposible for this because I recently changed FR the sorting function to treat something with version=x+1 as larger FR than something with version=x, instead of equal? FR (SVN r18230, change to EntityByTypeThenIdComparator...) FR Bye FR Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] GetPropertyAction in xml/common/ElementWriter
Some Mac users still require it. I think some older Macs will never get a Java6. But I think the existing 1.5 compatible version is probably enough, it lets them work with API 0.6 format files. It's up to you though, you can maintain a 1.5 compatible version if you wish. Brett On Tue, Dec 8, 2009 at 8:37 PM, Alexander Chmyr ach...@cloudmade.comwrote: Brett, Is there any issues here related to Java1.5 support branch, is it still required for Mac users, or they're already using Java6. Alexander Chmyr Senior Software Engineer Cogniance, Inc -- skype: alex.smudgy On Tue, Dec 8, 2009 at 2:52 AM, Hakan Tandogan ha...@gurkensalat.comwrote: Brett Henderson wrote: On Tue, Dec 8, 2009 at 11:01 AM, Hakan Tandogan ha...@gurkensalat.com mailto:ha...@gurkensalat.com wrote: Hi again, Is there any particular reason why ElementWriter uses new sun.security.action.GetPropertyAction(line.separator) instead of a simple System.getProperty(line.separator) ? Not that I can see. I've been meaning to take a look at that myself. Done. Regards, Hakan ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] use classworlds-Loader fort start script?
On Tue, Dec 8, 2009 at 7:39 PM, marcus.wolsc...@googlemail.com wrote: On Tue, 08 Dec 2009 02:04:55 +0100, Hakan Tandogan ha...@gurkensalat.com wrote: Brett Henderson wrote: If it isn't too invasive it should be fine. I guess the best way is to try it out and see if any problems crop up. I committed my changes to use the classworlds launcher a few hours ago. The configuration is in the file config/plexus.conf and instructs the launcher to load all jars from lib and lib/default ; I also modified the windows and unix scripts to use the classworlds launcher. Is this just in SVN or a part of 0.32? It's just SVN for now. Once the tag transform patch is in (I haven't had time to check it in tonight, and need to fix the bzip2 ivy problem first), this classworlds launcher doesn't cause any complaints, and I get around to documenting all the new tasks on the wiki then a new 0.33 release could be made. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Postgis is in repo1.maven.org
[ivy:resolve] http://repo1.maven.org/maven2/org/postgis/postgis/1.3.2/postgis-1.3.2.pom [ivy:resolve] -- artifact org.postgis#postgis;1.3.2!postgis.jar: [ivy:resolve] http://repo1.maven.org/maven2/org/postgis/postgis/1.3.2/postgis-1.3.2.jar [ivy:resolve] :: [ivy:resolve] :: UNRESOLVED DEPENDENCIES :: [ivy:resolve] :: [ivy:resolve] :: org.postgis#postgis;1.3.2: not found [ivy:resolve] :: [ivy:resolve] [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS BUILD FAILED /home/mapnik/osmosis/build-ivy.xml:56: impossible to resolve dependencies: resolve failed - see output for details Okay, the postgis library has a different name on the public repo. I've tried updating the ivy dependency descriptor as follows (note addition of -jdbc in the name): dependency org=org.postgis name=postgis-jdbc rev=1.3.2 conf=compile-default/ This doesn't work for a couple of reasons. 1. It also pulls in a postgis-stubs module. I don't know what the purpose of this is but it breaks all PostgreSQL functionality. Presumably it stubs out the PGConnection object for some reason, but I don't know why the postgis-jdbc pom includes it as a transitive dependency. 2. The postgis jdbc extensions aren't registered. This breaks classes such as PGGeometry such that any PostGIS functionality breaks (ie. --pgsql-* tasks). I can work around problem 1 by excluding postgis-stubs explicitly. It works well enough. I suspect this stubs jar should only be part of test scope or somesuch but I could be wrong. dependency org=org.postgis name=postgis-jdbc rev=1.3.2 conf=compile-default exclude module=postgis-stubs/ /dependency However it doesn't solve the postgis extensions not being registered. I've compared the original postgis.jar with the new postgis-jdbc.jar and there is one key difference. The file org/postgresql/driverconfig.properties is missing in the postgis-jdbc.jar. Hakan, I assume you produced this jar yourself using your own maven setup? Are you able to fix it to include this missing file? Or is there another dependency I need to include in order to get this file from somewhere else? When I get a chance I'll revert the delete of postgis jar from the local repo and use that until the public one works. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Postgis is in repo1.maven.org
On Sat, Dec 12, 2009 at 12:52 AM, Hakan Tandogan ha...@gurkensalat.comwrote: If you want to wait a few moments more, you should have a fixed postgis jar latest tomorrow. I did the revert to get the build running because switching backwards and forwards between jars is straightforward. Let me know when you're done though and we can switch it back to your implementation. The build and all tests are completing successfully at the moment. On a related note, I also made some classworlds launcher fixes so I think that is working properly as well. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Bzip2-issues
It looks like the commons-compress library changes how it handles the file
prefix. The old libraries didn't use a prefix so I had hacks in a class
called CompressionActivator like the following to make it compatible with
command line bzip2.
// Command line BZip2 adds BZ to the start of the file which
// we must strip out.
if (sourceStream.read() != 'B' || sourceStream.read() != 'Z') {
throw new OsmosisRuntimeException(
The source stream must start with the characters BZ if it is to be
read as a BZip2 stream.
);
}
These should be able to be removed now but I'll do some tests to make sure.
Brett
On Mon, Dec 14, 2009 at 12:17 AM, Marcus Wolschon mar...@wolschon.bizwrote:
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to
instantiate a BZip2 compression stream.
at
org.openstreetmap.osmosis.core.xml.common.CompressionActivator.createCompressionInputStream(CompressionActivator.java:110)
at
org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:102)
at org.openstreetmap.osm.io.FileLoader.parseOsm(FileLoader.java:127)
at
org.openstreetmap.travelingsalesman.actions.LoadMapFileActionListener.loadMapFile(LoadMapFileActionListener.java:1233)
at
org.openstreetmap.travelingsalesman.actions.LoadMapFileActionListener$1.run(LoadMapFileActionListener.java:1010)
Caused by: java.io.IOException: Stream is not BZip2 formatted:
expected 'B' as first byte but got 'h'
at
org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.checkMagicChar(BZip2CompressorInputStream.java:238)
at
org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream.init(BZip2CompressorInputStream.java:218)
at org.apache.commo...)
The file clearly starts with B and 7zip can decompress it as bzip2.
This is Osmosis 0.32.1 with the supplied commons-compress-1.0.jar .
I wanted to release a new version of Traveling Salesman today but this
issue is blocking it.
Marcus
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Bzip2-issues
0.32 should be fine because these library changes were made after that was released. Version 0.32.1 is just a development release and shouldn't be treated as a stable version. In fact the 0.32.1 build on http://dev.openstreetmap.org/~bretth/osmosis-build/ will get overwritten when the next nightly build kicks off. So in your case I'd suggest targeting 0.32 and sticking with it until 0.33 is released. I can see that the current numbering scheme is very confusing though, perhaps it should be changed. Any suggestions? On Mon, Dec 14, 2009 at 8:16 PM, marcus.wolsc...@googlemail.com wrote: Thanks. I'll test it. Do you want to do a 0.32.2 -release with this? I imagine the latest osmosis-release not being able to read map-dumps is quite a showshopper. Marcus On Mon, 14 Dec 2009 19:22:09 +1100, Brett Henderson br...@bretth.com wrote: Hi Marcus, I've checked a fix into svn. Let me know if you see any more problems. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] download Osmosis 0.32
Anything's possible. What would you like to see? The osmosis-latest download links are actually symbolic links that point to the osmosis-0.32 release files anyway. The build directory contains everything. http://dev.openstreetmap.org/~bretth/osmosis-build/ The wiki only points to the osmosis-release files to save updating the wiki when a new release is made, but perhaps that is a bad idea. Feel free to update the wiki to make it clearer. As I've mentioned before, I'm open to suggestions on how to better number the releases. The fact that 0.32 is a full release and 0.32.1 is an unstable dev release is not obvious. Brett On Thu, Dec 17, 2009 at 11:58 PM, Jan Tappenbeck o...@tappenbeck.net wrote: is there a possiblity to see the right number outside ??? in changes.txt you can the only the milestones ... 0.31, 0.32 regards Jan :-) Brett Henderson schrieb: On Tue, Dec 15, 2009 at 4:32 PM, Marcus Wolschon mar...@wolschon.biz mailto:mar...@wolschon.biz wrote: Where can I download Osmosis 0.32 ? The wiki-page only contains links for 0.31.1 and changing the version-number in these does not work. Which wiki page are you referring to? I think the links here are correct. http://wiki.openstreetmap.org/index.php/Osmosis#Downloading The links should point to here: http://bretth.dev.openstreetmap.org/osmosis-build/ 0.32 is available there. ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Fwd: download Osmosis 0.32
Forwarding back to the list ... -- Forwarded message -- From: Brett Henderson br...@bretth.com Date: Fri, Dec 18, 2009 at 9:22 AM Subject: Re: [osmosis-dev] download Osmosis 0.32 To: Jan Tappenbeck o...@tappenbeck.net Cc: osmosis-dev@openstreetmap.org, Marcus Wolschon public-marcus-jln08e+kluoanpmfv7s...@lo.gmane.orgpublic-marcus-jln08e%2bkluoanpmfv7s...@lo.gmane.org, osmosis-dev public-osmosis-dev-3+rWM/WnaLOn4i5uJCXUsti2O/ jbr...@lo.gmane.org Anything's possible. What would you like to see? The osmosis-latest download links are actually symbolic links that point to the osmosis-0.32 release files anyway. The build directory contains everything. http://dev.openstreetmap.org/~bretth/osmosis-build/http://dev.openstreetmap.org/%7Ebretth/osmosis-build/ The wiki only points to the osmosis-release files to save updating the wiki when a new release is made, but perhaps that is a bad idea. Feel free to update the wiki to make it clearer. As I've mentioned before, I'm open to suggestions on how to better number the releases. The fact that 0.32 is a full release and 0.32.1 is an unstable dev release is not obvious. Brett On Thu, Dec 17, 2009 at 11:58 PM, Jan Tappenbeck o...@tappenbeck.net wrote: is there a possiblity to see the right number outside ??? in changes.txt you can the only the milestones ... 0.31, 0.32 regards Jan :-) Brett Henderson schrieb: On Tue, Dec 15, 2009 at 4:32 PM, Marcus Wolschon mar...@wolschon.biz mailto:mar...@wolschon.biz wrote: Where can I download Osmosis 0.32 ? The wiki-page only contains links for 0.31.1 and changing the version-number in these does not work. Which wiki page are you referring to? I think the links here are correct. http://wiki.openstreetmap.org/index.php/Osmosis#Downloading The links should point to here: http://bretth.dev.openstreetmap.org/osmosis-build/ 0.32 is available there. ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] download Osmosis 0.32
On Fri, Dec 18, 2009 at 9:40 AM, Hakan Tandogan ha...@gurkensalat.comwrote: Brett Henderson wrote: Anything's possible. What would you like to see? As I've mentioned before, I'm open to suggestions on how to better number the releases. The fact that 0.32 is a full release and 0.32.1 is an unstable dev release is not obvious. O.k., I am pretty strongly brainwashed by maven by now, but how about adopting the maven way? 0.32-SNAPSHOT would be the current unstable dev 0.32 would be the stable release 0.33-SNAPSHOT would be the next unstable after 0.32 Okay, sounds good to me. We could set up some big reminder that -SNAPSHOT versions are always prone to breakage, and hudson would copy the SNAPSHOT builds to the deployment web space... Okay. I'll update the build.xml so that the next nightly build on dev.openstreetmap.*org* produces SNAPSHOT builds from now on. I probably should delete any existing nightly builds in the download directory to avoid confusion. Is there anything that needs to be done for maven or that managed via hudson somehow. Whatever we do, we just have to keep in mind users who download the source and build Osmosis themselves. For that reason I think we'll have to keep the current version in the build files in svn. Within hudson itself we can do whatever we want. For the ant build at least, you can use Hudson to set the properly project.version and that will override whatever is in the ant build.xml file. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problem opening .osm files in JOSM after osmosis-0.32 processing
Hi Holger,
I don't have much net connectivity at the moment, and don't have JOSM
installed.
I'm assuming that osmosis is creating node id 289490930 in such a way that
JOSM doesn't like it.
Can you provide the xml element for node with id=289490930 from both the
original unterfraken.osm.bz2 file, and the osmosis modified files? Then we
can compare the two and try to figure out what might be causing the problem.
Brett
On Mon, Dec 28, 2009 at 9:06 AM, Holger Blum ho...@gmx.net wrote:
Hello!
I want to extract all data within a polygon out of an .osm file. Some
time ago I've already used an older version of osmosis for this task,
but this was with API 0.5 which obviously doens't work anymore, so I've
downloaded osmosis 0.32.
The processing was successful, but I'm not able to open the generated
file in JOSM.
The exact proceeding was:
* download the data
http://download.geofabrik.de/osm/europe/germany/bayern/unterfranken.osm.bz2
* edit a simple polygon file
wue_rect.poly
1
9.90 49.75
9.90 49.80
9.99 49.80
9.99 49.75
9.90 49.75
END
END
* run osmosis
D:\GPS\osmosis-0.32\binosmosis.bat --read-xml file=unterfranken.osm
--bounding-polygon file=wue_rect.poly --write-xml file=wue_rect_ufr.osm
* open the file in JOSM
The error message in JOSM is
org.openstreetmap.josm.io.IllegalDataException:
org.openstreetmap.josm.data.osm.
DataIntegrityProblemException: Objekt {Node id=289490930 version=0 IV }
kann nicht zum Datensatz hinzugef³gt werden, weil es bereits im
Datensatz enthalten ist
I've tried with different .osm files, different polygons in this area
and both JOSM latest and stable, it always fails. The original .osm file
loads without error, so I guess that there's something wrong with the
processing in osmosis.
Does anyone have an idea what could be wrong? The files from geofabrik
are still made with 0.31 and are ok, but I wasn't able to make this
version work on my XP to try it myself.
Extracts of other areas do work, for example an extract of the data
within a polygon based on relation 162360. The main difference is that
this is a rural area with lower node density and simpler tagging.
I think this behaviour should be easily reproduceable, if not I'd
willingly supply my files for analysis.
Holger
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] planet diff to polygon extract?
On Wed, Dec 30, 2009 at 3:56 AM, David Carmean d...@halibut.com wrote: For the moment I'm working with just the Cloudmade California extract (in PostGIS simple schema). Is it possible to perform a polygon clip/extract of the daily planet diffs (changefile)? Failing that, what happens if a planet-wide diff is applied to an incomplete planet base file? You can't perform a polygon extract of a diff file (the diff file doesn't contain enough info to determine the location of ways whose nodes haven't changed). However you can perform a polygon extract on the merged result. In other words: 1. Apply the full diff to your extract first. 2. Run the result through the bounding polygon task again with the California coordinates. You can do it all in a single osmosis incantation if necessary. Something like: osmosis --read-xml-change diff.osc --read-xml california-orig.osm --apply-change --bounding-polygon file=polygonfile.txt --write-xml california-updated.osm Also curious: how much diskspace does the postGIS simple schema use for a complete planet file? I have no idea ... lots :-) Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] completeRelations flag for --bp/--bb not working
On Sat, Jan 9, 2010 at 9:15 PM, Frederik Ramm frede...@remote.org wrote: Hi, consider this test case (run with 0.31.3): osm version=0.6 node id=1 version=1 timestamp=2007-10-31T23:48:54Z uid=1 user=fred changeset=1 lat=53 lon=10/ node id=2 version=1 timestamp=2008-09-28T17:44:53Z uid=1 user=fred changeset=1 lat=54 lon=10 / node id=3 version=1 timestamp=2008-09-28T17:44:53Z uid=1 user=fred changeset=1 lat=55 lon=10 / relation id=1 version=1 timestamp=2005-07-05T02:14:07Z uid=1 user=fred changeset=1 member type=node role= ref=1 / member type=node role= ref=2 / /relation relation id=2 version=1 timestamp=2005-07-05T02:14:07Z uid=1 user=fred changeset=1 member type=node role= ref=3 / /relation /osm Cutting out from this a bbox that only contains node #1, such as osmosis --rx test.osm --bb left=1 right=90 top=53.5 bottom=52.5 --wx - one would expect to find in the output: * without extra flags: node #1, relation #1 referring to nodes #1,#2 -- pass * with clipIncompleteEntities: node #1, relation #1 refering to node #1 -- pass * with completeRelations: nodes #1,#2, relation #1 referring to both -- fail; instead, you get node #1 only, but relations #1 and #2. It seems that completeRelations always acts as copy all relations from input to output no matter what and has no influence on handling nodes and ways. Re-writing the same example to use ways, and the trying it with completeWays=true, yields the expected result. The Wiki documentation on completeRelations says: Include all available relations which are members of relations which have at least one member in the bounding polygon. This describes neither the behaviour I had expected, nor the behaviour I have observed. I believe the intended behaviour is really Include all available *objects* ...- no? My expectation is much the same as yours. I have no idea what's going on there. I'll check it out when I get time. If you have the time to patch it to give you the expected behaviour I'd appreciate it. Thanks for the detailed example, that should make it much easier to debug :-) Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Crash on processing nodes?
On Mon, Jan 11, 2010 at 8:38 PM, Brett Henderson br...@bretth.com wrote: On Mon, Jan 11, 2010 at 6:40 PM, Frederik Ramm frede...@remote.orgwrote: The problem was that in switching from --rci to --rri I didn't add the --simc filter, which led to duplicate objects in the extracts. It should be fixed now. I don't think that it was related to the Osmosis version, but switched to 0.32 nonetheless. I probably should fix the --apply-change so that it either fails or handles history diffs properly ... it's fairly non-obvious. I've checked in a change to detect this scenario and fail now. It's only in svn for now. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Crash on processing nodes?
On Wed, Jan 20, 2010 at 11:24 PM, Simon Nuttall simon.nutt...@gmail.comwrote: On one of the CycleStreets machines it takes 17.3 hours to extract and merge data from the Britain and Ireland GeoFabrik planets using: osmosis -v 100 --read-xml data/osm/downloads/great_britain.osm enableDateParsing=no --sort-0.6 type=TypeThenId --read-xml data/osm/downloads/ireland.osm enableDateParsing=no --sort-0.6 type=TypeThenId --merge --write-apidb dbType=mysql host=localhost database=britainOSM user=import password=xxx validateSchemaVersion=no (On our main machine the same only takes about 5 hours.) During that time it builds an 11 GB innodb file in /var/lib/mysql/ibdata1 Osmosis fills the britainOSM database, which I then treat as 'read only' to build the CycleStreets routing tables in a separate database. As I'm only filling a database and then treating it as read-only do I really need all the transaction processing that is provided by innodb? Is there a way of turning those features off to speed up all the processing and use less disk space? You could possibly add populateCurrentTables=no to the --write-apidb task and use the history tables for your queries. After a planet import they contain identical data to the current tables. Are the history tables still MyISAM or are they InnoDB? I haven't used MySQL for a while. If the history tables are MyISAM and you disable current table population you should get very large performance speedups. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] [RFE] Add bounding shape to OSM dumps
On Fri, Jan 22, 2010 at 5:17 AM, WanMil wmgc...@web.de wrote: Hi, Apollinaris Schoell wrote: osmosis supports 2 options to keep ways and relations intact. but geofabrik extracts don't use it as far as I know. completeWays completeRelations That is correct: These options exist, and are not used, because if we'd use them, the nightly build would take something like three days instead of six hours. Osmosis works very well in streaming mode and these options make it impossible to stream - Osmosis needs to create a temporary copy of the full dataset and is rather inefficient at it. We typically do something like osmosis --rx file.osm --tee 20 --bp file=country1.poly --wx country1.osm --bp file=country2.poly --wx country2.osm... which would causes Osmosis to make 20 temporary full copies of the data and write them to disk in completeWays mode. We use clipIncompleteEntities which means that Osmosis removes references to those nodes outside the polygon from any ways or relations. It would be good if Osmosis could somehow flag the clipped entities so that processing software could at least know that there is something wrong, or incomplete, with them. Adding the actual polygon used for clipping could of course be done but it will not automatically enable proper filling. Assume this: | +-|+-+ | | | | | |--- filled area +-|+-+ | |-- clipping boundary After clipping with clipIncompleteEntities, this will lead to | | +-+ | | | | | | | +-+ | | Even if you know where the clipping boundary is, you cannot extend the object towards that boundary properly because you are missing the nodes beyond the boundary. Bye Frederik Ok, we cannot reconstruct the 100% exact original shape if the first node outside the boundary is missing. But a reasonable workaround is to connect directly to the boundary in case there are nodes missing in a polygon and the last point is not too far away from the boundary. I think in most cases this will be ok. This is not the 100% solution and it would be better if ways and relations are added completely or at least the first point beyond boundary should be clear. But as long as this is too time-consuming the proposed solution to add the boundary to the OSM dump is an easy to realize and very helpful improvement. Flagging of clipped ways is another good and helpful improvement. Hi WanMil, Frederik has already given you most useful info, but I'll add some comments to the discussion in case you or others find it useful. ADDING GEOMETRY DETAILS TO OUTPUT FILE It is not simple to add additional geometry information to an output OSM file due to the way Osmosis works internally. The task that extracts boundary data (eg. --bounding-polygon) is independent of the --write-xml task producing the output file which means that the geometry information cannot be written to the output file. This is an intentional design design in the Osmosis design where each task can be implemented independently of all other tasks. The advantage of this approach is that tasks can be combined in various combinations according to what the end user requires. The disadvantage is that there is no simple way of passing additional information between tasks other than standard node/way/relation data. Osmosis is a generic tool supporting many use cases, which means it can't always provide the ideal solution. It is not impossible, but requires a fair bit of rework to implement. Simple bounding box information can be propagated through the pipeline so I believe the --bounding-box will cause a Bound element to be added to the output file. But more complex geometries such as those used by --bounding-polygon are not supported. This would need to be enhanced if we wanted to pass bounding box information. I don't have a lot of time for Osmosis so I can't tackle this one. CLIPPED WAYS Flagging clipped ways would be quite useful. It requires enhancing the Entity class within Osmosis to allow additional information to be attached. It also requires all existing tasks to be enhanced where necessary to manipulate this additional data. Again, I'm not likely to do this myself, but would support anybody trying to implement it themselves. COMPLETE GEOMETRIES As you've discovered, you can end up with missing data for ways or relations that sit outside the bounding box. For example, nodes that sit outside the area may be part of a way inside the area but will be left out. The --bounding- tasks are unlikely to ever be fixed in this regard. As Frederik points out there is just no way to efficiently do this in a streamy fashion because it requires random access to the entire dataset which isn't something Osmosis is good at. By the time you process ways, the nodes are not available in memory any more so you can't tell where each way is located. You can't go back and add extra nodes to the
Re: [osmosis-dev] extracting with --used-node BitSet misses certain objects
On Tue, Jan 19, 2010 at 2:59 PM, Maxim Dubinin s...@gis-lab.info wrote: Hi osmosis-devs, I've encountered rather strange behaviour for this test case: osmosis --rx file=data/planet-091230.osm --bp file=poly/tatar.poly idTrackerType=BitSet --used-node idTrackerType=BitSet --wx file=tatar.osm It produces OSM XML, that completely misses place=hamlet. If I run it like this: osmosis --rx file=data/planet-091230.osm --bp file=poly/tatar.poly idTrackerType=BitSet --wx file=tatar.osm They are back. Is it normal? I'm using Osmosis 0.33 (r19236) The --used-node task removes all nodes that aren't part of a way. Presumably your hamlet nodes are not part of any ways. However, if you can show me a hamlet node that *is* part of a way, but is being left out of the output file then please let me know. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Osmosis Release 0.34
Hi, I've created osmosis 0.34 which should fix the recent issues with a missing plexus.conf file, and problems with various other missing files in the tar distribution. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Slow to import into a postgres api schema
On Mon, Feb 22, 2010 at 12:31 AM, Kai Krueger kakrue...@gmail.com wrote: On 02/21/2010 11:43 AM, Brett Henderson wrote: Having a --write-api0.6-dump command might not be a bad idea. Although the --wd task is simpler from a users point of view if it can be made to be equally fast. Sorry, I wasn't clear there. I wasn't suggesting we create a --write-apidb-dump task. I was just suggesting that there may be some re-usable code in the --write-pgsql-dump task because it already formats a number of PostgreSQL data types into the COPY format. Although I haven't looked at the new JDBC COPY support so I'm not sure if that even makes sense. As a proof of concept, I added statements into disableIndexes to manually drop each index and then recreate them in enableIndexes. Together with using the Copy command (supported in the postgres 8.4 JDBC driver), my initial experiments show a speedup of 3 - 4 times on the initial population of the tables (i.e. without populating the current tables, but I suspect that this step can be similarly sped up). These numbers were obtained using small country extracts (e.g. 1 - 20 Mb in bz2 size), but I would guess that they hold up with the full planet imports too. The main benefit comes from disabling the indecies, and the copy command seems less important. The patch I have is quite ugly (and untested for correctness), as it breaks the levels of abstraction and has to hard code all the available indecies. So my question is, what would be the best way to do this in a clean way? Looking at the speedups obtained and the time involved in imports, it seems like it might be worth it. If it's truly 3-4 times faster then it's worth a lot of effort. I don't have a lot of time to get involved in this myself though, so if you have some time to write a maintainable patch, then I'd be very grateful. My only ask is that you stick around to get it working and provide support until it is proven stable. I have very little time myself at the moment. But perhaps I will be able to come up with something. Perhaps not too soon though. At least not a maintainable patch that integrates nicely with moduler structure of osmosis. Okay. See how you go, and post whatever you come up with back to the list. * I'd prefer to make this a new task within Osmosis. The current one is called --write-apidb I think. You could create a new one called something like --write-apidb-fast. If you can get it stable and it works well, then we can point --write-apidb at your task and delete the current one. Oh, that would mean I would actually have to understand this wonderful extensible structure of osmosis rather than just hack some random bits and pieces into it... ;-) In case you don't know already, creating a new task is a matter of modifying TaskRegistrar to register your task factory, creating a new task factory (use those called from TaskRegistrar as an example), then create a new Task implementing the Sink interface (use the existing DB tasks as an example). How much of the rest of Osmosis you choose to use is largely up to you, although more re-use is definitely a good thing from a maintenance perspective. Cheers, Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] keeping an AOI-API-DB up to date
Hi Marco, I'm not sure if you've received an answer elsewhere, but here goes. There are a number of ways your data can end up with referential integrity problems. The most likely is simply that by cutting an area out using the bounding polygon task you'll end up with missing nodes. Some ways will be included because they have nodes within the bounding polygon, but nodes that are part of the way but lie outside the polygon will not be included. You have a couple of choices: - Set clipIncompleteEntities=true on your --bp task. This will modify the ways and relations to fix referential integrity, but be aware that the entities no longer match the API database so they *MUST NOT* be re-uploaded. - Simply drop the offending constraint from the database. This is what I've done in the past. I can't guarantee it will work with rails, but it certainly used to in the past. Brett On Tue, Feb 9, 2010 at 2:50 AM, Marco Lechner - FOSSGIS e.V. marco.lech...@fossgis.de wrote: Hi I tried to use the described way to keep a AOI (area of interest) OSM-file up to date. The following errors or unexpected behaviour occured: 1. downloaded planetfile (planet-100120.osm.bz2) 2. osmosis (0.32) with --bp completeWays=yes and completeRelations=yes to cut out myAOI 3. downloaded daily-diffs (20100120-20100121.osc, ...) 4. merged the diffs using osmosis = 20100120-20100120.osc 5. applied merged diffs using: ./bin/osmosis --rxc 20100120-20100126.osc --rx aoi-100120.osm.bz2 --ac --bp file=aoi.pff completeWays=yes completeRelations=yes --wx file=aoi-100126.osm.bz2 6. tried to import both aoi-osm.bz2 into separate PostgreSQL-DBs using API-Scheme. Results: No Errors: - bzcat ./aoi-100120.osm.bz2 | ./bin/osmosis --rx file=- --wd database=osmAOI_api_1 host=localhost:5432 user=osmuser password=pass Error: - bzcat ./aoi-100126.osm.bz2 | ./bin/osmosis --rx file=- --wd database=osmAOI_api_2 host=localhost:5432 user=osmuser password=pass Error message: Unable to load current way nodes. at org.openstreetmap.osmosis.core.apidb.v0_6.ApidbWriter.complete(ApidbWriter.java:980) at org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:110) at java.lang.Thread.run(Thread.java:619) Caused by: org.postgresql.util.PSQLException: ERROR: Insert or Update in table »current_way_nodes« verletzt foreignkey-Constraint »current_way_nodes_node_id_fkey« Detail: key (node_id)=(581918002) does not exist in table »current_nodes«. The error message says it clear. My workflow to keep aoi.osm.bz2 up to date seems to produce an erroneous aoi.osm.bz2 or at least a osm-file that differs from a freshly cutted one. I expected that cutting an AOI from a planetfile should produce the same results than keeping my AOI up to date by applying planet*.osc and cutting these back to myAOI. Before trying to: bzcat ./aoi-100120.osm.bz2 | ./bin/osmosis --rx file=- --wd database=osmAOI_api_1 host=localhost:5432 user=osmuser password=pass ./bin/osmosis --rxc 20100120-20100126.osc --rd database=osmAOI_api_1 host=localhost:5432 user=osmuser password=pass --ac --bp file=aoi.pff completeWays=yes completeRelations=yes --wd database=osmAOI_api_1 host=localhost:5432 user=osmuser password=pass will this keep my apiDB of an AOI up to date? Marco P.S. I'm not sure if this is more or less a user-question and therefore using the wrong list, but I guess it moves quite next to the feature-wish section. Lennard schrieb: Marco Lechner - FOSSGIS e.V. wrote: Otheriwse I have to apply the diff onto a planetfile and cut it everyday or apply the planet diff to ma AOI.osm and cut it afterwards. Is this a proper workflow? Or what ist the best task to keep an AOI.osm (cutted by osmosis using --bp) up-to-date? Apply the diffs first, then pass it through the --bp task before writing it out again: osmosis --rxc 20100127-20100128.osc.gz --rx AIO-20100127.osm --ac --bp file=/path/to/your/polygon/file --wx AIO-20100128.osm ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] keeping an AOI-API-DB up to date
Hi Marco, On Tue, Feb 23, 2010 at 12:08 AM, Marco Lechner - FOSSGIS e.V. marco.lech...@fossgis.de wrote: Hi Brett, thanx for your answer. I thought using completeWays=yes and completeRelations=yes takes care of nodes outside my bounding polygon. Those options will fix the problem, but they are horrendously slow. If they work for you then that's fine, but most users avoid those options if at all possible. 1. If a node is created new it will be inside my merged osc and therefore be inside my patched AOI.osm and (if part of a way or relation) put in my recently cutted AOI.osm Correct. 2. If a node is deleted it will be deleted by patching the merged osc Also correct. 3. the only problem seems to exist if an existing node outside the polygon is changed and integrated into a way or relation inside my AOI. Because the node doesn't exist in my AOI (also not in the patched AOI before cutting it again), because it isn't created freshly. Am I right? Or are there situations I miss? Yes, also correct :-) Well kind of anyway. If the node is modified as well as the newly containing way or relation then it will be available for inclusion (if you're using completeWays or completeRelations anyway). But if the node is not modified but it is added to a new or existing way or relation then it cannot be included. However if you're not using the completeWays and completeRelations options then it will never be included regardless of scenario. I'm very interessted in creating a consistent regionally bordered OSM-DB. Creating a perfect bordered extract is not simple. But you do have a couple of options: 1. Load the complete planet (or at least an area somewhat larger than the one you require) into the Osmosis simple schema via the --write-pgsql-dump task, then use the --read-pgsql and --dataset-bounding-box task to extract a perfect extract including all required nodes. Keep the database up to date with minute changesets and the --write-pgsql-change task. 2. Create the extract in two parts. First part is to extract a region from the overall planet that is larger than your required area, but small enough to keep the result file small. Do this with the default --bounding-box task. Then run the --bounding-box task over the smaller result file with completeWays and completeRelations both enabled. I haven't tried this myself, but I think it should work. It's not a simple problem unfortunately, and unless you have a really really good reason to make the extract correct, it's much simpler to make some compromises. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] getting all the relations from source with completeRelations=yes
Frederik Ramm wrote about similar issues on January 9 (not sure if they're the same though). I haven't had a chance to look into them yet. I didn't write the completeRelations, completeWays functionality so I don't understand how they work at the moment. It's on my TODO list, but I'm finding it hard to get the time. If somebody else has the time to look into that area of the code I'd be very grateful ;-) Brett On Wed, Mar 10, 2010 at 10:30 AM, Maxim Dubinin s...@gis-lab.info wrote: Hi all, to follow up on http://www.mail-archive.com/osmosis-dev@openstreetmap.org/msg00279.html I'm using 0.34 and I'm doing: osmosis --read-xml file=input.osm --bounding-polygon file=bnd.poly completeWays=yes completeRelations=yes idTrackerType=BitSet --write-xml file=output.osm As a result, I'm having _all_ relations (more that 16 thousands) from input.osm dumped into output.osm. All these extra relations are empty ones, ways and nodes are not extracted. According to wiki: completeRelations Include all available relations which are members of relations which have at least one member in the bounding box. But some of extracted ones definitely do not have any members in bnd.poly. Any ideas? Maxim ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] problems with xapi, relations and osmosis
You need to check both of your OSM files downloaded from XAPI. The first one contains the full relation details as you expect. The second one contains an empty relation. Both have the same version number. The Osmosis *--merge* task is intentionally simple and doesn't alter relation contents in any way. In your case it is seeing that the relation exists in both inputs, it determines that both have the same version number, and picks the second one. The second one is written to the result file. The real question to ask is Why does XAPI return an empty relation for my second query?. But that will have to be asked on the dev mailing list ;-) Brett On Thu, Mar 11, 2010 at 6:22 AM, Jan Tappenbeck o...@tappenbeck.net wrote: hi ! i have following statement to download NODES for memorial-elements: http://xapi.openstreetmap.org/api/0.6/node[memorial:type=stolperstein][bbox=5.5,47.4,15.0,55.3] on the other way i have following statement for download the main-relation of this categorie: http://xapi.openstreetmap.org/api/0.6/relation[name=Stolpersteine][bbox=5.5,47.4,15.0,55.3] in the next moment i want to merge the files, and some other, with OSMOSIS. because there are some NODES in the first and the secound download-files. in the new file only this line exists for the relation relation id=407359 version=75 timestamp=2010-03-05T21:26:43Zuid=32786 user=Netzwolf changeset=4044847/ this is the osmosis-statement and the message: OSMOSIS_command: D:\DATEN\JAN\openstreetmap\osmosis\bin\osmosis.bat --read-xml xapi4osm_node_1.osm --sort-0.6 --read-xml xapi4osm_relation_1.osm --sort-0.6 --r ead-xml xapi4osm_node_2.osm --sort-0.6 --read-xml xapi4osm_relation_2.osm --sort -0.6 --merge --merge --merge --write-xml D:\DATEN\JAN\openstreetmap\osmosis\sto lpersteine.osm 10.03.2010 20:21:17 org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.32 10.03.2010 20:21:17 org.openstreetmap.osmosis.core.Osmosis run INFO: Preparing pipeline. 10.03.2010 20:21:17 org.openstreetmap.osmosis.core.Osmosis run INFO: Launching pipeline execution. 10.03.2010 20:21:17 org.openstreetmap.osmosis.core.Osmosis run INFO: Pipeline executing, waiting for completion. 10.03.2010 20:21:17 org.openstreetmap.osmosis.core.Osmosis run INFO: Pipeline complete. 10.03.2010 20:21:17 org.openstreetmap.osmosis.core.Osmosis run INFO: Total execution time: 468 milliseconds. did anybody now why the relation in the merge-file is only one line and so uncomplete ??? regards Jan :-) ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] problems with xapi, relations and osmosis
On Fri, Mar 12, 2010 at 11:38 PM, Jan Tappenbeck o...@tappenbeck.net wrote: Am 12.03.2010 12:30, schrieb Brett Henderson: You need to check both of your OSM files downloaded from XAPI. The first one contains the full relation details as you expect. The second one contains an empty relation. Both have the same version number. The Osmosis *--merge* task is intentionally simple and doesn't alter relation contents in any way. In your case it is seeing that the relation exists in both inputs, it determines that both have the same version number, and picks the second one. The second one is written to the result file. The real question to ask is Why does XAPI return an empty relation for my second query?. But that will have to be asked on the dev mailing list ;-) HI ! but i did not understand your new question ! * when you download by http://xapi.openstreetmap.org/api/0.6/node[memorial:type=stolperstein][bbox=5.5,47.4,15.0,55.3]then there are only NODES - RIGHT ! * when you download by http://xapi.openstreetmap.org/api/0.6/node[memorial:type=stolperstein][bbox=5.5,47.4,15.0,55.3]then there are ALL RELATIONS-Members (even Nodes) and the RELATION defintion. Both of the above URLs are identical. I think you meant: URL1 - http://xapi.openstreetmap.org/api/0.6/node[memorial:type=stolperstein][bbox=5.5,47.4,15.0,55.3]http://xapi.openstreetmap.org/api/0.6/node%5Bmemorial:type=stolperstein%5D%5Bbbox=5.5,47.4,15.0,55.3%5D and URL2 - http://xapi.openstreetmap.org/api/0.6/relation[name=Stolpersteine][bbox=5.5,47.4,15.0,55.3]http://xapi.openstreetmap.org/api/0.6/relation%5Bname=Stolpersteine%5D%5Bbbox=5.5,47.4,15.0,55.3%5D Unless I'm mistaken, URL1 returns a file containing the following entry: relation id='407359' user='Netzwolf' uid='32786' timestamp='2010-03-05T21:26:43Z' version='75' changeset='4044847' /relation ONLY the MERGE file include the RELATION-Definition by ONE LINE - so i did not see a Problem by XAPI. OSMOSIS lose the relation-MEMBERS - OSMOSIS-Question ! Can you please open your two OSM files and check them before you add more !!'s to your emails ;-) Cheers, Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] problems with xapi, relations and osmosis
On Sat, Mar 13, 2010 at 12:05 AM, Jan Tappenbeck o...@tappenbeck.net wrote: Am 12.03.2010 13:52, schrieb Brett Henderson: On Fri, Mar 12, 2010 at 11:38 PM, Jan Tappenbeck o...@tappenbeck.netwrote: Am 12.03.2010 12:30, schrieb Brett Henderson: You need to check both of your OSM files downloaded from XAPI. The first one contains the full relation details as you expect. The second one contains an empty relation. Both have the same version number. The Osmosis *--merge* task is intentionally simple and doesn't alter relation contents in any way. In your case it is seeing that the relation exists in both inputs, it determines that both have the same version number, and picks the second one. The second one is written to the result file. The real question to ask is Why does XAPI return an empty relation for my second query?. But that will have to be asked on the dev mailing list ;-) HI ! but i did not understand your new question ! * when you download by http://xapi.openstreetmap.org/api/0.6/node[memorial:type=stolperstein][bbox=5.5,47.4,15.0,55.3]then there are only NODES - RIGHT ! * when you download by http://xapi.openstreetmap.org/api/0.6/node[memorial:type=stolperstein][bbox=5.5,47.4,15.0,55.3]then there are ALL RELATIONS-Members (even Nodes) and the RELATION defintion. Both of the above URLs are identical. I think you meant: URL1 - http://xapi.openstreetmap.org/api/0.6/node[memorial:type=stolperstein][bbox=5.5,47.4,15.0,55.3]http://xapi.openstreetmap.org/api/0.6/node%5Bmemorial:type=stolperstein%5D%5Bbbox=5.5,47.4,15.0,55.3%5D and URL2 - http://xapi.openstreetmap.org/api/0.6/relation[name=Stolpersteine][bbox=5.5,47.4,15.0,55.3]http://xapi.openstreetmap.org/api/0.6/relation%5Bname=Stolpersteine%5D%5Bbbox=5.5,47.4,15.0,55.3%5D Unless I'm mistaken, URL1 returns a file containing the following entry: relation id='407359' user='Netzwolf' uid='32786' timestamp='2010-03-05T21:26:43Z' version='75' changeset='4044847' /relation ONLY the MERGE file include the RELATION-Definition by ONE LINE - so i did not see a Problem by XAPI. OSMOSIS lose the relation-MEMBERS - OSMOSIS-Question ! Can you please open your two OSM files and check them before you add more !!'s to your emails ;-) Cheers, Brett Hi Brett, the URL's are not identical !!! URL1 ask for NODE and URL2 ask for RELATION !!! Have you opened the files and actually checked them yet? I'm only going to say this once more. Download the OSM files from both URLs. OPEN! them both. Search for the text relation id='407359'. It appears in both files. One contains relation members, the other one doesn't. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Database is empty after five days of importing
It's not currently possible. The --truncate-apidb and --write-apidb tasks have to be run via separate Osmosis invocations (to ensure one completes before the other starts), and cannot be made to run inside a single transaction. Note that the import itself does run in a transaction, but doesn't include the truncate. On Fri, Mar 26, 2010 at 1:26 AM, Klemens toural...@gmail.com wrote: Thanks. Any hints on how this can be done with osmosis? On 24.03.2010 21:12, Peter Körner wrote: Import in a transaction. ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Error with osmosis.bat when using paths that include spaces
Hi Franz, Thanks for the error report. I haven't been able to test this out yet but I haven't forgotten. I'll get to it eventually and apply your changes. Brett On Sat, Mar 13, 2010 at 2:39 AM, Franz gr...@cip.ifi.lmu.de wrote: Hello, I just wanted to try osmosis and encountered an little error because I tried to launch it from my home directory (Windows). Exception in thread main java.lang.NoClassDefFoundError: und Caused by: java.lang.ClassNotFoundException: und at java.net.URLClassLoader$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) Could not find the main class: und. Program will exit. Well, the reason was, that the path contains whitespaces: C:\Dokumente und Einstellungen\user\Eigene Dateien\downloads\OSM\osmosis-0.34\bin Starting osmosis from c:\temp caused no problem (as expected). After changing the Set Exec-line in osmosis.bat to SET EXEC=%JAVACMD% %JAVACMD_OPTIONS% -cp %PLEXUS_CP% -Dapp.home=%MYAPP_HOME% -Dclassworlds.conf=%MYAPP_HOME%\config\plexus.conf %MAINCLASS% %OSMOSIS_OPTIONS% %* (just adding some around path variables) everything was fine. Just wanted to let you know. Thanks for the cool program anyways! Saved me quite some time. Regards Franz ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] extracting data in a polygon
On Tue, Mar 30, 2010 at 8:25 PM, Brett Henderson br...@bretth.com wrote: On Tue, Mar 30, 2010 at 10:04 AM, Nakor nakor@gmail.com wrote: On 03/28/2010 09:28 PM, Brett Henderson wrote: osmosis --read-xml enableDateParsing=no file=MI_limits_merged.osm --bounding-polygon file=mi_poly.txt --write-xml file=michigan.osm and am getting the following error message: SEVERE: Thread for task 1-read-xml failed java.lang.NoClassDefFoundError: org.openstreetmap.osmosis.core.filter.common.PolygonFileReader at java.lang.Class.initializeClass(libgcj.so.10) [...] Caused by: java.lang.ClassNotFoundException: java.awt.geom.Path2D$Double at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50) [...] What am I doing wrong? Can you give some more details about your installation? OS version, Java version (ie. java -version output), Osmosis version, etc. I am running Fedora 12 on a 64bit processor $ java -version java version 1.6.0_18 Java(TM) SE Runtime Environment (build 1.6.0_18-b07) Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode) INFO: Osmosis Version 0.34 Note that extracting with a bounding box works correctly here. A few things appear strange to me. - I'm also running Fedora 12 (albeit 32-bit) and don't have the same issue. - I don't think you're using the java version you think you are. I notice that the stack trace refers to libgcj.so.10. To the best of my knowledge that library is not used by a real Java VM, but by the GCJ version of java. - Your version output is considerably different to mine. Do you have the Fedora openjdk packages installed? Or have you installed java manually? Osmosis appears to be trying to run under the GCJ version of Java, and your non-openjdk java output makes me suspect a Java installation problem. There may be several reasons for this problem. - Do you have a /etc/osmosis or ~/.osmosis file lying around that is causing an alternative java to be invoked? - Have you installed java manually using a download from Sun/Oracle and not included it on the search path? Brett You can check which copy of Java Fedora thinks it is using with the following command: alternatives --display java It produces a lot of output, but the last line in my output is: Current `best' version is /usr/lib/jvm/jre-1.6.0-openjdk/bin/java. Yours will probably be different on a 64-bit OS, but unless you have a good reason for not using OpenJDK it should be similar. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problem using Osmosis34, Please Help
On Wed, Apr 21, 2010 at 5:03 PM, M Naveed Akram cmn...@gmail.com wrote: I can connect to my database explicitly, i think there is some problem with jdbc connection Sorry to ask, but did you run: mysql -h 127.0.0.1 -u root -p rose* *Or did you run:* *mysql -h 127.0.0.1 -u root -p rose *openstreetmap* The error seems to indicate that a successful connection is being made to the database server, but that that database schema doesn't exist. Can you verify that you can run a query against this schema? For example: *SELECT * from nodes;* I'm confused because I've never before seen a case where a successful connection is made to the server, but the schema can't be found. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problem using Osmosis34, Please Help
The MySQL support isn't well tested any more because PostgreSQL is now the preferred database. It is possible there's a problem there. Unfortunately I don't have a working MySQL installation to verify this. Do you have to use MySQL, or could you use PostgreSQL instead? Can anybody verify that Osmosis 0.34 still works with MySQL? On Wed, Apr 21, 2010 at 7:55 PM, M Naveed Akram cmn...@gmail.com wrote: i am using FreeBSD 7.1 Release i changed the db but same stuff..i have almost 4 dbs in place one with 0.5 schema and other with 0.6 schema i also tried with empty db.. On Wed, Apr 21, 2010 at 2:25 PM, Peter Körner osm-li...@mazdermind.dewrote: M Naveed Akram schrieb: It also gives the same error stack if i give wrong database information.. What OS are you running on? Are you using VMs or Jails or sth. like that? I thought about some strange SELinux error but that would prevent you from connecting. I also thought about a permission issue on the database-files but that would prevent mysql from working, too. Could you create another database and try with this? Or try to rename the openstreetmap database to any other name and try this? I'm not sure whether this is an osmosis problem or it's somewhere in the sql server. but the build failed.. saying one or more junit tests failed.. couple of failed tests are - org.openstreetmap.osmosis.core.pgsql.v0_6.PostgreSqlTest - org.openstreetmap.osmosis.extract.apidb.v0_6.DatabaseTimeLoader These tests always fail if you don't have the right postgres-db in place. Peter -- Regards M Naveed Akram http://www.google.com/profiles/cmnajs ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problem using Osmosis34, Please Help
I've just looked at your error message again. It appears to be using the PostgreSQL drivers instead of MySQL. That might explain the problem. Now I just need to figure out why it's not using the MySQL drivers ... On Wed, Apr 21, 2010 at 9:03 PM, Brett Henderson br...@bretth.com wrote: The MySQL support isn't well tested any more because PostgreSQL is now the preferred database. It is possible there's a problem there. Unfortunately I don't have a working MySQL installation to verify this. Do you have to use MySQL, or could you use PostgreSQL instead? Can anybody verify that Osmosis 0.34 still works with MySQL? On Wed, Apr 21, 2010 at 7:55 PM, M Naveed Akram cmn...@gmail.com wrote: i am using FreeBSD 7.1 Release i changed the db but same stuff..i have almost 4 dbs in place one with 0.5 schema and other with 0.6 schema i also tried with empty db.. On Wed, Apr 21, 2010 at 2:25 PM, Peter Körner osm-li...@mazdermind.dewrote: M Naveed Akram schrieb: It also gives the same error stack if i give wrong database information.. What OS are you running on? Are you using VMs or Jails or sth. like that? I thought about some strange SELinux error but that would prevent you from connecting. I also thought about a permission issue on the database-files but that would prevent mysql from working, too. Could you create another database and try with this? Or try to rename the openstreetmap database to any other name and try this? I'm not sure whether this is an osmosis problem or it's somewhere in the sql server. but the build failed.. saying one or more junit tests failed.. couple of failed tests are - org.openstreetmap.osmosis.core.pgsql.v0_6.PostgreSqlTest - org.openstreetmap.osmosis.extract.apidb.v0_6.DatabaseTimeLoader These tests always fail if you don't have the right postgres-db in place. Peter -- Regards M Naveed Akram http://www.google.com/profiles/cmnajs ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problem using Osmosis34, Please Help
It should be fixed now. Try a continuous integration build here: http://dev.openstreetmap.de:23457/hudson/job/osmosis-SNAPSHOT-ant/lastSuccessfulBuild/artifact/ It will be included in the 0.35 release when that occurs. On Wed, Apr 21, 2010 at 9:48 PM, Brett Henderson br...@bretth.com wrote: I've just looked at your error message again. It appears to be using the PostgreSQL drivers instead of MySQL. That might explain the problem. Now I just need to figure out why it's not using the MySQL drivers ... On Wed, Apr 21, 2010 at 9:03 PM, Brett Henderson br...@bretth.com wrote: The MySQL support isn't well tested any more because PostgreSQL is now the preferred database. It is possible there's a problem there. Unfortunately I don't have a working MySQL installation to verify this. Do you have to use MySQL, or could you use PostgreSQL instead? Can anybody verify that Osmosis 0.34 still works with MySQL? On Wed, Apr 21, 2010 at 7:55 PM, M Naveed Akram cmn...@gmail.com wrote: i am using FreeBSD 7.1 Release i changed the db but same stuff..i have almost 4 dbs in place one with 0.5 schema and other with 0.6 schema i also tried with empty db.. On Wed, Apr 21, 2010 at 2:25 PM, Peter Körner osm-li...@mazdermind.dewrote: M Naveed Akram schrieb: It also gives the same error stack if i give wrong database information.. What OS are you running on? Are you using VMs or Jails or sth. like that? I thought about some strange SELinux error but that would prevent you from connecting. I also thought about a permission issue on the database-files but that would prevent mysql from working, too. Could you create another database and try with this? Or try to rename the openstreetmap database to any other name and try this? I'm not sure whether this is an osmosis problem or it's somewhere in the sql server. but the build failed.. saying one or more junit tests failed.. couple of failed tests are - org.openstreetmap.osmosis.core.pgsql.v0_6.PostgreSqlTest - org.openstreetmap.osmosis.extract.apidb.v0_6.DatabaseTimeLoader These tests always fail if you don't have the right postgres-db in place. Peter -- Regards M Naveed Akram http://www.google.com/profiles/cmnajs ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Error with osmosis.bat when using paths that include spaces
HI Franz, I've applied the fixes now. Sorry for the delay. An interim release is available here: http://dev.openstreetmap.de:23457/hudson/job/osmosis-SNAPSHOT-ant/lastSuccessfulBuild/artifact/ Or at least it will be as soon as this build completes: http://dev.openstreetmap.de:23457/hudson/job/osmosis-SNAPSHOT-ant/59/ Brett On Mon, Mar 29, 2010 at 11:31 AM, Brett Henderson br...@bretth.com wrote: Hi Franz, Thanks for the error report. I haven't been able to test this out yet but I haven't forgotten. I'll get to it eventually and apply your changes. Brett On Sat, Mar 13, 2010 at 2:39 AM, Franz gr...@cip.ifi.lmu.de wrote: Hello, I just wanted to try osmosis and encountered an little error because I tried to launch it from my home directory (Windows). Exception in thread main java.lang.NoClassDefFoundError: und Caused by: java.lang.ClassNotFoundException: und at java.net.URLClassLoader$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) Could not find the main class: und. Program will exit. Well, the reason was, that the path contains whitespaces: C:\Dokumente und Einstellungen\user\Eigene Dateien\downloads\OSM\osmosis-0.34\bin Starting osmosis from c:\temp caused no problem (as expected). After changing the Set Exec-line in osmosis.bat to SET EXEC=%JAVACMD% %JAVACMD_OPTIONS% -cp %PLEXUS_CP% -Dapp.home=%MYAPP_HOME% -Dclassworlds.conf=%MYAPP_HOME%\config\plexus.conf %MAINCLASS% %OSMOSIS_OPTIONS% %* (just adding some around path variables) everything was fine. Just wanted to let you know. Thanks for the cool program anyways! Saved me quite some time. Regards Franz ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Osmosis Release 0.35
Hi All, I just released Osmosis 0.35. There are no major new features, just an assortment of bugs and improvements that should be useful to users. The key changes are listed below. Cheers, Brett *Changes* - Registered --fast-read-xml as a task instead of requiring --fast-read-xml-0.6 to be specified. - Fixed the completeRelations option on the --bounding-box and --bounding-polygon tasks to work as expected and documented. - Updated the --fast-read-xml task to ignore changeset elements. - Various enhancements to the Ivy processing including publishing of all artefacts. - Fixed apidb tasks to correctly select the MySQL drivers instead of always using PostgreSQL drivers. - Fixed the osmosis.bat launcher to handle spaces in the installation path. ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Removed 0.5 Support
Hi All, With the 0.6 API having been released for 12 months now, the time has come to remove 0.5 support from Osmosis. While 0.5 support hasn't caused much pain, I'm keen to simplify things where possible. Some before and after stats. 0.35 Lines of Java Code: 91,602 Number of Java Classes: 843 Size of osmosis.jar: 1,147,843 bytes 0.36-SNAPSHOT Lines of Java Code: 60,730 Number of Java Classes: 558 Size of osmosis.jar: 756,598 Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis Release 0.35
On Thu, Apr 22, 2010 at 11:17 PM, Frederik Ramm frede...@remote.org wrote: Brett, Brett Henderson wrote: There are no major new features, just an assortment of bugs That's how we love our Osmosis ;-) Hehe, oops. Bug *fixes* :-) Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] problems importing .osm files into postgis
On Sun, May 2, 2010 at 10:12 PM, _-_ txp_unimatri...@web.de wrote: Hi there, I've got a problem with importing .osm files into a postgis database. I think i set up the database correcly is, I used the pgsql_simple_schema_0.6.sql schema from osmosis. I also used the 900913.sql file. What is the 900913.sql file for? When i try to import files like this: osmosis --read-xml file=madagascar.osm --write-apidb host=localhost database=osm5 user=postgres password=12345 I'll get the following output/Error messages: # SEVERE: Thread for task 1-read-xml failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to create resultset. at org.openstreetmap.osmosis.core.apidb.common.DatabaseContext.executeQuery(DatabaseContext.java:429) at org.openstreetmap.osmosis.core.apidb.v0_6.impl.SchemaVersionValidator.validateDBVersion(SchemaVersionValidator.java:82) at org.openstreetmap.osmosis.core.apidb.v0_6.impl.SchemaVersionValidator.validateVersion(SchemaVersionValidator.java:55) at org.openstreetmap.osmosis.core.apidb.v0_6.ApidbWriter.initialize(ApidbWriter.java:288) at org.openstreetmap.osmosis.core.apidb.v0_6.ApidbWriter.process(ApidbWriter.java:1046) at org.openstreetmap.osmosis.core.xml.v0_6.impl.BoundElementProcessor.end(BoundElementProcessor.java:85) at org.openstreetmap.osmosis.core.xml.v0_6.impl.OsmHandler.endElement(OsmHandler.java:107) at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source) at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(SAXParser.java:195) at org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:108) at java.lang.Thread.run(Thread.java:636) Caused by: org.postgresql.util.PSQLException: ERROR: relation schema_migrations does not exist at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1592) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1327) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:451) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:336) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:235) at org.openstreetmap.osmosis.core.apidb.common.DatabaseContext.executeQuery(DatabaseContext.java:424) ... 20 more You are using the wrong task to write to the database. It doesn't support that schema. So don't use these tasks: http://wiki.openstreetmap.org/wiki/Osmosis/DetailedUsage#API_Database_Tasks Use these tasks instead: http://wiki.openstreetmap.org/wiki/Osmosis/DetailedUsage#PostGIS_Tasks Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Can't load osm file to PostGRESql
On Tue, May 4, 2010 at 12:30 AM, Ibrahim Bouchrika ibrahim_bouchr...@hotmail.com wrote: Hi, I'm trying to load a osm file to my postgresql database. I'm working on a windows xp system with dual core and 3GB of RAM and about 10GB of free space. I downloaded the netherlands.osm file from cloudmade.com, and extracted Amsterdam (which is now slightly bigger than 6MB) using Osmosis' latest version 0.35 : C:\osmosis-0.35\binosmosis --read-xml file=netherlands.osm --bb left=4.8828 top=52.3927 right=4.9501 bottom=52.3654 clipIncompleteEntities=true idTrackerType=BitSet --write-xml amsterdam.osm This worked without issues. I then proceeded loading the 0.6 schema and the linestring schema into a PostGIS database. After that I tried using the command line to load the extracted city into this database, using next command: C:\osmosis-0.35\binosmosis --read-xml amsterdam.osm --write-pgsql host= 127.0.0.1:5432 database=Amsterdam user=postgres password=1kilnmouthenableLinestringBuilder=yes nodeLocationStoreType=TempFile This used to work without issues (Last time I used it was a few months less then a year ago, but now it doesn't seem to work anymore, while the only thing that changed is the country/city i'm trying to load. I used to use version 0.31.1, which back then worked, but now doesn't seem to do the trick anymore. That's why I switched to the newest release, which doesn't seem to solve the problem. I copied the output below (excuse me for posting the whole printout, but I don't know for sure which information is relevant and which is not) Could anyone tell me how I can get by this problem? 'cause it's really bugging me that something which worked before now all of the sudden doesn't cut it. SEVERE: Thread for task 1-read-xml failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to execute statem ent. at org.openstreetmap.osmosis.core.pgsql.common.DatabaseContext.executeSt atement(DatabaseContext.java:107) at org.openstreetmap.osmosis.core.pgsql.v0_6.PostgreSqlWriter.complete(P ostgreSqlWriter.java:835) at org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java: 110) at java.lang.Thread.run(Thread.java:619) Caused by: org.postgresql.util.PSQLException: ERROR: could not create unique ind ex pk_ways Detail: Table contains duplicated values. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryEx ecutorImpl.java:1592) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutor The error message above is stating that you are trying to insert a way twice. This could be because your database already contains data, or that your data file contains multiple copies of the same way. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Can't load osm file to PostGRESql
On Tue, May 4, 2010 at 10:50 PM, Brett Henderson br...@bretth.com wrote: On Tue, May 4, 2010 at 5:56 PM, Ibrahim Bouchrika ibrahim_bouchr...@hotmail.com wrote: The database i'm using is a fresh made one, so the error must be duplicate entries in the osm. I have the same problem with a duplicate relation in the new york city file i extracted from a cloudmade map. Could the duplicate values be a result of a bug while extracting a bbox with osmosis? I doubt it, extracting a bbox should never create multiple instances of the same id. It's more likely to be caused by incorrect use of the --apply-change task when applying minute or hourly diff files. It was possible with older versions of osmosis to apply a full history diff containing multiple changes for a single entity which would result in multiple versions of the same entity being written to the result file. Is there a way around those duplicate entries, like perhaps overwriting duplicate data? Going through all those entries manually seems almost impossible. Hmm, I'm not sure about the best way to tackle this ... I seem to remember Frederik having a clever way of removing duplicates. I think he did something like the following: 1. Convert the entire file to an osc file by wrapping the entire contents of the file in osmChange element with an action of create or modify. 2. Feed the osc file through the --simplify-change task. 3. Create an empty osm file, then use the --apply-change task to add the entire change file to it. Found it. His steps were: quote Just for laughs - and wonder if you can come up with something better? - this is what I did to remove the duplicate objects from an existing OSM file named faulty.osm: echo osm/osm empty.osm osmosis --rx faulty.osm --rx empty.osm --dc --sort-change-0.6 --simc --wxc good.osc osmosis --rxc good.osc --rx empty.osm --ac --wx good.osm (The two osmosis steps could be written as one but that would probably make it more confusing.) /quote Can you try these steps on your cloudmade file and see if that fixes the problem? If it does, Cloudmade has a problem with duplicate data in their files. If it doesn't fix it, then we'll have to dig further :-) Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Can't load osm file to PostGRESql
Can you try downloading this file? http://www.bretth.com/empty.osm And then follow Frederik's instructions as below. osmosis --rx faulty.osm --rx empty.osm --dc --sort-change-0.6 --simc --wxc good.osc osmosis --rxc good.osc --rx empty.osm --ac --wx good.osm On Mon, May 10, 2010 at 8:38 PM, Ibrahim Bouchrika ibrahim_bouchr...@hotmail.com wrote: I tried finding another source of osm-data, but I haven't found anythig yet. openstreetmap itself only alows dumps of less then 50k nodes, and downloading the whole planet file is a bit of overkill (+ it wouldn't fit on my hard drive anymore). So I guess i'm stuck trying with the cloudmade file, in which case brett's proposal still is the best idea yet...if it would work properly. I still haven't figured out how to get that idea working, and I hope someone could help me further out on this! -- From: ibrahim_bouchr...@hotmail.com To: osmosis-dev@openstreetmap.org Date: Tue, 4 May 2010 14:29:51 + Subject: Re: [osmosis-dev] Can't load osm file to PostGRESql I deleted the first line of the file, which started with xml. The first line is now osm version=0.6 generator=Osmosis 0.31.1, but the same error persists. Date: Tue, 4 May 2010 16:09:50 +0200 From: frede...@remote.org To: ibrahim_bouchr...@hotmail.com CC: osmosis-dev@openstreetmap.org Subject: Re: [osmosis-dev] Can't load osm file to PostGRESql Hi, Ibrahim Bouchrika wrote: Your idea seams nifty, although it's giving an error on the --rx command. I don't know Osmosis well enough to figure out where the error comes from. Is it just an error in the syntax or is it because of the osm-file? Caused by: org.xml.sax.SAXParseException: Content is not allowed in prolog. Can you make sure your empty.osm and your amsterdam.osm begin with the characters osm and nothing else? Bye Frederik -- The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail. Get busy.http://www.windowslive.com/campaign/thenewbusy?tile=multiaccountocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4 -- Hotmail is redefining busy with tools for the New Busy. Get more from your inbox. See how.http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_2 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] build problem with 0.35 release
On Sun, May 16, 2010 at 6:05 AM, datendelphin mailingl...@osm.datendelphin.net wrote: Hello I tried to make an ebuild for gentoo for the recent osmosis 0.35. But ant fails building with ant build_binary Buildfile: build.xml BUILD FAILED /var/tmp/portage/sci-geosciences/osmosis-0.35/work/osmosis-0.35/build.xml:13: Cannot find build_support/build-ivy.xml imported from /var/tmp/portage/sci-geosciences/osmosis-0.35/work/osmosis-0.35/build.xml I tried the svn version, and there it works, because the build_support directory is included. It seems that it is missing in the source tarball. Hi Detendelphin, Thanks for the info. I've fixed this issue in SVN now. Are you able to use the current SVN version to build your package? Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Unexpected Database Migration
Thanks Shaun, I don't think it was in git when I checked last time. I didn't realise I sent this to the osmosis-dev list. I meant to send it to the main dev list. Oh well :-) Anyway, now I can fix the Osmosis on horntail and stop the warning emails occurring every minute ... Brett On Tue, May 18, 2010 at 8:25 AM, Shaun McDonald sh...@shaunmcdonald.me.ukwrote: Looks like it's just an index: http://git.openstreetmap.org/?p=rails.git;a=blob_plain;f=db/migrate/20100513171259_add_user_date_index_to_changeset.rb;hb=8657cd4724619b22a7c99b2400a0faa36229ae71 Shaun On 14 May 2010, at 02:07, Brett Henderson wrote: Can somebody tell me what migration 20100513171259 does in the production database? I can't see it in the git repo. I'm getting warnings from the Osmosis replication and before I turn them off for each migration I verify that the db change isn't in an area I care about. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] question about --merge
On Wed, May 19, 2010 at 4:29 AM, Christoph Wagner freemaps@googlemail.com wrote: Am 18.05.2010 15:18, schrieb Brett Henderson: To summarise, if you can draw a graph of the data flows in Osmosis, ensure that you never have data being split then recombined at a later point. In your case the only option is to split it into two steps using temporary XML files as you already appear to be doing. It would be nice if Osmosis could detect this situation and throw an error, but it would take a lot of effort and add a lot of complexity that I don't really wish to attempt. Ok, thanks for your reply. I think now I understand the situation better. While reading your answer I got an idea to solve the problem without writing to disk. I use a very big buffer now: osmosis-0.35.1/bin/osmosis --rx input.osm --t outPipe.1=points \ --wk keyList=addr:interpolation,addr:housenumber --un --s --b 1 \ --nk inPipe.0=points keyList=addr:housenumber --s --b 1 \ --m --wx addr.osm This works very well if you have enough RAM and set the bufferCapacity high enough. Yes, that will also work if you're prepared to experiment a bit with buffer sizes. Another possibility would be a better tagfilter that allows me to filter POIs and ways at once. The task would be for example: Give me all ways that match this way-filter and all nodes that match this node-filter or are part of the way that matches the way-filter The --tag-filter task is about the most comprehensive task currently available. To add a full boolean expression language is certainly possible, but not trivial. If somebody wants to take on the challenge of creating a complex yet generic tag analysis task then I'll do my best to help them out. I'll be honest though, it's not likely to be something I'll get around to implementing myself. Thinking about your case a bit more, you could avoid the temp file if you simply read the input file twice so you avoid the --tee. Although that might be slower than creating a temp file and invoking Osmosis twice. If it was me I'd simply create a wrapper shell script to tie several osmosis commands together and do some checking of the Osmosis return code to ensure it hasn't failed at each step. The --buffer approach might work, but the memory consumption is likely to bite you at some point. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] rails_port and Mapnik with one database
Hi Klemens, I believe the answer is no. Both schemas are serve different purposes and use a very different implementation. Perhaps not the ideal list for this question though given that neither schema is defined by Osmosis. Brett On Thu, Jun 10, 2010 at 1:34 AM, Klemens toural...@gmail.com wrote: Hi, This has probably been asked before but I haven't found an answer yet: I need both the OSM rails_port (for xml exporting) and the static maps from Mapnik on one server. What I've done so far is to use two databases, one importing data with osmosis, the other importing with osm2pgsql. I only need read-only databases and so far I have used only Europe and country .osm extracts. Now if I have two databases for the full planet.osm this takes up really a lot (too much) of disk space. Is there a way that both can use the same database? I estimate that the osmosis DB will have about 410 GB, and the osm2pgsql another ~115 GB. Thanks, Klemens ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] [OSM-dev] Problem in rake db:migrate with postgre
The Osmosis error message seems to indicate that the active column on the users table no longer exists. I haven't checked the recent migrations to figure out exactly what has changed. Currently when populating a datatabase, Osmosis creates any required users and makes them active (active = 1 I think). If active no longer exists, the replacement column will need to be populated with an equivalent value. Brett On Fri, Jun 11, 2010 at 6:43 PM, M Naveed Akram cmn...@gmail.com wrote: Brett: the issue is still there in the latest build of osmosis.. can u please guide a little if i want to resolve it On Mon, Jun 7, 2010 at 12:44 PM, M Naveed Akram cmn...@gmail.com wrote: can anyone tell me that to which schema migration version number the new osmosis works fine On Sun, Jun 6, 2010 at 7:35 AM, Brett Henderson br...@bretth.com wrote: On Fri, Jun 4, 2010 at 10:38 PM, Ævar Arnfjörð Bjarmason ava...@gmail.com wrote: On Fri, Jun 4, 2010 at 09:40, M Naveed Akram cmn...@gmail.com wrote: thanks to all UTF8 issue resolved. but now its osmosis Last I checked osmosis hadn't been fixed to work with the latest schema changes in the rails port, I run the Debian osmosis version, but it wasn't fixed in trunk either (at least the newest user group thing). Maybe that's what's going on, I just use an older version of the rails port for my importing purposes. Due to the way osmosis does imports it's fairly susceptible to breaking like this due to relatively small schema changes. I wasn't aware of the breakage until now. The database I use for unit tests is ancient. I won't get to this for a while, but if anybody else wants to take a look I'd appreciate it muchly. Brett -- Regards M Naveed Akram http://www.google.com/profiles/cmnajs -- Regards M Naveed Akram http://www.google.com/profiles/cmnajs ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Project Restructure
Hi All, I just created a breakup branch. I had several hours in an airport without much to do so started experimenting with a new way of packaging osmosis. One problem I have with the current single project approach is that every time a new feature is added, a whole bunch of new libraries need to be added and the distribution is starting to get fairly large. It's also reached the limit of what I can maintain myself given the limited time I have to spend on it. I'm hesitant to allow too many new tasks to be added because it gets hard to ensure that they're maintained properly and they're hard to remove once they're in there. So far the branch has the following projects: - core - Most of the existing project. - apidb - All of the apidb tasks (with postgresql dependencies), depends on core. - pgsnapshot - All of the pgsql tasks (with postgresql and postgis dependencies), depends on core. - package - Depends on the above three projects and adds the launch scripts and config files. The apidb and pgsnapshot tasks are no longer loaded directly via the TaskRegistrar class, but instead defined via a osmosis-plugins.conf file in the package project which specifies their plugin class names. The idea is that all tasks will be factored out of the core and into plugins which are bundled into the main distribution so that the end user doesn't see any difference. It doesn't compile yet due to some class dependency issues in the replication code, but I'll sort that out next. I aim to move the vast majority of existing tasks out into plugin projects. The plugins will loosely be aligned to the task groups defined on the detailed usage page on the wiki. http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage I'm hoping this will allow people to develop new plugins and incorporate them into the main build and distribution without too much effort, and lower the current barrier to entry by eliminating the need to pass my code review to make it into core. By keeping them out of the core, I can focus on core and the tasks I maintain and rely on others to keep their tasks up to date. If they fall behind and aren't maintained they can be removed from the build and distribution by changing a couple of config files instead of having to delete them entirely. The existing plugin methods (ie. raw plugins specified with -p on command line, and JPF plugins) will both continue to work so external plugins can still be developed if desired. Once I get it working I'll create a simple No-Op plugin task that provides an example of how the plugin projects are created. So far it looks like it should work fairly well without adding too much complexity. The ant builds for each project are becoming much smaller and more well defined which should be a good thing. Let me know if you see any major issues with this approach. Cheers, Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] error caused by Caused by: org.postgresql.util.PSQLException: ERROR: column active of relation users does not exist
Hi Michael, There were some changes made to the schema recently that have broken Osmosis. Fixing Osmosis is on my list of things to do, but I'm not sure when I'll get to do it. Brett On Wed, Jun 23, 2010 at 7:10 PM, Michael musset mickamus...@gmail.comwrote: Hi, I've a france.osm from geofabrik i'm trying to import it into my database with osmosis with this command : ./osmosis --read-xml file=/home/mickael/planet/france.osm --write-apidb populateCurrentTables=yes host=localhost database=openstreetmap user=openstreetmap validateSchemaVersion=no the osmosis come from : wget http://dev.openstreetmap.org/~bretth/osmosis-build/osmosis-bin-latest.tgz http://dev.openstreetmap.org/%7Ebretth/osmosis-build/osmosis-bin-latest.tgz tar xvfz osmosis-bin-latest.tgz cd osmosis-0.35 chmod a+x bin/osmosis bin/osmosis my railsport config is : test: adapter: postgresql database: osm_test # username: osm_test # password: osm_test host: localhost encoding: utf8 production: adapter: postgresql database: osm # username: osm # password: osm host: localhost encoding: utf8 development: adapter: postgresql database: openstreetmap # username: openstreetmap # password: openstreetmap host: localhost encoding: utf8 when i've created the database i've done : createdb -E UTF8 -O openstreetmap openstreetmap createdb -E UTF8 -O openstreetmap osm_test createdb -E UTF8 -O openstreetmap osm and psql -d openstreetmap /usr/share/postgresql/8.4/contrib/btree_gist.sql + rake db:migrate env RAILS_ENV=production rake db:migrate I don't understand what is wrong ! thx, ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] How to merge 2 osm files
Hi Sam, On Tue, Jul 20, 2010 at 6:09 AM, Sam Vekemans acrosscanadatra...@gmail.comwrote: Hi there, I was wondering how i can use osmosis to merge 2 osm files. I use Windows XP and want to use the command line for it. My guess is this osmosis --rx file2.osm --rx file1.osm --m --wx merged.osm That looks correct. Yet it does not work on my computer. Why? What is going wrong? Are you seeing an error? Are you getting an unexpected result file? When preforming this task, i need to make file1.osm and file2.osm in the 'osmosis' directory. Perhaps i need to use java -jar osmosis.jar --rx file2.osm --rx file1.osm --m --wx merged.osm java -jar won't work. You definitely need to use the osmosis.bat launcher. and i get a 'noClassDefFoundError similar to the errors in this thread http://www.mail-archive.com/newb...@openstreetmap.org/msg01252.html Unfortunatly, the wiki page tells users to download a planet.osm file. But that is just 1 function of the osmosis program, and not what im looking for. I dont have a planet.osm extract, nor a local PostgreSQL rails port database. http://wiki.openstreetmap.org/wiki/Osmosis Im just looking to take 2 .osm files and bring them into 1 .osm file. You're on the right track with the --m (also --merge) command. We just need to find out why that isn't working for you. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] postgresql base for osmosis
Hi Nakor,
You're getting an error earlier in the script:
psql:/tmp/pgsql_simple_schema_0.6.sql:42: ERROR: AddGeometryColumns() -
invalid SRID
CONTEXT: SQL statement SELECT AddGeometryColumn('','', $1 , $2 , $3 , $4 ,
$5 )
PL/pgSQL function addgeometrycolumn line 4 at SQL statement
You don't appear to have the PostGIS SRID tables populated. I don't have
details handy on how to do this, but your PostGIS distribution should
include a script for doing this ...
I'll look again later when I have a PostgreSQL system available.
Brett
On Thu, Jul 22, 2010 at 10:16 AM, Nakor nakor@gmail.com wrote:
Hello,
Is there a good documentation for how to setup a postgresql base for
osmosis. I've read thru OpenStrettMap wiki and it is quite hard to
understand waht exact schema you need to use.
I tired to use pgsql_simple_schema_0.6.sql but got errors (see below) and
then when I run osmosis it complains about index idx_nodes_geom not
existing.
Thanks in advance,
N.
-bash-4.1$ psql -d local -f /tmp/pgsql_simple_schema_0.6.sql
[...]
psql:/tmp/pgsql_simple_schema_0.6.sql:42: ERROR: AddGeometryColumns() -
invalid SRID
CONTEXT: SQL statement SELECT AddGeometryColumn('','', $1 , $2 , $3 , $4
, $5 )
PL/pgSQL function addgeometrycolumn line 4 at SQL statement
[...]
INSERT 0 1
psql:/tmp/pgsql_simple_schema_0.6.sql:111: NOTICE: ALTER TABLE / ADD
PRIMARY KEY will create implicit index pk_schema_info for table
schema_info
ALTER TABLE
psql:/tmp/pgsql_simple_schema_0.6.sql:113: NOTICE: ALTER TABLE / ADD
PRIMARY KEY will create implicit index pk_users for table users
ALTER TABLE
psql:/tmp/pgsql_simple_schema_0.6.sql:115: NOTICE: ALTER TABLE / ADD
PRIMARY KEY will create implicit index pk_nodes for table nodes
ALTER TABLE
psql:/tmp/pgsql_simple_schema_0.6.sql:117: NOTICE: ALTER TABLE / ADD
PRIMARY KEY will create implicit index pk_ways for table ways
ALTER TABLE
psql:/tmp/pgsql_simple_schema_0.6.sql:119: NOTICE: ALTER TABLE / ADD
PRIMARY KEY will create implicit index pk_way_nodes for table way_nodes
ALTER TABLE
psql:/tmp/pgsql_simple_schema_0.6.sql:121: NOTICE: ALTER TABLE / ADD
PRIMARY KEY will create implicit index pk_relations for table relations
ALTER TABLE
CREATE INDEX
psql:/tmp/pgsql_simple_schema_0.6.sql:126: ERROR: column geom does not
exist
[...]
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] postgresql base for osmosis
On Sat, Jul 24, 2010 at 2:44 AM, Nakor nakor@gmail.com wrote: On Fedora, the script to run against your database is /usr/share/pgsql/contrib/spatial_ref_sys.sql Brett, Thanks that helps. I was able to import data into the DB. BTW what are the other schema useful for? What other schema are you referring to? Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] other schemas delivered with osmosis (was postgresql base for osmosis)
On Sun, Jul 25, 2010 at 1:46 AM, Nakor nakor@gmail.com wrote: On 07/24/2010 04:59 AM, Brett Henderson wrote: On Sat, Jul 24, 2010 at 2:44 AM, Nakor nakor@gmail.com What other schema are you referring to? Brett The ones provided with osmosis like pgsql_simple_schema_0.6_bbox.sql They're all documented here: http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#PostGIS_Tasks Let me know if any of the descriptions aren't clear. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] How to merge 2 osm files
On Sun, Jul 25, 2010 at 3:23 AM, Sam Vekemans acrosscanadatra...@gmail.comwrote: cool, to start, the .bat file was actually in the sub directory called 'bin', i was expecting it to be in the root directory, instead of the .jar file which maybe should be in a sub folder? That is where the batch file should be. That's not a problem. i looked in the .bat file and i couldnt figure out what to replace. i'm not familar with the % syntax used. I can't help you until you tell me what error you're encountering :-) i'll move the .bat file over to the root directory, along with the .osm files and what should these files be named in order for it to merge to a single output.osm file? Moving the .bat file is not necessary. Neither is moving the .osm files. The .osm files can be called whatever you like so the names aren't important. If you provide the full command line that you're typing, along with the entire output of the Osmosis command I may be able to assist. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Patch: ReplicationLagReader
Hi Peter, You seem to have left my name as the author in some of the source code. Presumably that should be your name ;-) Can you let me know what the purpose of this is? Okay, I understand that this is for replication lag, but what is it comparing? On first read it appears to be comparing the latest server timestamp from the local timestamp, but I can't see the local timestamp ever being set. Is that correct? Brett On Sun, Jul 25, 2010 at 10:34 AM, Peter Körner osm-li...@mazdermind.dewrote: Hi To learn a bit about the internal structure of osmosis I developed a new task in the replication section. It's basicly the same as the little replag script [1] I use on my servers. I also added a munin plugin that plots the replication lag into a graph. On some systems the plugin needs some extra time (upper munin-node timeout) because of javas slow startup. I don't know if this task should be comitted to svn or if it's too simple (its task can also be fulfilled by that simple shell-script). I also don't know where to place the muinin plugin in the source tree and how to make it installed into the right place when using debuild to make a debian package. Please find the patch attached. Peter [1] http://svn.toolserver.org/svnroot/mazder/diff-import/replag ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Changes to Osmosis Pgsql Schema
Hi David, On Sat, Aug 7, 2010 at 10:10 PM, David Carmean d...@halibut.com wrote: Brett, et al: Please consider leaving the current simple schema intact (with the addition of the index clustering) and instead create a third schema for the HSTORE version. Losing the current schema would break at least two of my use cases: 1: I use postGIS as back-end storage for GIS client software, and use views and tables derived from queries against the OSM simple schema to create various layers of OSM data in the GIS client, e.g. roads, footpaths/trails/bike routes, hydrography, landuse, structures (buildings), etc. This is such an obvious technique that I'd be very surprised not to find a lot of others doing the same. It's great to hear you're finding this schema useful. I don't hear a lot from people using it, so getting feedback is always appreciated. Is it enough to have the existing 0.36 version of Osmosis to do this? Is it possible to change your queries to use the hstore data instead? I'm hoping to avoid maintaining lots of different schemas if possible. 2: I am beginning a project to parallelize OSM data processing with Hadoop, and the postgreSQL copy-format output is perfect for loading into HDFS. (If this goes well, I'd want to discuss ideas for adapting Osmosis to talk to Hadoop, eventually.) I also think that the simple schema lends itself better to research uses, especially when the input is a country/state extract or smaller. That said, I can also see cases where it would be useful to have the option to create a simple++ schema, with the separate tags table *and* the HSTORE columns as well. I did think about adding the hstore columns as an optional add-on to the existing schema. I already have a couple of cases of this (ie. action table, way linestring column, way bbox column). But this change is fairly invasive and given that I have to re-work a large amount of existing code I'd end up having to implement everything twice. If the only bit you need here is the COPY format, you may be better off with a new task that creates a file in exactly the format you want. The CLUSTER operation is indeed a big performance booster. Also, creating the optional ways.bbox column is at least two orders of magnitude faster when performed against an indexed ways.linestring column rather than as-shipped, which uses the nodes geometry before that has been indexed. The general intent is that you only create one of the linestring or bbox columns. If you already have a linestring column is there any point creating the bbox column? As for using the node geometry before indexing, you shouldn't have to do that. The fastest way to create the linestring or bbox columns will be to use the enableLinestringBuilder or enableBboxBuilder options on the various write-pgsql tasks. Cheers, Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Reading OSM History dumps
Hi Peter, This all sounds very interesting and will no doubt have many uses that I can't anticipate. I can't give you much assistance but will try to answer any specific questions you have. My wife is going to give birth sometime within the next month which means my priorities are about to change drastically ;-) You seem to have thought about most of the complexities of the problem already so you know what you're dealing with. You mentioned the problem of obtaining test data. I'd suggest using: http://planet.openstreetmap.org/history/ That is a full history from day one of the project up until now. It is already in the OSM change format that Osmosis understands. Cutting bounding boxes out of full history data is a difficult (but not impossible) problem that you may have to solve in order to move forward. In order to build way linestrings for all way versions and for all node versions impacting the way you will have to solve a similar problem to understanding how to cut bbox data so you may be able to kill a couple of birds with one stone. One thing to note is that I'm currently changing the simple schema a bit to improve performance. I've moved the tags into hstore columns, and have duplicated the way_node table info into a nodes array column on the way table. This improves bounding box style query performance by several times on large datasets. I don't think it will impact you too much. Good luck! Cheers, Brett On Sun, Aug 22, 2010 at 12:18 AM, Peter Körner osm-li...@mazdermind.dewrote: Hi I during the last week I thought intensively about the new full history dump and how we could use it. I wrote some kind of paper and also some demo code to check how we could get osm history information into a postgis database with linestings and all this delicate features osmosis offers. I've put it on the wiki at http://wiki.openstreetmap.org/wiki/User:MaZderMind/Reading_OSM_History_dumps I'd love to hear some comments about it. Peter ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Parsing the JOSM file format
Hi Michael, The JOSM format differs from the Osmosis format in a number of ways, not just version information. In order to open a JOSM file properly you really need to handle other information including the extra action attribute. I try to keep Osmosis pure wherever possible to avoid lots of special case coding, avoid weird corner cases, and keep the behaviour simple and predictable. For that reason I'm hesitant to add JOSM specific logic to the parser. I'd prefer to treat the JOSM file format separately and create a new task capable of reading JOSM (eg. --read-josm-xml). Brett On Sun, Aug 22, 2010 at 7:27 PM, Michael Zilske michael.zil...@gmail.comwrote: Hi, I'm using Osmosis for extracting road networks and public transit routes from OSM and converting them to our own file format with my own Sink. (We do transportation research.) We work on an API 0.6 planet extract .osm file which we sometimes open with JOSM to manually add some tags of our own, e.g. as conversion hints for our application. We're not checking this in, because it's of no use to anyone else. Now my problem is that the Osmosis FastXMLParser chokes on the .osm files saved by JOSM, because newly added entities get neither a version nor a changeset attribute in JOSM's interpretation of the .osm format. I helped myself by patching the Parser to simply ignore the fact that these mandatory attributes aren't there. Is there a better solution to this? Would it be an option to integrate this into Osmosis, or don't you want to parse these improper files? Cheers, Michael ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Reading OSM History dumps
On Tue, Aug 24, 2010 at 1:09 AM, Peter Körner osm-li...@mazdermind.dewrote: Am 23.08.2010 13:35, schrieb Brett Henderson: To create your own store implementation you can build on the Osmosis persistence support. All classes that are persistable implement the Storeable interface and have a constructor with StoreReader sr, StoreClassRegister scr arguments. The existing IndexedObjectStore assumes that the key is a long but provides a good example to start from. The underlying IndexStore it uses can support any type of key as long as it has a fixed width (ie. always persists to the same number of bytes). It would need a key of 96 bit (id long + version int). I was not aware of any type 64bit in java so I'm not sure how I could build a store with a 96bit index, but I think I have to take a deeper look into the IndexStore company. IndexStore just requires an IndexElement implementation that holds both the key and the value. You can define a key implementation class that holds as many individual long or int values as you like, so long as it persists through the Storeable interface to a fixed number of bytes. You also have to provide the IndexStore with a comparator that knows how to compare the order of two keys. The timestamp is just a 64bit long value, so the only problem is here to do the comparison but this is the easy past, i think. It may be possible to make the existing IndexedObjectStore more generic but I'd need to experiment with it. I'll try to keep the whole changes local to my project. Once its finished you can take classes over to core as they're needed. Hmm, but thinking more about your problem it may make more sense to stick with the IndexedObjectStore and store a list of Nodes as each element instead of single Nodes. I suspect in most cases you won't know the exact version you're looking for when you're loading a Node In the first phase when selecting the versions of the nodes used to create a version of a way I'll have a lot of timestamp searches (find the oldest node that is younger then the timestamp of the way) that need the timestamp index. later on, when the intermediate versions are calculated, i'll need a lookup for all versions of an id. a direct request for a known id/version will, as far as I see in this early stage, not be used too often (maybe during linestring building) (you'll only know node ids when looking at a way after all), and will only know a timestamp range. When looking up a specific node/version/timestamp combination you would have to load all versions of a node from the IndexedObjectStore then linearly search for a match in the (usually fairly limited) list of objects. You will possibly need to create you own Storeable list type to hold all versions of a particular Node because I don't think one exists. The main problem I see is, that such a list won't be of fixed size. When I write it to the store and later on add another version, it will grow bigger and have to be re-allocated in the store file, freing up space at the beginning. Basically a malloc/realloc/free in files. If you need the ability to write values randomly then it won't work. But if you have sorted input (ie. all versions of a node are together on input) then you can write them all to the store at once. IndexedObjectStore will allow you to write variable length objects to the store which is already necessary to hold entities with variable numbers of tags. Just keep in mind that Osmosis stores aren't particularly fast to query because they're based on very simple data structures. They tend to result in huge amounts of disk seeks when processing, so there may be libraries out there that perform better. The main reason they were originally developed was to minimise external library dependencies and I haven't revisited that decision since Osmosis put on weight (ie. it now relies on many third-party jars). Thinking about all this I find that we're re-inventing the wheel. I'll try to use a JavaDB as the backend store. It is entirely written in Java ant thus cross platform compatible, supports btree indexes on multiple fields an can reside both, in-memory and on-disk. If it shows that it's fast enough, it may be a good alternative to a custom binary file/memory store. I hope it works out because I've been down a similar path here. After I gave up on custom stores I tried Berkeley DB Java edition and performance was horrible. I finally bit the bullet and went the PostgreSQL path and created the pgsql tasks. I hope JavaDB works out though because requiring a full database server really complicates usage. Be very careful with btree indexes on multiple fields because they usually only work well when you're looking up values for specific values of indexed fields. If you ever need to use range queries (eg. timestamp range) involving multiple fields they tend to fall down. I suspect you'll be just as well off
Re: [osmosis-dev] Reading OSM History dumps
On Wed, Aug 25, 2010 at 11:14 PM, Peter Körner osm-li...@mazdermind.dewrote: Brett, the pgsql tasks currently write (in COPY mode) all data to temp files first. The process seems to be PlanetFile - NodeStoreTempFile - CopyFormatTempFile - PgsqlCopyImport in osm2pgsql the copy data is pushed to pgsql via unix pipes (5 or 6 COPY transactions running at the same time in different connections). This approach skips the CopyFormatTempFile stage. Is there any special reason this approach isn't used in the pgsnapshot package? Not too sure now :-) I think it was the simplest way to share code between both the --write-pgsql-dump task and what was then the --fast-write-pgsql (now simply --write-pgsql) task. In practice the COPY file creation and loading is fairly fast. The biggest downside is the extra disk space. The slowest parts of the whole process are the way geometry creation, index building, and the CLUSTER statements (in the newest schema). On relatively low-end hardware it takes many days to import an entire planet, only a small part of which is the COPY processing. In most cases I create the COPY files using --write-pgsql-dump and load them via the provided load script so that I can better monitor progress and resume if processing is interrupted. In short it just hasn't been a high priority to change it. While I'm on the topic, I've mostly completed the changes to the schema now. Performance is drastically improved over the old version for bounding box query processing. The --read-pgsql --dataset-bounding-box task combination would previously take approximately an hour to retrieve a 1x1 degree box in a populated area, now it is down to around 5 minutes due to far better disk layout. The biggest downside is that the table clustering takes a long time during initial database creation. ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Reading OSM History dumps
On Wed, Aug 25, 2010 at 11:33 PM, Peter Körner osm-li...@mazdermind.dewrote: Am 25.08.2010 15:26, schrieb Brett Henderson: In short it just hasn't been a high priority to change it. I was planning to share on FileInputStream/FileOutputStream level. You can feed a FileInputStream into the CopyManager as well as into a file, can't you? Sorry, I'm not sure what you mean. I think the only way to feed data into the CopyManager is via an InputStream. That InputStream can be a FileInputStream or a piped input stream or whatever you wish. But there are also classes like PGCopyOutputStream so perhaps you can use those directly to avoid using multiple threads. It's been a while since I looked at it. Maybe want to can copy the relevant bits later to pgsnapshot. Yep, sure. ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Reading OSM History dumps
On Thu, Aug 26, 2010 at 8:19 AM, Peter Körner osm-li...@mazdermind.dewrote: Hi Marco The first snapshot is out. Unfortunately the hstore migration progress Brett is still in let the pgsnapshot tests fail, which is why hudson is not providing nightly builds anymore. I hope to have this fixed over the next few days. I'm working with the server admins to get hstore support added to the database. Because of that you'll need to compile osmosis yourself. I attached instructions to this mail that also include the concrete plugin usage. If you wish to avoid compiling yourself you can also get nightly builds from: http://bretth.dev.openstreetmap.org/osmosis-build/ The 0.37.SNAPSHOT version in the above location is built via a cron job. No tests are run during the build. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] keyValueList option, and latest node versions
Hi Andrzej, I finally got around to checking out your patch. I've just checked it in. I changed the name of the task to --flatten or --f for short because I don't believe any other tasks have filter in the name. I notice that it assumes that data is sorted. Ideally it should utilise something like the SortedEntityPipeValidator class to ensure that data is sorted and to throw an error if it isn't. Do you have time to add that in? Brett On Wed, Aug 4, 2010 at 2:20 AM, andrzej zaborowski balr...@gmail.comwrote: On 6 July 2010 23:26, Peter Körner osm-li...@mazdermind.de wrote: Am 06.07.2010 19:41, schrieb Simon Nuttall: Are there osmosis options for just catching the latest versions of the nodes and ways? It's called --simplify-changeset: http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.35#--simplify-change_.28--simc.29 That's for change streams. For entity streams you can use the --ff filter I sent in a patch attached to http://lists.openstreetmap.org/pipermail/osmosis-dev/2010-May/000587.html It only works on sorted streams. Cheers ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] keyValueList option, and latest node versions
I've applied the patch, modified it to use a new SortedDuplicateEntityPipeValidator class, and checked it all in. I haven't tested it though ... On Fri, Sep 3, 2010 at 10:20 PM, Brett Henderson br...@bretth.com wrote: Hi Andrzej, On Thu, Sep 2, 2010 at 12:45 PM, andrzej zaborowski balr...@gmail.comwrote: Hi Brett, [I tried sending this yesterday with git-send-email but the list doesn't seem to accept, either way sorry if you get a duplicate] On 30 August 2010 13:15, Brett Henderson br...@bretth.com wrote: I finally got around to checking out your patch. I've just checked it in. I changed the name of the task to --flatten or --f for short because I don't believe any other tasks have filter in the name. Thanks. I seem to remember there were some but I may be talking about a terribly old version. (tagfilter?) Yes, there is a --tag-filter task. So there is one other task with filter in the name ... Oh well, there's less typing now :-) I notice that it assumes that data is sorted. Ideally it should utilise something like the SortedEntityPipeValidator class to ensure that data is sorted and to throw an error if it isn't. Do you have time to add that in? The attached diff makes it use SortedEntityPipeValidator, however SortedEntityPipeValidator bails out immediately on multiple objects with the same id so it doesn't work. Possibly I need a new validator, is my thinking correct? I'm sending the patch anyway to see if you're ok with using the validator this way. Yes, it looks like it will need a new validator that allows two versions of the same entity in a row. I don't mind how you use the validator. I'm not too fussy about the internal implementation of tasks. Your usage is fine. For an example of how I've used it in the past check out the org.openstreetmap.osmosis.dataset.v0_6.impl.DatasetStore class. But yours might be better actually. The reason I didn't add it in the first place was that I generally trust the data I throw at osmosis so don't want to use cycles for testing, I'm missing a way to globally turn off creation of the validators. Each of them adds at least a full function call per entity. I try to avoid adding options without good reason. If you can show that the additional validation slows things down then we can look at ways of disabling them, but I highly doubt that the performance impact will be measurable. I think that one extra function call per entity will be lost in the noise. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Boxing accuracy
Hi Zenon, On Wed, Sep 1, 2010 at 11:33 AM, Zenon Panoussis ora...@provocation.netwrote: Hi everyone I just subscribed and, as you might expect, I'm coming here with a problem. I think it's an osmosis bug, but I can't say that for sure. I find the planet.osm file completely unworkable because of its sheer size, so I used osmosis to split it into 720 more manageable parts, which I then imported into a postgis/mapserver-utils database. Looking at the result, I have lost about 2 meridian minutes of data along each meridian, one on each side of it. A couple of images explain it best: http://www.xs4all.nl/~oracle/gap_manaus.pnghttp://www.xs4all.nl/%7Eoracle/gap_manaus.png http://www.xs4all.nl/~oracle/gap_lima.pnghttp://www.xs4all.nl/%7Eoracle/gap_lima.png Both Lima and Manaus lie smack on a meridian. Those bands of nothingness in the middle of densly populated cities are not the result of some conspiracy among osm data contributors. Here's the original dataset: http://osm.org/go/NNZblPk-- http://osm.org/go/N2dmLuy-- Thus, my import ate up some of the data. This is what I did in the specific part of the world where Manaus lies: --rx planet.osm \ --tee 2 \ --bp file=S-78--77.pol \ --wx S-78--77.osm \ --bp file=S-77--76.pol \ --wx S-77--76.osm These are the polygon files referenced: S-77--76.pol 1 -77.0 -90.0 -77.0 0.0 -76.0 0.0 -77.0 -90.0 END END S-78--77.pol 1 -78.0 -90.0 -78.0 0.0 -77.0 0.0 -78.0 -90.0 END END As you can see, I was just slicing meridian cake triangles from the equator to the poles. Note that the overlap between the polygons is a one-dimensional line, not an area. Um, don't your polygons only touch at a single point (-77,0) and not along a one dimensional -77 longitude line as you suggest. The two polygons touch at the equator, drift apart as they travel south, then touch again at the south pole. There's a gap between them which would explain the missing data. Have I misinterpreted something here? The resulting .osm files were imported into postgres with for i in *.osm; do osm2pgsql -d osm -a -s -v -S /usr/share/osm2pgsql/default.style \ -k -G $i done The osm2pgsql binary was built out of yesterday's SVN with the insert-as-modify patch at http://www.mail-archive.com/d...@openstreetmap.org/msg10349.html . Theoretically, my meridian gap could be caused by osmosis, by osm2pgsq, or by any of their underlying libraries. My gut feeling though is that it's caused by a rounding error in osmosis or by an accuracy error in its underlying routines. Osmosis uses the http://download.oracle.com/javase/6/docs/api/java/awt/geom/Area.html class to manage the polygon, and it is constructed from the http://download.oracle.com/javase/6/docs/api/java/awt/geom/Path2D.Double.htmlclass which provides double precision coordinates. It will only treat objects lying along some edges of shapes as being inside the area (ie. left edge of a polygon is included, but right edge isn't). This should mean that in cases of touching polygons data should only be included in one of the two. See http://download.oracle.com/javase/6/docs/api/java/awt/Shape.htmlfor more details. That's not to say there isn't a bug in the Sun *cough* Oracle libraries, but it's certainly not custom code that I've written. On that note, I suspect that data along the international date line might encounter issues if you specify a polygon right edge of exactly 180 degrees because data on that line will then be excluded. But that's not your issue. The solution in that case is probably to specify a right edge of 181 degrees to work around the problem. Ideas anybody? Please check your polygons again. And if you still think they're right please let me know where I've gone wrong :-) Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problem with 0.36
On Mon, Sep 13, 2010 at 4:57 PM, Frederik Ramm frede...@remote.org wrote: Hi, Frederik Ramm wrote: I'll debug this further - currently unsure if there's something wrong with the input data or what. The problem occurs about 1h50m hours into processing the job. so it looks more like it is towards the end. Ok, that was easy; it looks like clipIncompleteEntities is broken in 0.36 because Way.getWayNodes is documented as returning a read-only list, and AreaFilter does this if clipIncompleteEntities is true: I should read all of my emails before responding :-) ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Planet import
On Tue, Sep 28, 2010 at 10:25 AM, Nakor nakor@gmail.com wrote: On 09/24/2010 09:52 AM, Nakor wrote: Brett, I should have read the whole documentation first. I am using --write-pgsql . I'll let it continue (as I am now over 200 Gb) and see how it goes. Next time I need to import I will probably just import the US vs the whole planet. Thanks for your input it gives a rough idea. Thanks, N. It finished in 7 days and 17hours. The DB is 260Gb. Note that some speed loss is probably due to the fact that the only disk where I had space to store the DB is encrypted. Okay, cool. Which version of the schema are you using? Is it the version 5 schema that comes with Osmosis 0.36, or the version 6 schema that comes with Osmosis 0.37-SNAPSHOT? The version 6 schema takes much longer to import if you have the optional bbox or linestring columns on the ways table, but the query performance is much improved due to table CLUSTERing. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problem with bounding-polygon and completeWays=yes
On Sat, Oct 2, 2010 at 3:06 PM, Brett Henderson br...@bretth.com wrote: On Sat, Oct 2, 2010 at 1:47 AM, Martin Kaistra ad...@djfun.de wrote: Hi, when I try this command ./osmosis --read-xml file=friedrichshafen_highways.osm --bounding-polygon file=Friedrichshafen.poly completeWays=yes clipIncompleteEntities=true --write-xml file=friedrichshafen_output.osm with Osmosis 0.36, there are missing ways in the output file. When I try the same command with Osmosis 0.35.1 it works as expected. This is the first time I use osmosis so I am not sure whether it is a bug or I am making a mistake somewhere.. The files can be found here: http://djfun.de/osm/friedrichshafen_highways.osm [5.1mb] http://djfun.de/osm/Friedrichshafen.poly http://djfun.de/osm/friedrichshafen_output_0_36.osm [2.6mb] http://djfun.de/osm/friedrichshafen_output_0_35_1.osm [2.9mb] For example way:14805086 is missing in friedrichshafen_output_0_36.osm It looks like a bug to me. If you specify the option idTrackerType=BitSet or idTrackerType=IdList the way shows up correctly. That is a workaround you can use for now. It might take me a while to figure this one out. There were a couple of bugs in there. The first was that it wasn't handling unsorted ids correctly. The input file has ids in non-sorted order. That caused your way to be left out. But the output still wasn't correct and I found another bug that was causing some nodes to be left out. Both have been fixed in the latest SVN version. So you can either use the latest dev version, or simply use the idTrackerType=BitSet for now. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis 0.37 Released
On Sat, Oct 16, 2010 at 9:41 PM, Jochen Topf joc...@remote.org wrote: On Sat, Oct 16, 2010 at 05:33:37PM +1100, Brett Henderson wrote: - Updated the pgsql schema (now version 6) to move all tags into hstore columns, add a ways.nodes column and CLUSTER the nodes and ways tables. Significant performance increases. Is there some magic involved or do you just call CLUSTER after the import? When I tested CLUSTER it helped with reads, but creating the cluster was very expensive. No magic unfortunately. The CLUSTER step is the most expensive part of the process because it changes the location on disk of every record in the database. It took several days for a full planet when I tested it out. I was performing bbox queries (similar ROMA or TRAPI but with my implementation that uses true way geometries instead of just node locations) and 1x1 degree queries for heavily populated areas (eg. Munich, London) were taking up to an hour to perform. After removing tags tables and performing CLUSTER statements the time reduced to somewhere between 5 and 10 minutes due to much reduced disk seeking. The database is kept up to date with hourly diffs so the CLUSTER step only has to be performed once (or once every few months perhaps) but the read benefits are ongoing. If you're performing regular full imports it may not make sense. If the overhead of performing the CLUSTER step outweighs the read benefits (depends on the lifetime of your database) then you can skip the CLUSTER step. It makes no functional difference that I'm aware of. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] lost relations while clipping
Hi Maxim, 2010/10/12 Maxim Dubinin s...@gis-lab.info Hi osmosis-devs, On our regional extraction system we've ran into what seems to be a bug. Relations, that are members by themselves are dropped while extracting. Example of such relation: http://www.openstreetmap.org/browse/relation/358215 Code and example to reproduce this behavior is attached. Thanks for the detailed example, it makes it much easier to diagnose your issue. The reason for the dropped relation is a limitation of how Osmosis performs bounding-box/bounding-polygon processing. Osmosis processes all data in a stream. First it processes all nodes one by one and keeps track of which nodes are in the bounding box. It then processes each way one by one and uses the node list to determine which ways are in the bounding box. Then it processes the relations one by one and uses the list of nodes, ways, and already selected relations to determine if a relation is in the bounding box. In your example, relation 795 is processed first and contains relation 792. But 792 hasn't been processed (in your example, the two ids are out of order) yet so Osmosis doesn't know that it will be inside the bounding box and ignores it. It then processes 792 which has two ways inside the bounding box so it includes that relation. But it never goes back to include relation 795 because it has forgotten about it. In your example, if you switched the order of relation 792 and 795 then 795 *would* get included. The only way to avoid this limitation is to replace your existing --bp step: --bp file=test.poly with --bp file=test.poly completeWays=yes completeRelations=yes This makes Osmosis run *much* slower and might not be acceptable. With those options enabled, it builds temporary stores of nodes, ways and relations so that it can go back and randomly access data. But the temporary data stores are quite slow which has a very large impact on performance. Hope that helps. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problem with --node-key-value
I've just taken a look at the code (I didn't write these tasks) for the --node-key-value task and it works slightly differently to the --way-key-value task. The --way-key-value task passes through node and relation objects unchanged, but the --node-key-value task drops them. That means that when you have a --node-key-value task followed by a --used-node task you are going to drop everything. Note that it doesn't make sense to use the --used-node task in this case anyway because place nodes will typically not be part of a way. So two suggestions: 1. Remove the --used-node task. 2. Use the --tag-filter task instead which is much more flexible. Cheers, Brett On Sun, Oct 17, 2010 at 1:15 AM, Flex Fauler faulerf...@gmx.de wrote: Hi, i am new here so sorry if i mail to the wrong place. I wanted an osm file with all places=* so city, town etc. But i am getting an empty out.osm. What am i donig wrong? With ways (--way-key-value) this works great. terminal command: osmosis --read-xml in.osm --node-key-value keyValueList=place.city,place.town,place.village,place.hamlet,place.suburb --used-node idTrackerType=BitSet --write-xml out.osm out.osm: ?xml version='1.0' encoding='UTF-8'? osm version=0.6 generator=Osmosis 0.36 bound box=49.39089,7.76757,51.66252,10.24398 origin=0.37-SNAPSHOT/ /osm regards faulerflex -- GMX DSL Doppel-Flat ab 19,99 euro;/mtl.! Jetzt auch mit gratis Notebook-Flat! http://portal.gmx.net/de/go/dsl ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis on Windows
Hi Nakor, Thanks for the report. On Fri, Nov 5, 2010 at 2:39 AM, Nakor nakor@gmail.com wrote: Hello, Lately I had to use osmosis on Windows. I found a few issues in osmosis.bat that I want to report. 1) lines 14 and 19 need to have extra quotes: IF EXIST %ALLUSERSPROFILE%\osmosis.bat CALL %ALLUSERSPROFILE%\osmosis.bat IF EXIST %USERPROFILE%\osmosis.bat CALL %USERPROFILE%\osmosis.bat I've made this change and checked it in. Is it enough to only quote the second half of the lines, or should they look like this? IF EXIST %ALLUSERSPROFILE%\osmosis.bat CALL %ALLUSERSPROFILE%\osmosis.bat IF EXIST %USERPROFILE%\osmosis.bat CALL %USERPROFILE%\osmosis.bat 2) around line 28-35, MYAPP_HOME will not be correct if the starting directory is not on the same drive where osmosis is: C:\tmpset SAVEDIR=C:\tmp C:\tmpset MYAPP_HOME=P:\OSM\osmosis-0.37\bin\.. C:\tmpREM Now make the MYAPP_HOME path absolute C:\tmpcd P:\OSM\osmosis-0.37\bin\.. C:\tmpset MYAPP_HOME=C:\tmp C:\tmpREM Change back to the original directory C:\tmpcd C:\tmp C:\tmpset MAINCLASS=org.codehaus.classworlds.Launcher C:\tmpset PLEXUS_CP=C:\tmp\lib\default\plexus-classworlds-2.2.2.jar C:\tmpSET EXEC=java -cp C:\tmp\lib\default\plexus-classworlds-2.2.2.jar -Dapp.home=C:\tmp -Dclassworlds.conf=C:\tmp\config\plexus.conf org.codehaus.classworlds.Launcher --read-pbf file=E:\Downloads\OSM\Canada_101103.osm.pbf --bounding-box left=-83.08 bottom=42.30 right=-83.03 top=42.33 --write-xml file=E:\Downloads\OSM\Windsor.osm C:\tmpjava -cp C:\tmp\lib\default\plexus-classworlds-2.2.2.jar -Dapp.home=C:\tmp -Dclassworlds.conf=C:\tmp\config\plexus.conf org.codehaus.classworlds.Launcher --read- pbf file=E:\Downloads\OSM\Canada_101103.osm.pbf --bounding-box left=-83.08 bottom=42.30 right=-83.03 top=42.33 --write-xml file=E:\Downloads\OSM\Windsor.osm Exception in thread main java.lang.NoClassDefFoundError: org/codehaus/classworlds/Launcher Caused by: java.lang.ClassNotFoundException: org.codehaus.classworlds.Launcher at java.net.URLClassLoader$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) Could not find the main class: org.codehaus.classworlds.Launcher. Program will exit. Do you have any idea how to fix this second issue? Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Improve relation extraction handling
Hi Frederik, On Sun, Nov 7, 2010 at 10:44 PM, Frederik Ramm frede...@remote.org wrote: Hi, I'm a major user of the --bp task in Osmosis, for the nightly Geofabrik country extracts. I cannot use the complete-something flags as that would take far too much time, and on the whole the clip-incomplete-entities mode seems not to be too bad for users. There's one thing that has popped up on this list however (lost relations while clipping) which is an increasing concern to our users and which I hope I can fix. The problem is that with cascading relations, Osmosis will only include a relation if one of its children has been included already, so if node N1 is inside the clipping area, relation R1 contains node N1, relation R2 contains relation R1 then relation R1 will alway be included in the output, but relation R2 will only be included if it comes after R1 in the input file. Now while using the temporary storage for nodes and ways would be too expensive for the nightly Geofabrik build, I think that temporarily storing relations would be acceptable (as there aren't too many). I'm thinking of adding a cascadingRelations=yes flag to the --bp/--bb tasks that would * process relations not containing other relations as usual, * throw all other relations onto a stack and process them in a suitable order afterwards I am aware that determining a suitable order could be difficult if you have many levels of cascasing (or even circular) relations, and that the output file would not be ordered properly, but I think such an option would mean a big improvement at a relatively low cost. 1. Any comments? It sounds good to me. 2. Anyone done something like that already? Not exactly, although it should be very similar to the existing completeXXX functionality. 3. Is any of Osmosis' internal storage mechanisms suitable for buffering relations in this way or do I have to invent something new? I'd try to use the existing completeXXX options as a starting point. The existing file system stores should work for your purposes. If their performance is not good enough then it might be necessary to investigate other options. In that case if you did come up with a higher performance design it might also be able to be used for the completeXXX options as well. Cheers, Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Changes to Plugin Loader
Hi Everybody, I've made a simple but significant improvement to the way Osmosis loads tasks and plugins. Previously there was a file called osmosis-plugins.conf in the config directory that listed out all the plugins to be loaded during startup. This was the way that the tasks for each individual sub-system (ie. xml, apidb, pgsql, dataset, etc) were loaded. In other words, only a single osmosis-plugins.conf file was loaded so only one could exist on the classpath. I've changed that now so that all instances of osmosis-plugins.conf on the classpath will be processed. This allows each project (and jar file) to provide its own osmosis-plugins.conf file. I've modified all projects to include their own copy, and have removed the global copy. The implication of this is that creating new plugins is as simple as creating a new jar file with your tasks, your plugin loader class, and an osmosis-plugins.conf file. Installing them is just a matter of adding the new jar to the classpath (ie. in the lib/default directory). Note that this has nothing to do with the JPF (Java Plugin Framework) support provided by Osmosis. JPF is a far more capable (and complex) system that is unchanged and will continue to be supported (although to be honest I have very little idea how it works ;-) Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis 0.37 and enableLinestringBuilder error
It looks like the wiki needs a fix up :-) Sorry about that. You don't need to specify that option. The code *should* (been a while since I looked at it) figure out what columns you have in the database dynamically. There is no need to specify such an option. Just remove the option and try again. On Thu, Nov 18, 2010 at 2:42 AM, Emilie Laffray emilie.laff...@gmail.comwrote: Hello, I have been trying to run the following command line (the host information was removed from the command line): osmosis --read-pbf file=roads.osm.pbf --bounding-box top=90 bottom=-90 left=-180 right=-126 completeWays=yes idTrackerType=BitSet --write-pgsql enableLinestringBuilder=yes nodeLocationStoreType=TempFile But I run into a problem as write-pgsql doesn't seem to support the parameter enableLinestringBuilder. The parameter as far as I can tell from the documentation is still here: http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--write-pgsql_.28--wp.29. I am getting the following error though. I am not quite sure what is wrong. Nov 17, 2010 4:27:00 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.37 Nov 17, 2010 4:27:00 PM org.java.plugin.registry.xml.ManifestParser init INFO: got SAX parser factory - org.apache.xerces.jaxp.saxparserfactoryi...@7bc9a690 Nov 17, 2010 4:27:00 PM org.java.plugin.registry.xml.PluginRegistryImpl configure INFO: configured, stopOnError=false, isValidating=true Nov 17, 2010 4:27:00 PM org.java.plugin.registry.xml.PluginRegistryImpl register INFO: plug-in and fragment descriptors registered - 1 Nov 17, 2010 4:27:00 PM org.java.plugin.standard.StandardPluginManager activatePlugin INFO: plug-in started - org.openstreetmap.osmosis.core.plugin.c...@0.37.0 Nov 17, 2010 4:27:00 PM org.openstreetmap.osmosis.core.Osmosis run INFO: Preparing pipeline. Nov 17, 2010 4:27:00 PM org.openstreetmap.osmosis.core.Osmosis main SEVERE: Execution aborted. org.openstreetmap.osmosis.core.OsmosisRuntimeException: Argument enableLinestringBuilder for task 5-write-pgsql was not recognised. at org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactory.createTaskManager(TaskManagerFactory.java:64) at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.buildTasks(Pipeline.java:50) at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.prepare(Pipeline.java:112) at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:86) at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) at org.codehaus.classworlds.Launcher.main(Launcher.java:31) Emilie Laffray ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Osmosis 0.38 Released
Hi All, I've just released Osmosis 0.38. http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.38.tgz http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-0.38.zip http://dev.openstreetmap.de:23457/hudson/job/osmosis-RELEASE-ant/38/ From changes.txt: - Updated internal storage to use long instead of int identifiers to remove 2^31 node id limit. - Updated apidb support to the latest schema 20100910084426. - Fixed the --write-pbf short option --wb to point to the writer implementation instead of the reader. - Added additional quoting to commands in the osmosis.bat launcher to fix issues with spaces in the installation path. - Modified plugin loader code to support multiple osmosis-plugins.conf files to allow each plugin to provide its own. Let me know if you see any issues. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problem with pbf import into a postgres DB
Hi Thomas, My German isn't great, but I'm guessing that the following message is complaining about a missing tags column in the nodes table. FEHLER: Spalte »tags« von Relation »nodes« existiert nicht I notice you're specifying the following option. validateSchemaVersion=no Why are you doing that? You need to use a schema version that matches your version of Osmosis. Brett On 16/12/2010, at 4:10, Thomas Schreiner schreiner.tho...@gmx.net wrote: nicht ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problem with pbf import into a postgres DB
Hi Thomas, I think you have the wrong schema installed. There is no node_tags table in the schema. There used to be in older versions of the schema. Can you try a couple of things please? - Remove the validateSchemaVersion=no option from your Osmosis command line. You don't need it and it can hide the real problem. - Re-create your schema from the 0.38 distribution. The 0.38 script doesn't create a node_tags table, it creates a hstore tags column on the nodes table. The below script should be the one you are using. http://svn.openstreetmap.org/applications/utils/osmosis/tags/0.38/package/script/pgsql_simple_schema_0.6.sql - Make sure that no errors occur during execution of the above script. - You may need to install hstore support in your database. Your PostgreSQL distribution should provide this script. - Note that the scripts are in the script directory, not in the script/contrib directory. Do NOT use the script/contrib/apidb_0.6.sql script. There are no step by step instructions on debugging Osmosis, however there are some basic instructions on configuring Eclipse. http://wiki.openstreetmap.org/wiki/Osmosis/Development#Eclipse_Setup If you do want to test it in Eclipse, run the existing JUnit tests in the pgsnapshot project from within Eclipse. Cheers, Brett On Thu, Dec 16, 2010 at 8:05 PM, Thomas Schreiner schreiner.tho...@gmx.netwrote: Hello Brett, my english is also not so well, but we can at least communicate. What I did is, I set up a postgres 8.4 database, create a user and a database. Run the sql script pgsql_simple_schema_0.6.sql from the 0.38 distribution out of the directory script/contrib. I downloaded a pbf file from download.geofabrik.de an try to import these pbf file. The first error message I got was about the schema version, so I turned the validation of the schema of. Then I got the error with the index, no idea why the index is not created in the sql script. Never then less I changed the code as described bevor. Now I get the error message that the node table has no tags column. That's right, because there is another table called node_tags. A node can have several tags, e.g. house number, telephone number, name, amenity, ... So I had also a look into the sql script which creates the postgres schema, but I find nothing strange. Can you tell me how to set up osmosis via svn in eclipse to be able to debug the application? Is there a step by step documentation? Thanks Thomas Original-Nachricht Datum: Thu, 16 Dec 2010 18:58:51 +1100 Von: Brett Henderson br...@bretth.com An: Thomas Schreiner schreiner.tho...@gmx.net CC: osmosis-dev@openstreetmap.org osmosis-dev@openstreetmap.org Betreff: Re: [osmosis-dev] Problem with pbf import into a postgres DB Hi Thomas, My German isn't great, but I'm guessing that the following message is complaining about a missing tags column in the nodes table. FEHLER: Spalte »tags« von Relation »nodes« existiert nicht I notice you're specifying the following option. validateSchemaVersion=no Why are you doing that? You need to use a schema version that matches your version of Osmosis. Brett On 16/12/2010, at 4:10, Thomas Schreiner schreiner.tho...@gmx.net wrote: nicht -- Thomas Schreiner Grasslweg 4 85049 Ingolstadt GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt auch mit gratis Notebook-Flat! http://portal.gmx.net/de/go/dsl ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Source for JAR archives in osmosis source
Hi Giovanni, The source for the osmbin jar is stored in a git repo on github. I don't have the URL handy (I'm on my phone) but I think a search for PBF or similar should find it on the osm wiki ... The source for osmosis-hstore is in the binary jar I think. I didn't bother setting up a proper project for it because it is relatively tiny. Let me know if you can't find them and I'll dig up details once I'm back on a PC. Brett On 24/12/2010, at 4:55, Giovanni Mascellani mascell...@poisson.phc.unipi.it wrote: Hi. I'm one of the people that maintains the package osmosis in the Debian (and Ubuntu) distribution. I'd like to update our package to version 0.38 of osmosis, but there are some JAR archives in the SVN repository for which I cannot find the source code: build-support/repo/crosby/osmbin/1.0-6d760534/jars/osmbin.jar build-support/repo/org.openstreetmap.osmosis/osmosis-hstore/0.1/jars/osmosis-hstore.jar build-support/repo/org.openstreetmap.osmosis/osmosis-hstore/0.2/jars/osmosis-hstore.jar Could you please point me to the corresponding source? Google wasn't able to help! :-) Thanks, Giovanni. -- Giovanni Mascellani mascell...@poisson.phc.unipi.it Pisa, Italy Web: http://poisson.phc.unipi.it/~mascellani Jabber: g.mascell...@jabber.org / giova...@elabor.homelinux.org ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Source for JAR archives in osmosis source
I've finally checked this in. There's now a new directory in the Osmosis source tree called hstore-jdbc. It is a project called osmosis-hstore which creates the osmosis-hstore.jar file. Note that the licence is BSD unlike the rest of Osmosis because it is a file obtained from the PostgreSQL forum that will (hopefully) eventually become part of the PostgreSQL JDBC driver. On Fri, Dec 31, 2010 at 9:07 AM, Brett Henderson br...@bretth.com wrote: On 24/12/2010, at 10:26, Brett Henderson br...@bretth.com wrote: On Fri, Dec 24, 2010 at 9:18 AM, Giovanni Mascellani mascell...@poisson.phc.unipi.it mascell...@poisson.phc.unipi.it wrote: Hi. On 23/12/2010 22:27, Brett Henderson wrote: Hi Giovanni, Thanks for the quick reply. :-) The source for the osmbin jar is stored in a git repo on github. I don't have the URL handy (I'm on my phone) but I think a search for PBF or similar should find it on the osm wiki ... Ok, found. Anyway, no license indication is present in the repository, and this is a problem for including the package in Debian, because technically it means that no one can do anything with it. Since this project is strictly tied with osmosis, I suppose that it's its author intention to release it in public domain like osmosis. Could this bit be made explicit in some file into the repository, something like the copying.txt present in osmosis main repository? (BTW, Scott Crosby reads this mailing list, doesn't he?) Yes, he should do. I'm glad you've asked the licence question actually. I should have sorted that out sooner but it slipped my mind. The source for osmosis-hstore is in the binary jar I think. I didn't bother setting up a proper project for it because it is relatively tiny. No, they're not. I can just see the .class files. Oops. Not sure what I've done there. I'll take a look when I get home, perhaps it's just sitting on my home PC. I need to get that checked in somewhere. Sorry for delay. I have found the source. Will try to get it checked in over the next few days. It's a single class I found on a forum somewhere. Relatively simple so I hope licensing isn't an issue. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] [patch] Suboptimal handling of the compressionMethod parameter in xml tasks
Hi Igor, Thanks for the patch. It looks good to me. I've applied it to trunk. Can you please update the documentation for the next version on the wiki to describe the new argument value? http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.39 Cheers, Brett On Sun, Feb 6, 2011 at 4:12 AM, Igor Podolskiy igor.podols...@vwi-stuttgart.de wrote: Hello everybody, first of all, thank you all for developing Osmosis! :) I currently have some issues with the compressionMethod parameter in the XML tasks. This is just a minor annoyance but nevertheless: The compressionMethod parameter of the --[fast]-read-xml task is currently handled somewhat differently than most other parameters in Osmosis. CompressionMethod has three official values (none, gzip, bzip2), but actually there is a fourth option (autodetect) which can only be specified implicitly by omitting compressionMethod altogether. Although this is documented in the wiki, it creates problems for people who build tools around Osmosis which read and/or generate Osmosis command lines automatically, and even for a human user, this is another special case which you need to think of (no other parameter in any task works this way). This can easily be fixed by creating an explicit fourth value (say, auto) for compressionMethod, and making it the default value. This change doesn't break old scripts and streamlines the parameters for new ones. The attached patch against current trunk does this (and the trunk still passes all unit tests with the patch applied). It would be cool if you could integrate this into the next version of Osmosis. If you have any questions or if the patch needs some improvement, I'll be here on the mailing list :). Best regards Igor ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Improving completeWays/completeRelations performance
On Fri, Feb 18, 2011 at 10:20 PM, Frederik Ramm frede...@remote.org wrote: Igor, On 02/18/11 11:40, Igor Podolskiy wrote: just a random thought: what's wrong with using --dataset-bounding-box? Importing the planet file into a database and doing a bunch of queries against is equivalent to creating a single disk buffer for all bb tasks (the database _is_ the disk buffer, if you want). This still isn't very elegant as it requires two passes (import into DB and export to PBFs) but is IMHO more elegant than selection lists. I currently use bounding polygons so I'd have to add a --dataset-bounding-polygon task for that but that should be possible. The major problem is finding a database that would be able to import the data, index it, and fulfil my requests, all within less than 24 hours. From what I hear, PostGIS takes a week for the import step alone but I want to produce fresh extracts every day. So I would have to import into PostGIS and then apply the daily diffs, but I could only do that if applying a daily diff and making all the bounding-polygon requests would take much less than a day and somehow I suspect that's not going to work. Don't get me wrong, maybe there's a reason why dataset tasks are unsuitable - but I've not found any... Maybe the dbb task itself needs improvement - but in this case, this should be the way to go. I doubt that a performant solution can ever be found with PostGIS but I'd love to hear from someone who can prove me wrong. Processing a full day's worth of daily diffs is doable in much less than a day. The one time I did this on a server with only average IO performance (data spread across two disks) it would take about 6 hours to process a day of data. Extracting continent sized chunks from the database will probably be a problem though. The schema works well for 1x1 or 2x2 degree bboxes, but gets rapidly slower as the size of returned data increases. Given that each of your extracts is extracting a large percentage of the overall planet, then doing multiple passes across the entire planet will be tough to beat performance wise. In other words, a database is best (and the only way possible) for small up-to-date extracts, but sequential reads are still best for large extracts where multi-hour time lags are acceptable. I suspect that lots of optimisation is possible for databases, because for one thing the PostGIS data types take up humungous amounts of disk space. Perhaps someone with lots of time and some custom data types could improve on this ... Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Unable to load current way nodes, when extracting and importing data from a osm file to a API BD
This is a known limitation of Osmosis bounding box processing. When you perform a bounding box extract, if nodes lie outside the bounding box, but are referenced by ways inside the bounding box they will not be included in the file. This is due to the way Osmosis processing works because it finishes processing nodes before it sees the ways. You have several options here: 1. Drop the constraint from the database. This might lead to other issues. 2. Add the clipIncompleteEntities=true option to the --bounding-box task. This will modify the ways so that they no longer reference the problematic ways. Obviously they will no longer be identical to the original ways from the input planet file. 3. Add the completeWays=true option to the --bounding-box task. This will include the missing nodes, but will slow down processing enormously. On Thu, Feb 17, 2011 at 12:07 AM, Egoitz Ormaetxea egoitz.ormaet...@gmail.com wrote: - Downloaded planet file planet-110202.osm.bz2 - Extracted a osm for the zone 43.5N -3.5W | 42N -1W with osmosis : bzcat planet-101027.osm.bz2 | osmosis --read-xml file=/dev/stdin enableDateParsing=no --bounding-box top=43.5 left=-3.5 bottom=42 right=-1 --write-xml file=- | bzip2 extracted.osm.bz2 It takes 2 hours to complete but it finishes with no errors. - Clear the API DB : dropdb openstreetmap createdb -E UTF8 -O openstreetmap openstreetmap dropdb osm_test createdb -E UTF8 -O openstreetmap osm_test dropdb osm createdb -E UTF8 -O openstreetmap osm psql -d openstreetmap /usr/share/postgresql/8.4/contrib/btreet_gist.sql rake db:migrate rake test - Import extracted osm file to the API DB with osmosis : bzcat extracted.osm.bz2 | osmosis --read-xml-0.6 file=- --write-apidb-0.6 populateCurrentTables=yes host=localhost database=openstreetmap user=openstreetmap password=xxx validateSchemaVersion=no Ends with error : org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to load current way nodes. ... Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on table current_way_nodes violates foreign key constraint current_way_nodes_node_id_fkey Detail: Key (node_id)=(490767853) is not present in table current_nodes ... The 'weird' thing is that 490767853 node is out of the bounding box http://www.openstreetmap.org/browse/node/490767853 : 42.3951781, -3.5017794http://www.openstreetmap.org/?lat=42.3951781lon=-3.5017794zoom=18 Am I doing the process to extract+import correctly ? Is there a way to avoid this error ? ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] [PATCH] Some issues with Osmosis launch scripts
Hi Igor,
On Wed, Feb 16, 2011 at 4:25 AM, Igor Podolskiy
igor.podols...@vwi-stuttgart.de wrote:
Hi,
one of my students and I managed to discover different issues with the
Osmosis launch scripts in the $OSMOSIS/bin directory, both on Windows and
Unix-like systems.
The shell script (osmosis) will fail or even worse, silently pass incorrect
parameters to the Java process if any of the parameter values contain
spaces. Consider the following command entered in an interactive shell:
osmosis --read-apidb database=osm user=os m password=os m
--write-null
This will fail with the Only one default (un-named) argument can exist per
task. message. Changing the quoting on the original command line doesn't
help.
The problem is with the way osmosis shell script executes the java runtime.
The relevant lines of the script read like this:
EXEC=$JAVACMD ... $MAINCLASS $OSMOSIS_OPTIONS $@
exec $EXEC
While interpolating the $@ in this expression, the shell discards all
original quoting. The solution I've found for this is doing the following:
EXEC=$JAVACMD ... $MAINCLASS $OSMOSIS_OPTIONS
exec $EXEC $@
Apparently, the exact sequence $@ instructs the shell to preserve
original quoting in the arguments, while foo $@ does not. It doesn't make
any sense to me, either, but it's probably some compatibility burden from
decades ago. Anyway, the attached patch osmosis-shell-quoting.patch makes
the necessary change (tested on Ubuntu 9.10 with bash 4.0.33 and /bin/sh
which is linked to dash 0.5.5.1 here).
I've applied your patch directly without any modifications. Thanks for the
patch, it seems to fix the problem nicely.
The Windows batch script gets the quoting right, but fails if Osmosis is
installed on a drive different from the current working directory of the
user. This is somewhat tricky, so I'll try to explain it step by step.
Assume that Osmosis is installed in c:\osmosis, PATH includes
c:\osmosis\bin, and the current working directory is d:\data. User types:
d:\data osmosis --read-xml dump.osm --write-null
The osmosis.bat script does the following:
set SAVEDIR=%CD% # SAVEDIR == d:\data
set MYAPP_HOME=%~dp0.. # MYAPP_HOME == c:\osmosis\bin\..
cd %MYAPP_HOME%# expands to cd c:\osmosis\bin\.. -- BUG!
# %CD% == d:\data
set MYAPP_HOME=%CD%# MYAPP_HOME == d:\data, which is clearly wrong.
# everything fails hereafter as the classpath points to d:\data\lib etc.
Ah, nice work.
The bug is that the current working directory (%CD%) didn't change after cd
%MYAPP_HOME% since the working directories are per-drive as they were in DOS
2.0 or so and we're on drive D: and did just change the CWD for drive C:.
The solution is really simple: all this %SAVEDIR% theater for getting the
absolute path isn't necessary at all, because %~dp0 already returns the
absolute path ('d' magic letter gets the drive, 'p' gets the full path, as
documented on TechNet), so
set MYAPP_HOME=%dp~0
is perfectly sufficient, AFAIK.
The attached osmosis-batch-cwd.patch does the necessary change, tested on
Windows 7 (32-bit).
Your solution looks okay. The only hesitation I have is that it leaves a
.. in the path it generates which the original version must have been
trying to avoid. It probably doesn't matter but I think I've solved it
anyway. The cd command accepts a /D argument which changes the current
drive as well as path. I've modified the two cd statements to use that
switch and it seems to work correctly now.
But thanks for the patch, it highlighted the cause of the problem which is
90% of the work.
Both patches apply to current trunk. I'd also like to thank Markus Bernd
Wagner from the University of Stuttgart for discovering the shell script
issue. If anything is wrong with the patches, just say so, I'll try to fix
it.
Best regards
Igor
Cheers,
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] using osmosis to import full history planet
Hi Martijn, The history tasks you're using are part of some experimental work that Peter Körner was working on and they're not part of the normal Osmosis distribution. Perhaps Peter can provide some assistance ... Brett On Fri, Mar 4, 2011 at 1:09 AM, Martijn van Exel mart...@geodan.nl wrote: Op 03-03-11 15:07, Martijn van Exel schreef: Hi all, We have been trying to import an extract (bbox) of the full history planet file into a PostGIS database using the osmosis db schema. Without much success so far. We described the steps we took here: https://docs.google.com/document/d/1n8g1skzgcvXQF9RtRYNtK2rKdQ_R4JYmC6TNnsGeA9o/edit?hl=en ..or the published version of the Google Doc for readability... https://docs.google.com/document/pub?id=1n8g1skzgcvXQF9RtRYNtK2rKdQ_R4JYmC6TNnsGeA9o -- Martijn van Exel Senior Researcher - Geodan SR President Kennedylaan 1 1079 MB Amsterdam (NL) - Tel: +31 (0)20 - 5711 318 Fax: +31 (0)20 - 5711 333 - E-mail: mart...@geodan.nl Website: www.geodan.nl KvK-nummer: 33 247475 Disclaimer: www.geodan.nl/disclaimer - ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] OSMembrane - GUI front-end for Osmosis
Hi Tobias, I just finished having a play with OSMembrane. Colour me impressed :-) It's a great little app, and provides a nicely polished user experience. I hope development on the tool remains active, or that it at least continues to be maintained. It should be a great complement to Osmosis. Cheers, Brett On Wed, Feb 16, 2011 at 11:05 PM, Tobias Kuhn tob...@osmembrane.de wrote: Hello OSM community, we're three students which had to develop a *Graphical Interface for Osmosis* during a practical task. This task now is finished and we'd like to release our product to the real world and see whether the world needs it or not. The program is written in Java with Swing and named OSMembrane, the name is a pun on Osmosis and semipermeable membranes. It is available at * http://www.osmembrane.de/* (the site is in English) http://osmembrane.de/There you can obtain the latest executable release as well as the source code and project managment. It's licensed under the GPL to be compatible with software already present in OSM. The Osmosis tasks are included via an XML file so it is extendible. Currently supported locales are English and German. We're primarily interested in feedback for OSMembrane: Do you like it? Do you need it? Will you use it in the future? What's good and what's bad? So please don't hesitate to post your thoughts, a lousy answer is better than none. :) And additionally, since we won't be able to spend too much additional time on this project we'd also like to offer the possibility to maintain the project and continue the development, if anyone likes to. Greetings ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Improving completeWays/completeRelations performance
Hi Frederik, On Fri, Feb 18, 2011 at 7:57 PM, Frederik Ramm frede...@remote.org wrote: But I've been thinking: With the the high performance of PBF reading, a two-pass operation should become possible. Simply read the input file twice, determining which objects to copy in pass 1, and actually copying them in pass 2. I'm just not sure how that could be made to fit in Osmosis. One way could be creating a special type of file, a selection list, from a given entity stream. A new task --write-seelction-list would dump the IDs of all nodes, ways, and relations that were either present or referenced in the entity stream: Your suggestion of creating a selection list file to support a two-pass approach seems quite reasonable, and would avoid making any changes to the pipeline design. I've often thought about ways of allowing tasks to receive their input data stream twice, but can't think of a way to do it without greatly complicating the entire pipeline. My latest idea was for Sink tasks (eg. your --multi-bb task) to throw an exception (eg. ResendDataException) that could be caught by Source tasks (eg. --read-xml) triggering them to re-start processing. It wouldn't be terribly difficult to add support for it to tasks like --read-xml, but it starts to get messy in the pipeline control tasks like --tee and --buffer. I'd have to re-write the DataPostbox class which forms the basis of all thread interaction to fix --buffer, and --tee would have to track which downstream pipes had requested a restart then trigger a single restart when all downstream pipes had finished with the first pass. One thing I like about the exception approach to restarts is that it could be rolled into existing tasks incrementally. Any tasks not supporting it would abort cleanly when they received the exception. I started experimenting with this idea this evening, but I've had to abandon the idea because I don't have the time to see it through. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Cannot represent -1 as a char.
Hi Scott, I've just modified the serialisation code to serialise the full 32-bit signed int instead of casting to an unsigned chararacter. This means that the internal serialisation format will be slightly less efficient but I suspect the difference will be immeasurable. Note that recently I updated the primary id to be stored as a 64-bit signed long instead of casting to a 32-bit signed int because some users of Osmosis were expected to run into the limit soon. That change has an even bigger impact to the serialised format (4 extra bytes versus 2) so this one isn't worth worrying about. Changing your code to 0 would have also worked, but at least this way there'll be no surprises for anybody else in the future who wishes to use negative ids. Cheers, Brett On Wed, Mar 9, 2011 at 3:39 PM, Scott Crosby sc...@sacrosby.com wrote: Brett, do you have any advice? In OsmosisBinaryParser, I'm using NOCHANGESET and NOVERSION when there's omitted metadata. Currently both are -1. The bug is because another serializer in osmosis is complaining that the version number is negative. We can set NOVERSION to 0, make that code more robust, or something else. Scott On Mon, Mar 7, 2011 at 8:01 AM, Nakor nakor@gmail.com wrote: Can you give me more of the stack trace? Scott Here you go: SEVERE: Thread for task 1-read-pbf failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Cannot represent -1 as a char. at org.openstreetmap.osmosis.core.util.IntAsChar.intToChar(IntAsChar.java:35) at org.openstreetmap.osmosis.core.domain.v0_6.CommonEntityData.store(CommonEntityData.java:186) at org.openstreetmap.osmosis.core.domain.v0_6.Entity.store(Entity.java:138) at org.openstreetmap.osmosis.core.domain.v0_6.Node.store(Node.java:184) at org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.store(NodeContainer.java:49) at org.openstreetmap.osmosis.core.store.BaseObjectWriter.writeObject(BaseObjectWriter.java:63) at org.openstreetmap.osmosis.core.store.SimpleObjectStore.add(SimpleObjectStore.java:106) at org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.process(UsedNodeFilter.java:78) at org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.process(NodeContainer.java:58) at org.openstreetmap.osmosis.tagfilter.v0_6.UsedNodeFilter.process(UsedNodeFilter.java:61) at org.openstreetmap.osmosis.tagfilter.v0_6.WayKeyValueFilter.process(WayKeyValueFilter.java:70) at org.openstreetmap.osmosis.core.container.v0_6.NodeContainer.process(NodeContainer.java:58) at org.openstreetmap.osmosis.tagfilter.v0_6.WayKeyValueFilter.process(WayKeyValueFilter.java:53) at crosby.binary.osmosis.OsmosisBinaryParser.parseDense(OsmosisBinaryParser.java:136) at crosby.binary.BinaryParser.parse(BinaryParser.java:107) at crosby.binary.BinaryParser.handleBlock(BinaryParser.java:51) at crosby.binary.file.FileBlock.process(FileBlock.java:117) at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:15) at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:36) at java.lang.Thread.run(Unknown Source) Mar 7, 2011 9:01:05 AM org.openstreetmap.osmosis.core.Osmosis main SEVERE: Execution aborted. org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed. at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146) at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92) at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) at org.codehaus.classworlds.Launcher.main(Launcher.java:31) ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Osmosis Maintenance
Hi All, I'm getting worse and worse at responding to any Osmosis issues. The primary reason is a recent addition to the family, but a busy day job isn't helping. This manifests as slow response to emails, slower response to patches, extra slow response to bugs, and total absence of new features (from me anyway). If there's anybody out there who wants to take a more active role in Osmosis maintenance (if not full development) let me know. This can be as simple as testing out the very occasional patch and creating releases, through to taking Osmosis in new directions without fear of me jumping in and saying I don't like it :-) Obviously I'd prefer somebody who I've already dealt with and have a reasonable level of trust in, but I'll consider all options. The main requirement is some familiarity with the existing code base. I suspect along with most of OSM that developers with spare time on their hands are in short supply and I won't be inundated with offers. If that's the case then I'm happy to continue with things in their current state, but just be aware that for all intents and purposes Osmosis is going to be fairly stagnant for the foreseeable future. I'll find time where I can, but Osmosis maintenance isn't going to get much attention. Cheers, Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] planet import
Hi Brian, Yes, you should definitely use the built-in box builder functionality. I've never tried to use the database to do it on a complete planet. I have no idea how long it will take. Brett On Sun, Apr 17, 2011 at 4:34 AM, Brian DeRocher br...@derocher.org wrote: On Saturday 09 April 2011 22:27:31 you wrote: I started a full planet import on the 29th, 11 days ago. I'm trying to get an idea how long this will take. I just want to know if this will take about 20 days or more like 40 days. I think i found my answer, thanks to Toby: enableBboxBuilder=yes: This tells the write-pgsql-dump task to calculate a bounding box for each way. This is stored in a geometry type column in postgres that can be used in spatial queries. This can be done in the database but you may not live to see it finish if you are processing the entire planet. Seriously. http://weait.com/content/build-your-own-openstreetmap-server Also i was using Osmosis 0.34 from Debian 6. I'll upgrade to the stable or devel release, or try svn. For the record this import is on day 17.5 and it's currently running this query for 15 days. UPDATE ways SET bbox = ( SELECT Envelope(Collect(geom)) FROM nodes JOIN way_nodes ON way_nodes.node_id = nodes.id WHERE way_nodes.way_id = ways.id ) Brian -- Brian DeRocher http://brian.derocher.org ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Proposal for Allowing Additional Data in Pipeline
Hi All, This email is just musings at this point. I'm not sure if I'll be able to implement anything anytime soon, but I'd be interested in people's thoughts on this. Until now I've intentionally kept the core data classes in Osmosis as simple as possible to simplify maintenance and ensure consistency across all tasks. I've only added attributes that are required to support basic OSM data and avoided any extensions from creeping in. However it can be quite limiting when there is no way of passing additional data through the pipeline. Examples of additional data might be: - A mutated flag of some kind to flag when a particular entity has been changed and shouldn't be uploaded to the main API. An example is when ways are clipped at bounding box boundaries. - A visible flag. I hesitate to include this one because Osmosis supports this via change streams, not optional visible attributes. - Header information to be attached to the Bound element such as replication timestamp information, source URLs, etc. - Custom data exchanged between specialised tasks. For example, a polygon processing task might add full geometric information to a way. To add some flexibility I'm thinking along the following lines: - Add a new collection to entities that can be optionally populated with String/Object pairs. Conceptually similar to a MapString, Object but possibly stored like existing Tag objects in a simple Collection (currently implemented as an ArrayList) for efficiency. - The collection may be null when no data is required to minimise overhead in the common case. Consumers would need to explicitly check for null which is a tad ugly but I think warranted here. - Modify key tasks such as XML tasks to support serialising these additional values as attributes on the entities themselves (eg. node id=1 version=1 ... mutated=true / ). Alternatively represent them as sub-elements (eg. metatag stored as node id=1...mtag k=mymtag v=myvalue/node) . The object would simply have the toString method called on it to get a string representation. Reading from XML would result in a String object. - Tasks not caring about the data would simply pass the objects on without modification. - Some Sink tasks such as PostgreSQL database tasks would ignore the additional data. - Some tasks such as --bounding-box could add a flag such as mutated. - Rename the existing Bound entity to something more generic like Header to allow more file attributes to be persisted. I think this approach would allow additional data to be attached to entities in a generic fashion without Osmosis itself having to add special support for it. It would keep the pipeline generic but allow specialised tasks to exchange their own custom data. I think representing the value part of data as an Object rather than String makes more sense because it allows custom tasks to exchange complete objects instead of forcing serialisation to and from String. The additional data could in theory be represented as Tags without changing the pipeline at all, but it gets messy mixing real data with metadata. I'm not sure if it makes sense to add support for this to the Bound object, or to simply allow Tag objects to be added instead. Perhaps tags make more sense here? The whole Bound concept has always fitted awkwardly in Osmosis, so I'm not sure how to tackle this one. Hmm, a somewhat rambling email :-) Any thoughts? Cheers Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] --used-node performance and a possible way to improve it
Hi Igor, You're describing a common problem with the Osmosis pipeline. Many scenarios would be improved if you could access the data stream twice rather than having to buffer all data within the task itself. Your solution of requiring two input streams is quite practical. I've often been tempted to do something similar. The main issue is that it relies on the user passing an identical input stream twice which is somewhat error prone and ... well just feels messy. Without a good solution I've just avoided solving the problem at all :-) I've dabbled with the idea of allowing pipeline restarts. The only way I can think to do it without requiring a more complicated task API is to allow tasks to throw a special type of exception (eg. RequestStreamRestartException) which could be caught by reader tasks (eg. --read-xml) and indicate to them to begin reading from the start of the stream again. Reader tasks not supporting the new exception would simply abort as per normal exception handling. This approach has the benefit that end-users could construct pipelines without consideration for whether data was read once or twice. The downside is a bunch of added complexity though. I haven't had time to investigate it properly. I'm a little hesitant to see this type of workaround become the norm, but I don't have a better option right now. Given that your tasks are independent of existing tasks I don't have any objection to them being added. Cheers, Brett On Thu, May 12, 2011 at 5:29 AM, Igor Podolskiy igor.podols...@vwi-stuttgart.de wrote: Hi Osmosis developers, recently I found myself filtering large extracts of data with fairly restrictive tag-based filters. Imagine a task like get all administrative boundaries in the state of Baden-Württemberg, Germany. It was taking a long time, lots of CPU, and lots of I/O. And I asked myself what it was doing all that time, actually. I had a pipeline like [1] (OSMembrane screenshot). Just the old-fashioned filter for ways and relations with a merge at the end (in the screenshot, top row is ways, bottom row is relations). Turns out that the main culprit is the --used-node task. As you surely know, it works like this: 1. Store all ways, nodes and relations coming in into a simple object store. 2. During this, records all node references. 3. Replay the simple object store to the output, filtering out unneeded nodes. Basically, in a workflow like read from disk - filter ways - get used nodes - write you basically write _everything_ you got from disk back to disk and then read it back again in --used-nodes. More than that, you only can control the filesystem the second write happens by setting the java.io.tmpdir property which isn't really intuitive. And you spend CPU time compressing and decompressing the intermediate store. So, in actual numbers for boundaries in Baden-Württemberg (128 MB PBF as of today) this workflow boils down to read 128 MB from disk, write ~180 MB gzipped serialized objects to disk, read ~180 MB from disk, write 2 MB PBF to disk. In the example pipeline shown above, those ~180 MB of gzipped serialized objects get written and read _twice_ because of two --used-node tasks. This seemed, well, a little wasteful to me. You should only pay for what you getting (the 2 MB), not for everything there is (the 128 MB), and surely not twice :) So I thought up an approach which avoids intermediate stores. It involves a task that takes two input streams and produces a single one. It works like this: 1. Read everything from the first stream and ignore all nodes, record the required ids for ways and relations in an id tracker just like --used-node, and pass the ways and relations downstream immediately. 2. Read everything from the second stream, ignore all ways and relations and only pass the required nodes (based on the id tracker) downstream. It's a bit like a merge with an id tracker. In terms of the complete workflow, it involves reading the source file from disk twice; the pipeline equivalent to [1] is shown in [2] (another OSMembrane screenshot). I implemented this task (named --fast-used-node, better name needed ;)) and made a couple of measurements for the example I mentioned above (admin boundaries in Baden-Württemberg). Pipeline [1] with --used-node: ~312+-10 seconds Pipeline [2] with --fast-used-node: ~140+-10 seconds A simple pipeline like read-filter ways-used-nodes-write takes about 140 seconds with --used-node, the --fast-used-node one takes ~90 seconds complete. All numbers on Pentium Dual-Core E5300 2.6 GHz, Win7 Pro 32-Bit, vanilla SATA disk, default 64MB heap size (irrelevant for this task). Both approaches seem to be CPU-bound (so the compression/decompression is more a problem than the IO in and of itself). Of course, everything has a price. First, you effectively need to read the source file twice from disk; just splitting up a stream and buffering it isn't
Re: [osmosis-dev] --used-node performance and a possible way to improve it
Hi Igor, On Sat, Jun 4, 2011 at 6:04 AM, Igor Podolskiy igor.podols...@vwi-stuttgart.de wrote: Hi Brett, hi @osmosis-dev, You're describing a common problem with the Osmosis pipeline. Many scenarios would be improved if you could access the data stream twice rather than having to buffer all data within the task itself. good to know I'm not alone ;) In the meantime I've been working quite a lot with Osmosis, with the standard tasks, and with my custom ones, and I've been following the discussions on this list, especially your posts about metadata... so now I think I have some clue about what actually lies behind these problems and solutions, mine included. In a way, this is also a reply to your recent post about metadata. Feel free to skip my ramblings :) Here's the TL;DR version: my --fast-used-* is really just a workaround and should really stay a plugin and out of the main tree. What I think we need is a much more generic solution, like a second type of communication channel between the tasks for flow control information. Full version: There seem to be many use cases for Osmosis which more or less blow up the current streaming principle, --used-* being only one of them. That's why we're talking on this list about additional metadata, and completeWays performance of --bbox, and performance of --used-node, and RestartExceptions. Even the bound handling is actually wrong at many places, and it can't be made right because of the stream ordering requirements. I somewhat fixed it in --merge, but it's more or less unfixable in, say, --apply-change without caching the whole stream which leads to the --used-node problem (lots of wasted CPU, I/O and disk space). As I used Osmosis over the last couple of weeks, and I often found myself thinking: can I do X with Osmosis? Can I optimize step Y of the pipeline? And the answer is almost always: no, it requires replaying the stream. No, it requires a bit of information from another task up or down the pipeline. No, it requires some coordination or synchronization between multiple tasks. In short, and I think that's the core problem: the answer is no because _the tasks do not know enough about each other_ both before and during processing. And as tempting as some metadata embedded in the data stream like you implemented in your last commits or a workaround implemented with my --fast-used-node might be - in my very humble and uninformed opinion it handles symptoms, not this core problem. What we have now, is very similar to the plain old analog telephone network: we transmit both data and control information over a single channel. And there's a reason the telecom networks switched to out-of-band signaling with separate channels for data and metadata like call setup: it is more flexible. Before that, you had crazy workaround stuff like hook flash where you submitted information by actually interrupting the channel with a particular timing. Maybe that's just me, but throwing an exception up the pipeline is in a way very much like a hook flash :) As to the RestartStreamException in particular, I have a feeling that it isn't really going to work well with --tee, --buffer, --merge, --apply-change and similar tasks. And even if, an exception really seriously messes up your control flow. Once you throw it, you cannot really make any assumptions as to what state you're currently in. And even if you work around it, that code going to be _really_ messy. So I think what we need is a full-blown control plane for communication between tasks. Like in telecommunications, it should be orthogonal to the data streams, otherwise you will always be having the problem that the bit of information that you need is in the wrong place of the stream. We see this with bound: you need to write it out at the start of the stream but can only you know it at the end in the current model. That's why you would need to cache a GB-sized stream which seems pretty wasteful for 4 doubles. I'd like a way for a task to know what is downstream and upstream of the pipeline. I'd like proper synchronization like locks and latches adn waits between tasks - yes, synchronization is complex and can lead to deadlocks, but we also can have deadlocks _now_ without the benefits of synchronization. I'd like a way for one task to be able to say what it is going to do to stream with respect to the sort ordering or the bounding box. And I like for other tasks to be able to reason about that in the context of the current pipeline. Enough I'd likes. :) If you have a good control plane, good things happen, that's it, actually ;) I very well understand that this is going to make the task API more complex. But I think it would pay off as you could use Osmosis for more tasks. Passing metadata in the data stream and exceptions and --fast-used-node make things more complex as well, and the payoff is less IMHO. You're right when you say that
Re: [osmosis-dev] bounding-polygon dismisses points at +180° longitude
On Sun, Jul 17, 2011 at 4:25 AM, WanMil wmgc...@web.de wrote: Hi, what need to be done to get this issue fixed? The following things were provided: * a simple testcase * a scenario in which the problem is a real issue (extracting russian border) * a simple patch based on Brett ideas I don't know what I can do now to speed up a bugfix. So please commit the patch or let me know what's wrong with it. Hi WanMil, I've applied the patch (plus checkstyle fixes). Let me know if you see any issues. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Version Control Full History and GIT Test
Hi All, I've been playing in a couple of areas with regards to Osmosis version control. - I'm pondering a move to GIT. Primarily because it gives people a better opportunity to experiment with the codebase. Secondly because I am experimenting with GIT anyway and wanted a real-world task to work with. - The svn history is disjointed due to the early history being hosted on my server, and subsequent history being on the osm svn server. In addition there have been some renames and re-organisation that are confusing and break potential migrations to other tools (ie. GIT). I decided I'd try to fix the SVN history and produce a single repo with complete history. What started out as a simple shell script with some grep and sed statements quickly got out of hand and became a full-blown java-based SVN dump file re-writer that looks quite a lot like Osmosis itself. Perhaps I should have written it as an Osmosis plugin :-) Anyway, it does the following major operations. - Creates a new empty repo with standard trunk/tags/branches structure at the root. - Extracts history from http://www.bretth.com/repos/main from revision 417-471 which was the initial period of Osmosis when it was called conduit. - Replaces the conduit path with trunk. - Removes the initial creation of conduit directory which isn't necessary as data will be imported into existing trunk. - Renumbers the revisions from 1 upwards. - Skips revision 472 which was a rename from conduit to osmosis. We don't need this because the project name isn't reflected in the new repo, the project is stored in the root. - Extracts history from revision 473-474 which are the only two revisions with code stored in the root of the osmosis project. - Replaces the osmosis path with trunk. - Renumbers the revisions. - Skips revision 475 which was a re-organisation into a trunk/tags/branches structure. This is already done in the new repository. - Extracts all remaining revisions in my repository from 475-HEAD. - Removes the osmosis prefix on paths. - Renumbers the revisions. - Skips 4742 because that's the original import which is already done in the new repo. - Extracts history from http://svn.openstreetmap.org/applications/utils/osmosis from 4743-12410. - Replaces the applications/utils/osmosis path with trunk. - Renumbers the revisions. - Skips 12411 which was a re-organisation into a trunk/tags/branches structure. - Extracts remaining history from 12412-HEAD. - Replaces a copyfrom path for creation of 0.28 and 0.29 tags which refer to revisions prior to the repo re-structure and thus have incorrect path structures in the dump file. - Removes the applications/utils/osmosis prefix on paths. - Renumbers the revisions. It seems to have all worked well. It provides a single repo with full history from day 1, gives a consistent directory layout for the entire history, and includes all tags that were ever created. I got part way there with shell scripts, but the real trick was the revision renumbering which had to be done because there are numerous occasions where revisions copy files from significantly older revisions where the main svn repo moved on between me copying a latest version file and actually doing a commit. Renumbering required knowledge of entire history and called for a better language. One thing I haven't fixed is the svn:mergeinfo tags (it's hard), so I'll probably remove them completely to avoid any confusion. If you want to try it out for yourself, the migration app is here: https://www.bretth.com/repos/main/osmosis-svn-to-git If you load it up in Eclipse, you can simply run the DumpFileMigrator class. From a command line you'd need to compile the app and run it with the shell scripts in your working directory. Note that it will take a *very* long time the first time you run it because it generates a complete snapshot of the main osm repository locally using svnsync. If you already have a replica you can copy it in because it continues to update the one it has each time you run it. Any thoughts re a possible GIT move? Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
