Re: [osmosis-dev] Licence ...
On Mon, Mar 09, 2009 at 12:35:48PM +0100, Stefan Baebler wrote: > I consider my contributions so minor that i hereby give you all the > power to relicense them as you please. Same from me. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Continuous Integration
On Fri, Mar 20, 2009 at 09:09:49AM +1100, Brett Henderson wrote: > It's not ideal running it on my home server because I don't want to give > out access to it. If anybody has other suggestions let me know. The > osm dev server may be more appropriate but I need a tomcat server to run > hudson under and I'm not sure how easy that will be to organise. Depending on what exactly you need we could also maybe run this on the server where openstreetmap.de is running on or use the server that the wikipedia people are buying for experimental OSM stuff. How much ressources would you need? Tomcat and the Osmosis dev stuff ? I guess the whole thing could be swapped out most of the time and only needs a few resources for the actual build every few hours? Do several people need access? root access? Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Changes to WayGeometryBuilder
I have made some changes to WayGeometryBuilder in r18117 (all backwards compatible): I added getNodeLocation() method to get the locations back out, changed createLineString() method to be public and added a createPoint() method. Those are all simple methods that I need in a plugin and they shouldn't affect the rest of the application. If somebody doesn't like the functions, please complain. :-) Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] CI-Server (Hudson) for Osmosis?
On Thu, Dec 10, 2009 at 11:24:00AM +0100, Hakan Tandogan wrote: > If it is possible, I'd love to share admin responsibilities on the box you > are already get access to. Send me your preferred login name and a ssh key and I'll create the account for you. All users are in the same group 'osm' and this way you can work on the same stuff. If needed we can also create sudo scripts, special users, etc. as needed. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis 0.37 Released
On Sat, Oct 16, 2010 at 05:33:37PM +1100, Brett Henderson wrote: >- Updated the pgsql schema (now version 6) to move all tags into hstore >columns, add a ways.nodes column and CLUSTER the nodes and ways tables. >Significant performance increases. Is there some magic involved or do you just call CLUSTER after the import? When I tested CLUSTER it helped with reads, but creating the cluster was very expensive. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] one node ways
On Fri, Apr 08, 2011 at 07:06:45PM +0200, David Paleino wrote: > On Fri, 8 Apr 2011 20:55:43 +0400, Kirill Bestoujev wrote: > > > You may have a look at way 35966868 to see the problem by yourself. > > That's non-sense to me. Why on Earth one would want to draw a one-node way? A > *way* is composed by at least two nodes. > > That's a mapping error, IMHO. Of course it is. But those errors tend to happen so you need to deal with them. And thats what this thread is about. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] one node ways
Hi! This has worked for me: I use the ST_isvalid() function in Postgis to find out if a geometry is valid. If not, I either delete the way or set the geometry to NULL and then do all further processing only on ways with geometry != NULL. Jochen On Fri, Apr 08, 2011 at 08:55:43PM +0400, Kirill Bestoujev wrote: > Date: Fri, 8 Apr 2011 20:55:43 +0400 > From: Kirill Bestoujev > To: Igor Podolskiy > Cc: osmosis-dev > Subject: Re: [osmosis-dev] one node ways > > Igor, > > the context is simple - we have a mirror of the planet, we update if with > replication with minute diffs. > > You may have a look at way 35966868 to see the problem by yourself. After > creation of geom by osmosis it looks like LINESTRING(2.2321042 > 48.7767191,2.2321042 48.7767191) which is not good. > > > > K. > > 2011/4/8 Igor Podolskiy > > > Hi Kirill, > > > > > > When a one node way is present in the database and osmosis creates a > >> linestring geometry for it the resulting geometry is something like > >> LINESTRING(139.386972 37.095865) > >> > >> When we process this linestring with any postgis function it gives us > >> an error - the linestring must contain 0 or > 1 nodes. > >> > > > > I think I understand the problem but I'm having a hard time understanding > > the context :) Could you maybe provide a simple example and/or a > > step-by-step guide to reproduce this? Like the osmosis command line you're > > using, the kind of data you're trying to put in the database - that would be > > very helpful. > > > > Regards > > Igor > > > ___ > osmosis-dev mailing list > osmosis-dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/osmosis-dev -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] one node ways
You can have several nodes on the same position which could also lead to a broken geometry but will not be caught be looking just at the nodes. Jochen On Sat, Apr 09, 2011 at 02:22:58PM +0400, Kirill Bestoujev wrote: > Date: Sat, 9 Apr 2011 14:22:58 +0400 > From: Kirill Bestoujev > To: osmosis-dev@openstreetmap.org > Subject: Re: [osmosis-dev] one node ways > > And in works! > > and (select count(1) from (select n.geom as geom, count(1) as node_count > from way_nodes wn join nodes n on n.id = wn.node_id where wn.way_id = > w.idgroup by 1) n_group) > 1 > > added to the request below works fine! > > I will look at it for a couple of days to make sure it works in all cases, > but looks like it is a solution. > > What is the process of merging it into osmosis distro? > > The pg_snapshot and pg_simple load scripts should also be updated! > > Kirill > > 2011/4/9 Kirill Bestoujev > > > Sure, I know this. But if I create an indes with, for example, a st_within > > condition, than osmosis falls already while replicating diffs... > > > > That's why I want to solve the problem in osmosis itself, not in the db. > > > > Yesterday on Russian irc channel we found a solution to add to NodeDao.java > > linesting update sql-command the following line: > > > > and (select count(1) from (select n.geom as geom, count(1) as node_count > > from way_nodes wn join nodes n on n.id = wn.node_id where wn.way_id = > > w.idgroup by 1) n_group) > 1 > > > > This solves the problem with ways having only duplicate nodes in them. > > > > But the recompiled version of osmosis falls on the following command: > > > > org.springframework.jdbc. > > UncategorizedSQLException: PreparedStatementCallback; uncategorized > > SQLException for SQL [UPDATE ways w SET linestring = ( SELECT > > MakeLine(c.geom) AS way_line FROM ( SELECT n.geom AS geom FROM nodes n INNER > > JOIN way_nodes wn ON n.id = wn.node_id WHERE (wn.way_id = w.id) and > > array_length(nodes, 1) > 1 ORDER BY wn.sequence_id ) c ) WHERE w.id = ?]; > > SQL state [XX000]; error code [0]; ERROR: Exception in LWGEOM2GEOS; nested > > exception is org.postgresql.util.PSQLException: ERROR: Exception in > > LWGEOM2GEOS > > > > the added code - and array_length(nodes, 1) > 1 - which I thought was > > solving the problem does not work OR there is an other problem - not one > > node ways. > > > > Kirill > > > > > > 2011/4/9 Jochen Topf > > > >> Hi! > >> > >> This has worked for me: I use the ST_isvalid() function in Postgis to find > >> out > >> if a geometry is valid. If not, I either delete the way or set the > >> geometry to > >> NULL and then do all further processing only on ways with geometry != > >> NULL. > >> > >> Jochen > >> > >> On Fri, Apr 08, 2011 at 08:55:43PM +0400, Kirill Bestoujev wrote: > >> > Date: Fri, 8 Apr 2011 20:55:43 +0400 > >> > From: Kirill Bestoujev > >> > To: Igor Podolskiy > >> > Cc: osmosis-dev > >> > Subject: Re: [osmosis-dev] one node ways > >> > > >> > Igor, > >> > > >> > the context is simple - we have a mirror of the planet, we update if > >> with > >> > replication with minute diffs. > >> > > >> > You may have a look at way 35966868 to see the problem by yourself. > >> After > >> > creation of geom by osmosis it looks like LINESTRING(2.2321042 > >> > 48.7767191,2.2321042 48.7767191) which is not good. > >> > > >> > > >> > > >> > K. > >> > > >> > 2011/4/8 Igor Podolskiy > >> > > >> > > Hi Kirill, > >> > > > >> > > > >> > > When a one node way is present in the database and osmosis creates a > >> > >> linestring geometry for it the resulting geometry is something like > >> > >> LINESTRING(139.386972 37.095865) > >> > >> > >> > >> When we process this linestring with any postgis function it gives us > >> > >> an error - the linestring must contain 0 or > 1 nodes. > >> > >> > >> > > > >> > > I think I understand the problem but I'm having a hard time > >> understanding > >> > > the context :) Could you maybe provide a simple example and/or a > >> > > step-by-step guide to reproduce this? Like the osmosis command line > >> you're > >> > > using, the kind of data you're trying to put in the database - that > >> would be > >> > > very helpful. > >> > > > >> > > Regards > >> > > Igor > >> > > > >> > >> > ___ > >> > osmosis-dev mailing list > >> > osmosis-dev@openstreetmap.org > >> > http://lists.openstreetmap.org/listinfo/osmosis-dev > >> > >> > >> -- > >> Jochen Topf joc...@remote.org http://www.remote.org/jochen/ > >> +49-721-388298 > >> > >> > > > ___ > osmosis-dev mailing list > osmosis-dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/osmosis-dev -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] one node ways
Sorry, that was a bit unclear. You can't just look at the node ids, you have to look at the geometries of those nodes, too. Jochen On Sat, Apr 09, 2011 at 02:36:36PM +0400, Kirill Bestoujev wrote: > Date: Sat, 9 Apr 2011 14:36:36 +0400 > From: Kirill Bestoujev > To: Jochen Topf > Cc: osmosis-dev@openstreetmap.org > Subject: Re: [osmosis-dev] one node ways > > Why? Where do I have to look if not at the nodes??? > > K. > > 2011/4/9 Jochen Topf > > > You can have several nodes on the same position which could also lead to a > > broken geometry but will not be caught be looking just at the nodes. > > > > Jochen > > > > On Sat, Apr 09, 2011 at 02:22:58PM +0400, Kirill Bestoujev wrote: > > > Date: Sat, 9 Apr 2011 14:22:58 +0400 > > > From: Kirill Bestoujev > > > To: osmosis-dev@openstreetmap.org > > > Subject: Re: [osmosis-dev] one node ways > > > > > > And in works! > > > > > > and (select count(1) from (select n.geom as geom, count(1) as node_count > > > from way_nodes wn join nodes n on n.id = wn.node_id where wn.way_id = > > > w.idgroup by 1) n_group) > 1 > > > > > > added to the request below works fine! > > > > > > I will look at it for a couple of days to make sure it works in all > > cases, > > > but looks like it is a solution. > > > > > > What is the process of merging it into osmosis distro? > > > > > > The pg_snapshot and pg_simple load scripts should also be updated! > > > > > > Kirill > > > > > > 2011/4/9 Kirill Bestoujev > > > > > > > Sure, I know this. But if I create an indes with, for example, a > > st_within > > > > condition, than osmosis falls already while replicating diffs... > > > > > > > > That's why I want to solve the problem in osmosis itself, not in the > > db. > > > > > > > > Yesterday on Russian irc channel we found a solution to add to > > NodeDao.java > > > > linesting update sql-command the following line: > > > > > > > > and (select count(1) from (select n.geom as geom, count(1) as > > node_count > > > > from way_nodes wn join nodes n on n.id = wn.node_id where wn.way_id = > > w.idgroup by 1) n_group) > 1 > > > > > > > > This solves the problem with ways having only duplicate nodes in them. > > > > > > > > But the recompiled version of osmosis falls on the following command: > > > > > > > > org.springframework.jdbc. > > > > UncategorizedSQLException: PreparedStatementCallback; uncategorized > > > > SQLException for SQL [UPDATE ways w SET linestring = ( SELECT > > > > MakeLine(c.geom) AS way_line FROM ( SELECT n.geom AS geom FROM nodes n > > INNER > > > > JOIN way_nodes wn ON n.id = wn.node_id WHERE (wn.way_id = w.id) and > > > > array_length(nodes, 1) > 1 ORDER BY wn.sequence_id ) c ) WHERE w.id = > > ?]; > > > > SQL state [XX000]; error code [0]; ERROR: Exception in LWGEOM2GEOS; > > nested > > > > exception is org.postgresql.util.PSQLException: ERROR: Exception in > > > > LWGEOM2GEOS > > > > > > > > the added code - and array_length(nodes, 1) > 1 - which I thought was > > > > solving the problem does not work OR there is an other problem - not > > one > > > > node ways. > > > > > > > > Kirill > > > > > > > > > > > > 2011/4/9 Jochen Topf > > > > > > > >> Hi! > > > >> > > > >> This has worked for me: I use the ST_isvalid() function in Postgis to > > find > > > >> out > > > >> if a geometry is valid. If not, I either delete the way or set the > > > >> geometry to > > > >> NULL and then do all further processing only on ways with geometry != > > > >> NULL. > > > >> > > > >> Jochen > > > >> > > > >> On Fri, Apr 08, 2011 at 08:55:43PM +0400, Kirill Bestoujev wrote: > > > >> > Date: Fri, 8 Apr 2011 20:55:43 +0400 > > > >> > From: Kirill Bestoujev > > > >> > To: Igor Podolskiy > > > >> > Cc: osmosis-dev > > > >> > Subject: Re: [osmosis-dev] one node ways > > > >> > > > > >> > Igor, > > > >> > > > > >> > the context is simple - we have a mirror of the planet,
Re: [osmosis-dev] --used-node performance and a possible way to improve it
On Fri, Jun 03, 2011 at 10:04:40PM +0200, Igor Podolskiy wrote: > [lots of good thinking] I have thoght about these things quite a lot, too. I agree with Igor that we are bumping into the limits of what the current Osmosis pipelining model can do. And it gets even more complicated: For lots of the things we want to do the best way of doing it depends on the kind of data we are working on: For working on a small excerpt a totally different approach might be needed than when working on the whole planet. For the excerpt we might be able to work completely in memory, for the planet we might need temporary files or multiple passes on the input. The idea of having some "control plane" that magically does all the right things is very tempting, but I think it is premature to try to define it. We don't know enough about the different things people are doing and want to do with OSM data yet. If we design such a control plane, it would probably end up a huge monster that can do everything, but nobody can actually work with it, because of its complexity. I have been trying a different approach with the Osmium framework that I have been working on for the last year or so: Don't try to solve every problem in one gigantic flexible program. Instead have lots of building blocks that a programmer can use to fit together a program that does exactly what he needs the way he needs it. There is no "control plane", but a programmer who decides to use this class and that driver and maybe a class he writes himself deriving it from some other class from the toolkit. What we loose is the ability for a "normal user" to build this by using a few command line options (or a GUI), what we gain is a lot of flexibility. And then, later, when we have a better idea of the typical uses, we can build this "control plane" on top of the library. Actually we will probably not just build one control plane but several. And together they will probably solve 95% of all use cases and for the rest you still go back to the source code. I started my own project, Osmium, to go that route, but I don't see why the same thing couldn't be done with Osmosis: Split Osmosis into two parts, one is the low-level code for OSM objects, readers and writers etc. And one is the "control plane" code that stitches pipelines together etc. Leave the Osmosis application as it is, it solves many many problems in a very nice and easy way. But also use the low-level blocks and stick them together in new ways in new applications that don't use the current pipelining model but other combinations. (I also encourage you to take a look at Osmium (https://github.com/joto/osmium either to use it or to steal ideas from. It can read and write XML and PBF, assemble multipolygons, do several passes over the input, filter data, create shapefiles, and many other things. :-) Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Proposal for Allowing Additional Data in Pipeline
On Wed, May 11, 2011 at 10:59:34PM +1000, Brett Henderson wrote: > This email is just musings at this point. I'm not sure if I'll be able to > implement anything anytime soon, but I'd be interested in people's thoughts > on this. > > Until now I've intentionally kept the core data classes in Osmosis as simple > as possible to simplify maintenance and ensure consistency across all > tasks. I've only added attributes that are required to support basic OSM > data and avoided any extensions from creeping in. > > However it can be quite limiting when there is no way of passing additional > data through the pipeline. Examples of additional data might be: > >- A "mutated" flag of some kind to flag when a particular entity has been >changed and shouldn't be uploaded to the main API. An example is when ways >are clipped at bounding box boundaries. >- A "visible" flag. I hesitate to include this one because Osmosis >supports this via change streams, not optional visible attributes. >- Header information to be attached to the Bound element such as >replication timestamp information, source URLs, etc. >- Custom data exchanged between specialised tasks. For example, a >polygon processing task might add full geometric information to a way. > > To add some flexibility I'm thinking along the following lines: > >- Add a new collection to entities that can be optionally populated with >String/Object pairs. Conceptually similar to a Map but >possibly stored like existing Tag objects in a simple Collection (currently >implemented as an ArrayList) for efficiency. >- The collection may be null when no data is required to minimise >overhead in the common case. Consumers would need to explicitly check for >null which is a tad ugly but I think warranted here. >- Modify key tasks such as XML tasks to support serialising these >additional values as attributes on the entities themselves (eg. version=1 ... mutated="true" /> ). Alternatively represent them as >sub-elements (eg. metatag stored as v="myvalue">) . The object would simply have the toString method >called on it to get a string representation. Reading from XML would result >in a String object. >- Tasks not caring about the data would simply pass the objects on >without modification. >- Some Sink tasks such as PostgreSQL database tasks would ignore the >additional data. >- Some tasks such as --bounding-box could add a flag such as "mutated". >- Rename the existing Bound entity to something more generic like Header >to allow more file attributes to be persisted. > > I think this approach would allow additional data to be attached to entities > in a generic fashion without Osmosis itself having to add special support > for it. It would keep the pipeline generic but allow specialised tasks to > exchange their own custom data. I think representing the value part of data > as an Object rather than String makes more sense because it allows custom > tasks to exchange complete objects instead of forcing serialisation to and > from String. > > The additional data could in theory be represented as Tags without changing > the pipeline at all, but it gets messy mixing real data with metadata. > > I'm not sure if it makes sense to add support for this to the Bound object, > or to simply allow Tag objects to be added instead. Perhaps tags make more > sense here? The whole Bound concept has always fitted awkwardly in Osmosis, > so I'm not sure how to tackle this one. > > Hmm, a somewhat rambling email :-) Any thoughts? A flexible mechanism like this would be very interesting and useful, but I also see a lot of potential for confusion. Some tasks can handle certain extra data, some tasks can't. Some would only work if extra data is present, some would silently do wrong things when expected data is not present, etc. Currently all tasks can be plugged together and if you are trying to combine them in a way that doesn't work (for instance a task reading change data instead of one reading plain data), Osmosis will complain. So I think this needs to be a bit more formalized so Osmosis can still make those checks. The Map could be connected to some registry for the strings (or the strings would be objects instead with some extra information.) For each one you would have at least the two options: If a tasks encounters an object of this type in the pipeline and doesn't understand it, * it will just ignore it and optionally pass it on * it has to complain. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Merge huge count of files
On Fri, Sep 30, 2011 at 07:21:22PM +0200, Igor Podolskiy wrote: > On 29.09.2011 15:28, Gubler, Ruediger wrote: > >I have to merge a huge count of files. Doing this in one osmosis call > >creates thousands of threads and stops the rest of the system working well. > >Is it possible and efficient to split the giant merge into smaller pieces? > >What is the best strategy to merge a huge count (e.g. 100x100 matrix) > >together with a minimum of needed memory? > >Must the whole dataset fit in the memory? > Memory isn't the problem with merges, the only thing worth > mentioning that merge stores in memory are the buffers. Those are > either very small (20 entities in 0.39 release) or can be set to a > more appropriate value on the command line (in current trunk, or > HEAD now that it's in git ;)). Other than that, --merge just looks > at the next entities on the input stream and chooses one of them to > pass through downstream. There can be a memory problem with PBF files though. PBF files contain blocks of entities, typically 8000 at a time. A PBF reader (or writer) needs to allocate buffers for those blocks and those can be quite large. The PBF spec says that an uncompressed block can be a maximum of 32 MBytes. Thats not the only buffer you need. I'd have to look a the source code to check the overall buffer size you need, but in any case its not negligable if you have many files open. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Streaming Replication
Very interesting work! How do you handle new incoming requests. They have to start from a known point so I guess you have to do an SQL query for each of them? Or do you just read the existing .osc files from disk and stream them out? This could take a long time... Jochen On Sat, Oct 13, 2012 at 03:43:32PM +1100, Brett Henderson wrote: > Date: Sat, 13 Oct 2012 15:43:32 +1100 > From: Brett Henderson > To: osmosis-dev > Subject: [osmosis-dev] Streaming Replication > > Hi All, > > For those of you who currently use the minute diffs to keep a local > database up to date, you may be interested to know that a new form of > replication has hit the street. > > The current replication system is based on a series of static replication > files that are placed on a web server for clients to download as described > here: > http://wiki.openstreetmap.org/wiki/Planet.osm/diffs#Using_the_replication_diffs > > It is a very simple mechanism and works well for the existing daily, hourly > and minutely replication feeds. Unfortunately it doesn't work well for > sub-minute replication because it becomes far too "chatty". On the server > side, the current feeds are generated from cron which also works well down > to one minute intervals, but the overhead of launching a new process and > connecting to the database for every replication interval also becomes too > inefficient for shorter intervals. > > To solve this, a new streaming replication mechanism has been developed. > Under the covers the same database queries are utilised, but the process > performing the queries runs continously and polls the database for changes > at a shorter interval. It is currently set to poll every 10 seconds, but > it can be reduced further if required. The network transport is also > continuous and holds a single HTTP connection open for the lifetime of > communication between the server and client. It is all implemented within > the latest version of Osmosis 0.41. If you wish to experiment with the > server-side tasks however, several bugs have been fixed in the latest > development version. Internally it uses the JBoss Netty framework which > means that it's all event-driven (ie. doesn't require a thread per client) > and should theoretically support a large number of concurrent clients. > > To quickly see this in action, point your browser at this URL and you > should see new replication "state" data become available approximately > every 10 seconds. > http://planet.openstreetmap.org/replication/streaming/replicationState/current/tail > > New Osmosis tasks have been developed to consume this data. For some basic > instructions to help you get started, refer to this link: > http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Client-side_Streaming > > If you don't wish to use Osmosis, some basic documentation on the wire > protocol is available here: > http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Streaming_Replication_Wire_Protocol > > This is very much experimental and bugs will undoubtedly be encountered to > please be wary about trusting it to update your database if you've just > spent two weeks importing a planet file. However, I'd love to see it get > some usage and would welcome any feedback. This is not intended for use in > updating a local planet file as the existing daily files are better suited > to that. For databases that can tolerate a minute delay, the existing > mechanism is very simple and has proven to be fairly reliable. But if you > really need current access to data, and can cope with the additional > complexity, this should be useful. The current 10 second delay is not a > lower limit, but is a good starting point for now. > > Cheers, > Brett > ___ > osmosis-dev mailing list > osmosis-dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/osmosis-dev -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Single node ways
On Wed, Nov 14, 2012 at 10:33:43PM +1100, Brett Henderson wrote: > Jochen Topf was the one who originally introduced the checks in > WayGeometryBuilder to ensure a Way contained at least two nodes. He might > have some thoughts on whether we can remove the checks. Perhaps it was > simply introduced to avoid the additional overheads of having to do > st_isvalid() checks? I don't remember doing that but it might well have been me, I guess Brett looked at the svn log, so I believe him. :-) Doing checks later in the database is much more expensive so it makes sense to do them in Osmosis. But that doesn't work when you want to update the database later, of course. But you have to take into account that for almost everything you want to do with the geometries you'll get no end of problems if you have the occasional bad geometry in there. This discussion here highlights again the fact that there is no once-size-fits-all solution for storing OSM data in a database. It very much depends on what you are doing whether and when you want to check for geometry problems. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Using Osmosis for non OSM data and in other projections - remove validation checks?
Hi! There is a whole world of Open Source tools out there that deal with any kind of GIS data in any projection which allows you to do all sorts of things. I suggest you use those true and testes libaries and programs for all your non-OSM needs. That seems like much less work than adding all this functionality to Osmosis. For a start look at OGR (www.gdal.org/ogr) and all the stuff you'll find from http://www.osgeo.org/ . Jochen On Sat, Dec 08, 2012 at 11:29:39AM +, Nick Whitelegg wrote: > Date: Sat, 8 Dec 2012 11:29:39 + > From: Nick Whitelegg > To: osmosis-dev@openstreetmap.org > Subject: [osmosis-dev] Using Osmosis for non OSM data and in other > projections - remove validation checks? > > Hi, > > As part of preparing data for a England-and-Wales wide Android walking app > (opentrail) which uses the mapsforge library, I am making extensive use of > Osmosis to select the OSM data I need and to merge OSM data with Ordnance > Survey LandForm Panorama contours, converted to OSM format. Because OS data > is supplied as British National Grid (BNG) tiles, I need to prepare BNG tiles > of OSM data so I can easily combine the OSM data with the OS contours. > > However, osmosis rejects coordinates outside of the range -180 to 180. I can > get round this by using units of tens of thousands of metres rather than > metres, but I still think that osmosis has a lot of potential uses with data > in other projections. What would people's attitudes be to removing the > restriction that lat/lon has to be between -180 and 180, due to the > possibility that the data is in another projection? > > Another validation that could be removed is the check on the timestamp and > version. Again as part of the above process, I prepare OSM files of the OS > LandForm Panorama data and have to give them a fake timestamp and version but > it would be nice not to have to do this. Opinions on this? > > Thanks, > Nick > ___ > osmosis-dev mailing list > osmosis-dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/osmosis-dev -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Example code for PBF reading
On Di, Mär 18, 2014 at 01:29:57 -0600, Martijn van Exel wrote: > Are there any examples known to you that use the osmosis code in > another project? I am specifically interested in using the PBF reading > code. Or is there perhaps an easier way to add OSM PBF reading to my > own (Java) project? There is some example PBF reading code in the OSM-binary repository https://github.com/scrosby/OSM-binary/blob/master/src.java/crosby/binary/test/ReadFileExample.java Also I think mkgmap (http://www.mkgmap.org.uk/) has its own PBF reading code based on OSM-binary. How good those implementations are, I don't know. Reading PBF files is not so simple, there are undocumented details that can trip you up. So using either of these might be more difficult than with Osmosis. Jochen -- Jochen Topf joc...@remote.org http://www.jochentopf.com/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Building/Debian Package
Hi! The Debian package of Osmosis is really old and doesn't work on current unstable any more. Since then Osmosis has switched build systems from ant to gradle, so we basically have to set up the Debian package build from scratch. I think it is important we get Osmosis in a current version back into Debian and Ubuntu and want to help get this going. Debian doesn't allow getting any binaries from external sources during the building process. Build has to be done from scratch and using only other Debian packages as dependencies. I have already figured out that I can use the Debian "gradle" package and call the gradle from it instead of the "gradlew" script. But it downloads a bunch of files from Maven: Download http://repo1.maven.org/maven2/com/puppycrawl/tools/checkstyle/5.6/checkstyle-5.6.pom Download http://repo1.maven.org/maven2/org/sonatype/oss/oss-parent/5/oss-parent-5.pom Download http://repo1.maven.org/maven2/antlr/antlr/2.7.7/antlr-2.7.7.pom ... I don't know gradle at all. Is there a list of those dependencies somewhere? Any suggestions how to disable all this and tell gradle where to search for its dependecies locally? Jochen -- Jochen Topf joc...@remote.org http://www.jochentopf.com/ +49-721-388298 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Proposal for a multithreaded PBF reader
On Di, Apr 28, 2015 at 06:56:23 -0600, Martijn van Exel wrote:
> Not sure if this has been discussed recently, but we've been thinking
> about improving osmosis PBF reading performance over at Telenav. My
> colleague Jon (cc) has come up with a suggestion that I want to put
> forward for discussion. I'm posting this to both osmosis-dev as well
> as dev because it affects the PBF format definition.
>
> When reading a large PBF resource from a random access file (as
> opposed to a stream), it might be possible to significantly increase
> throughput by reading data of the same entity type from multiple
> threads simultaneously, making use of an optional directory structure
> to locate valid blocks of nodes, ways and relations for threads to
> consume.
>
> To support parallel access, an optional directory_offset might be
> added to the HeaderBlock:
>
> message HeaderBlock {
> …
> optional int64 directory_offset
> }
>
> The directory_offset field would be the seek location in the file of a
> Directory message which is written at the end of the file (since the
> directory is flexible in length and all offsets are only known after
> writing all data to the PBF file). The directory itself is simply a
> list of valid read offsets for each entity type. Threads can read data
> from a given offset in the list to the next offset. The best chunk
> size for blocks in the directory can be determined through
> experimentation, although something around 1MB might be a good first
> guess.
>
> message Directory {
> repeated int64 node_block_offsets;
> repeated int64 way_block_offsets;
> repeated int64 relation_block_offsets;
> }
>
> Before we explore this further, I'd like to know if this has been
> attempted before, and what concerns there may be.
PBF files already come in blocks with a length header in front of every
block. Osmium reads this length header in one thread and then puts the
data of each block into a work queue to be parsed by as many threads as
you want. This way you already get a nice speedup without any changes to
the file format.
Jochen
--
Jochen Topf joc...@remote.org http://www.jochentopf.com/ +49-351-31778688
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] [OSM-dev] Proposal for a multithreaded PBF reader
PBF reading is definitely not IO bound. It is actually quite expensive to parse PBF, so, yes, it about everything you do multi-threaded reading is going to be a win. Jochen On Do, Jun 04, 2015 at 11:04:48 +0200, Andrew Byrd wrote: > Can anyone provide anecdotes of use cases where multi-threaded PBF reading > significantly speeds up processing? Generally I would expect PBF reading to > be IO bound rather than processor bound, but I still need to make more > accurate measurements. > > Of course actually processing the OSM data once the PBF is decoded can be > quite CPU intensive, but that would imply buffering decoded data and > parallelizing geometric operations for example, not the reading. > > I’d appreciate any data points and example use cases you might have, as I’m > currently working on related tooling. > > Andrew Byrd > > > On 04 Jun 2015, at 05:57, Brett Henderson wrote: > > > > On 30 April 2015 at 03:27, Paul Norman > <mailto:penor...@mac.com>> wrote: > > On 4/29/2015 9:55 AM, Martijn van Exel wrote: > > If osmosis is the reference implementation, is there a reason why it > > doesn't seem to leverage this block structure to speed up reading? Or > > does it? > > Osmosis has the --read-pbf-fast task which allows multiple worker threads. > > > > That's right. I forget how the PBF structure works off the top of my head, > > but the file is already split into blocks. The main --read-pbf-fast thread > > simply grabs the outer protobuf blocks from file and then distributes them > > to worker threads who parse out the OSM entities from within the block. > > After extraction, the entities within each block are passed to the > > downstream task in original file order. I'm not sure I see the need to > > modify the PBF file format. > > > > ___ > > osmosis-dev mailing list > > osmosis-dev@openstreetmap.org > > https://lists.openstreetmap.org/listinfo/osmosis-dev > > ___ > osmosis-dev mailing list > osmosis-dev@openstreetmap.org > https://lists.openstreetmap.org/listinfo/osmosis-dev -- Jochen Topf joc...@remote.org http://www.jochentopf.com/ +49-351-31778688 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Cutting PBF file into 1° tiles
On Mo, Apr 18, 2016 at 10:10:06 +0200, Sylvain Melin wrote: > My plan is to : > - exploit a planet sized pbf file > - cut it into 1° tiles using osmosis > - filter and extract the data from these tiles as shapefiles using libosmium If you are writing your own program anyway to create those shapefiles, why don't you do the splitting in this step *after* creating the geometries and before writing them into shapefiles? That is probably much easier to do than based on the PBF due to the structure of the OSM data files. Jochen -- Jochen Topf joc...@remote.org http://www.jochentopf.com/ +49-351-31778688 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Cutting PBF file into 1° tiles
On Mo, Apr 18, 2016 at 11:52:24 +0200, Sylvain Melin wrote:
> On 18/04/2016 11:00, Jochen Topf wrote:
> >On Mo, Apr 18, 2016 at 10:10:06 +0200, Sylvain Melin wrote:
> >>My plan is to :
> >>- exploit a planet sized pbf file
> >>- cut it into 1° tiles using osmosis
> >>- filter and extract the data from these tiles as shapefiles using libosmium
> >If you are writing your own program anyway to create those shapefiles, why
> >don't you do the splitting in this step *after* creating the geometries and
> >before writing them into shapefiles? That is probably much easier to do than
> >based on the PBF due to the structure of the OSM data files.
> >
> >Jochen
> Maybe I'm wrong but because I don't want to parse the full planet.osm.pbf
> every time I want to extract a small set of data.
> The processing time seems to grow exponentially with the size of source file
The time of what processing exactly? I don't see anything in what you are doing
that should scale worse then linearly. Of course if you don't have enough memory
you'll run into problems.
> so having an intermediate level with 1° sized pbf containing everything
> seems very practical to me.
In theory yes, but, as you noticed, you'll have to handle all objects specially
that straddle tile boundaries.
> Also, my osmium program loops over the target tile and parse the appropriate
> pbf :
>
> /for each j in [-90,89]//
> //{//
> //for each i in [-180,179]//
> //{//
> //create osmium::handler//
> //parse i_j.pbf with osmium::io::Reader//
> //extract data to single handler with osmium::apply//
> //}//
> }/
>
> Do you think it would be more efficient to have a single big PBF and extract
> data to several handlers ?
It will probably be most efficient to just do everything in one go. And only at
the moment where you are writing out the finished feature into the shapefile,
decide in which shapefile it should belong. You'll only have one handler, but
180*360 output shapefiles.
> Is it even possible without filling the RAM ?
Depends on how much RAM you have. You'll need 32GB RAM for the node location
store. And you'll need same RAM to buffer the output, because you can't write
to 180*360 files at the same time efficiently. Maybe fewer files would be
better? (Also you'll have not only one shape file for each tile, but probably
dozens for all the different layers of data, which makes this problem worse.)
So if you don't have this kind of memory, you have a problem.
You can also have a look at
https://github.com/joto/osm-history-splitter
which should be more efficient at splitting a planet into smaller files than
Osmosis. But people have reported some issues with this software. It is on my
TODO list to look at this and fix them, but that will take a while.
Jochen
--
Jochen Topf joc...@remote.org http://www.jochentopf.com/ +49-351-31778688
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problems applying changesets
On Mon, Apr 03, 2017 at 02:52:59PM -0400, David Niklas wrote: > Unfortunately, I'm getting an error: > SEVER: Thread for task 1-read-xml-change failed > org.openstreetmap.osmosis.core.OsmosisRuntimeException: This does not > appear to be an OSM Change XML file. > > > Here is my (minimal) command line (I checked the hashes and decompressed > the changesets before hand), (I also checked the hash of the planet > file): > lbzip2 -dkc planet-160718.osm.bz2 | pv -WcarbB 100m | osmosis > --read-xml-change file=changesets-160725.osm compressionMethod=none > --read-xml file=/dev/stdin compressionMethod=none --apply-change > --write-xml file=- compressionMethod=none | pv -WcarbB 100m | lbzip2 > -z4c5 > planet-160725.osm.bz2 > > In the unlikely event that decompression resulted in a corrupted file, I > did try using the compressed and md5sum verified originals, but still got > the same error. You are confusing "changeset" files with "change" files. Those are different beasts. Change files contain the actual changes, they end in .osc or .osc.gz. Changeset files contain the meta information about the changesets, they can't be used to update a planet file. Jochen -- Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations
On Mon, Mar 05, 2018 at 05:18:00PM +, Locke, Jonathan wrote: > Only one technical question for you: is there any way to detect from > the header of the PBF file whether the file contains way node > locations? It would be nice not to have to read nodes for 45 minutes > before discovering that the PBF file doesn't have way node locations. > Once there's an osmosis release with the way node location feature in > it (and ideally a data drop with way node locations), we hope to > switch to consuming only PBF files with way node locations and we'd > remove support from our apps for PBF files without this information > (thus the need to detect if the file has way nodes). Osmium sets "LocationsOnWays" as an "optional feature" string in the header to signify that there are locations on the ways. Jochen -- Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Osmosis and Osmium-enhanced PBF files with way node locations
On Sun, Mar 11, 2018 at 10:19:21AM +, Brett Henderson wrote: > *osmformat.proto* > Are these changes mastered somewhere else? In other words, are these new > fields the same ones used by Osmium in its implementation? I'm wondering > if we need to re-sync from the main OSM-binary project. The osmosis > version is the same as that in https://github.com/scrosby/OSM-binary. The > repo https://github.com/brettch/OSM-binary is a fork of that, and the > osmosis branch there *is* the same code as the osmosis-osm-binary directory > in the osmosis repo, just with some re-arranging to fit inside the Osmosis > structure and fit inside the Osmosis java package namespace. > > *Jochen* (if you're reading), where does Osmium source its osmformat.proto > file from? The PBF implementation in Osmium doesn't need the .proto file, so this didn't come up there. > Long story short, rather than make changes directly to the file in Osmosis > and create a fork, should we apply them to upstream first and then re-sync > Osmosis with that? Yes, we should do this upstream in https://github.com/scrosby/OSM-binary. If somebody creates a pull request, I'll merge it there... Jochen -- Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] feature request: displaying the bounding box
Hi! Osmosis isn't developed further, so it is unlikely that somebody implements your request. But Osmium can do something like what you want with the `fileinfo` command: https://osmcode.org/osmium-tool/ https://docs.osmcode.org/osmium/latest/osmium-fileinfo.html Jochen On Sun, Feb 24, 2019 at 06:57:07PM +0800, andrew goh via osmosis-dev wrote: > Date: Sun, 24 Feb 2019 18:57:07 +0800 > From: andrew goh via osmosis-dev > To: osmosis-dev@openstreetmap.org > Subject: [osmosis-dev] feature request: displaying the bounding box > > hi this is a feature request > > I'm using the Mapsforge Map-Writer plugin to convert osm and pbf maps into > mapforge format > https://github.com/mapsforge/mapsforge/blob/master/docs/Getting-Started-Map-Writer.md > > however, there is one frustration is that Mapsforge Map-Writer plugin > requires that the user specify the bounding box. > > could osmosis provide an option to simply run through the entities and > display the bounding box when it is done? > i've tried the likes of these but there isn't any bounding box info as > relates to the input file being used > > * 3.12 Dataset Tasks > > <https://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.47#Dataset_Tasks> > > o 3.12.1 --dataset-bounding-box (--dbb) > > <https://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.47#--dataset-bounding-box_.28--dbb.29> > o 3.12.2 --dataset-dump (--dd) > > <https://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.47#--dataset-dump_.28--dd.29> > * 3.13 Reporting Tasks > > <https://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.47#Reporting_Tasks> > > o 3.13.1 --report-entity (--re) > > <https://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.47#--report-entity_.28--re.29> > o 3.13.2 --report-integrity (--ri) > > <https://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.47#--report-integrity_.28--ri.29> > > the trouble is often that i've map extracts done either in josm or some > planet osm country / region extracts. > but that after initially setting up the bounding box, i did not store that > anywhere and later 'forget' about it. > i'd later make some guesses from the map and get a new bounding box > coordinates. > > hence, it would be really helpful if we can have a reporting option like > --report-statistics in which osmosis runs through all the entities > and print some basic statistics like the number of different type of > entities and the bounding box of all those entities > > thanks much in advance, > Andrew > ___ > osmosis-dev mailing list > osmosis-dev@openstreetmap.org > https://lists.openstreetmap.org/listinfo/osmosis-dev -- Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Full History pbf's DenseInfo Values are Doubled for Timestamp, Changeset and Uid
Hi! What you are seeing as doubling might be the zigzag-encoding of signed integers. See here: https://developers.google.com/protocol-buffers/docs/encoding#signed-integers Jochen On Thu, Sep 26, 2019 at 01:14:03PM -0700, Pbf Crasher wrote: > Date: Thu, 26 Sep 2019 13:14:03 -0700 > From: Pbf Crasher > To: osmosis-dev@openstreetmap.org > Subject: [osmosis-dev] Full History pbf's DenseInfo Values are Doubled for > Timestamp, Changeset and Uid > > Hi, > > I was reading full history pbf published on planet website and realized > that timestamp, changeset and uid values (which are delta coded) in > DenseInfo are all doubled. Anybody aware of this? > > For example, if you read DenseNodes in history-190916.osm.pbf > <https://planet.openstreetmap.org/pbf/full-history/history-190916.osm.pbf>. > > *First DenseNodes block:* > https://www.openstreetmap.org/node/1/history > DenseNode[0]: id: 1, lat: 0, lon: 0 > DenseInfo[0]: timestamp: 2294571334 (1147285667 x 2 => May 10, 2006 6:27:47 > PM), changeset: 18514 (9257 x 2), uid: 2596 (1298 x 2 > https://www.openstreetmap.org/api/0.6/user/1298), version: 1 > > DenseNode[1]: id: 1, lat: 2000, lon: 2000 > DenseInfo[1]: timestamp: 184876220 ((2294571334 + 184876220) / 2 = > 1239723777), changeset: 1030752 ((18514 + 1030752) / 2 = 524633), uid: 7732 > ((2596 + 7732) / 2 = 5164 => https://www.openstreetmap.org/api/0.6/user/5164), > version: 3 > > *Second DenseNodes block:* > https://www.openstreetmap.org/node/80923/history > DenseNode[0]: id: 80923, lat: 584375459, lon: 155271244 > DenseInfo[0]: timestamp: 2845668844 (1422834422 x 2 => February 1, 2015 > 11:47:02 PM), changeset: 57117152 (28558576 x 2), uid: 57550 (28775 x 2 => > https://www.openstreetmap.org/api/0.6/user/28775), version: 4 > > DenseNode[1]: id: 80923, lat: 584374940, lon: 155271345 > DenseInfo[1]: timestamp: 80492406 ((80492406 + 2845668844) / 2 = 1463080625 > => May 12, 2016 7:17:05 PM), changeset: 21434884 ((57117152 + 21434884) / 2 > = 39276018), uid: 109452 ((57550 + 109452) / 2 = 83501 => > https://www.openstreetmap.org/api/0.6/user/83501), version: 5 > ___ > osmosis-dev mailing list > osmosis-dev@openstreetmap.org > https://lists.openstreetmap.org/listinfo/osmosis-dev -- Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
