Re: [OSM-dev] lua+osm2pgsql=?
On 05/10/2013 02:14 AM, Peter Wendorff wrote: Hi Kai. I'm not sure, but as far as I understand you didn't say anything about the use case of dropping tags from a relation to members. Let's say motorway A44 is tagged as a relation and the individual ways don't carry the ref-tag (ref=A44). A common use case IMHO is to drop the tags down to the ways for rendering, because that's far more easy than to draw highway shields out of the relations attributes. Is that possible in your lua implementation, too? No, that is currently not possible. Given the internal workings of osm2pgsql it is also unlikely to be added any time soon, as the levels of abstractions aren't really designed to handle that case. For your example, you could probably go the other way, i.e. elevate all of the way tags to the relation, then add the relation as a line entity and filter out the individual ways to not render things twice. However, that only works if all of the tags you care about are the same for all member ways. This is probably the case for rendering and motorways, but likely doesn't work for other route relations, as e.g. the name of a primary road might change many times for the same ref, so you can't replace the individual ways with a single relation line string. Kai regards Peter Am 09.05.2013 23:02, schrieb Kai Krueger: Hello everyone, In response to the previous thread, I have cleaned up the initial proof of concept patch, expanded it somewhat and I have now committed it into trunk of osm2pgsql. As mentioned before, this allows to filter and transform (sets of) tags on nodes, ways and relations before continuing processing in osm2pgsql. Hopefully this can be used for a variety of things, like normalising tagging, or more sophisticated relation processing. By default it still uses the old tag processing pipeline, but using the command line switch --tag-transform-script /path/to/tagprocessing_script.lua should enable the new more flexible pathway. At the moment it only works in the rendering output, but it shouldn't be too difficult to extend it to the gazetteer output if needed / wished. The lua script you pass into osm2pgsql needs to implement 4 functions function filter_tags_node function filter_tags_way function filter_basic_tags_rel function filter_tags_relation_member The first three each take a set of tags as a lua key-value table and return a transformed (or unchanged) set of tags back. They also return a flag to say if the entity(way/node/relation) should be filtered out and not added to the database (they will still end up in the slim mode tables, but not in the rendering tables). The filter_tags_way function furthermore returns a flag if the way should be treated as a line or as a polygon. The function filter_tags_relation_member is a bit more complex and allows to deal with more advanced relation tagging, such as multi-polygons that take can take their tags from the member ways. This function therefore takes the set of tags from the relation, as well as the set of tags and role for each of the member ways (member relations and nodes get ignored). It then returns a transformed (and combined) set of tags to be applied to the relation in later processing. It furthermore returns a couple of additional information. First of all, one can specify for each member way, if it has already been dealt with, or needs to (potentially) have its own entry. E.g. outer ways in multi-polygon relations are superseded by the multi-polygon geometry. Tagged inner ways on the other hand still need to be processed as separate entries. Secondly, one can specify if the relation should be processed as a line, a polygon, or both (e.g. administrative boundaries). Thirdly, one can again specify to discard the entity from further processing. There is a sample tag transform lua script in the repository as an example, which (nearly) replicates current processing and can hopefully be used as a template for ones own scripts. Performance wise, the lua tag transform is slower than the C based one, however, it is probably not as bad as I feared. I haven't done an extensive performance analysis yet though and it will likely heavily depend on the complexity of the tag transform script. Everything should hopefully work, but as with any new feature and particularly as I don't have a immediate use case for it myself, there is a non negligible chance that there are bugs in the committed code, so treat it with caution to begin with. As to support this a fair amount of refactoring was necessary, there is also a possibility that the default c-based tag transform got new bugs as well. I don't have the resources to test it on a full planet import, but on all of my test extracts, the non lua code produced the same database as before the commit. So hopefully things are OK. For production systems, you should stick with the tagged 0.82.0 version of osm2pgsql. But I would very much appreciate any feedback on the new lua tag-transfo
Re: [OSM-dev] lua+osm2pgsql=?
Hi Kai. I'm not sure, but as far as I understand you didn't say anything about the use case of dropping tags from a relation to members. Let's say motorway A44 is tagged as a relation and the individual ways don't carry the ref-tag (ref=A44). A common use case IMHO is to drop the tags down to the ways for rendering, because that's far more easy than to draw highway shields out of the relations attributes. Is that possible in your lua implementation, too? regards Peter Am 09.05.2013 23:02, schrieb Kai Krueger: > Hello everyone, > > In response to the previous thread, I have cleaned up the initial proof of > concept patch, expanded it somewhat and I have now committed it into trunk > of osm2pgsql. > > As mentioned before, this allows to filter and transform (sets of) tags on > nodes, ways and relations before continuing processing in osm2pgsql. > Hopefully this can be used for a variety of things, like normalising > tagging, or more sophisticated relation processing. > > By default it still uses the old tag processing pipeline, but using the > command line switch --tag-transform-script /path/to/tagprocessing_script.lua > should enable the new more flexible pathway. At the moment it only works in > the rendering output, but it shouldn't be too difficult to extend it to the > gazetteer output if needed / wished. > > The lua script you pass into osm2pgsql needs to implement 4 functions > > function filter_tags_node > function filter_tags_way > function filter_basic_tags_rel > function filter_tags_relation_member > > The first three each take a set of tags as a lua key-value table and return > a transformed (or unchanged) set of tags back. They also return a flag to > say if the entity(way/node/relation) should be filtered out and not added to > the database (they will still end up in the slim mode tables, but not in the > rendering tables). The filter_tags_way function furthermore returns a flag > if the way should be treated as a line or as a polygon. > > The function filter_tags_relation_member is a bit more complex and allows to > deal with more advanced relation tagging, such as multi-polygons that take > can take their tags from the member ways. > > This function therefore takes the set of tags from the relation, as well as > the set of tags and role for each of the member ways (member relations and > nodes get ignored). It then returns a transformed (and combined) set of tags > to be applied to the relation in later processing. It furthermore returns a > couple of additional information. First of all, one can specify for each > member way, if it has already been dealt with, or needs to (potentially) > have its own entry. E.g. outer ways in multi-polygon relations are > superseded by the multi-polygon geometry. Tagged inner ways on the other > hand still need to be processed as separate entries. Secondly, one can > specify if the relation should be processed as a line, a polygon, or both > (e.g. administrative boundaries). Thirdly, one can again specify to discard > the entity from further processing. > > There is a sample tag transform lua script in the repository as an example, > which (nearly) replicates current processing and can hopefully be used as a > template for ones own scripts. > > Performance wise, the lua tag transform is slower than the C based one, > however, it is probably not as bad as I feared. I haven't done an extensive > performance analysis yet though and it will likely heavily depend on the > complexity of the tag transform script. > > Everything should hopefully work, but as with any new feature and > particularly as I don't have a immediate use case for it myself, there is a > non negligible chance that there are bugs in the committed code, so treat it > with caution to begin with. As to support this a fair amount of refactoring > was necessary, there is also a possibility that the default c-based tag > transform got new bugs as well. I don't have the resources to test it on a > full planet import, but on all of my test extracts, the non lua code > produced the same database as before the commit. So hopefully things are OK. > > For production systems, you should stick with the tagged 0.82.0 version of > osm2pgsql. But I would very much appreciate any feedback on the new lua > tag-transform possibilities. > > Kai > > > > > > > Kai Krueger wrote >> In response to Richard's suggestion, I hacked up a proof of concept >> implementation of a tag filter in lua last week. >> >> Currently it allows you to write a filter function (one for nodes, one for >> ways and one for relations) in lua that takes in a set of tags and returns >> a (potentially) transformed set of tags and a boolean flag, if the object >> should be processed further. For ways, it also determines if it should be >> treated as a line or polygon. This should allow for things like >> normalising the data from "oneway=yes/1/true" or rewriting things like >> "highway=footway" into "highway=path, foot=de
Re: [OSM-dev] lua+osm2pgsql=?
Hello everyone, In response to the previous thread, I have cleaned up the initial proof of concept patch, expanded it somewhat and I have now committed it into trunk of osm2pgsql. As mentioned before, this allows to filter and transform (sets of) tags on nodes, ways and relations before continuing processing in osm2pgsql. Hopefully this can be used for a variety of things, like normalising tagging, or more sophisticated relation processing. By default it still uses the old tag processing pipeline, but using the command line switch --tag-transform-script /path/to/tagprocessing_script.lua should enable the new more flexible pathway. At the moment it only works in the rendering output, but it shouldn't be too difficult to extend it to the gazetteer output if needed / wished. The lua script you pass into osm2pgsql needs to implement 4 functions function filter_tags_node function filter_tags_way function filter_basic_tags_rel function filter_tags_relation_member The first three each take a set of tags as a lua key-value table and return a transformed (or unchanged) set of tags back. They also return a flag to say if the entity(way/node/relation) should be filtered out and not added to the database (they will still end up in the slim mode tables, but not in the rendering tables). The filter_tags_way function furthermore returns a flag if the way should be treated as a line or as a polygon. The function filter_tags_relation_member is a bit more complex and allows to deal with more advanced relation tagging, such as multi-polygons that take can take their tags from the member ways. This function therefore takes the set of tags from the relation, as well as the set of tags and role for each of the member ways (member relations and nodes get ignored). It then returns a transformed (and combined) set of tags to be applied to the relation in later processing. It furthermore returns a couple of additional information. First of all, one can specify for each member way, if it has already been dealt with, or needs to (potentially) have its own entry. E.g. outer ways in multi-polygon relations are superseded by the multi-polygon geometry. Tagged inner ways on the other hand still need to be processed as separate entries. Secondly, one can specify if the relation should be processed as a line, a polygon, or both (e.g. administrative boundaries). Thirdly, one can again specify to discard the entity from further processing. There is a sample tag transform lua script in the repository as an example, which (nearly) replicates current processing and can hopefully be used as a template for ones own scripts. Performance wise, the lua tag transform is slower than the C based one, however, it is probably not as bad as I feared. I haven't done an extensive performance analysis yet though and it will likely heavily depend on the complexity of the tag transform script. Everything should hopefully work, but as with any new feature and particularly as I don't have a immediate use case for it myself, there is a non negligible chance that there are bugs in the committed code, so treat it with caution to begin with. As to support this a fair amount of refactoring was necessary, there is also a possibility that the default c-based tag transform got new bugs as well. I don't have the resources to test it on a full planet import, but on all of my test extracts, the non lua code produced the same database as before the commit. So hopefully things are OK. For production systems, you should stick with the tagged 0.82.0 version of osm2pgsql. But I would very much appreciate any feedback on the new lua tag-transform possibilities. Kai Kai Krueger wrote > In response to Richard's suggestion, I hacked up a proof of concept > implementation of a tag filter in lua last week. > > Currently it allows you to write a filter function (one for nodes, one for > ways and one for relations) in lua that takes in a set of tags and returns > a (potentially) transformed set of tags and a boolean flag, if the object > should be processed further. For ways, it also determines if it should be > treated as a line or polygon. This should allow for things like > normalising the data from "oneway=yes/1/true" or rewriting things like > "highway=footway" into "highway=path, foot=designated", which could > hopefully simplify this processing out of the mapnik stylesheets. > > The current proof of concept implementation only works for the rendering > output so far, not the gazeteer output. > > The question now is what to do with this? Is there enough interest for it > to be worth cleaning it up and committing it? Should it replace the > current C filters, or be an additional option? How much performance hit is > acceptable for it to become the default option? Should it not be committed > in this form and rather worked on a more generic solution to include the > gazeteer backend? > Are there other things that would be good to be able to script outside o
Re: [OSM-dev] lua+osm2pgsql=?
Sven Geggus wrote > Kai Krueger < > kakrueger@ > > wrote: > >> The question now is what to do with this? Is there enough interest for it >> to >> be worth cleaning it up and committing it? > > I once did a somewhat ugly hardcoded hack for converting width and height > tags to useful floating point values by doing a lot of heuristics > (output-pgsql.c line 460ff.) > > Am I right in the asumption, that we can get rid at least of parts of > this as well using your code? I think you should be able to. (The current patch didn't include that and I haven't checked it though). Your code seems to work at a later stage of the processing pipeline at the stage of actually creating the sql statements. But if I understood correctly what you are trying to do, it should be possible to do this in the early stages of processing too, by transforming all of the tags into a normalised form, and then one only needs to deal with that normalised form during the later sql generation stage. Kai -- View this message in context: http://gis.19327.n5.nabble.com/lua-osm2pgsql-tp5757509p5758915.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] lua+osm2pgsql=?
Kai Krueger wrote: > The question now is what to do with this? Is there enough interest for it to > be worth cleaning it up and committing it? I once did a somewhat ugly hardcoded hack for converting width and height tags to useful floating point values by doing a lot of heuristics (output-pgsql.c line 460ff.) Am I right in the asumption, that we can get rid at least of parts of this as well using your code? Sven -- Why are there so many Unix-haters-handbooks and not even one Microsoft-Windows-haters handbook? Gurer vf ab arrq sbe n unaqobbx gb ungr Zvpebfbsg Jvaqbjf! /me is giggls@ircnet, http://sven.gegg.us/ on the Web ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] lua+osm2pgsql=?
Kai Krueger wrote: > The question now is what to do with this? Is there enough interest > for it to be worth cleaning it up and committing it? Looks superb to me - really impressed you've done this. It'll remove an entire post-processing stage from my setup: at present I munge the tags in the database after import, but I only get away with that because I'm just running a weekly import of a UK extract. +1 to committing this. cheers Richard -- View this message in context: http://gis.19327.n5.nabble.com/lua-osm2pgsql-tp5757509p5758855.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] lua+osm2pgsql=?
In response to Richard's suggestion, I hacked up a proof of concept implementation of a tag filter in lua last week. Currently it allows you to write a filter function (one for nodes, one for ways and one for relations) in lua that takes in a set of tags and returns a (potentially) transformed set of tags and a boolean flag, if the object should be processed further. For ways, it also determines if it should be treated as a line or polygon. This should allow for things like normalising the data from "oneway=yes/1/true" or rewriting things like "highway=footway" into "highway=path, foot=designated", which could hopefully simplify this processing out of the mapnik stylesheets. The current proof of concept implementation only works for the rendering output so far, not the gazeteer output. The question now is what to do with this? Is there enough interest for it to be worth cleaning it up and committing it? Should it replace the current C filters, or be an additional option? How much performance hit is acceptable for it to become the default option? Should it not be committed in this form and rather worked on a more generic solution to include the gazeteer backend? Are there other things that would be good to be able to script outside of the source code with lua? How does it interact with the current styling of the columns in the osm2pgsql schema? For anyone interested, they can find the current patch at http://pastebin.com/Jrh9tA8i Kai Richard Fairhurst wrote > One of the many wondrous things about OSRM is that you handle the speed > impact of different tags (e.g. highway=motorway vs highway=unclassified) > with plugins written in Lua, a fast but easy-to-understand scripting > language. > > Wouldn't it be great to have the same capability in osm2pgsql? > > Think: path rendering. Right now, you have to potentially weigh up > highway=, access=, bicycle/foot/horse/etc.=, designation, and surface= > tags. That's a whole bunch of Mapnik rules (or whatever) - slow to > write, slow to run. Remapping the tags on database import with osm2pgsql > would fix this. > > Adding this to osm2pgsql is way beyond my poor brane, I'm afraid, though > I'd love to do it. But it would make a great GSoC project: > > http://wiki.openstreetmap.org/wiki/Google_Summer_of_Code/2013/Project_Ideas#Data_processing > > or maybe someone might feel inspired to just code it. ;) > > (Thanks to lonvia and Gnonthgol in #osm-dev for suggestions leading to > this!) > > cheers > Richard > > > ___ > dev mailing list > dev@ > http://lists.openstreetmap.org/listinfo/dev -- View this message in context: http://gis.19327.n5.nabble.com/lua-osm2pgsql-tp5757509p5758794.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] lua+osm2pgsql=?
One of the many wondrous things about OSRM is that you handle the speed impact of different tags (e.g. highway=motorway vs highway=unclassified) with plugins written in Lua, a fast but easy-to-understand scripting language. Wouldn't it be great to have the same capability in osm2pgsql? Think: path rendering. Right now, you have to potentially weigh up highway=, access=, bicycle/foot/horse/etc.=, designation, and surface= tags. That's a whole bunch of Mapnik rules (or whatever) - slow to write, slow to run. Remapping the tags on database import with osm2pgsql would fix this. Adding this to osm2pgsql is way beyond my poor brane, I'm afraid, though I'd love to do it. But it would make a great GSoC project: http://wiki.openstreetmap.org/wiki/Google_Summer_of_Code/2013/Project_Ideas#Data_processing or maybe someone might feel inspired to just code it. ;) (Thanks to lonvia and Gnonthgol in #osm-dev for suggestions leading to this!) cheers Richard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
