Re: [Neo4j] neo4j spatial - dynamic styling
Hi David, I think the problem on my side was simply lack of experience with SLD's. Since FOSS4G I've not spent any time on SLD's either, but instead on some updates to the OSM data model. However, I'm very keen to get the SLD work moving again, so right now I'm trying to turn your test code into a unit test in neo4j-spatial so I can start playing with it, and then integrate it with some code I got from Jesse for embedding the SLD in a uDig datastore so when we connect uDig to OSM in Neo4j it automatically gets great rendering :-) One question for you: the code has new dependencies for org.geotools.map and org.geotools.render. Where can I find these? A quick google search said org.geotools.map was found in gt-build, but that does not seem to exist. I'll look a little further, but if you have a quick answer, that would be great. And you last point about performance and style composition are very important. I'm assuming that the more advanced the styling we apply the slower the rendering, so I've pretty sure I'm going to be getting into deep water there and will certainly benefit from the advice of a real expert in SLD's. Of course, my plan has been to start by using the mapnik2geotools generated SLD's as a starting point, since I'm sure you did a great job with that :-) Regards, Craig On Thu, Oct 7, 2010 at 4:03 AM, David Winslow cdwins...@gmail.com wrote: Hi all, My name's David Winslow and I talked to Craig Taverner a few weeks ago at FOSS4G about Neo4j-spatial. I work with GeoTools and GeoServer and was interested in the demo I saw of OSM data stored and rendered from a Neo4j database instead of one of the more traditional storage systems. We also talked a bit about problems with styling in GeoTools. So, I played around with neo4j-spatial a bit last week to see what was going wrong and styling seems to work ok with the layers I tried. What I did was to run just the TestDynamicLayers suite and then write a small program to load it in GeoTools and render to PNG. http://gist.github.com/614421 is the precise code I used. Since I didn't run into any problems actually doing the styling, I guess the issues we discussed are a bit higher-level (performance, style composition, that kind of thing). Does anyone have the time to explain these problems in a bit more detail? ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j spatial - dynamic styling
I found the correct geotools jar. It is gt-render. On Thu, Oct 7, 2010 at 10:18 AM, Craig Taverner cr...@amanzi.com wrote: Hi David, I think the problem on my side was simply lack of experience with SLD's. Since FOSS4G I've not spent any time on SLD's either, but instead on some updates to the OSM data model. However, I'm very keen to get the SLD work moving again, so right now I'm trying to turn your test code into a unit test in neo4j-spatial so I can start playing with it, and then integrate it with some code I got from Jesse for embedding the SLD in a uDig datastore so when we connect uDig to OSM in Neo4j it automatically gets great rendering :-) One question for you: the code has new dependencies for org.geotools.map and org.geotools.render. Where can I find these? A quick google search said org.geotools.map was found in gt-build, but that does not seem to exist. I'll look a little further, but if you have a quick answer, that would be great. And you last point about performance and style composition are very important. I'm assuming that the more advanced the styling we apply the slower the rendering, so I've pretty sure I'm going to be getting into deep water there and will certainly benefit from the advice of a real expert in SLD's. Of course, my plan has been to start by using the mapnik2geotools generated SLD's as a starting point, since I'm sure you did a great job with that :-) Regards, Craig On Thu, Oct 7, 2010 at 4:03 AM, David Winslow cdwins...@gmail.com wrote: Hi all, My name's David Winslow and I talked to Craig Taverner a few weeks ago at FOSS4G about Neo4j-spatial. I work with GeoTools and GeoServer and was interested in the demo I saw of OSM data stored and rendered from a Neo4j database instead of one of the more traditional storage systems. We also talked a bit about problems with styling in GeoTools. So, I played around with neo4j-spatial a bit last week to see what was going wrong and styling seems to work ok with the layers I tried. What I did was to run just the TestDynamicLayers suite and then write a small program to load it in GeoTools and render to PNG. http://gist.github.com/614421 is the precise code I used. Since I didn't run into any problems actually doing the styling, I guess the issues we discussed are a bit higher-level (performance, style composition, that kind of thing). Does anyone have the time to explain these problems in a bit more detail? ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] An attempt at documenting the internals and architecture of Neo
Hi Chris, Absolutely awesome stuff you have written. As a new hire in Neo Technologies, I'm reading your blog posts with great interest. I like how you take code and make a story out of it - it's much easier to understand the code after reading your posts on it. Thank you! I have two wishes that would make your posts even more awesomer (for me at least). 1. I'd love it if you could point to the code. In this post, you write about getNodeById() - why not make the method name a link to the class file in https://svn.neo4org? 2. I think that it would be interesting to have a post about how backups are done. Personally, I have always been interested in how you make a backup of a live database without using huge locks. But maybe that's just me... :) You are doing a great job Chris. If you have any questions, ask away here on the list. Andrés On Thu, Oct 7, 2010 at 1:43 PM, Chris Gioran chris.gio...@gmail.com wrote: Hi list, I have started a series of small articles that comprise an attempt to provide an exposition of the internals of Neo, explaining some of the inner workings and the functions of some core classes. So far I have taken a shot at the startup/base class creation and the caching subsystem. You can find them here: http://digitalstain.blogspot.com/2010/10/inside-neo4j-hello-world.html http://digitalstain.blogspot.com/2010/10/neo4j-internals-caching.html This effort begun as a personal documentation project to keep notes as I was going through the code, but judging the result as potentially useful to a broader audience I thought it might be a good idea to share. I am, however, not convinced that the format is explanatory enough or deep enough (or correct, for that matter :), so I was hoping for some community feedback for improvement. I have still to do the persistence layer, the traversal framework and, of course, transactions. Is there anything else I should cover (or is this a complete waste of time)? Thank you for your time. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j spatial - dynamic styling
The most important thing you can do to improve performance is to ensure that your style renders an appropriate number of features (if you want render every driveway at state level it will take a while) and that your datastore can provide fast access to those features (ie, well-indexed). Beyond that, most styling options have negligible impact on rendering speed. (However, with as many different style rules as you have with OSM data things will start to add up.) wrt to mapnik2geotools - Mapnik allows styles to specify a postgres * subquery* in the table name field (so generated requests come out like select foo from (select foo, bar, baz from quux as roads) which is totally valid). GeoTools doesn't support this option, so my approach so far has been to inspect the datastore parameters and for every table that looks like a subquery, I generate an actual table - create table roads as (select foo, bar, baz from quux). While expedient, this approach doesn't translate well to other datastores. So, I think the first barrier to using mapnik2geotools with neo4j-spatial will be coming up with a solution to this problem that works with neo4j as the database. -- David Winslow On Thu, Oct 7, 2010 at 4:47 AM, Craig Taverner cr...@amanzi.com wrote: I found the correct geotools jar. It is gt-render. On Thu, Oct 7, 2010 at 10:18 AM, Craig Taverner cr...@amanzi.com wrote: Hi David, I think the problem on my side was simply lack of experience with SLD's. Since FOSS4G I've not spent any time on SLD's either, but instead on some updates to the OSM data model. However, I'm very keen to get the SLD work moving again, so right now I'm trying to turn your test code into a unit test in neo4j-spatial so I can start playing with it, and then integrate it with some code I got from Jesse for embedding the SLD in a uDig datastore so when we connect uDig to OSM in Neo4j it automatically gets great rendering :-) One question for you: the code has new dependencies for org.geotools.map and org.geotools.render. Where can I find these? A quick google search said org.geotools.map was found in gt-build, but that does not seem to exist. I'll look a little further, but if you have a quick answer, that would be great. And you last point about performance and style composition are very important. I'm assuming that the more advanced the styling we apply the slower the rendering, so I've pretty sure I'm going to be getting into deep water there and will certainly benefit from the advice of a real expert in SLD's. Of course, my plan has been to start by using the mapnik2geotools generated SLD's as a starting point, since I'm sure you did a great job with that :-) Regards, Craig On Thu, Oct 7, 2010 at 4:03 AM, David Winslow cdwins...@gmail.com wrote: Hi all, My name's David Winslow and I talked to Craig Taverner a few weeks ago at FOSS4G about Neo4j-spatial. I work with GeoTools and GeoServer and was interested in the demo I saw of OSM data stored and rendered from a Neo4j database instead of one of the more traditional storage systems. We also talked a bit about problems with styling in GeoTools. So, I played around with neo4j-spatial a bit last week to see what was going wrong and styling seems to work ok with the layers I tried. What I did was to run just the TestDynamicLayers suite and then write a small program to load it in GeoTools and render to PNG. http://gist.github.com/614421 is the precise code I used. Since I didn't run into any problems actually doing the styling, I guess the issues we discussed are a bit higher-level (performance, style composition, that kind of thing). Does anyone have the time to explain these problems in a bit more detail? ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] An attempt at documenting the internals and architecture of Neo
On Thu, Oct 7, 2010 at 2:59 PM, Andres Taylor andres.tay...@neotechnology.com wrote: Hi Chris, Absolutely awesome stuff you have written. As a new hire in Neo Technologies, I'm reading your blog posts with great interest. I like how you take code and make a story out of it - it's much easier to understand the code after reading your posts on it. Thank you! I am happy to know that it proved useful to someone. Thank you. I have two wishes that would make your posts even more awesomer (for me at least). 1. I'd love it if you could point to the code. In this post, you write about getNodeById() - why not make the method name a link to the class file in https://svn.neo4org? Fair enough. I had decided against it because I don't like to litter the text with links and, given the expectation that there would be many references, I feared it would be distracting. Moreover, if someone is reading my posts I assumed she has already downloaded the code and set it up in her IDE. However, I think I should add links to at least some of the referenced code, to save some trouble and link back to the project website. 2. I think that it would be interesting to have a post about how backups are done. Personally, I have always been interested in how you make a backup of a live database without using huge locks. But maybe that's just me... :) Nice idea. Added in the TODO list. You are doing a great job Chris. If you have any questions, ask away here on the list. Appreciate it. Andrés CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Architecturing neo4j for scalability
Sorry Peter for the late reply (and the clumsy first email). I drew a very very simple diagram exemplifying the architecture, I don't know if the list will support the attachment so if it doesn't, warn me :P As you can see by the diagram, my neo4j DB is running inside the REST app that will read and write from the DB and also perform. The Reader is an application which reads out a queue, checks relationships of an user making some action and send some messages based on those relationships. The website is indeed a back-end application which the real website is based on. My question was: when I need to scale the REST app (because of the huge load that the mentioned website will generate) what should I do with the neo4j embedded in it? Thanks for your attention again. On Mon, Oct 4, 2010 at 10:17 AM, Peter Neubauer peter.neuba...@neotechnology.com wrote: Victor, sorry, I am not entirely understanding your domain so I can't say anything for sure. Do you have some architectural diagram and/or some example flow in order to better understand? Sorry for the inconvenience Cheers, /peter neubauer VP Product Development, Neo Technology GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Tue, Sep 28, 2010 at 4:47 AM, Victor Augusto de Campos piv...@gmail.com wrote: Hey folks, I'm actually developing an application that requires maintaining following relationships between nodes, another application reads out those relationships and take the steps to perform notifications about actions to some entities. That said, I got a doubt about the best design to maintain all things going well. As I've benchmarked neo4j alone can easily handle the reading/writing load but as I'm running it as a embedded database into the application who also provides a RESTful API to let other applications notify about actions I'm afraid that when I need to scale out this application I'll have troubles scaling neo4j embedded in it. So, what would be the best approach? Decouple neo4j from that application and let it run as a separate app (or use neo4j-rest, although I haven't looked into its API and don't know if it provides a official client, but well) or if it would be simple to scale out neo4j DB with the application providing that RESTful API. I don't know if I exposed my question clear, any doubts I'll be glad to answer ^^ Best regards, Victor Augusto de Campos ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user attachment: diagram.png___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j spatial - dynamic styling
You may also find the styling section in the GeoServer manual useful while
getting up to speed on SLD:
http://docs.geoserver.org/stable/en/user/styling/index.html
wrt filtering out small features, I think that is generally done in the
stylesheet (GeoTools combines the filters of all the style rules together
and sends them to the database so that GeoTools fetches only what will
actually be rendered from the database.) This avoids having a hard-coded
decision about the size of the important features in the rendering code,
which as you note does not work very well for point data.
Is there documentation about the JSON query language beyond what is
available in the JavaDoc for DynamicLayer? In the short term, I am mainly
curious about (not) NULL tests, OR, and testing for specific values of
properties since these are the most used predicates in the MapQuest styles;
for example:
CREATE TABLE landmark AS
SELECT way, aeroway, amenity, landuse, leisure,
man_made, military, natural, power, shop, tourism, name,
religion
FROM planet_osm_polygon
WHERE landuse is not null or leisure is not null
or shop is not null
or aeroway in ('apron', 'aerodrome', 'taxiway', 'runway')
or amenity in ('parking', 'university', 'college', 'school',
'hospital',
'kindergarten', 'grave_yard')
or military in ('barracks', 'danger_area')
or natural in ('field', 'beach', 'heath', 'mud', 'wood')
or power in ('station', 'sub_station')
or tourism in ('attraction', 'camp_site', 'caravan_site',
'picnic_site', '
ORDER BY z_order, way_area DESC;
--
David Winslow
On Thu, Oct 7, 2010 at 3:52 PM, Craig Taverner cr...@amanzi.com wrote:
Hi David,
The most important thing you can do to improve performance is to ensure
that
your style renders an appropriate number of features (if you want
render
every driveway at state level it will take a while) and that your
datastore
can provide fast access to those features (ie, well-indexed). Beyond
that,
most styling options have negligible impact on rendering speed. (However,
with as many different style rules as you have with OSM data things will
start to add up.)
Seems like a useful option is for the DataStore to only return geometries
that are 'big enough' to matter at the resolution in question. This could
be
a clever modification to the spatial index itself, so that it is passed a
resolution hint during query time and simply ignores data that is 'too
small'. The concept of 'too small' is the tricky part, because it is not a
purely spatial concept. Some points are too small, while others are not,
depending on the tags. Points representing the location of cities are not
too small, while points representing the location of trees certainly. Of
course this can also be solved client side by putting such different
objects
in different layers and setting zoom level filters on the entire layer.
wrt to mapnik2geotools - Mapnik allows styles to specify a postgres *
subquery* in the table name field (so generated requests come out like
select
foo from (select foo, bar, baz from quux as roads) which is totally
valid).
GeoTools doesn't support this option, so my approach so far has been to
inspect the datastore parameters and for every table that looks like a
subquery, I generate an actual table - create table roads as (select
foo,
bar, baz from quux). While expedient, this approach doesn't translate
well
to other datastores.
So, I think the first barrier to using mapnik2geotools with neo4j-spatial
will be coming up with a solution to this problem that works with neo4j
as
the database.
My plan is for the DynamicLayers to support this kind of thing. You could
view these as database views, which is similar to your idea of generating
an
actual table, minus the data duplication, of course. Right now my dynamic
layers only filter on combinations of OSM tags, but there is not reason why
not to filter on much more complex things. Today the filter is, in fact, a
graph matching algorithms, specified in JSON. The JSON document describes
how to traverse the graph checking for attributes on various nodes. The
original reason for such a complex solution was because the OSM tags node
is
two steps away from the geometry node. However, now that it exists I can
hopefully use it for much more powerful queries of sub-graphs.
By the way, I added a modified version of your code to a unit test in
neo4j-spatial (not yet commited, but soon), and also played around with the
SLD to see if I could get some more interesting effects. I manage to
generate the attached image, which has three levels of road rendered with
different colors, all with dark borders. The bridges are highlighted, as
you
had, but now also with glowing red labels. I got the labeling from the SLD
tutorial on the geoserver wiki at
Re: [Neo4j] neo4j spatial - dynamic styling
So far the only docs for the JSON syntax are in the code. But you are
welcome to read the JSON parsing code itself. It is very short, albeit
cryptic.
I'm sure I support 'not null', but I am also sure I do not support OR. I
need to think a bit harder about how to support OR, since the current
compact code was designed around supporting AND. It should be possible. But
it is 1am now, and I definitely will not figure that out tonight ;-)
On Fri, Oct 8, 2010 at 1:18 AM, David Winslow cdwins...@gmail.com wrote:
You may also find the styling section in the GeoServer manual useful while
getting up to speed on SLD:
http://docs.geoserver.org/stable/en/user/styling/index.html
wrt filtering out small features, I think that is generally done in the
stylesheet (GeoTools combines the filters of all the style rules together
and sends them to the database so that GeoTools fetches only what will
actually be rendered from the database.) This avoids having a hard-coded
decision about the size of the important features in the rendering code,
which as you note does not work very well for point data.
Is there documentation about the JSON query language beyond what is
available in the JavaDoc for DynamicLayer? In the short term, I am mainly
curious about (not) NULL tests, OR, and testing for specific values of
properties since these are the most used predicates in the MapQuest styles;
for example:
CREATE TABLE landmark AS
SELECT way, aeroway, amenity, landuse, leisure,
man_made, military, natural, power, shop, tourism, name,
religion
FROM planet_osm_polygon
WHERE landuse is not null or leisure is not null
or shop is not null
or aeroway in ('apron', 'aerodrome', 'taxiway', 'runway')
or amenity in ('parking', 'university', 'college', 'school',
'hospital',
'kindergarten', 'grave_yard')
or military in ('barracks', 'danger_area')
or natural in ('field', 'beach', 'heath', 'mud', 'wood')
or power in ('station', 'sub_station')
or tourism in ('attraction', 'camp_site', 'caravan_site',
'picnic_site', '
ORDER BY z_order, way_area DESC;
--
David Winslow
On Thu, Oct 7, 2010 at 3:52 PM, Craig Taverner cr...@amanzi.com wrote:
Hi David,
The most important thing you can do to improve performance is to ensure
that
your style renders an appropriate number of features (if you want
render
every driveway at state level it will take a while) and that your
datastore
can provide fast access to those features (ie, well-indexed). Beyond
that,
most styling options have negligible impact on rendering speed.
(However,
with as many different style rules as you have with OSM data things
will
start to add up.)
Seems like a useful option is for the DataStore to only return geometries
that are 'big enough' to matter at the resolution in question. This could
be
a clever modification to the spatial index itself, so that it is passed a
resolution hint during query time and simply ignores data that is 'too
small'. The concept of 'too small' is the tricky part, because it is not
a
purely spatial concept. Some points are too small, while others are not,
depending on the tags. Points representing the location of cities are not
too small, while points representing the location of trees certainly. Of
course this can also be solved client side by putting such different
objects
in different layers and setting zoom level filters on the entire layer.
wrt to mapnik2geotools - Mapnik allows styles to specify a postgres *
subquery* in the table name field (so generated requests come out like
select
foo from (select foo, bar, baz from quux as roads) which is totally
valid).
GeoTools doesn't support this option, so my approach so far has been
to
inspect the datastore parameters and for every table that looks like a
subquery, I generate an actual table - create table roads as (select
foo,
bar, baz from quux). While expedient, this approach doesn't translate
well
to other datastores.
So, I think the first barrier to using mapnik2geotools with
neo4j-spatial
will be coming up with a solution to this problem that works with neo4j
as
the database.
My plan is for the DynamicLayers to support this kind of thing. You could
view these as database views, which is similar to your idea of generating
an
actual table, minus the data duplication, of course. Right now my dynamic
layers only filter on combinations of OSM tags, but there is not reason
why
not to filter on much more complex things. Today the filter is, in fact,
a
graph matching algorithms, specified in JSON. The JSON document describes
how to traverse the graph checking for attributes on various nodes. The
original reason for such a complex solution was because the OSM tags node
is
two steps away from the geometry node. However,
