Re: Solr Geospatial Polygon Indexing/Querying Issue
On Tue, Jul 30, 2019 at 4:41 PM Sanders, Marshall (CAI - Atlanta) <
marshall.sande...@coxautoinc.com> wrote:
> I’ll explain the context around the use case we’re trying to solve and
> then attempt to respond as best I can to each of your points. What we have
> is a list of documents that in our case the location is sometimes a point
> and sometimes a circle. These basically represent (in our case) inventory
> at a physical location (point) or inventory that can be delivered to you
> within X km (configurable per document) which represents the circle use
> case. We want to be able to allow a user to say I want all documents
> within X distance of my location, but also all documents that are able to
> be delivered to your point where the delivery distance is defined on the
> inventory (creating the circle).
>
That background info helps me understand things!
> This is why we were actually trying to combine both point based data and
> poly/circle data into a single geospatial field, since I don’t believe you
> could do something like fq=geofilt(latlng, x, y, d) OR
> geofilt(latlngCircle, x, y, 1) but perhaps we’re just not getting quite the
> right syntax, etc.
>
Oh quite possible :-). It would look something like this: fq= {!geofilt
sfield=latLng d=queryDistance} OR {!geofilt sfield=latLngCircle
d=0}&pt=myLocation
Notice the space after the fq= which is critical so that the first
local-params (i.e. first geofilt) does not "own" the entire filter query
string end to end. Due to the space, the whole thing is parsed by the
default lucene/standard query parser, and then we have the two clauses
clearly there. The second geofilt has distance 0; it'd be nice if it
internally optimized to a point but nonetheless it's fine. Alternatively
there's another syntax to embed WKT where you can specify a point
explicitly... something like this: ... {!field f=latLngCircle
v="Intersects(POINT(x y))"}
That said, it's also just fine to do as you were planning -- have one RPT
based field for the shape representation (mixture of points and circles),
and one LLPSF field purely for the center point that is used for sorting.
That LLPSF field would be indexed=false docValues=true since you wouldn't
be filtering on it.
>
> * Generally RptWithGeometrySpatialField should be used over
> SpatialRecursivePrefixTreeFieldType unless you want heatmaps or are willing
> to make trade-offs in higher index size and lossy precision in order to get
> faster search. It's up to you; if you benchmark both I'd love to hear how
> it went.
>
> -We may explore both but typically we’re more interested in speed
> than accuracy, benchmarking it may be a very interesting exercise however.
> For sorting for instance we’re actually using sqedist instead of geodist
> because we’re not overly concerned about sorting accuracy.
>
Okay... though geodist on a LLPSF field is remarkably optimized.
> * I see you are using Geo3D, which is not the default. Geo3D is strict
> about the coordinate order -- counter-clickwise. Your triangle is
> clockwise and thus it has an inverted interpretation -- thus it's a shape
> that covers nearly the whole globe. I recently documented this
> https://issues.apache.org/jira/browse/SOLR-13467 but it's not published
> yet since it's so new.
>
> - Thanks for this clarification as well. I had read this in the
> WKT docs too, again something we tried but really weren’t sure about what
> the right answer was and had been going back and forth on. The
> documentation seems to specify that you need to specify either JTS or
> Geo3d, but doesn’t provide much info/guidance about which to use when and
> since JTS required adding another jar manually and therefore complicates
> our build process significantly (at least vs using Geo3D) we tried Geo3D.
> I’d love to hear more about the tradeoffs and other considerations between
> the two, but sounds like we should switch to JTS (the default, correct?)
>
The default spatialContextFactory is something internal; not JTS or Geo3D.
Based on your requirements, you needn't specify either JTS or Geo3D, mostly
because you don't actually need polygons. I wouldn't bother specifying it
unless you want to experiment with some benchmarking. JTS would give you
nothing here but Geo3D + prefixTree=S2 (in Solr 8.2) might be faster.
> * You can absolutely index a circle in Solr -- this is something cool and
> somewhat unique. And you don't need format=legacy. The documentation needs
> to call t out better, though it at least refers to circles as a "buffered
> point" which is the currently supported way of representing it, and it does
> have one example. Search for "BUFFER" and you'll see a WKT-like syntax to
> do it. BUFFER is not standard WKT; it was added on to do this. The first
> arg is a X Y center, and 2nd arg is a distance in decimal degrees (not
> km). BTW Geo3D is a good choice here but not essential either.
>
> - This sounds very promising and we’ll defin
Re: Solr Geospatial Polygon Indexing/Querying Issue
David, Firstly, thanks for putting together such a thorough email it helps a lot to understand some of the things we were just guessing at because (as you mentioned a few times) the documentation around all of this is rather sparse. I’ll explain the context around the use case we’re trying to solve and then attempt to respond as best I can to each of your points. What we have is a list of documents that in our case the location is sometimes a point and sometimes a circle. These basically represent (in our case) inventory at a physical location (point) or inventory that can be delivered to you within X km (configurable per document) which represents the circle use case. We want to be able to allow a user to say I want all documents within X distance of my location, but also all documents that are able to be delivered to your point where the delivery distance is defined on the inventory (creating the circle). This is why we were actually trying to combine both point based data and poly/circle data into a single geospatial field, since I don’t believe you could do something like fq=geofilt(latlng, x, y, d) OR geofilt(latlngCircle, x, y, 1) but perhaps we’re just not getting quite the right syntax, etc. * Personally, I find it highly confusing to have a field named "latlng" and have it be anything other than a simple point -- it's all you have if given a single latitude longitude pair. If you intend for the data to be a circle (either exactly or approximated) then perhaps call it latLngCircle - This is happening because we’re trying to combine two different use cases into a single field, since I don’t think we have that option from the query side. The name is really just us re-using our current field for this exploration, but would probably end up being named something different. * geodist() and for that matter any other attempt to get the distance to a non-point shape is not going to work -- either error or confusing results; I forget. This is hard to do and the logic isn't there for it, and probably wouldn't perform to user's expectations if it did. This ought to be documented but seems not to be. -Good to know, so no matter what we’ll have to have a point value stored somewhere for each document and calculate geodist on that. * Generally RptWithGeometrySpatialField should be used over SpatialRecursivePrefixTreeFieldType unless you want heatmaps or are willing to make trade-offs in higher index size and lossy precision in order to get faster search. It's up to you; if you benchmark both I'd love to hear how it went. -We may explore both but typically we’re more interested in speed than accuracy, benchmarking it may be a very interesting exercise however. For sorting for instance we’re actually using sqedist instead of geodist because we’re not overly concerned about sorting accuracy. * In WKT format, the ordinate order is "X Y" (thus longitude then latitude). Looking at your triangle, it is extremely close to Antarctica, and I'm skeptical you intended that. This is not directly documented AFAICT but it's such a common mistake that it ought to be called out in the docs. -Definitely did not intend it to be close to Antarctica, I think we tried both but probably went back to lat,long and was definitely more common in our (failed) testing. * I see you are using Geo3D, which is not the default. Geo3D is strict about the coordinate order -- counter-clickwise. Your triangle is clockwise and thus it has an inverted interpretation -- thus it's a shape that covers nearly the whole globe. I recently documented this https://issues.apache.org/jira/browse/SOLR-13467 but it's not published yet since it's so new. - Thanks for this clarification as well. I had read this in the WKT docs too, again something we tried but really weren’t sure about what the right answer was and had been going back and forth on. The documentation seems to specify that you need to specify either JTS or Geo3d, but doesn’t provide much info/guidance about which to use when and since JTS required adding another jar manually and therefore complicates our build process significantly (at least vs using Geo3D) we tried Geo3D. I’d love to hear more about the tradeoffs and other considerations between the two, but sounds like we should switch to JTS (the default, correct?) * You can absolutely index a circle in Solr -- this is something cool and somewhat unique. And you don't need format=legacy. The documentation needs to call this out better, though it at least refers to circles as a "buffered point" which is the currently supported way of representing it, and it does have one example. Search for "BUFFER" and you'll see a WKT-like syntax to do it. BUFFER is not standard WKT; it was added on to do this. The first arg is a X Y center, and 2nd arg is a distance in decimal degrees (not km). BTW Geo3D is a good choice here but no
Re: Solr Geospatial Polygon Indexing/Querying Issue
Hello Marshall,
I worked on a lot of this functionality. I have lots to say:
* Personally, I find it highly confusing to have a field named "latlng" and
have it be anything other than a simple point -- it's all you have if given
a single latitude longitude pair. If you intend for the data to be a
circle (either exactly or approximated) then perhaps call it latLngCircle
* geodist() and for that matter any other attempt to get the distance to a
non-point shape is not going to work -- either error or confusing results;
I forget. This is hard to do and the logic isn't there for it, and
probably wouldn't perform to user's expectations if it did. This ought to
be documented but seems not to be.
* Generally RptWithGeometrySpatialField should be used
over SpatialRecursivePrefixTreeFieldType unless you want heatmaps or are
willing to make trade-offs in higher index size and lossy precision in
order to get faster search. It's up to you; if you benchmark both I'd love
to hear how it went.
* In WKT format, the ordinate order is "X Y" (thus longitude then
latitude). Looking at your triangle, it is extremely close to Antarctica,
and I'm skeptical you intended that. This is not directly documented AFAICT
but it's such a common mistake that it ought to be called out in the docs.
* I see you are using Geo3D, which is not the default. Geo3D is strict
about the coordinate order -- counter-clickwise. Your triangle is
clockwise and thus it has an inverted interpretation -- thus it's a shape
that covers nearly the whole globe. I recently documented this
https://issues.apache.org/jira/browse/SOLR-13467 but it's not published yet
since it's so new.
* You can absolutely index a circle in Solr -- this is something cool and
somewhat unique. And you don't need format=legacy. The documentation needs
to call this out better, though it at least refers to circles as a
"buffered point" which is the currently supported way of representing it,
and it does have one example. Search for "BUFFER" and you'll see a
WKT-like syntax to do it. BUFFER is not standard WKT; it was added on to
do this. The first arg is a X Y center, and 2nd arg is a distance in
decimal degrees (not km). BTW Geo3D is a good choice here but not
essential either.
Back to your core requirement -- you want to index circles and sort results
by distance. Can you please elaborate better on this... distance to the
outer ring of the circle or the center point? Center point is easy to do
simply by putting the center point additionally in a field using
LatLonPointSpatialField and use geodist referring to that. Also,
FYI geodist() is a function that can take arguments directly which makes
more sense when multiple spatial fields are in play. Sadly this aspect is
not documented. Suffice it to say, if you do geodist(latLng) (maybe
quoted?) then it'll use that field, and parse "pt" param from the request.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Tue, Jul 23, 2019 at 2:32 PM Sanders, Marshall (CAI - Atlanta) <
marshall.sande...@coxautoinc.com> wrote:
> We’re trying to index a polygon into solr and then filter/calculate
> geodist on the polygon (ideally we actually want a circle, but it looks
> like that’s not really supported officially by wkt/geojson and instead you
> have to switch format=”legacy” which seems like something that might be
> removed in the future so don’t want to rely on it).
>
> Here’s the info from schema:
> multiValued="true"/>
>
> class="solr.SpatialRecursivePrefixTreeFieldType"
>geo="true" distErrPct="0.025" maxDistErr="0.09"
> distanceUnits="kilometers"
> spatialContextFactory="Geo3D"/>
>
>
> We’ve tried indexing some different data, but to keep it as simple as
> possible we started with a triangle (will eventually add more points to
> approximate a circle). Here’s an example document that we’ve added just
> for testing:
>
> {
> "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
> 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> "ID": "284598223"
> }
>
>
> However, it seems like filtering/distance calculations aren’t working (at
> least not the way we are used to doing it for points). Here’s an example
> query where the pt is several hundred kilometers away from the polygon, yet
> the document still returns. Also, it seems that regardless of origin point
> or polygon location the calculated geodist is always 20015.115
>
> Example query:
>
> select?d=1&fl=ID,latlng,geodist()&fq=%7B!geofilt%7D&indent=on&pt=33.9798087,-94.3286133&q=*:*&sfield=latlng&wt=json
>
> Example documents coming back anyway:
> "docs": [
> {
> "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
> 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> "ID": "284598223",
> "geodist()": 20015.115
> },
> {
> "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
> 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> "ID
Re: Solr Geospatial Polygon Indexing/Querying Issue
That didn't seem to work either. I think there must be something wrong with
how we're indexing/storing the polygon and/or how we've configured the
field/querying it. The docs are so sparse on this (
Here's the response:
{
"responseHeader":{
"status":0,
"QTime":1,
"params":{
"q":"*:*",
"fl":"latlng,ID",
"fq":"{!geofilt sfield=latlng pt=33.3786,-94.8985 d=1}",
"rows":"2",
"_":"1564065725241"}},
"response":{"numFound":10,"start":0,"docs":[
{
"latlng":["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
"ID":"284598223"},
{
"latlng":["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
"ID":"284600596"}]
}}
On 7/25/19, 2:51 AM, "Ere Maijala" wrote:
Oops, sorry! Don't know how I missed that.
Have you tested if it makes any difference if you put the sfield
parameter inside the fq like in the example
(https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_8-5F1_spatial-2Dsearch.html-23geofilt-29-3F&d=DwIDaQ&c=hrETxhO8sRCXAcJITi-bu62jJ43QQVS6-BatTNT-3bs&r=3lL1Fjs6t-l8MLo9jYFBo7cXQNBxZBB5BXFvpvXk4cU&m=JR0_KNI-GjB0_I3qC1jsCqb3SySydbHO0e6W5SeYKH4&s=76D0RQHnWeh9KYT1Kx4Q4rz3lMgPR3krYF8uuKVtFaU&e=
We actually put pt and d in there too, e.g.
{!geofilt+sfield%3Dlocation_geo+pt%3D61.2%2C24.9+d%3D1}
--Ere
Sanders, Marshall (CAI - Atlanta) kirjoitti 24.7.2019 klo 16.33:
> My example query has d=1 as the first parameter, so none of the results
should be coming back, but they are which makes it seem like it's not doing any
geofiltering for some reason.
>
> On 7/24/19, 2:06 AM, "Ere Maijala" wrote:
>
> I think you might be missing the d parameter in geofilt. I'm not sure
if
> geofilt actually does anything useful without it.
>
> Regards,
> Ere
>
> Sanders, Marshall (CAI - Atlanta) kirjoitti 23.7.2019 klo 21.32:
> > We’re trying to index a polygon into solr and then filter/calculate
geodist on the polygon (ideally we actually want a circle, but it looks like
that’s not really supported officially by wkt/geojson and instead you have to
switch format=”legacy” which seems like something that might be removed in the
future so don’t want to rely on it).
> >
> > Here’s the info from schema:
> >
> >
> > >geo="true" distErrPct="0.025"
maxDistErr="0.09" distanceUnits="kilometers"
> > spatialContextFactory="Geo3D"/>
> >
> >
> > We’ve tried indexing some different data, but to keep it as simple
as possible we started with a triangle (will eventually add more points to
approximate a circle). Here’s an example document that we’ve added just for
testing:
> >
> > {
> > "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611
-84.4028091, 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> > "ID": "284598223"
> > }
> >
> >
> > However, it seems like filtering/distance calculations aren’t
working (at least not the way we are used to doing it for points). Here’s an
example query where the pt is several hundred kilometers away from the polygon,
yet the document still returns. Also, it seems that regardless of origin point
or polygon location the calculated geodist is always 20015.115
> >
> > Example query:
> >
select?d=1&fl=ID,latlng,geodist()&fq=%7B!geofilt%7D&indent=on&pt=33.9798087,-94.3286133&q=*:*&sfield=latlng&wt=json
> >
> > Example documents coming back anyway:
> > "docs": [
> > {
> > "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611
-84.4028091, 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> > "ID": "284598223",
> > "geodist()": 20015.115
> > },
> > {
> > "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611
-84.4028091, 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> > "ID": "284600596",
> > "geodist()": 20015.115
> > }
> > ]
> >
> >
> > Anyone who has experience in this area can you point us in the
right direction about what we’re doing incorrectly with either how we are
indexing the data and/or how we are querying against the polygons.
> >
> > Thank you,
> >
> >
> > --
> > Marshall Sanders
> > Principal Software Engineer
> > Autotrader.com
> >
marshall.sande...@coxautoinc.com
> >
> >
>
> --
> Ere Maijala
> Kansalliskirjasto / The National Library of Finland
>
>
--
Re: Solr Geospatial Polygon Indexing/Querying Issue
Oops, sorry! Don't know how I missed that.
Have you tested if it makes any difference if you put the sfield
parameter inside the fq like in the example
(https://lucene.apache.org/solr/guide/8_1/spatial-search.html#geofilt)?
We actually put pt and d in there too, e.g.
{!geofilt+sfield%3Dlocation_geo+pt%3D61.2%2C24.9+d%3D1}
--Ere
Sanders, Marshall (CAI - Atlanta) kirjoitti 24.7.2019 klo 16.33:
> My example query has d=1 as the first parameter, so none of the results
> should be coming back, but they are which makes it seem like it's not doing
> any geofiltering for some reason.
>
> On 7/24/19, 2:06 AM, "Ere Maijala" wrote:
>
> I think you might be missing the d parameter in geofilt. I'm not sure if
> geofilt actually does anything useful without it.
>
> Regards,
> Ere
>
> Sanders, Marshall (CAI - Atlanta) kirjoitti 23.7.2019 klo 21.32:
> > We’re trying to index a polygon into solr and then filter/calculate
> geodist on the polygon (ideally we actually want a circle, but it looks like
> that’s not really supported officially by wkt/geojson and instead you have to
> switch format=”legacy” which seems like something that might be removed in
> the future so don’t want to rely on it).
> >
> > Here’s the info from schema:
> > multiValued="true"/>
> >
> > class="solr.SpatialRecursivePrefixTreeFieldType"
> >geo="true" distErrPct="0.025" maxDistErr="0.09"
> distanceUnits="kilometers"
> > spatialContextFactory="Geo3D"/>
> >
> >
> > We’ve tried indexing some different data, but to keep it as simple as
> possible we started with a triangle (will eventually add more points to
> approximate a circle). Here’s an example document that we’ve added just for
> testing:
> >
> > {
> > "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
> 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> > "ID": "284598223"
> > }
> >
> >
> > However, it seems like filtering/distance calculations aren’t working
> (at least not the way we are used to doing it for points). Here’s an example
> query where the pt is several hundred kilometers away from the polygon, yet
> the document still returns. Also, it seems that regardless of origin point
> or polygon location the calculated geodist is always 20015.115
> >
> > Example query:
> >
> select?d=1&fl=ID,latlng,geodist()&fq=%7B!geofilt%7D&indent=on&pt=33.9798087,-94.3286133&q=*:*&sfield=latlng&wt=json
> >
> > Example documents coming back anyway:
> > "docs": [
> > {
> > "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
> 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> > "ID": "284598223",
> > "geodist()": 20015.115
> > },
> > {
> > "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
> 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> > "ID": "284600596",
> > "geodist()": 20015.115
> > }
> > ]
> >
> >
> > Anyone who has experience in this area can you point us in the right
> direction about what we’re doing incorrectly with either how we are indexing
> the data and/or how we are querying against the polygons.
> >
> > Thank you,
> >
> >
> > --
> > Marshall Sanders
> > Principal Software Engineer
> > Autotrader.com
> >
> marshall.sande...@coxautoinc.com
> >
> >
>
> --
> Ere Maijala
> Kansalliskirjasto / The National Library of Finland
>
>
--
Ere Maijala
Kansalliskirjasto / The National Library of Finland
Re: Solr Geospatial Polygon Indexing/Querying Issue
My example query has d=1 as the first parameter, so none of the results should
be coming back, but they are which makes it seem like it's not doing any
geofiltering for some reason.
On 7/24/19, 2:06 AM, "Ere Maijala" wrote:
I think you might be missing the d parameter in geofilt. I'm not sure if
geofilt actually does anything useful without it.
Regards,
Ere
Sanders, Marshall (CAI - Atlanta) kirjoitti 23.7.2019 klo 21.32:
> We’re trying to index a polygon into solr and then filter/calculate
geodist on the polygon (ideally we actually want a circle, but it looks like
that’s not really supported officially by wkt/geojson and instead you have to
switch format=”legacy” which seems like something that might be removed in the
future so don’t want to rely on it).
>
> Here’s the info from schema:
>
>
> geo="true" distErrPct="0.025" maxDistErr="0.09"
distanceUnits="kilometers"
> spatialContextFactory="Geo3D"/>
>
>
> We’ve tried indexing some different data, but to keep it as simple as
possible we started with a triangle (will eventually add more points to
approximate a circle). Here’s an example document that we’ve added just for
testing:
>
> {
> "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> "ID": "284598223"
> }
>
>
> However, it seems like filtering/distance calculations aren’t working (at
least not the way we are used to doing it for points). Here’s an example query
where the pt is several hundred kilometers away from the polygon, yet the
document still returns. Also, it seems that regardless of origin point or
polygon location the calculated geodist is always 20015.115
>
> Example query:
>
select?d=1&fl=ID,latlng,geodist()&fq=%7B!geofilt%7D&indent=on&pt=33.9798087,-94.3286133&q=*:*&sfield=latlng&wt=json
>
> Example documents coming back anyway:
> "docs": [
> {
> "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> "ID": "284598223",
> "geodist()": 20015.115
> },
> {
> "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> "ID": "284600596",
> "geodist()": 20015.115
> }
> ]
>
>
> Anyone who has experience in this area can you point us in the right
direction about what we’re doing incorrectly with either how we are indexing
the data and/or how we are querying against the polygons.
>
> Thank you,
>
>
> --
> Marshall Sanders
> Principal Software Engineer
> Autotrader.com
> marshall.sande...@coxautoinc.com
>
>
--
Ere Maijala
Kansalliskirjasto / The National Library of Finland
Re: Solr Geospatial Polygon Indexing/Querying Issue
I think you might be missing the d parameter in geofilt. I'm not sure if
geofilt actually does anything useful without it.
Regards,
Ere
Sanders, Marshall (CAI - Atlanta) kirjoitti 23.7.2019 klo 21.32:
> We’re trying to index a polygon into solr and then filter/calculate geodist
> on the polygon (ideally we actually want a circle, but it looks like that’s
> not really supported officially by wkt/geojson and instead you have to switch
> format=”legacy” which seems like something that might be removed in the
> future so don’t want to rely on it).
>
> Here’s the info from schema:
> multiValued="true"/>
>
> class="solr.SpatialRecursivePrefixTreeFieldType"
>geo="true" distErrPct="0.025" maxDistErr="0.09"
> distanceUnits="kilometers"
> spatialContextFactory="Geo3D"/>
>
>
> We’ve tried indexing some different data, but to keep it as simple as
> possible we started with a triangle (will eventually add more points to
> approximate a circle). Here’s an example document that we’ve added just for
> testing:
>
> {
> "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
> 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> "ID": "284598223"
> }
>
>
> However, it seems like filtering/distance calculations aren’t working (at
> least not the way we are used to doing it for points). Here’s an example
> query where the pt is several hundred kilometers away from the polygon, yet
> the document still returns. Also, it seems that regardless of origin point
> or polygon location the calculated geodist is always 20015.115
>
> Example query:
> select?d=1&fl=ID,latlng,geodist()&fq=%7B!geofilt%7D&indent=on&pt=33.9798087,-94.3286133&q=*:*&sfield=latlng&wt=json
>
> Example documents coming back anyway:
> "docs": [
> {
> "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
> 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> "ID": "284598223",
> "geodist()": 20015.115
> },
> {
> "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
> 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> "ID": "284600596",
> "geodist()": 20015.115
> }
> ]
>
>
> Anyone who has experience in this area can you point us in the right
> direction about what we’re doing incorrectly with either how we are indexing
> the data and/or how we are querying against the polygons.
>
> Thank you,
>
>
> --
> Marshall Sanders
> Principal Software Engineer
> Autotrader.com
> marshall.sande...@coxautoinc.com
>
>
--
Ere Maijala
Kansalliskirjasto / The National Library of Finland
Re: solr geospatial / spatial4j
On Wed, Mar 7, 2012 at 7:25 AM, Matt Mitchell wrote: > Hi, > > I'm researching options for handling a better geospatial solution. I'm > currently using Solr 3.5 for a read-only "database", and the > point/radius searches work great. But I'd like to start doing point in > polygon searches as well. I've skimmed through some of the geospatial > jira issues, and read about spaitial4j, which is very interesting. I > see on the github page that this will soon be part of lucene, can > anyone confirm this? perhaps -- see the discussion on: https://issues.apache.org/jira/browse/LUCENE-3795 This will involve a few steps before it is actually integrated with the lucene project -- and then a few more to be usable from solr > > I attempted to build the spatial4j demo but no luck. It had problems > finding lucene 4.0-SNAPSHOT, which I guess is because there are no > 4.0-SNAPSHOT nightly builds? If anyone knows how I can get around > this, please let me know! > ya they are published -- you just have to specify where you want to pull them from. If you use the 'updateLucene' profile, it will pull them from: https://repository.apache.org/content/groups/snapshots/ use: mvn clean install -P updateLucene > Other than spatial4j, is there a way to do point in polgyon searches > with solr 3.5.0 right now? Is there some tricky indexing/querying > strategy that would allow this? > I don't know of anything else -- and note that polygon stuff has a ways to go before it is generally ready for prime-time. ryan
Re: solr geospatial / spatial4j
Yes, there are trunk nightly builds, see: https://builds.apache.org//view/S-Z/view/Solr/job/Solr-trunk/ But I don't think LSP is in trunk at this point, so that's not useful. The code branch is on (I think) http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_3795_ls_spatial_playground but I confess I haven't tried to get and build it all, I'm not quite sure what's needed Best Erick On Wed, Mar 7, 2012 at 10:25 AM, Matt Mitchell wrote: > Hi, > > I'm researching options for handling a better geospatial solution. I'm > currently using Solr 3.5 for a read-only "database", and the > point/radius searches work great. But I'd like to start doing point in > polygon searches as well. I've skimmed through some of the geospatial > jira issues, and read about spaitial4j, which is very interesting. I > see on the github page that this will soon be part of lucene, can > anyone confirm this? > > I attempted to build the spatial4j demo but no luck. It had problems > finding lucene 4.0-SNAPSHOT, which I guess is because there are no > 4.0-SNAPSHOT nightly builds? If anyone knows how I can get around > this, please let me know! > > Other than spatial4j, is there a way to do point in polgyon searches > with solr 3.5.0 right now? Is there some tricky indexing/querying > strategy that would allow this? > > Thanks! > > - Matt
Re: SOLR geospatial
I would be more than happy to help with any of the spatial testing you are working on. adam On Sun, Dec 12, 2010 at 3:08 PM, Dennis Gearon wrote: > We're in Alpha, heading to Alpha 2. Our requirements are simple: radius > searching, and distance from center. Solr Spatial works and is current. > GeoSpatial is almost there, but we're going to wait until it's released to > spend > time with it. We have other tasks to work on and don't want to be part of > the > debugging process of any project right now. > > Dennis Gearon > > > Signature Warning > > It is always a good idea to learn from your own mistakes. It is usually a > better > idea to learn from others’ mistakes, so you do not have to make them > yourself. > from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036' > > > EARTH has a Right To Life, > otherwise we all die. > > > > - Original Message > From: Erick Erickson > To: solr-user@lucene.apache.org > Sent: Sun, December 12, 2010 11:18:03 AM > Subject: Re: SOLR geospatial > > By and large, spatial solr is being replaced by geospatial, see: > http://wiki.apache.org/solr/SpatialSearch. I don't think the old > spatial contrib is still included in the trunk or 3.x code bases, but > I could be wrong > > That said, I don't know whether what you want is on the roadmap > there either. Here's a place to start if you want to see the JIRA > discussions: https://issues.apache.org/jira/browse/SOLR-1568 > > Best > Erick > > > On Sun, Dec 12, 2010 at 11:23 AM, Adam Estrada >wrote: > > > I am particularly interested in storing and querying polygons. That sort > of > > thing looks like its on their roadmap so does anyone know what the status > > is > > on that? Also, integration with JTS would make this a core component of > any > > GIS. Again, anyone know what the status is on that? > > > > *What’s on the roadmap of future features?* > > > > Here are some of the features and henhancements we're planning for SSP: > > > > - > > > > Performance improvements for larger data sets > > - > > > > Fixing of known bugs > > - > > > > Distance facets: Allowing Solr users to be able to filter their results > > based on the calculated distances. > > - > > > > Search with regular polygons, and groups of shapes > > - > > > > Integration with JTS > > - > > > > Highly optimized distance calculation algorithms > > - > > > > Ranking results by distance > > - > > > > 3D dimension search > > > > > > Adam > > > > On Sun, Dec 12, 2010 at 12:01 AM, Markus Jelsma > > wrote: > > > > > That smells like: http://www.jteam.nl/news/spatialsolr.html > > > > > > > My partner is using a publicly available plugin for GeoSpatial. It is > > > used > > > > both during indexing and during search. It forms some kind of > gridding > > > > system and puts 10 fields per row related to that. Doing a Radius > > search > > > > (vs a bounding box search which is faster in almost all cases in all > > > > GeoSpatial query systems) seems pretty fast. GeoSpatial was our > > project's > > > > constraint. We've moved past that now. > > > > > > > > Did I mention that it returns distance from the center of the radius > > > based > > > > on units supplied in the query? > > > > > > > > I would tell you what the plugin is, but in our division of labor, I > > have > > > > kept that out of my short term memory. You can contact him at: > > > > Danilo Unite ; > > > > > > > > Dennis Gearon > > > > > > > > > > > > Signature Warning > > > > > > > > It is always a good idea to learn from your own mistakes. It is > usually > > a > > > > better idea to learn from others’ mistakes, so you do not have to > make > > > > them yourself. from > > > > 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036' > > > > > > > > > > > > EARTH has a Right To Life, > > > > otherwise we all die. > > > > > > > > > > > > > > > > - Original Message > > > > From: George Anthony > > > > To: solr-user@lucene.apache.org > > > > Sent: Fri, December 10, 2010 9:23:18 AM > > > > Subject: SOLR geospatial > > > > > > > > In looking at some of the docs support for geospatial search. > > > > > > > > I see this functionality is mostly scheduled for upcoming release 4.0 > > > (with > > > > some > > > > > > > > playing around with backported code). > > > > > > > > > > > > I note the support for the bounding box filter, but will "bounding > box" > > > be > > > > one of the supported *data* types for use with this filter? For > > example, > > > > if my lat/long data describes the "footprint" of a map, I'm curious > if > > > > that type of coordinate data can be used by the bounding box filter > (or > > > in > > > > any other way for similar limiting/filtering capability). I see it > can > > > > work with point type data but curious about functionality with > bounding > > > > box type data (in contrast to simple point lat/long data). > > > > > > > > Thanks, > > > > George > > > > > > >
Re: SOLR geospatial
We're in Alpha, heading to Alpha 2. Our requirements are simple: radius searching, and distance from center. Solr Spatial works and is current. GeoSpatial is almost there, but we're going to wait until it's released to spend time with it. We have other tasks to work on and don't want to be part of the debugging process of any project right now. Dennis Gearon Signature Warning It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036' EARTH has a Right To Life, otherwise we all die. - Original Message From: Erick Erickson To: solr-user@lucene.apache.org Sent: Sun, December 12, 2010 11:18:03 AM Subject: Re: SOLR geospatial By and large, spatial solr is being replaced by geospatial, see: http://wiki.apache.org/solr/SpatialSearch. I don't think the old spatial contrib is still included in the trunk or 3.x code bases, but I could be wrong That said, I don't know whether what you want is on the roadmap there either. Here's a place to start if you want to see the JIRA discussions: https://issues.apache.org/jira/browse/SOLR-1568 Best Erick On Sun, Dec 12, 2010 at 11:23 AM, Adam Estrada wrote: > I am particularly interested in storing and querying polygons. That sort of > thing looks like its on their roadmap so does anyone know what the status > is > on that? Also, integration with JTS would make this a core component of any > GIS. Again, anyone know what the status is on that? > > *What’s on the roadmap of future features?* > > Here are some of the features and henhancements we're planning for SSP: > > - > > Performance improvements for larger data sets > - > > Fixing of known bugs > - > > Distance facets: Allowing Solr users to be able to filter their results > based on the calculated distances. > - > > Search with regular polygons, and groups of shapes > - > > Integration with JTS > - > > Highly optimized distance calculation algorithms > - > > Ranking results by distance > - > > 3D dimension search > > > Adam > > On Sun, Dec 12, 2010 at 12:01 AM, Markus Jelsma > wrote: > > > That smells like: http://www.jteam.nl/news/spatialsolr.html > > > > > My partner is using a publicly available plugin for GeoSpatial. It is > > used > > > both during indexing and during search. It forms some kind of gridding > > > system and puts 10 fields per row related to that. Doing a Radius > search > > > (vs a bounding box search which is faster in almost all cases in all > > > GeoSpatial query systems) seems pretty fast. GeoSpatial was our > project's > > > constraint. We've moved past that now. > > > > > > Did I mention that it returns distance from the center of the radius > > based > > > on units supplied in the query? > > > > > > I would tell you what the plugin is, but in our division of labor, I > have > > > kept that out of my short term memory. You can contact him at: > > > Danilo Unite ; > > > > > > Dennis Gearon > > > > > > > > > Signature Warning > > > > > > It is always a good idea to learn from your own mistakes. It is usually > a > > > better idea to learn from others’ mistakes, so you do not have to make > > > them yourself. from > > > 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036' > > > > > > > > > EARTH has a Right To Life, > > > otherwise we all die. > > > > > > > > > > > > - Original Message > > > From: George Anthony > > > To: solr-user@lucene.apache.org > > > Sent: Fri, December 10, 2010 9:23:18 AM > > > Subject: SOLR geospatial > > > > > > In looking at some of the docs support for geospatial search. > > > > > > I see this functionality is mostly scheduled for upcoming release 4.0 > > (with > > > some > > > > > > playing around with backported code). > > > > > > > > > I note the support for the bounding box filter, but will "bounding box" > > be > > > one of the supported *data* types for use with this filter? For > example, > > > if my lat/long data describes the "footprint" of a map, I'm curious if > > > that type of coordinate data can be used by the bounding box filter (or > > in > > > any other way for similar limiting/filtering capability). I see it can > > > work with point type data but curious about functionality with bounding > > > box type data (in contrast to simple point lat/long data). > > > > > > Thanks, > > > George > > >
Re: SOLR geospatial
By and large, spatial solr is being replaced by geospatial, see: http://wiki.apache.org/solr/SpatialSearch. I don't think the old spatial contrib is still included in the trunk or 3.x code bases, but I could be wrong That said, I don't know whether what you want is on the roadmap there either. Here's a place to start if you want to see the JIRA discussions: https://issues.apache.org/jira/browse/SOLR-1568 Best Erick On Sun, Dec 12, 2010 at 11:23 AM, Adam Estrada wrote: > I am particularly interested in storing and querying polygons. That sort of > thing looks like its on their roadmap so does anyone know what the status > is > on that? Also, integration with JTS would make this a core component of any > GIS. Again, anyone know what the status is on that? > > *What’s on the roadmap of future features?* > > Here are some of the features and henhancements we're planning for SSP: > > - > > Performance improvements for larger data sets > - > > Fixing of known bugs > - > > Distance facets: Allowing Solr users to be able to filter their results > based on the calculated distances. > - > > Search with regular polygons, and groups of shapes > - > > Integration with JTS > - > > Highly optimized distance calculation algorithms > - > > Ranking results by distance > - > > 3D dimension search > > > Adam > > On Sun, Dec 12, 2010 at 12:01 AM, Markus Jelsma > wrote: > > > That smells like: http://www.jteam.nl/news/spatialsolr.html > > > > > My partner is using a publicly available plugin for GeoSpatial. It is > > used > > > both during indexing and during search. It forms some kind of gridding > > > system and puts 10 fields per row related to that. Doing a Radius > search > > > (vs a bounding box search which is faster in almost all cases in all > > > GeoSpatial query systems) seems pretty fast. GeoSpatial was our > project's > > > constraint. We've moved past that now. > > > > > > Did I mention that it returns distance from the center of the radius > > based > > > on units supplied in the query? > > > > > > I would tell you what the plugin is, but in our division of labor, I > have > > > kept that out of my short term memory. You can contact him at: > > > Danilo Unite ; > > > > > > Dennis Gearon > > > > > > > > > Signature Warning > > > > > > It is always a good idea to learn from your own mistakes. It is usually > a > > > better idea to learn from others’ mistakes, so you do not have to make > > > them yourself. from > > > 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036' > > > > > > > > > EARTH has a Right To Life, > > > otherwise we all die. > > > > > > > > > > > > - Original Message > > > From: George Anthony > > > To: solr-user@lucene.apache.org > > > Sent: Fri, December 10, 2010 9:23:18 AM > > > Subject: SOLR geospatial > > > > > > In looking at some of the docs support for geospatial search. > > > > > > I see this functionality is mostly scheduled for upcoming release 4.0 > > (with > > > some > > > > > > playing around with backported code). > > > > > > > > > I note the support for the bounding box filter, but will "bounding box" > > be > > > one of the supported *data* types for use with this filter? For > example, > > > if my lat/long data describes the "footprint" of a map, I'm curious if > > > that type of coordinate data can be used by the bounding box filter (or > > in > > > any other way for similar limiting/filtering capability). I see it can > > > work with point type data but curious about functionality with bounding > > > box type data (in contrast to simple point lat/long data). > > > > > > Thanks, > > > George > > >
Re: SOLR geospatial
I am particularly interested in storing and querying polygons. That sort of thing looks like its on their roadmap so does anyone know what the status is on that? Also, integration with JTS would make this a core component of any GIS. Again, anyone know what the status is on that? *What’s on the roadmap of future features?* Here are some of the features and henhancements we're planning for SSP: - Performance improvements for larger data sets - Fixing of known bugs - Distance facets: Allowing Solr users to be able to filter their results based on the calculated distances. - Search with regular polygons, and groups of shapes - Integration with JTS - Highly optimized distance calculation algorithms - Ranking results by distance - 3D dimension search Adam On Sun, Dec 12, 2010 at 12:01 AM, Markus Jelsma wrote: > That smells like: http://www.jteam.nl/news/spatialsolr.html > > > My partner is using a publicly available plugin for GeoSpatial. It is > used > > both during indexing and during search. It forms some kind of gridding > > system and puts 10 fields per row related to that. Doing a Radius search > > (vs a bounding box search which is faster in almost all cases in all > > GeoSpatial query systems) seems pretty fast. GeoSpatial was our project's > > constraint. We've moved past that now. > > > > Did I mention that it returns distance from the center of the radius > based > > on units supplied in the query? > > > > I would tell you what the plugin is, but in our division of labor, I have > > kept that out of my short term memory. You can contact him at: > > Danilo Unite ; > > > > Dennis Gearon > > > > > > Signature Warning > > > > It is always a good idea to learn from your own mistakes. It is usually a > > better idea to learn from others’ mistakes, so you do not have to make > > them yourself. from > > 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036' > > > > > > EARTH has a Right To Life, > > otherwise we all die. > > > > > > > > - Original Message > > From: George Anthony > > To: solr-user@lucene.apache.org > > Sent: Fri, December 10, 2010 9:23:18 AM > > Subject: SOLR geospatial > > > > In looking at some of the docs support for geospatial search. > > > > I see this functionality is mostly scheduled for upcoming release 4.0 > (with > > some > > > > playing around with backported code). > > > > > > I note the support for the bounding box filter, but will "bounding box" > be > > one of the supported *data* types for use with this filter? For example, > > if my lat/long data describes the "footprint" of a map, I'm curious if > > that type of coordinate data can be used by the bounding box filter (or > in > > any other way for similar limiting/filtering capability). I see it can > > work with point type data but curious about functionality with bounding > > box type data (in contrast to simple point lat/long data). > > > > Thanks, > > George >
Re: SOLR geospatial
That smells like: http://www.jteam.nl/news/spatialsolr.html > My partner is using a publicly available plugin for GeoSpatial. It is used > both during indexing and during search. It forms some kind of gridding > system and puts 10 fields per row related to that. Doing a Radius search > (vs a bounding box search which is faster in almost all cases in all > GeoSpatial query systems) seems pretty fast. GeoSpatial was our project's > constraint. We've moved past that now. > > Did I mention that it returns distance from the center of the radius based > on units supplied in the query? > > I would tell you what the plugin is, but in our division of labor, I have > kept that out of my short term memory. You can contact him at: > Danilo Unite ; > > Dennis Gearon > > > Signature Warning > > It is always a good idea to learn from your own mistakes. It is usually a > better idea to learn from others’ mistakes, so you do not have to make > them yourself. from > 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036' > > > EARTH has a Right To Life, > otherwise we all die. > > > > - Original Message > From: George Anthony > To: solr-user@lucene.apache.org > Sent: Fri, December 10, 2010 9:23:18 AM > Subject: SOLR geospatial > > In looking at some of the docs support for geospatial search. > > I see this functionality is mostly scheduled for upcoming release 4.0 (with > some > > playing around with backported code). > > > I note the support for the bounding box filter, but will "bounding box" be > one of the supported *data* types for use with this filter? For example, > if my lat/long data describes the "footprint" of a map, I'm curious if > that type of coordinate data can be used by the bounding box filter (or in > any other way for similar limiting/filtering capability). I see it can > work with point type data but curious about functionality with bounding > box type data (in contrast to simple point lat/long data). > > Thanks, > George
Re: SOLR geospatial
My partner is using a publicly available plugin for GeoSpatial. It is used both during indexing and during search. It forms some kind of gridding system and puts 10 fields per row related to that. Doing a Radius search (vs a bounding box search which is faster in almost all cases in all GeoSpatial query systems) seems pretty fast. GeoSpatial was our project's constraint. We've moved past that now. Did I mention that it returns distance from the center of the radius based on units supplied in the query? I would tell you what the plugin is, but in our division of labor, I have kept that out of my short term memory. You can contact him at: Danilo Unite ; Dennis Gearon Signature Warning It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036' EARTH has a Right To Life, otherwise we all die. - Original Message From: George Anthony To: solr-user@lucene.apache.org Sent: Fri, December 10, 2010 9:23:18 AM Subject: SOLR geospatial In looking at some of the docs support for geospatial search. I see this functionality is mostly scheduled for upcoming release 4.0 (with some playing around with backported code). I note the support for the bounding box filter, but will "bounding box" be one of the supported *data* types for use with this filter? For example, if my lat/long data describes the "footprint" of a map, I'm curious if that type of coordinate data can be used by the bounding box filter (or in any other way for similar limiting/filtering capability). I see it can work with point type data but curious about functionality with bounding box type data (in contrast to simple point lat/long data). Thanks, George
