[jira] [Comment Edited] (LUCENE-10654) New companion doc value format for LatLonShape and XYShape field types
[
https://issues.apache.org/jira/browse/LUCENE-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578674#comment-17578674
]
Nick Knize edited comment on LUCENE-10654 at 8/11/22 9:08 PM:
--
Nightly test failure on XY bounding box:
{code:java}
Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.document.TestShapeDocValues.testXYPolygonBBox"
-Ptests.jvms=4 -Ptests.haltonfailure=false
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=ABDF070B81479950
-Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false
-Ptests.gui=true -Ptests.file.encoding=ISO-8859-1
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-NightlyTests-main/test-data/enwiki.random.lines.txt
{code}
{code:java}
1 tests failed.
FAILED: org.apache.lucene.document.TestShapeDocValues.testXYPolygonBBox
Error Message:
java.lang.AssertionError: expected:<-2.028229934961692E32> but
was:<-2.026382696309321E32>
{code}
This is caused because the {{TestUtil.nextPolygon}} is producing a polygon with
an extruded colinear self intersecting vertex and the {{BaseXYShapeTestCase}}
is not throwing this as an invalid polygon because the test case uses
{{randomBoolean}}. The simple fix is to switch the TestCase to always throw an
exception on invalid polygons so we never test with a non-compliant polygon.
This passed the queries because the tessellator would filter out the dirty
vertext. This test failed because the dirty vertext just happened to be the
minimum X value. So this does expose an inconsistency where an invalid polygon
will have a bounding box inconsistent with the raw geometry. I think that's
okay because we have API guardrails to enable or disable strict validation and
I don't think that should be removed.
I will open a PR to switch the base test cases over to strict geometry
validation instead of random validation.
was (Author: nknize):
Nightly test failure on XY bounding box:
{code:java}
Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.document.TestShapeDocValues.testXYPolygonBBox"
-Ptests.jvms=4 -Ptests.haltonfailure=false
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=ABDF070B81479950
-Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false
-Ptests.gui=true -Ptests.file.encoding=ISO-8859-1
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-NightlyTests-main/test-data/enwiki.random.lines.txt
{code}
{code:java}
1 tests failed.
FAILED: org.apache.lucene.document.TestShapeDocValues.testXYPolygonBBox
Error Message:
java.lang.AssertionError: expected:<-2.028229934961692E32> but
was:<-2.026382696309321E32>
{code}
This is caused because the {{{TestUtil.nextPolygon}}} is producing a polygon
with an extruded colinear self intersecting vertex and the
{{BaseXYShapeTestCase}} is not throwing this as an invalid polygon because the
test case uses {{randomBoolean}}. The simple fix is to switch the TestCase to
always throw an exception on invalid polygons so we never test with a
non-compliant polygon. This passed the queries because the tessellator would
filter out the dirty vertext. This test failed because the dirty vertext just
happened to be the minimum X value. So this does expose an inconsistency where
an invalid polygon will have a bounding box inconsistent with the raw geometry.
I think that's okay because we have API guardrails to enable or disable strict
validation and I don't think that should be removed.
I will open a PR to switch the base test cases over to strict geometry
validation instead of random validation.
> New companion doc value format for LatLonShape and XYShape field types
> --
>
> Key: LUCENE-10654
> URL: https://issues.apache.org/jira/browse/LUCENE-10654
> Project: Lucene - Core
> Issue Type: New Feature
>Reporter: Nick Knize
>Priority: Major
> Fix For: 9.4
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> {{XYDocValuesField}} provides doc value support for {{XYPoint}}.
> {{LatLonDocValuesField}} provides docvalue support for {{LatLonPoint}}.
> However, neither {{LatLonShape}} nor {{XYShape}} currently have a docvalue
> format.
> This lack of doc value support for shapes means facets, aggregations, and
> IndexOrDocValues queries are currently not possible for Shape field types.
> This gap needs be closed in lucene.
> To support IndexOrDocValues queries along with various geometry aggregations
> and facets, the ability to compute the spatial relation with the doc value is
> needed. This is straightforward with {{XYPoint}} and {{LatLonPoint}} since
> the doc value encoding is nothing more than a simple 2D integer encoding of
> the x,y and lat,lon dimensional components. Accomplishing the same with a
>
[jira] [Commented] (LUCENE-10654) New companion doc value format for LatLonShape and XYShape field types
[
https://issues.apache.org/jira/browse/LUCENE-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578674#comment-17578674
]
Nick Knize commented on LUCENE-10654:
-
Nightly test failure on XY bounding box:
{code:java}
Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.document.TestShapeDocValues.testXYPolygonBBox"
-Ptests.jvms=4 -Ptests.haltonfailure=false
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=ABDF070B81479950
-Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false
-Ptests.gui=true -Ptests.file.encoding=ISO-8859-1
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-NightlyTests-main/test-data/enwiki.random.lines.txt
{code}
{code:java}
1 tests failed.
FAILED: org.apache.lucene.document.TestShapeDocValues.testXYPolygonBBox
Error Message:
java.lang.AssertionError: expected:<-2.028229934961692E32> but
was:<-2.026382696309321E32>
{code}
This is caused because the {{{TestUtil.nextPolygon}}} is producing a polygon
with an extruded colinear self intersecting vertex and the
{{BaseXYShapeTestCase}} is not throwing this as an invalid polygon because the
test case uses {{randomBoolean}}. The simple fix is to switch the TestCase to
always throw an exception on invalid polygons so we never test with a
non-compliant polygon. This passed the queries because the tessellator would
filter out the dirty vertext. This test failed because the dirty vertext just
happened to be the minimum X value. So this does expose an inconsistency where
an invalid polygon will have a bounding box inconsistent with the raw geometry.
I think that's okay because we have API guardrails to enable or disable strict
validation and I don't think that should be removed.
I will open a PR to switch the base test cases over to strict geometry
validation instead of random validation.
> New companion doc value format for LatLonShape and XYShape field types
> --
>
> Key: LUCENE-10654
> URL: https://issues.apache.org/jira/browse/LUCENE-10654
> Project: Lucene - Core
> Issue Type: New Feature
>Reporter: Nick Knize
>Priority: Major
> Fix For: 9.4
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> {{XYDocValuesField}} provides doc value support for {{XYPoint}}.
> {{LatLonDocValuesField}} provides docvalue support for {{LatLonPoint}}.
> However, neither {{LatLonShape}} nor {{XYShape}} currently have a docvalue
> format.
> This lack of doc value support for shapes means facets, aggregations, and
> IndexOrDocValues queries are currently not possible for Shape field types.
> This gap needs be closed in lucene.
> To support IndexOrDocValues queries along with various geometry aggregations
> and facets, the ability to compute the spatial relation with the doc value is
> needed. This is straightforward with {{XYPoint}} and {{LatLonPoint}} since
> the doc value encoding is nothing more than a simple 2D integer encoding of
> the x,y and lat,lon dimensional components. Accomplishing the same with a
> naive integer encoded binary representation for N-vertex shapes would be
> costly.
> {{ComponentTree}} already provides an efficient in memory structure for
> quickly computing spatial relations over Shape types based on a binary tree
> of tessellated triangles provided by the {{Tessellator}}. Furthermore, this
> tessellation is already computed at index time. If we create an on-disk
> representation of {{ComponentTree}} 's binary tree of tessellated triangles
> and use this as the doc value {{binaryValue}} format we will be able to
> efficiently compute spatial relations with this binary representation and
> achieve the same facet/aggregation result over shapes as we can with points
> today (e.g., grid facets, centroid, area, etc).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10667) Multi-Value Support for Binary DocValues
[ https://issues.apache.org/jira/browse/LUCENE-10667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Knize updated LUCENE-10667: Summary: Multi-Value Support for Binary DocValues (was: Multi-Value Support for Binary DocValus) > Multi-Value Support for Binary DocValues > > > Key: LUCENE-10667 > URL: https://issues.apache.org/jira/browse/LUCENE-10667 > Project: Lucene - Core > Issue Type: New Feature >Reporter: Nick Knize >Priority: Major > > LUCENE-10654 introduces a binary doc value format for shapes. Since the > geometries are decomposed into triangles the binary docvalue encoding can > technically support multi shapes and geometry collections in a single doc > value format, however it feels this is hacking around the limitation of > supporting multi-values for binary doc values. With multi-value binary > individual geometries can be stored in their own binary doc value w/ multiple > binary per doc. I'd like to open this issue to explore adding multi-value > support to binary doc values. Are there concerns, limitations, traps? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10666) Multi-Value Support for Binary DocValues
[ https://issues.apache.org/jira/browse/LUCENE-10666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Knize updated LUCENE-10666: Summary: Multi-Value Support for Binary DocValues (was: Multi-Value Support for Binary DocValus) > Multi-Value Support for Binary DocValues > > > Key: LUCENE-10666 > URL: https://issues.apache.org/jira/browse/LUCENE-10666 > Project: Lucene - Core > Issue Type: New Feature >Reporter: Nick Knize >Priority: Major > > LUCENE-10654 introduces a binary doc value format for shapes. Since the > geometries are decomposed into triangles the binary docvalue encoding can > technically support multi shapes and geometry collections in a single doc > value format, however it feels this is hacking around the limitation of > supporting multi-values for binary doc values. With multi-value binary > individual geometries can be stored in their own binary doc value w/ multiple > binary per doc. I'd like to open this issue to explore adding multi-value > support to binary doc values. Are there concerns, limitations, traps? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-10667) Multi-Value Support for Binary DocValus
[ https://issues.apache.org/jira/browse/LUCENE-10667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Knize resolved LUCENE-10667. - Resolution: Duplicate Not sure why Jira created more than one... closing > Multi-Value Support for Binary DocValus > --- > > Key: LUCENE-10667 > URL: https://issues.apache.org/jira/browse/LUCENE-10667 > Project: Lucene - Core > Issue Type: New Feature >Reporter: Nick Knize >Priority: Major > > LUCENE-10654 introduces a binary doc value format for shapes. Since the > geometries are decomposed into triangles the binary docvalue encoding can > technically support multi shapes and geometry collections in a single doc > value format, however it feels this is hacking around the limitation of > supporting multi-values for binary doc values. With multi-value binary > individual geometries can be stored in their own binary doc value w/ multiple > binary per doc. I'd like to open this issue to explore adding multi-value > support to binary doc values. Are there concerns, limitations, traps? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10666) Multi-Value Support for Binary DocValus
Nick Knize created LUCENE-10666: --- Summary: Multi-Value Support for Binary DocValus Key: LUCENE-10666 URL: https://issues.apache.org/jira/browse/LUCENE-10666 Project: Lucene - Core Issue Type: New Feature Reporter: Nick Knize LUCENE-10654 introduces a binary doc value format for shapes. Since the geometries are decomposed into triangles the binary docvalue encoding can technically support multi shapes and geometry collections in a single doc value format, however it feels this is hacking around the limitation of supporting multi-values for binary doc values. With multi-value binary individual geometries can be stored in their own binary doc value w/ multiple binary per doc. I'd like to open this issue to explore adding multi-value support to binary doc values. Are there concerns, limitations, traps? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10667) Multi-Value Support for Binary DocValus
Nick Knize created LUCENE-10667: --- Summary: Multi-Value Support for Binary DocValus Key: LUCENE-10667 URL: https://issues.apache.org/jira/browse/LUCENE-10667 Project: Lucene - Core Issue Type: New Feature Reporter: Nick Knize LUCENE-10654 introduces a binary doc value format for shapes. Since the geometries are decomposed into triangles the binary docvalue encoding can technically support multi shapes and geometry collections in a single doc value format, however it feels this is hacking around the limitation of supporting multi-values for binary doc values. With multi-value binary individual geometries can be stored in their own binary doc value w/ multiple binary per doc. I'd like to open this issue to explore adding multi-value support to binary doc values. Are there concerns, limitations, traps? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10654) New companion doc value format for LatLonShape and XYShape field types
[
https://issues.apache.org/jira/browse/LUCENE-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Knize updated LUCENE-10654:
Fix Version/s: 9.4
(was: 9.3)
> New companion doc value format for LatLonShape and XYShape field types
> --
>
> Key: LUCENE-10654
> URL: https://issues.apache.org/jira/browse/LUCENE-10654
> Project: Lucene - Core
> Issue Type: New Feature
>Reporter: Nick Knize
>Priority: Major
> Fix For: 9.4
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> {{XYDocValuesField}} provides doc value support for {{XYPoint}}.
> {{LatLonDocValuesField}} provides docvalue support for {{LatLonPoint}}.
> However, neither {{LatLonShape}} nor {{XYShape}} currently have a docvalue
> format.
> This lack of doc value support for shapes means facets, aggregations, and
> IndexOrDocValues queries are currently not possible for Shape field types.
> This gap needs be closed in lucene.
> To support IndexOrDocValues queries along with various geometry aggregations
> and facets, the ability to compute the spatial relation with the doc value is
> needed. This is straightforward with {{XYPoint}} and {{LatLonPoint}} since
> the doc value encoding is nothing more than a simple 2D integer encoding of
> the x,y and lat,lon dimensional components. Accomplishing the same with a
> naive integer encoded binary representation for N-vertex shapes would be
> costly.
> {{ComponentTree}} already provides an efficient in memory structure for
> quickly computing spatial relations over Shape types based on a binary tree
> of tessellated triangles provided by the {{Tessellator}}. Furthermore, this
> tessellation is already computed at index time. If we create an on-disk
> representation of {{ComponentTree}} 's binary tree of tessellated triangles
> and use this as the doc value {{binaryValue}} format we will be able to
> efficiently compute spatial relations with this binary representation and
> achieve the same facet/aggregation result over shapes as we can with points
> today (e.g., grid facets, centroid, area, etc).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10654) New companion doc value format for LatLonShape and XYShape field types
[
https://issues.apache.org/jira/browse/LUCENE-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571655#comment-17571655
]
Nick Knize commented on LUCENE-10654:
-
As per discussion on the PR I think this is too late for 9.3 so I'd like to
move forward for 9.4 and iterating the "nice to haves" (visitor access pattern)
in a follow up.
> New companion doc value format for LatLonShape and XYShape field types
> --
>
> Key: LUCENE-10654
> URL: https://issues.apache.org/jira/browse/LUCENE-10654
> Project: Lucene - Core
> Issue Type: New Feature
>Reporter: Nick Knize
>Priority: Major
> Fix For: 9.3
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> {{XYDocValuesField}} provides doc value support for {{XYPoint}}.
> {{LatLonDocValuesField}} provides docvalue support for {{LatLonPoint}}.
> However, neither {{LatLonShape}} nor {{XYShape}} currently have a docvalue
> format.
> This lack of doc value support for shapes means facets, aggregations, and
> IndexOrDocValues queries are currently not possible for Shape field types.
> This gap needs be closed in lucene.
> To support IndexOrDocValues queries along with various geometry aggregations
> and facets, the ability to compute the spatial relation with the doc value is
> needed. This is straightforward with {{XYPoint}} and {{LatLonPoint}} since
> the doc value encoding is nothing more than a simple 2D integer encoding of
> the x,y and lat,lon dimensional components. Accomplishing the same with a
> naive integer encoded binary representation for N-vertex shapes would be
> costly.
> {{ComponentTree}} already provides an efficient in memory structure for
> quickly computing spatial relations over Shape types based on a binary tree
> of tessellated triangles provided by the {{Tessellator}}. Furthermore, this
> tessellation is already computed at index time. If we create an on-disk
> representation of {{ComponentTree}} 's binary tree of tessellated triangles
> and use this as the doc value {{binaryValue}} format we will be able to
> efficiently compute spatial relations with this binary representation and
> achieve the same facet/aggregation result over shapes as we can with points
> today (e.g., grid facets, centroid, area, etc).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10654) New companion doc value format for LatLonShape and XYShape field types
[
https://issues.apache.org/jira/browse/LUCENE-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Knize updated LUCENE-10654:
Fix Version/s: 9.3
> New companion doc value format for LatLonShape and XYShape field types
> --
>
> Key: LUCENE-10654
> URL: https://issues.apache.org/jira/browse/LUCENE-10654
> Project: Lucene - Core
> Issue Type: New Feature
>Reporter: Nick Knize
>Priority: Major
> Fix For: 9.3
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{XYDocValuesField}} provides doc value support for {{XYPoint}}.
> {{LatLonDocValuesField}} provides docvalue support for {{LatLonPoint}}.
> However, neither {{LatLonShape}} nor {{XYShape}} currently have a docvalue
> format.
> This lack of doc value support for shapes means facets, aggregations, and
> IndexOrDocValues queries are currently not possible for Shape field types.
> This gap needs be closed in lucene.
> To support IndexOrDocValues queries along with various geometry aggregations
> and facets, the ability to compute the spatial relation with the doc value is
> needed. This is straightforward with {{XYPoint}} and {{LatLonPoint}} since
> the doc value encoding is nothing more than a simple 2D integer encoding of
> the x,y and lat,lon dimensional components. Accomplishing the same with a
> naive integer encoded binary representation for N-vertex shapes would be
> costly.
> {{ComponentTree}} already provides an efficient in memory structure for
> quickly computing spatial relations over Shape types based on a binary tree
> of tessellated triangles provided by the {{Tessellator}}. Furthermore, this
> tessellation is already computed at index time. If we create an on-disk
> representation of {{ComponentTree}} 's binary tree of tessellated triangles
> and use this as the doc value {{binaryValue}} format we will be able to
> efficiently compute spatial relations with this binary representation and
> achieve the same facet/aggregation result over shapes as we can with points
> today (e.g., grid facets, centroid, area, etc).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10654) New companion doc value format for LatLonShape and XYShape field types
[
https://issues.apache.org/jira/browse/LUCENE-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Knize updated LUCENE-10654:
Description:
{{XYDocValuesField}} provides doc value support for {{XYPoint}}.
{{LatLonDocValuesField}} provides docvalue support for {{LatLonPoint}}.
However, neither {{LatLonShape}} nor {{XYShape}} currently have a docvalue
format.
This lack of doc value support for shapes means facets, aggregations, and
IndexOrDocValues queries are currently not possible for Shape field types. This
gap needs be closed in lucene.
To support IndexOrDocValues queries along with various geometry aggregations
and facets, the ability to compute the spatial relation with the doc value is
needed. This is straightforward with {{XYPoint}} and {{LatLonPoint}} since the
doc value encoding is nothing more than a simple 2D integer encoding of the x,y
and lat,lon dimensional components. Accomplishing the same with a naive integer
encoded binary representation for N-vertex shapes would be costly.
{{ComponentTree}} already provides an efficient in memory structure for quickly
computing spatial relations over Shape types based on a binary tree of
tessellated triangles provided by the {{Tessellator}}. Furthermore, this
tessellation is already computed at index time. If we create an on-disk
representation of {{ComponentTree}} 's binary tree of tessellated triangles and
use this as the doc value {{binaryValue}} format we will be able to efficiently
compute spatial relations with this binary representation and achieve the same
facet/aggregation result over shapes as we can with points today (e.g., grid
facets, centroid, area, etc).
was:
{{XYDocValuesField}} provides doc value support for {{XYPoint}}.
{{LatLonDocValuesField}} provides docvalue support for {{LatLonPoint}}.
However, neither {{LatLonShape}} nor {{XYShape}} currently have a docvalue
format.
This lack of doc value support for shapes means facets, aggregations, and
IndexOrDocValues queries are currently not possible for Shape field types. This
gap needs be closed in lucene.
To support IndexOrDocValues queries along with various geometry aggregations
and facets, the ability to compute the spatial relation with the doc value is
needed. This is straightforward with {{XYPoint}} and {{LatLonPoint}} since the
doc value encoding is nothing more than a simple 2D integer encoding of the x,y
and lat,lon dimensional components. Accomplishing the same with a naive integer
encoded binary representation for N-vertex shapes would be costly.
{{ComponentTree}} already provides an efficient in memory structure for quickly
computing spatial relations over Shape types based on a binary tree of
tessellated triangles provided by the {{Tessellator}}. Furthermore, this
tessellation is already computed at index time. If we create an on-disk
representation of {{ComponentTree}}s binary tree of tessellated triangles and
use this as the doc value {{binaryValue}} format we will be able to efficiently
compute spatial relations with this binary representation and achieve the same
facet/aggregation result over shapes as we can with points today (e.g., grid
facets, centroid, area, etc).
> New companion doc value format for LatLonShape and XYShape field types
> --
>
> Key: LUCENE-10654
> URL: https://issues.apache.org/jira/browse/LUCENE-10654
> Project: Lucene - Core
> Issue Type: New Feature
>Reporter: Nick Knize
>Priority: Major
>
> {{XYDocValuesField}} provides doc value support for {{XYPoint}}.
> {{LatLonDocValuesField}} provides docvalue support for {{LatLonPoint}}.
> However, neither {{LatLonShape}} nor {{XYShape}} currently have a docvalue
> format.
> This lack of doc value support for shapes means facets, aggregations, and
> IndexOrDocValues queries are currently not possible for Shape field types.
> This gap needs be closed in lucene.
> To support IndexOrDocValues queries along with various geometry aggregations
> and facets, the ability to compute the spatial relation with the doc value is
> needed. This is straightforward with {{XYPoint}} and {{LatLonPoint}} since
> the doc value encoding is nothing more than a simple 2D integer encoding of
> the x,y and lat,lon dimensional components. Accomplishing the same with a
> naive integer encoded binary representation for N-vertex shapes would be
> costly.
> {{ComponentTree}} already provides an efficient in memory structure for
> quickly computing spatial relations over Shape types based on a binary tree
> of tessellated triangles provided by the {{Tessellator}}. Furthermore, this
> tessellation is already computed at index time. If we create an on-disk
> representation of {{ComponentTree}} 's binary tree of tessellated triangles
> and use this as the doc
[jira] [Created] (LUCENE-10654) New companion doc value format for LatLonShape and XYShape field types
Nick Knize created LUCENE-10654:
---
Summary: New companion doc value format for LatLonShape and
XYShape field types
Key: LUCENE-10654
URL: https://issues.apache.org/jira/browse/LUCENE-10654
Project: Lucene - Core
Issue Type: New Feature
Reporter: Nick Knize
{{XYDocValuesField}} provides doc value support for {{XYPoint}}.
{{LatLonDocValuesField}} provides docvalue support for {{LatLonPoint}}.
However, neither {{LatLonShape}} nor {{XYShape}} currently have a docvalue
format.
This lack of doc value support for shapes means facets, aggregations, and
IndexOrDocValues queries are currently not possible for Shape field types. This
gap needs be closed in lucene.
To support IndexOrDocValues queries along with various geometry aggregations
and facets, the ability to compute the spatial relation with the doc value is
needed. This is straightforward with {{XYPoint}} and {{LatLonPoint}} since the
doc value encoding is nothing more than a simple 2D integer encoding of the x,y
and lat,lon dimensional components. Accomplishing the same with a naive integer
encoded binary representation for N-vertex shapes would be costly.
{{ComponentTree}} already provides an efficient in memory structure for quickly
computing spatial relations over Shape types based on a binary tree of
tessellated triangles provided by the {{Tessellator}}. Furthermore, this
tessellation is already computed at index time. If we create an on-disk
representation of {{ComponentTree}}s binary tree of tessellated triangles and
use this as the doc value {{binaryValue}} format we will be able to efficiently
compute spatial relations with this binary representation and achieve the same
facet/aggregation result over shapes as we can with points today (e.g., grid
facets, centroid, area, etc).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10162) Add IntField, LongField, FloatField and DoubleField classes to index both points and doc values
[ https://issues.apache.org/jira/browse/LUCENE-10162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17517113#comment-17517113 ] Nick Knize commented on LUCENE-10162: - +1 I also like the idea of specifying their usage as "advanced" to encourage the new, easier to use, fields; but not removing the points and dv classes completely for these more advanced scenarios. > Add IntField, LongField, FloatField and DoubleField classes to index both > points and doc values > --- > > Key: LUCENE-10162 > URL: https://issues.apache.org/jira/browse/LUCENE-10162 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Attachments: LUCENE-10162-1.patch > > > Currently we have IntPoint, LongPoint, FloatPoint and DoublePoint on the one > hand, and NumericDocValuesField and SortedNumericDocValuesField on the other > hand. > When we introduced these classes, this distinction made sense: use the > XXXPoint classes if you want your numeric fields to be searchable and the > XXXDocValuesField classes if you want your numeric fields to be > sortable/aggregatable. > However since then, we introduced logic to take advantage of doc values for > filtering (IndexOrDocValuesQuery) and enhanced sorting to take advantage of > the Points index to skip non-competitive documents. So even if you only need > searching, or if you only need sorting, it's likely a good idea to index both > with points *and* doc values. > Could we make this easier on users by having XXXField classes that > automatically do it as opposed to requiring users to add both an XXXPoint and > an XXXDocValuesField for every numeric field to their index? This could also > make consuming these fields easier, e.g. factory methods for range queries > could automatically use IndexOrDocValuesQuery. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10162) Add IntField, LongField, FloatField and DoubleField classes to index both points and doc values
[
https://issues.apache.org/jira/browse/LUCENE-10162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514866#comment-17514866
]
Nick Knize commented on LUCENE-10162:
-
+1
bq. The way I was thinking of addressing this issue would consist of
implementing these LongField/IntField/... classes from scratch without trying
to reuse LongPoint/IntPoint/... or
NumericDocValuesField/SortedNumericDocValuesField.
+1 as I assume \{Long | Int | ...\}Point and NumericDocValuesField will still
be available and public facing and (at this point) there's no intention of
removing the option to index individually?
> Add IntField, LongField, FloatField and DoubleField classes to index both
> points and doc values
> ---
>
> Key: LUCENE-10162
> URL: https://issues.apache.org/jira/browse/LUCENE-10162
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Attachments: LUCENE-10162-1.patch
>
>
> Currently we have IntPoint, LongPoint, FloatPoint and DoublePoint on the one
> hand, and NumericDocValuesField and SortedNumericDocValuesField on the other
> hand.
> When we introduced these classes, this distinction made sense: use the
> XXXPoint classes if you want your numeric fields to be searchable and the
> XXXDocValuesField classes if you want your numeric fields to be
> sortable/aggregatable.
> However since then, we introduced logic to take advantage of doc values for
> filtering (IndexOrDocValuesQuery) and enhanced sorting to take advantage of
> the Points index to skip non-competitive documents. So even if you only need
> searching, or if you only need sorting, it's likely a good idea to index both
> with points *and* doc values.
> Could we make this easier on users by having XXXField classes that
> automatically do it as opposed to requiring users to add both an XXXPoint and
> an XXXDocValuesField for every numeric field to their index? This could also
> make consuming these fields easier, e.g. factory methods for range queries
> could automatically use IndexOrDocValuesQuery.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9981) CompiledAutomaton.getCommonSuffix can be extraordinarily slow, even with default maxDeterminizedStates limit
[
https://issues.apache.org/jira/browse/LUCENE-9981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355256#comment-17355256
]
Nick Knize commented on LUCENE-9981:
bq. Even with no API break, I don't want these changes rushed in to meet some
arbitrary 8.9 deadline, I'm really concerned about that.
+1 for letting this bake in main. Consumers of the API should handle the lack
of auth separately from the fix. Many thanks to [~rmuir] and [~mikemccand] for
the hard work on this.
> CompiledAutomaton.getCommonSuffix can be extraordinarily slow, even with
> default maxDeterminizedStates limit
>
>
> Key: LUCENE-9981
> URL: https://issues.apache.org/jira/browse/LUCENE-9981
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Robert Muir
>Priority: Major
> Attachments: LUCENE-9981.patch, LUCENE-9981.patch, LUCENE-9981.patch,
> LUCENE-9981.patch, LUCENE-9981.patch, LUCENE-9981_nfaprefix.patch,
> LUCENE-9981_test.patch, three-repeats-reverse-det.png, three-repeats.png
>
>
> We have a {{maxDeterminizedStates = 1}} limit designed to keep
> regexp-type queries from blowing up.
> But we have an adversary that will run for 268s on my laptop before hitting
> exception, first reported here:
> https://github.com/opensearch-project/OpenSearch/issues/687
> When I run the test and jstack the threads, this what I see:
> {noformat}
> "TEST-TestOpensearch687.testInteresting-seed#[4B9C20A027A9850C]" #15 prio=5
> os_prio=0 cpu=56960.04ms elapsed=57.49s tid=0x7fff7006ca80 nid=0x231c8
> runnable [0x7fff8b7f]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.lucene.util.automaton.SortedIntSet.decr(SortedIntSet.java:106)
> at
> org.apache.lucene.util.automaton.Operations.determinize(Operations.java:769)
> at
> org.apache.lucene.util.automaton.Operations.getCommonSuffixBytesRef(Operations.java:1155)
> at
> org.apache.lucene.util.automaton.CompiledAutomaton.(CompiledAutomaton.java:247)
> at
> org.apache.lucene.search.AutomatonQuery.(AutomatonQuery.java:104)
> at
> org.apache.lucene.search.AutomatonQuery.(AutomatonQuery.java:82)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:138)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:114)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:72)
> at org.apache.lucene.search.RegexpQuery.(RegexpQuery.java:62)
> at
> org.apache.lucene.TestOpensearch687.testInteresting(TestOpensearch687.java:42)
> {noformat}
> This is really sad, as {{getCommonSuffixBytesRef()}} is only supposed to be
> an "up-front" optimization to make the actual subsequent terms-intensive part
> of the query faster. But it makes the whole query run for nearly 5 minutes
> before it does anything.
> So I definitely think we should improve {{getCommonSuffixBytesRef}} to be
> more "best-effort". For example, it can reduce the lower bound to {{1000}}
> and catch the exception like such:
> {code}
> try {
>// this is slow, and just an opto anyway, so don't burn cycles on it for
> some crazy worst-case.
>// if we don't set this common suffix, the query will just run a bit
> slower, that's all.
>int limit = Math.min(1000, maxDeterminizedStates);
>BytesRef suffix = Operations.getCommonSuffixBytesRef(binary, limit);
>... (setting commonSuffixRef)
> } catch (TooComplexTooDeterminizeException notWorthIt) {
> commonSuffixRef = null;
> }
> {code}
> Another, maybe simpler option, is to just check that input state/transitions
> accounts don't exceed some low limit N.
> Basically this opto is geared at stuff like leading wildcard query of "*foo".
> By computing that the common suffix is "foo" we can spend less CPU in the
> terms dictionary because we can first do a memcmp before having to run data
> thru any finite state machine. It's really a microopt and we shouldn't be
> spending whole seconds of cpu on it, ever.
> But I still don't quite understand how the current limits are giving the
> behavior today, maybe there is a bigger issue and I don't want to shove
> something under the rug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9552) Add a LatLonPoint query that accepts an array of LatLonGeometries
[
https://issues.apache.org/jira/browse/LUCENE-9552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243456#comment-17243456
]
Nick Knize commented on LUCENE-9552:
Note [~ivera], [Reproducible
failure|https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-Check-master/980/console]
{code:java}
2> NOTE: reproduce with: gradlew test --tests
TestLatLonPointQueries.testLowCardinality -Dtests.seed=B77EDE3ECA6A5F79
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true
-Dtests.locale=seh-MZ -Dtests.timezone=America/Juneau -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
2> NOTE: test params are: codec=Lucene90,
sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=seh-MZ,
timezone=America/Juneau
2> NOTE: Linux 5.4.0-54-generic amd64/Oracle Corporation 15.0.1
(64-bit)/cpus=8,threads=1,free=239814848,total=270532608
2> NOTE: All tests run in this JVM: [TestLatLonPointQueries]
{code}
{code:java}
> Task :lucene:core:test
org.apache.lucene.search.TestLatLonPointQueries > testLowCardinality FAILED
java.lang.AssertionError
at
__randomizedtesting.SeedInfo.seed([B77EDE3ECA6A5F79:FCE542B87CE96F0B]:0)
at org.apache.lucene.geo.Rectangle.(Rectangle.java:58)
at
org.apache.lucene.geo.GeoEncodingUtils.createComponentPredicate(GeoEncodingUtils.java:185)
at
org.apache.lucene.document.LatLonPointInGeometryQuery.createWeight(LatLonPointInGeometryQuery.java:147)
at
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:849)
at
org.apache.lucene.search.AssertingIndexSearcher.createWeight(AssertingIndexSearcher.java:57)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:528)
at
org.apache.lucene.geo.BaseGeoPointTestCase.searchIndex(BaseGeoPointTestCase.java:1126)
at
org.apache.lucene.geo.BaseGeoPointTestCase.verifyRandomGeometries(BaseGeoPointTestCase.java:1068)
at
org.apache.lucene.geo.BaseGeoPointTestCase.verify(BaseGeoPointTestCase.java:776)
at
org.apache.lucene.geo.BaseGeoPointTestCase.testLowCardinality(BaseGeoPointTestCase.java:454)
{code}
> Add a LatLonPoint query that accepts an array of LatLonGeometries
> -
>
> Key: LUCENE-9552
> URL: https://issues.apache.org/jira/browse/LUCENE-9552
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Ignacio Vera
>Assignee: Ignacio Vera
>Priority: Major
> Fix For: 8.8
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> LatLonPoint currently support three types of queries, against a bound box, an
> array of polygons or by distance (circle). Therefore if you currently want to
> combine a query with two of those geometries, a user will need to combine the
> query with a boolean OR.
> It would be a nice addition to have a query that accepts an array of
> LatLonGeometries and generates that boolean OR query in a single query.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9149) Increase data dimension limit in BKD
[
https://issues.apache.org/jira/browse/LUCENE-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Knize resolved LUCENE-9149.
Resolution: Implemented
> Increase data dimension limit in BKD
>
>
> Key: LUCENE-9149
> URL: https://issues.apache.org/jira/browse/LUCENE-9149
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Nick Knize
>Priority: Major
> Attachments: LUCENE-9149.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> LUCENE-8496 added selective indexing; the ability to designate the first K <=
> N dimensions for driving the construction of the BKD internal nodes. Follow
> on work stored the "data dimensions" for only the leaf nodes and only the
> "index dimensions" are stored for the internal nodes. While
> {{maxPointsInLeafNode}} is still important for managing the BKD heap memory
> footprint (thus we don't want this to get too large), I'd like to propose
> increasing the {{MAX_DIMENSIONS}} limit (to something not too crazy like 16;
> effectively doubling the index dimension limit) while maintaining the
> {{MAX_INDEX_DIMENSIONS}} at 8.
> Doing this will enable us to encode higher dimension data within a lower
> dimension index (e.g., 3D tessellated triangles as a 10 dimension point using
> only the first 6 dimensions for index construction)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9150) Restore support for dynamic PlanetModel in Geo3D
Nick Knize created LUCENE-9150:
--
Summary: Restore support for dynamic PlanetModel in Geo3D
Key: LUCENE-9150
URL: https://issues.apache.org/jira/browse/LUCENE-9150
Project: Lucene - Core
Issue Type: Improvement
Reporter: Nick Knize
LUCENE-7072 removed dynamic planet model support in Geo3D. This was logical at
the time (given the state of Lucene and spatial projections and coordinate
reference systems). Since then, however, there have been a lot of new
developments within the OGC community around [Coordinate Reference
Systems|https://docs.opengeospatial.org/as/18-005r4/18-005r4.html], [Dynamic
Coordinate Reference
Systems|http://docs.opengeospatial.org/DRAFTS/18-058.html], and [Updated ISO
Standards|https://www.iso.org/obp/ui/#iso:std:iso:19111:ed-3:v1:en].
It would be useful for Geo3D (and eventually LatLon*) to support different
geographic datums to make lucene a viable option for indexing/searching in
different spatial reference systems (e.g., more accurately computing query
shape relations to BKD's internal nodes using datum consistent with the spatial
projection). This would also provide an alternative to other limitations of the
{{LatLon*/XY*}} implementation (e.g., pole/dateline crossing, quantization of
small polygons).
I'd like to propose keeping the current WGS84 static datum as the default for
Geo3D but adding back the constructors to accept custom planet models. Perhaps
this could be listed as an "expert" API feature?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9149) Increase data dimension limit in BKD
[
https://issues.apache.org/jira/browse/LUCENE-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Knize updated LUCENE-9149:
---
Attachment: LUCENE-9149.patch
Status: Open (was: Open)
Attached patch:
* refactors most {{numDataDim}} variables to more accurate name {{numDims}}
* increases {{MAX_DIMENSIONS}} to 16, keeps {{MAX_INDEX_DIMENSIONS}} at 8
* updates random test suites to test with new increased limit
Will open a PR for review
> Increase data dimension limit in BKD
>
>
> Key: LUCENE-9149
> URL: https://issues.apache.org/jira/browse/LUCENE-9149
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Nick Knize
>Priority: Major
> Attachments: LUCENE-9149.patch
>
>
> LUCENE-8496 added selective indexing; the ability to designate the first K <=
> N dimensions for driving the construction of the BKD internal nodes. Follow
> on work stored the "data dimensions" for only the leaf nodes and only the
> "index dimensions" are stored for the internal nodes. While
> {{maxPointsInLeafNode}} is still important for managing the BKD heap memory
> footprint (thus we don't want this to get too large), I'd like to propose
> increasing the {{MAX_DIMENSIONS}} limit (to something not too crazy like 16;
> effectively doubling the index dimension limit) while maintaining the
> {{MAX_INDEX_DIMENSIONS}} at 8.
> Doing this will enable us to encode higher dimension data within a lower
> dimension index (e.g., 3D tessellated triangles as a 10 dimension point using
> only the first 6 dimensions for index construction)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9149) Increase data dimension limit in BKD
Nick Knize created LUCENE-9149:
--
Summary: Increase data dimension limit in BKD
Key: LUCENE-9149
URL: https://issues.apache.org/jira/browse/LUCENE-9149
Project: Lucene - Core
Issue Type: Improvement
Reporter: Nick Knize
LUCENE-8496 added selective indexing; the ability to designate the first K <= N
dimensions for driving the construction of the BKD internal nodes. Follow on
work stored the "data dimensions" for only the leaf nodes and only the "index
dimensions" are stored for the internal nodes. While {{maxPointsInLeafNode}} is
still important for managing the BKD heap memory footprint (thus we don't want
this to get too large), I'd like to propose increasing the {{MAX_DIMENSIONS}}
limit (to something not too crazy like 16; effectively doubling the index
dimension limit) while maintaining the {{MAX_INDEX_DIMENSIONS}} at 8.
Doing this will enable us to encode higher dimension data within a lower
dimension index (e.g., 3D tessellated triangles as a 10 dimension point using
only the first 6 dimensions for index construction)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8369) Remove the spatial module as it is obsolete
[ https://issues.apache.org/jira/browse/LUCENE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Knize updated LUCENE-8369: --- Resolution: Resolved Status: Resolved (was: Patch Available) > Remove the spatial module as it is obsolete > --- > > Key: LUCENE-8369 > URL: https://issues.apache.org/jira/browse/LUCENE-8369 > Project: Lucene - Core > Issue Type: Task > Components: modules/spatial >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Attachments: LUCENE-8369.patch > > Time Spent: 20m > Remaining Estimate: 0h > > The "spatial" module is at this juncture nearly empty with only a couple > utilities that aren't used by anything in the entire codebase -- > GeoRelationUtils, and MortonEncoder. Perhaps it should have been removed > earlier in LUCENE-7664 which was the removal of GeoPointField which was > essentially why the module existed. Better late than never. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
