[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875108#comment-15875108
]
Ishan Chattopadhyaya commented on SOLR-5944:
The other usages of were on a per field definition, whereas I wanted to use it
on a per fieldType definition.
The other usages, e.g.
{code}
{code}
seemed like mis-named. "pfloat" here refers to a named fieldType definition,
and not a "floatClass". Hence, I chose the other convention of
"FloatPointField" being referred to as "solr.tests.floatClassName".
To standardize, I think we should rename these two as: (a)
"solr.tests.floatFieldType" as pfloat or float, when used on a per field basis,
or (b) "solr.tests.floatClassName" as FloatPointField, when used in a fieldType
definition.
WDYT?
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
> Fix For: 6.5, master (7.0)
>
> Attachments: defensive-checks.log.gz,
> demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch,
> DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png,
> regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874986#comment-15874986
]
Varun Thacker commented on SOLR-5944:
-
Hi Ishan,
While working on integrating points in the collapse query parser on SOLR-9994 ,
I was looking at how the existing tests were randomizing points /trie fields.
Most tests use the "solr.tests.intClass" system property including how it's
randomized in SolrTestCaseJ4
In this feature we use the "solr.tests.intClassName" system property. Although
it means the same thing a small improvement here will be to use the same
property names to make our tests more consistent?
I think we just need to rename these fields in {{schema-inplace-updates.xml}}
and it's usages?
{code}
{code}
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
> Fix For: 6.5, master (7.0)
>
> Attachments: defensive-checks.log.gz,
> demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch,
> DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png,
> regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873575#comment-15873575 ] ASF subversion and git services commented on SOLR-5944: --- Commit ea19bf5101817bae5b7b133a7d9d40ab41aac6ec in lucene-solr's branch refs/heads/master from [~ichattopadhyaya] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ea19bf5 ] Move solr/CHANGES.txt entries to appropriate sections after backporting SOLR-5944 and SOLR-10114 > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya > Fix For: 6.5, master (7.0) > > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873549#comment-15873549 ] ASF subversion and git services commented on SOLR-5944: --- Commit 476cea57e8e55832878c3b4c8efe1cf6f113b3c4 in lucene-solr's branch refs/heads/branch_6x from [~ichattopadhyaya] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=476cea5 ] SOLR-5944: Cleanup comments and logging, use NoMergePolicy instead of LogDocMergePolicy Conflicts: solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873548#comment-15873548 ] ASF subversion and git services commented on SOLR-5944: --- Commit 4de140bf8b5e621ad70a07cb272ddc7135346eaf in lucene-solr's branch refs/heads/branch_6x from [~ichattopadhyaya] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4de140b ] SOLR-5944: Use SolrCmdDistributor's synchronous mode for in-place updates > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873553#comment-15873553 ] ASF subversion and git services commented on SOLR-5944: --- Commit 0689b2a7fd05c86c7fc2f1d1adffdd631d671ba1 in lucene-solr's branch refs/heads/branch_6x from [~ichattopadhyaya] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0689b2a ] SOLR-5944: Suppress PointFields for TestSegmentSorting > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873545#comment-15873545 ] ASF subversion and git services commented on SOLR-5944: --- Commit e9d3bdd02ada23ad09bcc6fc7ff3661880dd45bc in lucene-solr's branch refs/heads/branch_6x from [~ichattopadhyaya] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e9d3bdd ] SOLR-5944: In-place updates of Numeric DocValues Conflicts: solr/CHANGES.txt solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java solr/core/src/test-files/solr/collection1/conf/schema.xml solr/core/src/test/org/apache/solr/cloud/TestSegmentSorting.java > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873445#comment-15873445 ] ASF subversion and git services commented on SOLR-5944: --- Commit 6358afbea66239436a6c0c52e088eeecebac1f65 in lucene-solr's branch refs/heads/master from [~ichattopadhyaya] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6358afb ] SOLR-5944: Cleanup comments and logging, use NoMergePolicy instead of LogDocMergePolicy > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873405#comment-15873405 ] ASF subversion and git services commented on SOLR-5944: --- Commit d5ef026f844a22cf56e56c86956b1c2a11a0a751 in lucene-solr's branch refs/heads/master from [~ichattopadhyaya] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d5ef026 ] SOLR-5944: Use SolrCmdDistributor's synchronous mode for in-place updates > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869838#comment-15869838 ] Ishan Chattopadhyaya commented on SOLR-5944: [~manokovacs], I shall be able to backport this some time mid next week (perhaps by 23rd February). Once that is done, I can backport SOLR-10114. Otherwise, in case you'd like SOLR-10114 to be backported sooner, I can work around it while backporting SOLR-5944. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869724#comment-15869724 ] Mano Kovacs commented on SOLR-5944: --- [~ichattopadhyaya], may I ask if the commit 5375410 (SOLR-5944: In-place updates of Numeric DocValues) will be backported to 6x? I am interested because the patch for SOLR-10114 based on your changes, and I would create a backport if this this one will not be backported. Thanks! > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855877#comment-15855877 ] ASF subversion and git services commented on SOLR-5944: --- Commit 0d760aeddbc95f2de977dcaed781547f2cbf715f in lucene-solr's branch refs/heads/master from [~ichattopadhyaya] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0d760ae ] SOLR-5944: Reverting the previous fix for SolrCmdDistributor > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855837#comment-15855837 ] ASF subversion and git services commented on SOLR-5944: --- Commit 3574404e3d3d5758c0d07f73fda72decc97c07d2 in lucene-solr's branch refs/heads/master from [~ichattopadhyaya] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3574404 ] SOLR-5944: Use SolrCmdDistributor's synchronous mode for in-place updates > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839375#comment-15839375 ] ASF subversion and git services commented on SOLR-5944: --- Commit 5375410807aecf3cc67f82ca1e9ee591f39d0ac7 in lucene-solr's branch refs/heads/apiv2 from [~ichattopadhyaya] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5375410 ] SOLR-5944: In-place updates of Numeric DocValues > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838969#comment-15838969 ] Ishan Chattopadhyaya commented on SOLR-5944: Planning to backport to 6x after SOLR-8396 is backported. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838967#comment-15838967 ] ASF subversion and git services commented on SOLR-5944: --- Commit 5375410807aecf3cc67f82ca1e9ee591f39d0ac7 in lucene-solr's branch refs/heads/master from [~ichattopadhyaya] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5375410 ] SOLR-5944: In-place updates of Numeric DocValues > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png, > regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833767#comment-15833767
]
Ishan Chattopadhyaya commented on SOLR-5944:
{code}
In this version of the script [0], the exact same 20K documents are added, and
the script still does
50 iterations of 5K updates – but this time only to the stored+indexed long
field...
{code}
Hoss' results, as of c21d8a005387eae4d0f0adde4de7e4e465fb73c8, on his laptop:
||Branch||Adds||stored+indexed Updates||
|master|49|521|
|5944|50|527|
My results:
||Branch||Adds||stored+indexed Updates||
|master|37|603|
|5944|38|637|
Seems like a 1% (Hoss) and 5% (my) slowdown in the regular non-dv updates from
master to 5944. Will try to dig deeper to see if this slowdown can be
avoided/reduced.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: defensive-checks.log.gz,
> demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch,
> DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833574#comment-15833574
]
Ishan Chattopadhyaya commented on SOLR-5944:
Hoss did some initial single threaded benchmarks on the performance impact of
doing in-place updates as compared to regular atomic updates.
{code}
The script [0] adds 20K docs containing a stored+indexed text field, a DVO long
field, and a stored+indexed text field. It then does 50 iterations of 5K
updates to each of the 2 long fields (500K updates total, 250K on the DVO
field, 250K on the stored+indexed field), recording the cumulative total times
for the udpates to each type of field.
{code}
Here are some results (times are in seconds):
Hoss' run, as of c21d8a005387eae4d0f0adde4de7e4e465fb73c8, on his laptop:
||Branch||Adds||DVO Updates||stored+indexed Updates||
|master|52|531|543|
|5944|51|352|503|
My run, as of right now:
||Branch||Adds||DVO Updates||stored+indexed Updates||
|master|40|682|663|
|5944|36|295|694|
Seems like Hoss observed a 30% speedup on 5944 branch (in-place update vs.
regular update), and I observed a 57% speedup on the same usecase. More
benchmarks need to done to analyse the performance in multi-threaded indexing,
and also to ensure there's no performance regression for non-dv updates.
[0] - https://gist.github.com/anonymous/ae8bf7db3c713bf9d45937ec0aa1cfae
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: defensive-checks.log.gz,
> demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch,
> DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831619#comment-15831619 ] Ishan Chattopadhyaya commented on SOLR-5944: Linking another PointsField issue, SOLR-10011, which affects this branch. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831170#comment-15831170 ] Ishan Chattopadhyaya commented on SOLR-5944: There are some new failures on the jira/solr-5944 branch after SOLR-8396 was merged to master. One of the failures is due to SOLR-9996, but there seem to be some other issue as well. I am looking into them. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806566#comment-15806566
]
Ishan Chattopadhyaya commented on SOLR-5944:
{quote}
We're either going to need to change things as part of SOLR-9941 first, or
deal with out of order DBQs differently as part of this issue
{quote}
I have really tried to think of various ways to "deal with out of order DBQs
differently", but haven't found anything other than the current fetch from
leader logic. I've even looked at ways to "undelete" a recently DBQ'd document,
but that didn't look so promising. There is likely no clean way, in a replica,
to retro-actively decide to ditch the partial update and instead do a full
update (since that decision has already been taken in the past by the leader,
so only going back to the leader for a full update/document can suffice here).
Hence, I would think we need to address SOLR-9941.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz,
> demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806457#comment-15806457
]
Hoss Man commented on SOLR-5944:
while reviewing some of ishan's recent commits to the jira/solr-5944 branch I
was confused by how exactly
{{TestRecovery.testLogReplayWithInPlaceUpdatesAndDeletes}} works and what it
was doing, and asked ishan about it offline -- while he walked me through the
test, we realized that at one point while writting it he had assumed some weird
behavior he had seen relating to the update log replay was an artifact of the
test harness, and he included this line to work around it...
{code}
// Clearing the updatelog as it would happen after a fresh node restart
h.getCore().getUpdateHandler().getUpdateLog().deleteAll();
{code}
..but the more we talked about it the more it seemed like a legitimate bug --
either in the new code, or in the existing log replay code.
I've been investigating and it seems like this is intentional, but weird, code
on master, see SOLR-9941.
The net result being that the test as original written (w/o that call to
{{getUpdateLog().deleteAll();}} really was finding a problematic situation with
log recovery on the branch, because of how the existing DUH2 code tries to
pre-emptively apply DBQs during log recovery. We're either going to need to
change things as part of SOLR-9941 first, or deal with out of order DBQs
differently as part of this issue, because the current approach of "re-fetch
whole doc from leader" won't work if the leader (or a single node install) is
itself recovering from it's tlog.
Here's some simple steps to demonstrate the problem as it stands on the
jira/solr-5944 branch (as of 5db04fd)...
{noformat}
bin/solr -e techproducts
curl -X POST http://localhost:8983/solr/techproducts/config -H 'Content-Type:
application/json' --data-binary
'{"set-property":{"updateHandler.autoCommit.maxTime":"-1"}}'
curl -X POST -H 'Content-type:application/json' --data-binary '{
"add-field":{
"name":"foo_dvo",
"type":"int",
"stored":false,
"indexed":false,
"docValues":true }
}' http://localhost:8983/solr/techproducts/schema
curl -X POST http://localhost:8983/solr/techproducts/update -H 'Content-Type:
application/json' --data-binary '{
"add":{
"doc": {
"id": "DOCX",
"foo_dvo": 41
}
},
"delete": { "query":"foo_dvo:42" },
"delete": { "query":"foo_dvo:43" },
"add":{
"doc": {
"id": "DOCX",
"foo_dvo": { "inc" : "1" }
}
},
"delete": { "query":"foo_dvo:41" },
"delete": { "query":"foo_dvo:43" },
"add":{
"doc": {
"id": "DOCX",
"foo_dvo": { "inc" : "1" }
}
},
"delete": { "query":"foo_dvo:41" },
"delete": { "query":"foo_dvo:42" }
}'
# verify the in-place atomic updates were applied correctly...
curl 'http://localhost:8983/solr/techproducts/get?wt=json=DOCX'
{
"doc":
{
"id":"DOCX",
"_version_":1555823554278195200,
"foo_dvo":43}}
# crash the node...
kill -9 PID # use whatever PID you get from "ps -ef | grep start.jar | grep
techproducts"
# restart and let recovery from log replay happen...
bin/solr start -s example/techproducts/solr/
# because of how the DUH2/UpdateLog code interacts, the doc is deleted during
one of the DBQs and the inplace update code can't recover it from anywhere
else...
curl 'http://localhost:8983/solr/techproducts/get?wt=json=DOCX'
{
"doc":null}
{noformat}
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
>
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15744004#comment-15744004
]
Ishan Chattopadhyaya commented on SOLR-5944:
{quote}
...none of those comments make sense now that there's no dynamic field checking
{quote}
Indeed, removed in my latest commit.
{quote}
...isn't that comment now bogus? aren't you explicitly requesting stored fields
w/true ?
{quote}
That false -> true change was for the resolveFullDocument parameter, so the
comment (meant for avoiding the retrieval of stored fields) remains valid.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz,
> demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742777#comment-15742777
]
Hoss Man commented on SOLR-5944:
bq. e528b46c6aa1824aa96f2a47a0d55f085967e02a
{code}
@@ -234,36 +233,27 @@ public class AtomicUpdateDocumentMerger {
...
+ if (! fieldNamesFromIndexWriter.contains(fieldName) ) {
+// nocommit: this comment is not usefull - doesn't explain *WHY*
+return Collections.emptySet(); // if dynamic field and this field
doesn't exist, DV update can't work
+ }
{code}
...none of those comments make sense now that there's no dynamic field checking
{code}
@@ -284,42 +274,29 @@ public class AtomicUpdateDocumentMerger {
updatedFields.add(DistributedUpdateProcessor.VERSION_FIELD); // add the
version field so that it is fetched too
SolrInputDocument oldDocument =
RealTimeGetComponent.getInputDocument(cmd.getReq().getCore(),
- idBytes, true, updatedFields,
false); // avoid stored fields from index
+ idBytes, true, updatedFields,
true); // avoid stored fields from index
{code}
...isn't that comment now bogus? aren't you explicitly requesting stored fields
w/{{true}} ?
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz,
> demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742173#comment-15742173 ] Ishan Chattopadhyaya commented on SOLR-5944: Hoss, I've fixed the test failure mentioned in the comment [0] in the commit [1]. Also, the presence or absence of a default value for a DV should not affect how it is dealt with. [0] - (the testReplay* failures) https://issues.apache.org/jira/browse/SOLR-5944?focusedCommentId=15727716=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15727716 [1] - e528b46c6aa1824aa96f2a47a0d55f085967e02a > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735756#comment-15735756 ] Hoss Man commented on SOLR-5944: bq. So, therefore, I think we should be checking if the field exists or not, irrespective of explicit or dynamic. And if the field doesn't exist, we should just delegate to a regular atomic update. Do you think it makes sense? IIUC, in a nutshell: all of my questions about why we're treating dynamicFields as special will no longer be relevent, because dynamicFields will no longer be considered special, because they were never really special to begin with, they just seemed that way because the only testing of non-dynamic fields assumed/required them to have a schema default. we'll remove that assumption from both the tests and code, and treat all fields the same. If my understanding is correct, then yes you're plan to move forward seems sound. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735138#comment-15735138
]
Ishan Chattopadhyaya commented on SOLR-5944:
{quote}
Specifically: when I tried to remove it, I started seeing NPEs from
SolrIndexSearcher.decorateDocValueFields in various tests:
{quote}
I tried to chase this bug down, and I think it has led me to some learnings
regarding the explicit vs dynamic fields (why dynamic fields are any different)
issue.
{quote}
The reason for this is that when a document is created, DV fields for all
explicit fields are created. But when a non-existent dynamic field is attempted
to be updated subsequently, then the underlying DV field doesn't exist.
{quote}
I think I missed the fact that "DV fields for all explicit fields are created"
*only if* they have a default value set on them. Otherwise, explicit fields are
no different from dynamic fields.
So, therefore, I think we should be checking if the field exists or not,
irrespective of explicit or dynamic. And if the field doesn't exist, we should
just delegate to a regular atomic update. Do you think it makes sense?
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz,
> demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735091#comment-15735091
]
Ishan Chattopadhyaya commented on SOLR-5944:
{quote}
In general, i think there's a gap in the DUP logic related to how the results
of fetchFullUpdateFromLeader are used when replicas never get dependent updates
{quote}
{quote}
// nocommit: INNER ELSE -- I'm concerned/confused by this
code happening here...
//
// at this point, we're replacing the data in our existing
"in-place" update (cmd) so it
// becomes a normal "add" using the full SolrInputDocument
fetched from the leader
//
// but this if/else is itself is wrapped in a bigger if (grep
for "OUTER IF" above
// and "OUTER ELSE" below) where the "else" clause does some
sanity checking / processing
// logic for "// non inplace update, i.e. full document
update" -- all of which is
// skipped for our modified "cmd"
//
// shouldn't the logic in that outer else clause also be
applied to our
// "no longer really an in-place" update?
{quote}
I think that the code there should be refactored more nicely. However, as it is
currently in the branch, I think the same checks within the outer else are
fulfilled; with the exception of {{checkDeleteByQueries = true}}, which doesn't
seem to be used anywhere. However, even if the current code (in the branch)
works correctly, it is very confusing and complicated and I shall try to
refactor it in such a way that the code in that outer else block gets
explicitly executed somehow.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz,
> demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735064#comment-15735064
]
Ishan Chattopadhyaya commented on SOLR-5944:
{quote}
The concern i have is that AFAICT rebuilding the Arrays.asList(tlog,
prevMapLog, prevMapLog2) in every iteration of the while loop seems to be at
cross purposes with the reason why the while loop might have more then one
iteration.
{quote}
Your explanation makes complete sense. I think we should pull it out. I've yet
to look deeply into your suggestion of having it all within the same
synchronized block, but it makes sense to me.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz,
> demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733820#comment-15733820
]
Hoss Man commented on SOLR-5944:
I've pushed an update to {{TestInPlaceUpdatesDistrib}} that refactors some
randomized index building into a new {{buildRandomIndex}} helper method which
is now used by most of the "test" methods in that class.
It's *NOT* currently used by {{docValuesUpdateTest()}} even though it was
designed to be -- I made several aborted attempts to try and switch that method
to use {{buildRandomIndex}} knowing that many assertions in that test would
need other tweaks to account for more docs in the index, but i kept running
into weird failures that took me a while to explain.
Ultimately I realized the problem is that currently,
{{schema-inplace-updates.xml}} is configured with {{inplace_updatable_float}}
having a {{default="0"}} setting -- which (besides making most of our testing
using hta field much weaker then i realized) means that the initial sanity
checks in {{docValuesUpdateTest()}} are even less useful then i originally
thought.
[~ichattopadhyaya]: do you remember why this default is set on
{{inplace_updatable_float}} (and {{inplace_updatable_int}})
{{schema-inplace-updates.xml}} ? ... i see {{TestInPlaceUpdatesDistrib}} doing
a preliminary sanity check assertion that the defaults exists in the schema,
but I don't see any test that seems to care/expect that default to work, and it
seems to weaken our test coverage of the more common case...
Specifically: when I tried to remove it, I started seeing NPEs from
{{SolrIndexSearcher.decorateDocValueFields}} in various tests:
{noformat}
[junit4] ERROR 0.05s J2 |
TestInPlaceUpdatesStandalone.testUpdateTwoDifferentFields <<<
[junit4]> Throwable #1: java.lang.NullPointerException
[junit4]>at
__randomizedtesting.SeedInfo.seed([29D61963E75459C5:26F189B6F032D44A]:0)
[junit4]>at
org.apache.solr.search.SolrIndexSearcher.decorateDocValueFields(SolrIndexSearcher.java:810)
[junit4]>at
org.apache.solr.handler.component.RealTimeGetComponent.getInputDocument(RealTimeGetComponent.java:599)
[junit4]>at
org.apache.solr.update.processor.AtomicUpdateDocumentMerger.doInPlaceUpdateMerge(AtomicUpdateDocumentMerger.java:286)
[junit4]>at
org.apache.solr.update.processor.DistributedUpdateProcessor.getUpdatedDocument(DistributedUpdateProcessor.java:1414)
[junit4]>at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1072)
[junit4]>at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:751)
[junit4]>at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorF
{noformat}
...while it certainly makes sense to have some testing of inplace updates when
there is a schema specified {{default}} that's *non-zero* (although see the
previously mentioned SOLR-9838 for some issues with doing that currently), i'm
now concerned about how much of the code may *only* be working _because_ these
fields have an explicit {{default="0"}} ?
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730122#comment-15730122 ] Hoss Man commented on SOLR-5944: bq. I suspect there is some code path in the regular atomic update logic where docs read from the tlog get any default field values added to them, and that isn't happening in the inplace code path case? ... not certain ... might be another bug to look into? Yup. I committed some new tests cases to AtomicUpdatesTest demonstrating 2 classes of problems: * testFieldsWithDefaultValuesWhenAtomicUpdatesAgainstTlog ** checks that after doing an (inplace or regular) atomic update against an uncommited doc, that all the expected "default" field values are returned by RTG ** this test passes fine on master, but on the SOLR-5944 branch the field that supports inplace updates breaks the test. * testAtomicUpdateOfFieldsWithDefaultValue ** attempts an "inc" against a field whose value comes from a schema default ** currently disabled completely because it's broken on master for regular atomic updates: SOLR-9838 > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729839#comment-15729839
]
Ishan Chattopadhyaya commented on SOLR-5944:
{quote}
IIRC comments about commits to jira/* branches are suppressed intentionally as
noise, because it's expected that there will be lots of iteration on the
branches, some of which might be thrown away, and for posterity what matters is
only commits to main line dev branches
{quote}
I see. I was looking at SOLR-8029, and assumed that it is the norm for those
notifications to make it to the JIRA issue by default, but I guess those may
have been enabled explicitly for that issue. :-)
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729798#comment-15729798
]
Ishan Chattopadhyaya commented on SOLR-5944:
[~hossman], I can confirm that this indeed is a genuine bug! I could trace the
culprit to the region where I'm doing the following in AUDM:
{code}
// If oldDocument doesn't have a field that is present in updatedFields,
// then fetch the field from RT searcher into oldDocument.
// This can happen if the oldDocument was fetched from tlog, but the DV
field to be
// updated was not in that document.
{code}
I think something like a full resolve (following documents using the
prevPointer up to the last full document) is in order, and my trying to
optimize there is the culprit in that case. Doing so without rewriting a bunch
of code (that does resolve) and trying to maximize code reuse
(resolveFullDocument()?) would be a challenge to get right, and I'll update the
branch once I can do this.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729698#comment-15729698
]
Hoss Man commented on SOLR-5944:
bq. ...looks like a genuine bug to me: ...
FWIW:
I made a half assed attempt to quickly reproduce this with {{bin/solr -e
schemaless}} and some curl commands to create fields and add docs -- and i
couldn't reproduce this.
It's possible I screwed something up with the commands, and or varried the
input slightly in a way that didn't tickle the bug; or it's possible that with
the default schemaless configs, some (default?) autocommit setting caued the
tlogs to get flushed/committed in a way that bypasses the bug.
In any case, I then attempted to more systematically demo the bug using the
same configs as the test -- and was able to easily reproduce...
{noformat}
mkdir -p /tmp/solr_test_home/cores/inplace_testing/conf
cp server/solr/solr.xml /tmp/solr_test_home/
touch /tmp/solr_test_home/cores/inplace_testing/core.properties
cp core/src/test-files/solr/collection1/conf/schema-inplace-updates.xml
/tmp/solr_test_home/cores/inplace_testing/conf/schema.xml
cp core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
/tmp/solr_test_home/cores/inplace_testing/conf/solrconfig.xml
cp
core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
/tmp/solr_test_home/cores/inplace_testing/conf/
# normally these are randomized by the test harness
# i just picked these values arbitrarily since it seemed to reproduce
regardless of seed
bin/solr start -f -s /tmp/solr_test_home/ -Dsolr.tests.maxBufferedDocs=1000
-Dsolr.tests.ramBufferSizeMB=100
-Dsolr.tests.mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler
-Dsolr.tests.useMergePolicy=false
-Dsolr.tests.mergePolicyFactory=org.apache.solr.index.TieredMergePolicyFactory
curl -X POST -H 'Content-Type: application/json' --data-binary '[
{ "id": "1","inplace_l_dvo": { "set": 555 } },
{ "id": "2","not_inplace_l": { "set": 555 } }
]'
'http://localhost:8983/solr/inplace_testing/update?commit=true=true'
curl -X POST -H 'Content-Type: application/json' --data-binary '[
{ "id": "1","regular_l": 666 },
{ "id": "2","regular_l": 666 }
]' 'http://localhost:8983/solr/inplace_testing/update'
curl -X POST -H 'Content-Type: application/json' --data-binary '[
{ "id": "1","inplace_l_dvo": { "inc": -77 } },
{ "id": "2","not_inplace_l": { "inc": -77 } }
]' 'http://localhost:8983/solr/inplace_testing/update'
curl
'http://localhost:8983/solr/inplace_testing/get?wt=json=true=1,2'
{noformat}
Final RTG Output...
{noformat}
{
"response":{"numFound":2,"start":0,"docs":[
{
"id":"1",
"inplace_l_dvo":478,
"_version_":1553084875040358400,
"regular_l":666},
{
"id":"2",
"regular_l":666,
"not_inplace_l":-77,
"_version_":155308487504242,
"copyfield1_src__both_updateable":0,
"inplace_updatable_float":0.0,
"copyfield2_src__only_src_updatable":0,
"inplace_updatable_int":0}]
}}
{noformat}
...note that unlike in the test where we only deal with a single document, I
used 2 docs here -- giving them identical updates -- to show that while the
problem reproduces when using a DVO field that gets inplace updates (doc id #1)
it doesn't reproduce when using a regular stored+indexed field that gets a
classic atomic update (doc id #2).
side note: i was confused by all those fields with a value of "0" in doc id#2,
and thought that might be somehow related to the bug -- ie: DVO fields getting
added with the default 0 in some code path -- but then i realized they all have
defaultValues in the schema.xml -- i'm not entirely sure why only doc#2 shows
them in this RTG, but doc#1 gets those field values as well once a commit
happens.
I suspect there is some code path in the regular atomic update logic where docs
read from the tlog get any default field values added to them, and that isn't
happening in the inplace code path case? ... not certain ... might be another
bug to look into?
perhaps the {{// Note: We don't need to add default fields ... would've
happened during the full indexing initially}} isn't necessarily true for
something that so far only exists in the tlog? ... need to thin about this some
more.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727716#comment-15727716
]
Hoss Man commented on SOLR-5944:
bq. I saw consistent failures on...
I'm seeing consistent failures from most of the randomized tests.
bq. I haven't looked deep into why it could be failing, but a preliminary look
at the logs lead me to believe that it is a test problem.
can you elaborate on what in the logs gave you that impression?
If it were a test bug -- ie: a bug in tracking the model state compared to the
inplace atomic updes -- I would expect the failures to reproduce if you
switched the test to use a regular (indexed+stored) long field instead of a DVO
field -- ie: use the classic atomic update code instead of the inplace update
code.
But when i tried toggling the field used (see comments in
{{checkRandomReplay}}) I couldn't reproduce any failures.
I added some hackish logging to {{checkRandomReplay}} to get it to dump a short
sequence that failed and turned that into a new test method
({{testReplay_nocommit}}) and then i distilled what seems to be the key
problematic bits into an even shorter test:
{{testReplay_SetOverriddenWithNoValueThenInc}} ...
{code}
public void testReplay_SetOverriddenWithNoValueThenInc() throws Exception {
final String inplaceField = "inplace_l_dvo";
// final String inplaceField = "inplace_nocommit_not_really_l"; //
nocommit: "inplace_l_dvo"
checkReplay(inplaceField,
//
sdoc("id", "1", inplaceField, map("set", 555L)),
SOFTCOMMIT,
sdoc("id", "1", "regular_l", 666L), // NOTE: no inplaceField,
regular add w/overwrite
sdoc("id", "1", inplaceField, map("inc", -77)),
HARDCOMMIT);
}
{code}
...all of that is now on the branch.
If you toggle the above code to use regular atomic updates, then it passes --
but as written, so it uses the new inplace update code paths, it fails like
so...
{noformat}
[junit4] FAILURE 0.54s |
TestInPlaceUpdatesStandalone.testReplay_SetOverriddenWithNoValueThenInc <<<
[junit4]> Throwable #1: java.lang.AssertionError: expected:<-77> but
was:<478>
[junit4]>at
__randomizedtesting.SeedInfo.seed([9D6E895FCBA28315:6DDD2091B324AFF2]:0)
[junit4]>at
org.apache.solr.update.TestInPlaceUpdatesStandalone.checkReplay(TestInPlaceUpdatesStandalone.java:920)
[junit4]>at
org.apache.solr.update.TestInPlaceUpdatesStandalone.testReplay_SetOverriddenWithNoValueThenInc(TestInPlaceUpdatesStandalone.java:590)
{noformat}
...looks like a genuine bug to me: when a regular update overwrites a doc that
had a DVO field value, a subsequent "inc" operation on the DVO fields is
picking up the _old_ value instead of operating against an implicit default of
0.
(This kind of corner case is what makes randomized testing totally worth the
time and effort)
bq. Btw, do you know how to enable commit notifications to show up here for the
jira/solr-5944 branch?
IIRC comments about commits to jira/* branches are suppressed intentionally as
noise, because it's expected that there will be lots of iteration on the
branches, some of which might be thrown away, and for posterity what matters is
only commits to main line dev branches
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
>
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727635#comment-15727635 ] Ishan Chattopadhyaya commented on SOLR-5944: [~hossman], I've created a branch (jira/solr-5944) for this, and committed your latest patch to the branch. Also, I've fixed the ClassCastException bug that you saw and also fixed a javadoc bug and removed unused imports. (https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8c7a2a6). However, I saw consistent failures on TestInPlaceUpdatesStandalone#testReplay_Random_FewDocsManyShortSequences which you recently added. I haven't looked deep into why it could be failing, but a preliminary look at the logs lead me to believe that it is a test problem. Btw, do you know how to enable commit notifications to show up here for the jira/solr-5944 branch? > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664306#comment-15664306 ] Ishan Chattopadhyaya commented on SOLR-5944: Replied to review comments inline here: https://issues.apache.org/jira/browse/SOLR-5944?focusedCommentId=15547986=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15547986 > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15617617#comment-15617617
]
Ishan Chattopadhyaya commented on SOLR-5944:
Just discovered another, more common, problem with reordered DBQs and in-place
updates working together. The earlier discussed problem, of resurrecting a
document, is very similar. So, here's a description of both:
SCENARIO 1:
{code}
Imagine, updates on leader are:
ADD (id=1, updateable_field=1, title="mydoc1", version=100)
INP-UPD (id=1, updateable_field=2, version=200, prevVersion=100)
DBQ (q="updateable_field:1", version=300)
The same on the replica (forwarded):
ADD (id=1, updateable_field=1, title="mydoc1", version=100)
DBQ (q="updateable_field:1", version=300)
INP-UPD (id=1, updateable_field=2, version=200, prevVersion=100)
The expected net effect is that no document is deleted, and id=1 document
exists with updateable_field=2.
Here, the DBQ was reordered. When they are executed on the replica, the
version=200 update cannot be applied since there is no document with
(id=1,prevVersion=100). What is required is a resurrection of the document that
was deleted by the DBQ, so that other stored/indexed fields are not lost.
{code}
SCENARIO 2:
{code}
Imagine, updates on leader are:
ADD (id=1, updateable_field=1, title="mydoc1", version=100)
INP-UPD (id=1, updateable_field=2, version=200, prevVersion=100)
DBQ (q="id:1", version=300)
The same on the replica (forwarded):
ADD (id=1, updateable_field=1, title="mydoc1", version=100)
DBQ (q="id:1", version=300)
INP-UPD (id=1, updateable_field=2, version=200, prevVersion=100)
The expected net effect is that the document with id=1 be deleted. But again,
the DBQ is reordered. When executed on replica, update version=200 cannot be
applied, since the id=1 document has been deleted. What is required is for this
update (version=200) to be dropped silently.
{code}
Scenario 1 is rare, scenario 2 would be more common. At the point when the
inplace update (version=200 in both cases) is applied, the replica has no way
to know if the update requires a resurrection of the document, or requires to
be dropped.
Till now, I hadn't considered scenario 2, but for the rare scenario 1, I
resorted to throwing an error so as to throw the replica in LIR. Clearly, in
view of scenario 2, this looks like a bad idea. Here are two potential
solutions that come to mind:
Solution 1:
{code}
In a replica, while applying an in-place update, if the required prevVersion
update cannot be found in tlog or index (due to these reordered DBQs), then
fetch from the leader an update that contains the full document with the
version (for which the update failed at replica). Downside to this approach is
that unstored/non-dv fields will get dropped (as is the case with regular
atomic updates today).
{code}
Solution 2:
{code}
Ensure that DBQs are never reordered from leader -> replica. One approach can
be SOLR-8148. Another could be to block, on the leader, all updates newer than
a DBQ, that has been sent through a different thread, until the DBQ is
processed on leader and all the replicas, and only then process the other
updates.
{code}
Solution 1 seems easier to implement now than solution 2, but solution 2 (if
implemented correctly) seems cleaner. Any thoughts?
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz,
> hoss.62D328FA1DEA57FD.fail.txt,
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582054#comment-15582054 ] Yonik Seeley commented on SOLR-5944: It doesn't seem to buy us much to deprecate that, and it would impose the burden on users to changing in order to upgrade (making it less likely that a user of EFF would upgrade). > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582028#comment-15582028 ] Erick Erickson commented on SOLR-5944: -- Deprecate ExternalFileField when this is committed? > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477688#comment-15477688
]
Ishan Chattopadhyaya commented on SOLR-5944:
Here's more on the problem with the re-ordered DBQs that Shalin pointed out.
When a document requiring in-place update cannot be "resurrected" when the
original full indexed document has been deleted by an out of order DBQ. As per
this patch, expected behaviour in this case is to throw the replica into LIR
(this situation itself is rare). Here's an example of the situation:
{code}
ADD(id=x, val=5, ver=1)
UPD(id=x, val=10, ver = 2)
DBQ(q=val:10, v=4)
DV(id=x, val=5, ver=3)
{code}
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417210#comment-15417210 ] Ishan Chattopadhyaya commented on SOLR-5944: bq. The easiest correctness workaround would perhaps just be to force open a realtime searcher before the DBQ as well. This seems to be working correctly for the few test cases I've tried out. Seems like this is the easiest workaround for LUCENE-7344 for now. I'll include this fix, and the tests in my next patch. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415981#comment-15415981 ] Hoss Man commented on SOLR-5944: bq. I'd still suggest that instead of introducing yet another delay mechanism via the DebugFilter in JettySolrRunner, we should use the fault injection facilities that we already have in Solr. I'm not sure how that would really work via TestInjection -- what Ishan's patch currently adds to DebugFilter allows for targeting specific requests (by sequence in the future) _on specific nodes_ with delays. The way TestInjection is designed is to be used at very low levels in the (production) code as asserts and completely randomized based on static "odds" that test methods can set. I have no idea how you would tell TestInjection "The HTTP request should be stalled for 10 seconds" and have that only apply to a _single_ specific node. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415326#comment-15415326
]
Yonik Seeley commented on SOLR-5944:
bq. Document that when inplace updates are configured/requested, hard commits
may happen
s/hard/soft commits?
Thinking about solr-cloud mode only (and the DBQ issue w/ updates that don't
change the internal doc id): solr already blocks updates, issues the DBQ, and
then forces open a new realtime searcher.
The easiest correctness workaround would perhaps just be to force open a
realtime searcher before the DBQ as well.
Basically, in DUH2.deleteByQuery():
{code}
synchronized (solrCoreState.getUpdateLock()) {
ulog.openRealtimeSearcher();
{code}
If Lucene could tell us if there were any unresolved updates (of the variety
that change the doc w/o changing it's internal docid), then we could
conditionally call that.
Lucene throwing an exception may be a non-starter... at the point in time that
a DBQ is issued, there may not be any numeric updates?
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415219#comment-15415219 ] Shalin Shekhar Mangar commented on SOLR-5944: - I hadn't had a chance to review the patch after Ishan incorporated my last few review comments. So I took a look again. # Hoss's suggestion of `bucket.wait(waitTimeout.timeLeft(TimeUnit.MILLISECONDS));` instead of `wait(5000)` is much better and should be implemented. # Under no circumstances should we we calling `forceUpdateCollection` in the indexing code path. It is just too dangerous in the face of high indexing rates. Instead we should do what the DUP is already doing i.e. calling getLeaderRetry to figure out the current leader. If the current replica is partitioned from the leader then we have other mechanisms for taking care of it and the replica has no business trying to determine this. # I'd still suggest that instead of introducing yet another delay mechanism via the DebugFilter in JettySolrRunner, we should use the fault injection facilities that we already have in Solr. # The executor service in the new tests should be shutdown in a finally clause. You can use ExecutorUtil.shutdownAndAwaitTermination for that. Hoss has given some excellent comments on the new tests that you added. Please incorporate them into your next patch. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414049#comment-15414049
]
Hoss Man commented on SOLR-5944:
It seems like solutions for dealing with LUCENE-7344 in Solr could fall into 3
broad categories:
{panel:title=IndexWriter changes}
These are solutions that might involve changes to IndexWriter
* The ideal solution would be to "fix" LUCENE-7344 at the IndexWriter level, so
that it magically does hte correct thing -- but I certainly have no idea how to
do that, and McCandless didn't think it was viable. So this is an unlikely
solution unless/untill Shai has more time to work through his ideas.
* An improvement on the overall situation might be if
{{IndexWriter.deleteDocuments}} could at least *detect* when the Query involves
a DocValues field which has non-committed updates, and in that case throw a
distinct Exception (ie: {{CanNotDeleteOnUpdatedDocValues extends
InvalidStateException}}
** Solr could then, at a minimum, propogate that exception up to the user, and
documentation could make it clera that if you want to mix atomic updates of
docvalue with DBQs on those docvalues, you need to commit or you will get this
exception
{panel}
{panel:title=Workaround's in Solr: Invisible To Users}
This would be any type of solution that let's us maintain the current approach
in the existing patches of making docvalue updates be a complete implementation
detail that happens automatically behind the scenes when the field types &
updates support it.
* One possibility would be to track somewhere (at the solr level) if/when
inplace updates have been used since the last commit, and do an implicit commit
if/when a DBQ comes in.
** to prevent excessive unexpected commits, we could completley disable all
inplace updates unless autocommit was enabled, since that should be the common
case for most "real time" applications of solr anywhere, where inplace updates
will be the most helpful
* The previous idea could be combined with the idea of
{{IndexWriter.deleteDocuments}} throwing a {{CanNotDeleteOnUpdatedDocValues}}
exception;
** if/when Solr encounters a {{CanNotDeleteOnUpdatedDocValues}}, we could do an
implicit commit and retry the DBQ
{panel}
{panel:title=Workaround's in Solr: involving API Changes/Additions to current
patch}
These are solutions that would involved explicit changes to the options offered
to users so they can explicitly indicate "I want an inplace update" with the
understanding that tha may prevent DBQs from working properly
* As ishan suggested, one possibility would be to require fields have explicit
{{inplaceUpdates="true"}} configuration in schema.xml to indicate that they may
be updated inplace
** at a later date, we could always increase the default schema {{version}} and
change the default value of this new field property to automatically be true
based on the {{docValues}} and other field attributes
** the biggest downside I see to this approach is that it doesn't really do
anything to prevent the underlying problem:
*** it doesn't help "enforce" that DBQs won't be used against these fields, and
it doesn't give the user any clear indication of a problem if they try --
Solr/Lucene will continue to silently delete the wrong docs.
* Alternatively, we could make this notion of requiring that inplaceupdates be
explicitly requested be something that can be specified on a per request basis,
along the lines of...{code}
{ "id": "12345",
"price": { "inc": 42,
"inplace": true }}
{code}
** again, the default could be "false" and at a later date we could always
change the default to be based on the {{docValues}} and other field attributes
** this approach seems _slightly_ better to me then a schema setting because:
*** the {{"inplace"}} directive is front and center for clients sending
updates, so they are more likeley to be aware of the caveats around using it
with DBQ (even if they haven't read the schema first hand)
*** clients can mix and match inplace atomic updates with "regular" atomic
updates on the same field (or diff fields definied by the same
{{}}, if/when they do/don't care about being able to reliably do
a DBQ on that field.
** But ultimately this still has the same scary behavior: Solr can silently
delete teh wrong docs w/o immediate/obvious reason why if the user doing the
delete doesn't know/care if/when/why a doc may have had an inplace atomic
update.
{panel}
My current feeling is that a hybrid solution may be the best bet:
* Require users to explicitly configure/request "inplace" updates in some way
* Document that when inplace updates are configured/requested, hard commits may
happen implicitly if/when a DBQ is executed to ensure correct documents are
deleted
* Only do the hard commit if neccessary because IndexWriter threw
{{CanNotDeleteOnUpdatedDocValues}} the first time we tried the DBQ,
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413974#comment-15413974 ] Shai Erera commented on SOLR-5944: -- [~ichattopadhyaya], I've continued the discussion in LUCENE-7344. I am looking into fixing the bug, though it's a hairy one... > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412124#comment-15412124
]
Ishan Chattopadhyaya commented on SOLR-5944:
Regarding LUCENE-7344,
{quote}
We still need some sort of solution to this problem – the suggested
requirement/documentation workarround suggested by McCandless in that issue
doesn't really fit with how all work on this jira to date has kept the the
decision of when/if to do an "in-place" update a low level implementation
detail that would have to radically change if we wanted to "pass the buck"
up to the user to say "you an't use DBQ on a docvalue field that you also use
for in place updates"
so what's our Solr solution / workaround? do we have any?
{quote}
I tried to look for a workaround in Solr, but couldn't figure out a viable
workaround. Given that we're looking to have in-place updates replace the
current atomic updates mechanism transparently when certain fields types are
involved (i.e. non-stored, non-indexed, single valued docvalues), LUCENE-7344
is a blocker for that: i.e. if a user updates a dv field, and issues a DBQ
trying to use that updated value without a commit in between, then the DBQ
doesn't actually use the updated value, but uses the older (last committed)
value.
One approach is to not have in-place updates be a transparent replacement to
atomic updates and be enabled or disabled (by default) on certain fields. By
doing this, we can document that enabling in-place updates has the downside
that DBQs on the updated values wouldn't work unless there's an explicit or
auto commit between the update and the dbq.
[~shaie] Would love to hear your thoughts on this, and your suggestions on how
should we go about dealing with LUCENE-7344.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412122#comment-15412122 ] Ishan Chattopadhyaya commented on SOLR-5944: bq. the secondary fine-grained lock is the per-bucket lock. The full updates use that . So,it should take the lock on both to mitigate the concurrency issues. Eventually, when Lucene exposes DocumentsWriter.updateDocValues(DocValuesUpdate... updates) we probably should be able to get rid of the global lock? Indeed, I realized that we can use IndexWriter#updateDocValues(Term term, Field[] updates) instead of trying to lock at Solr. I shall include that fix in the next patch. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409383#comment-15409383
]
Noble Paul commented on SOLR-5944:
--
bq.Do we need to either wrap all writer.updateDocument calls in the updateLock,
or use some sort of secondary fineer grained "per-uniqueKey" locking as well to
prevent bad stuff from happening in sequences like this?
the secondary fine-grained lock is the per-bucket lock. The full updates use
that . So,it should take the lock on both to mitigate the concurrency issues.
Eventually, when Lucene exposes
{{DocumentsWriter.updateDocValues(DocValuesUpdate... updates)}} we probably
should be able to get rid of the global lock?
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399807#comment-15399807
]
Hoss Man commented on SOLR-5944:
Ok -- it took a while, but here's my notes after reviewing the latest patch
{panel:title=DistributedUpdateProcessor}
* waitForDependentUpdates
** I know you & shalin went back and forth a bit on the wait call (ie:
wait(100) with max retries vs wait(5000)) but i think the way things settled
out {{bucket.wait(waitTimeout.timeLeft(TimeUnit.MILLISECONDS));}} would be
better then a generic {{wait(5000)}}
*** consider the scenerio where: the dependent update is never going to come; a
spurious notify/wake happens during the first "wait" call @ 4950ms; the
lookupVersion call takes 45ms. Now we've only got 5ms left on our original
TimeOut, but we _could_ wind up "wait"ing another full 5s (total of 10s) unless
we get another spurrious notify/wake inthe mean time.
** {{log.info("Fetched the update: " + missingUpdate);}} that's a really good
candidate for templating since the AddUpdateCommand.toString() could be
expensive if log.info winds up being a no-op (ie: {{log.info("Fetched the
update: \{\}", missingUpdate);}})
* fetchMissingUpdateFromLeader
** In response to a previous question you said...{quote}
[FIXED. Initially, I wanted to fetch all missing updates, i.e. from what we
have till what we want. Noble suggested that fetching only one at a time makes
more sense.]
{quote} ... but from what i can tell skimming RTGC.processGetUpdates() it's
still possible that multiple updates will be returned, notably in the case
where: {{// Must return all delete-by-query commands that occur after the first
add requested}}. How is that possibility handled in the code paths that use
fetchMissingUpdateFromLeader?
*** that seems like a scenerio that would be really easy to test for -- similar
to how outOfOrderDeleteUpdatesIndividualReplicaTest works
** {{assert ((List) missingUpdates).size() == 1: "More than 1 update ...}}
*** based on my skimming of the code, an empty list is just as possible, so the
assertion is missleading (ideally it should say how many updates it got, or
maybe toString() the whole List ?)
{panel}
{panel:title=AtomicUpdateDocumentMerger}
* isSupportedFieldForInPlaceUpdate
** javadocs
* getFieldNamesFromIndex
** javadocs
** method name seems VERY missleading considering what it does
* isInPlaceUpdate
** javadocs should be clear what hapens to inPlaceUpdatedFields if result is
false (even if answer is "undefined"
** based on usage, wouldn't it be simplier if instead of returning a boolean,
this method just returned a (new) Set of inplace update fields found, and if
the set is empty that means it's not an in place update?
** isn't getFieldNamesFromIndex kind of an expensive method to call on every
AddUpdateCommand ?
*** couldn't this list of fields be created by the caller and re-used at least
for the entire request (ie: when adding multiple docs) ?
** {{if (indexFields.contains(fieldName) == false &&
schema.isDynamicField(fieldName))}}
*** why does it matter one way or the other if it's a dynamicField?
** the special {{DELETED}} sentinal value still isn't being checked against the
return value of {{getInputDocumentFromTlog}}
** this method still seems like it could/should do "cheaper" validation (ie:
not requiring SchemaField object creation, or tlog lookups) first. (Ex: the
set of supported atomic ops are checked after isSupportedFieldForInPlaceUpdate
& a possible read from the tlog).
*** My suggested rewrite would be something like...{code}
Set candidateResults = new HashSet<>();
// first pass, check the things that are virtually free,
// and bail out early if anything is obviously not a valid in-place update
for (String fieldName : sdoc.getFieldNames()) {
if (schema.getUniqueKeyField().getName().equals(fieldName)
|| fieldName.equals(DistributedUpdateProcessor.VERSION_FIELD)) {
continue;
}
Object fieldValue = sdoc.getField(fieldName).getValue();
if (! (fieldValue instanceof Map) ) {
// not even an atomic update, definitely not an in-place update
return Collections.emptySet();
}
// else it's a atomic update map...
for (String op : ((Map)fieldValue).keySet()) {
if (!op.equals("set") && !op.equals("inc")) {
// not a supported in-place update op
return Collections.emptySet();
}
}
candidateResults.add(fieldName);
}
if (candidateResults.isEmpty()) {
return Collections.emptySet();
}
// now do the more expensive checks...
Set indexFields = getFieldNamesFromIndex(cmd.getReq().getCore());
SolrInputDocument rtDoc = null; // will lazy read from tlog if needed
for (String fieldName : candidateResults) {
SchemaField schemaField = schema.getField(fieldName);
// TODO: check isSupportedFieldForInPlaceUpdate
// TODO: check copyfields
// TODO: check indexFields, if not there...
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385215#comment-15385215
]
Shalin Shekhar Mangar commented on SOLR-5944:
-
bq. Btw, I've surrounded only the lookupVersion() calls with the acquiring and
releasing of the lock, instead of surrounding the entire wait loop with the
acquiring/releasing of the lock: I reasoned that while we are waiting in that
wait loop, other threads need to have indexed the update that we're waiting on,
and hence I released the lock as soon as it was not needed, only to re-acquire
it after 100ms. Does that sound like a valid reason?
The read lock is for safe publication of fields in update log and it is
acquired by indexing threads who only want to read stuff from update log. Also
read locks can be held by multiple readers. Therefore, acquiring this lock does
not prevent other threads from indexing.
Also, please be very careful when changing the order of acquiring locks because
it can result in deadlocks. It is good practice to do them in the same sequence
as everywhere else in the code. So synchronizing on bucket before
vinfo.lockForUpdate for a small optimization doesn't seem worthwhile to me.
bq. Since this method enters the wait loop for every in-place update that has
arrived out of order at a replica (an event, that I think is frequent under
multithreaded load), I don't want every such update to be waiting for the full
timeout period (5s here), but instead check back from time to time. In most of
the cases, the dependent update would've been written (by another thread)
within the first 100ms, after which we can bail out. Do you think that makes
sense?
You misunderstand. A wait(5000) does not mean that you are waiting the full 5
seconds. Any notifyAll() will wake up the waiting thread and when it does, it
will check the lastFoundVersion and proceed accordingly. In practice wait(100)
may not be so bad but if an update doesn't arrive for more than 100ms the
thread will wake up and lookup the version needlessly with your current patch.
A few more comments:
# In your latest patch, acquiring the read lock to call versionAdd is not
necessary -- it will do that anyway. You can re-acquire it for reading the
version after the method call returns.
# I don't think the case of {{vinfo.lookupVersion}} returning a negative value
(for deletes) is handled here at all.
# What happens if the document has been deleted already (due to reordering on
the replica) when you enter waitForDependentUpdates? i.e. what if re-ordering
leads to new_doc (v1) -> del_doc (v3) -> dv_update (v2) on the replica?
# Similarly, how do we handle the case when the doc has been deleted on the
leader when you execute fetchMissingUpdateFromLeader. Does RTG return the
requested version even if the doc has been deleted already? I suspect it does
but be nice to confirm.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384773#comment-15384773
]
Shalin Shekhar Mangar commented on SOLR-5944:
-
Thanks Ishan. This is great progress since the last time I reviewed this patch.
I've only skimmed the latest patch but in particular I find a few problems in
DistributedUpdateProcessor#waitForDependentUpdates:
# This method doesn't look correct. All places which call vinfo.lookupVersion()
do it after acquiring a read lock by calling vinfo.lockForUpdate(). Looking at
the code for vinfo.lookupVersion() it accesses the map and prevMap in updateLog
which can be modified by a writer thread while waitForDependentUpdates is
reading their values. So first of all we need to ensure that we acquire and
release the read lock. Acquiring this lock and then waiting on a different
object (the "bucket") will not cause a deadlock condition because it is a read
lock (which can be held by multiple threads).
# Secondly, this method can be made more efficient. It currently wakes up every
100ms and reads the new "lastFoundVersion" from the update log or index. This
is wasteful. A better way would be to wait for the timeout period directly
before calling {{vinfo.lookupVersion()}} inside the synchronized block.
# Similar to #1 -- calling {{vinfo.lookupVersion()}} after
{{fetchMissingUpdateFromLeader}} should be done after acquiring a read lock.
# There is no reason to synchronize on bucket when calling the {{versionAdd}}
method again because it will acquire the monitor anyway.
# DistributedUpdateProcessor#waitForDependentUpdates uses wrong javadoc tag
'@returns' instead of '@return'
# The debug log message should be moved out of the loop instead of introducing
a debugMessagePrinted boolean flag
# Use the org.apache.solr.util.TimeOut class for timed wait loops
# Method can be made private
I've attempted to write a better wait-loop here (warning: not tested):
{code}
long prev = cmd.prevVersion;
long lastFoundVersion = 0;
TimeOut timeOut = new TimeOut(5, TimeUnit.SECONDS);
vinfo.lockForUpdate();
try {
synchronized (bucket) {
lastFoundVersion = vinfo.lookupVersion(cmd.getIndexedId());
while (lastFoundVersion < prev && !timeOut.hasTimedOut()) {
if (log.isDebugEnabled()) {
log.debug("Re-ordered inplace update. version=" + (cmd.getVersion()
== 0 ? versionOnUpdate : cmd.getVersion()) +
", prevVersion=" + prev + ", lastVersion=" + lastFoundVersion +
", replayOrPeerSync=" + isReplayOrPeersync);
}
try {
bucket.wait(5000);
} catch (InterruptedException ie) {
throw new RuntimeException(ie);
}
lastFoundVersion = vinfo.lookupVersion(cmd.getIndexedId());
}
}
} finally {
vinfo.unlockForUpdate();
}
// check lastFoundVersion against prev again and handle all conditions
{code}
However I think that since the read lock and bucket monitor has to be acquired
by this method anyway, it might be a good idea to just call it from inside
versionAdd after acquiring those monitors. Then this method can focus on just
waiting for dependent updates and nothing else.
A random comment on the changes made to DebugFilter: The setDelay mechanism
introduced here may be a good candidate for Mark's new
TestInjection#injectUpdateRandomPause?
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368262#comment-15368262
]
Ishan Chattopadhyaya commented on SOLR-5944:
{quote}
But there's a fundemental difference betwen params like commitWithin and
overwrite and the new prevVersion param...
commitWithin and overwrite are client specified options specific to the
xml/javabin update format(s). The fact that they can be specified as request
params is an implementation detail of the xml/javabin formats that they happen
to have in common, but are not exclusively specifyied as params – for example
the XMLLoader only uses the params as defaults, they can be psecified on a per
basis.
The new prevVersion param however is an implementation detail of DUP ... DUP is
the only code that should have to know/care that prevVersion comes from a
request param.
{quote}
Sure, it makes sense. I'll fix it.
bq. We should have a comment to these affects (literally we could just paste
that text directly into a comment) when declaring the prevPointer variable in
this method.
I had put this comment there:
{code}
@return If cmd is an in-place update, then returns the pointer (in the tlog) of
the previous update that the given update depends on. Returns -1 if this is not
an in-place update, or if we can't find a previous entry in the tlog.
{code}
But now I have updated it to make it even more detailed:
{code}
@return If cmd is an in-place update, then returns the pointer (in the tlog) of
the previous update that the given update depends on.
*Returns -1 if this is not an in-place update, or if we can't find a
previous entry in the tlog. Upon receiving a -1, it
*should be clear why it was -1: if the command's
flags|UpdateLog.UPDATE_INPLACE is set, then this
*command is an in-place update whose previous update is in the index
and not in the tlog; if that flag is not set, it is not an in-place
*update at all, and don't bother about the prevPointer value at all
(which is -1 as a dummy value).)
{code}
{quote}
Hmm... that makes me wonder – we should make sure we have a test case of doing
atomic updates on numeric dv fields which have copyfields to other numeric
fields. ie: lets make sure our "is this a candidate for inplace updates" takes
into acount that the value being updated might need copied to another field.
(in theory if both the source & dest of the copy field are single valued dv
only then we can still do the in place updated as long as the copyField
happens, but even if we don't have that extra bit of logic we need a test that
the udpates are happening consistently)
{quote}
Sure, I'll add such a test. The latest patch incorporates the behaviour you
suggested: if any of the copy field targets is not a in-place updateable field,
then the entire operation is not an in-place update (but a traditional atomic
update instead). But, if copy field targets of an updated field is also
supported for an updateable dv, then it is updated as well.
{quote}
Hmmm... is that really the relevant question though?
I'm not sure how the existing (non-inplace) atomic update code behaves if you
try to "inc" a date, but why does it matter for the isSupportedForInPlaceUpdate
method?
if date "inc" is supported in the existing atomic update code, then
whatever that code path looks like (to compute the new value) it should be the
same for the new inplace update code.
if date "inc" is not supported in the existing atomic update code, then
whatever the error is should be the same in the new inplace update code
Either way, I don't see why isSupportedForInPlaceUpdate should care – or if it
is going to care, then it should care about the details (ie: return false for
(dv only) date field w/ "inc", but true for (dv only) date field with "set")
{quote}
For now I've removed date field totally out of scope of this patch. If there is
a update to date that is needed, it falls back to traditional atomic update. I
can try to deal with the trie date field, if you suggest.
bq. let's put those details in a comment where this Exception is thrown ... or
better yet, try to incorporate it into the Exception msg?
I had put this exception in the patch: {{Unable to resolve the last full doc in
tlog fully, and document not found in index even after opening new rt searcher.
}} but now I'll chang it to: {{Unable to resolve the last full doc in tlog
fully, and document not found in index even after opening new rt searcher. If
the doc was deleted, then there shouldn't have been an attempt to resolve to a
previous document by that id.}}
bq. Ah, ok ... good point – can we go ahead and add some javadocs to that
method as well making that clear?
Sure, I'll update the javadocs for that existing method as well.
> Support updates of numeric DocValues
>
>
>
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368165#comment-15368165
]
Hoss Man commented on SOLR-5944:
I've not had a chance to look at the latest patch, but here's some comment
responses...
bq. Since commitWithin and overwrite was being set here, I thought this is an
appropriate place to set the prevVersion to the cmd
But there's a fundemental difference betwen params like {{commitWithin}} and
{{overwrite}} and the new {{prevVersion}} param...
{{commitWithin}} and {{overwrite}} are _client_ specified options specific to
the xml/javabin update format(s). The fact that they can be specified as
request params is an implementation detail of the xml/javabin formats that they
happen to have in common, but are not exclusively specifyied as params -- for
example the XMLLoader only uses the params as defaults, they can be psecified
on a per {{}} basis.
The new {{prevVersion}} param however is an implementation detail of DUP ...
DUP is the _only_ code that should have to know/care that {{prevVersion}} comes
from a request param.
bq. Yes, this was intentional, and I think it doesn't make any difference. If
an "id" isn't found in any of these maps, it would mean that the previous
update was committed and should be looked up in the index.
bq. I think we don't need to worry. Upon receiving a prevPointer=-1 by whoever
reads this LogPtr, it should be clear why it was -1: if the command's
{{flags|UpdateLog.UPDATE_INPLACE}} is set, then this command is an in-place
update whose previous update is in the index and not in the tlog; if that flag
is not set, it is not an in-place update at all, and don't bother about the
prevPointer value at all (which is -1 as a dummy value).
We should have a comment to these affects (literally we could just paste that
text directly into a comment) when declaring the prevPointer variable in this
method.
bq. ... This was needed because the lucene document that was originally being
returned had copy fields targets of id field, default fields, multiple Field
per field (due to FieldType.createFields()) etc., which are not needed for
in-place updates.
Hmm... that makes me wonder -- we should make sure we have a test case of doing
atomic updates on numeric dv fields which have copyfields to other numeric
fields. ie: lets make sure our "is this a candidate for inplace updates" takes
into acount that the value being updated might need copied to another field.
(in theory if both the source & dest of the copy field are single valued dv
only then we can still do the in place updated as long as the copyField
happens, but even if we don't have that extra bit of logic we need a test that
the udpates are happening consistently)
bq. I wasn't sure how to deal with inc for dates, so left dates out of this
for simplicity for now
Hmmm... is that really the relevant question though?
I'm not sure how the existing (non-inplace) atomic update code behaves if you
try to "inc" a date, but why does it matter for the
{{isSupportedForInPlaceUpdate}} method?
* if date "inc" is supported in the existing atomic update code, then whatever
that code path looks like (to compute the new value) it should be the same for
the new inplace update code.
* if date "inc" is _not_ supported in the existing atomic update code, then
whatever the error is should be the same in the new inplace update code
Either way, I don't see why {{isSupportedForInPlaceUpdate}} should care -- or
if it is going to care, then it should care about the details (ie: return false
for (dv only) date field w/ "inc", but true for (dv only) date field with "set")
bq. I think this is fatal, since if the doc was deleted, then there shouldn't
have been an attempt to resolve to a previous document by that id. I think this
should never be triggered.
let's put those details in a comment where this Exception is thrown ... or
better yet, try to incorporate it into the Exception msg?
bq. I'm inclined to keep it to Long/null instead of long/-1, since
versionInfo.getVersionFromIndex() is also Long/null
Ah, ok ... good point -- can we go ahead and add some javadocs to that method
as well making that clear?
bq. ... I've changed this to now use the UpdateShardHandler's httpClient.
Ok, cool ... Yeah, that probably makes more sense in general.
bq. Not sure what needs to be done more here
Yeah, sorry -- that was a vague comment that even i don't know what i ment by,
was probably ment to be part of the note about the switch statemenet default.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
>
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337711#comment-15337711 ] Ishan Chattopadhyaya commented on SOLR-5944: As I was incorporating Hoss' suggestions, I wrote a test for DV updates with DBQ on updated values. This was failing if there was no commit between the update and the DBQ. I think this is due to LUCENE-7344. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330923#comment-15330923
]
Hoss Man commented on SOLR-5944:
I know Ishan has been working on improving the tests based on my last batch of
feedback -- since then I've been reviewing the _non test_ changes in the
lastest patch.
Here's my notes about specific class/methods as I reviewed them in
individually
{panel:title=JettySolrRunner}
* javadocs, javadocs, javadocs
{panel}
{panel:title=XMLLoader + JavabinLoader}
* why is this param checks logic duplicated in these classes?
* why not put this in DUP (which already has access to the request params) when
it's doing it's "FROMLEADER" logic?
{panel}
{panel:title=AddUpdateCommand}
* these variables (like all variables) should have javadocs explaining what
they are and what they mean
** people skimming a class shouldn't have to grep the code for a variable name
to understand it's purpose
* having 2 variables here seems like it might be error prone? what does it
mean if {{prevVersion < 0 && isInPlaceUpdate == true}} ? or {{0 < prevVersion
&& isInPlaceUpdate == false}} ?
** would it make more sense to use a single {{long prevVersion}} variable and
have a {{public boolean isInPlaceUpdate()}} that simply does {{return (0 <
prevVersion); }} ?
{panel}
{panel:title=TransactionLog}
* javadocs for both the new {{write}} method and the existig {{write}} method
** explain what "prevPointer" means and note in the 2 arg method what the
effective default "prevPoint" is.
* we should really have some "int" constants for refering to the List indexes
involved in these records, so instead of code like {{entry.get(3)}} sprinkled
in various classes like UpdateLog and PeerSync it can be smething more readable
like {{entry.get(PREV_VERSION_IDX)}}
{panel}
{panel:title=UpdateLog}
* javadocs for both the new {{LogPtr}} constructure and the existing constructor
** explain what "prevPointer" means and note in the 2 arg constructure what the
effective default "prevPoint" is.
* {{add(AddUpdateCommand, boolean)}}
** this new code for doing lookups in {{map}}, {{prevMap}} and {{preMap2}}
seems weird to me (but admitedly i'm not really an expert on UpdateLog in
general and how these maps are used
** what primarily concerns me is what the expected behavior is if the "id"
isn't found in any of these maps -- it looks like prevPointer defaults to "-1"
regardless of whether this is an inplace update ... is that intentional? ... is
it possible there are older records we will miss and need to flag that?
** ie: do we need to worry about distinguising here between "not an in place
update, therefore prePointer=-1" vs "is an in place update, but we can't find
the prevPointer" ??
** assuming this code is correct, it might be a little easier to read if it
were refactored into something like:{code}
// nocommit: jdocs
private synchronized long getPrevPointerForUpdate(AddUpdateCommand cmd) {
// note: sync required to ensure maps aren't changed out form under us
if (cmd.isInPlaceUpdate) {
BytesRef indexedId = cmd.getIndexedId();
for (Map currentMap : Arrays.asList(map, prevMap,
prevMap2)) {
LogPtr prevEntry = currentMap.get(indexedId);
if (null != prevEntry) {
return prevEntry.pointer;
}
}
}
return -1; // default when not inplace, or if we can't find a previous entry
}
{code}
* {{applyPartialUpdates}}
** it seems like this method would be a really good candidate for some direct
unit testing?
*** ie: construct a synthetic UpdateLog, and confirm applyPartialUpdates does
the right thing
** the sync block in this method, and how the resulting {{lookupLogs}} list is
used subsequently, doesn't seem safe to me -- particularly the way
{{getEntryFromTLog}} calls incref/decref on each TransactionLog as it loops
over that list...
*** what prevents some other thread from decref'ing one of these TransactionLog
objects (and possibly auto-closing it) in between the sync block and the incref
in getEntryFromTLog?
(most existing usages of TransactionLog.incref() seem to be in blocks that
sync on the UpdateLog -- and the ones that aren't in sync blocks look sketchy
to me as well)
*** in general i'm wondering if {{lookupLogs}} should be created outside of the
while loop, so that there is a consistent set of "logs" for the duration of the
method call ... what happens right now if some other thread changes
tlog/prevMapLog/prevMapLog2 in between iterations of the while loop?
** shouldn't we make some sanity check assertions about the results of
getEntryFromTLog? -- there's an INVALID_STATE if it's not an ADD or a list of 5
elements, but what about actually asserting that it's either an ADD or an
UPDATE_INPLACE? ... what about asserting the doc's uniqueKey matches?
*** (because unless i'm missing something, it's possible for 2 docs to have the
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323805#comment-15323805 ] Hoss Man commented on SOLR-5944: bq. I had a question come up today that I wanted to ask for posterity. What gets returned between the time we update one of these and a commit occurs? The old value or the new one? I'd assumed the old one but wanted to be sure. in theory, it's exactly identical to the existing behavior of an atomic update: searching returns only the committed values, an RTG will return the "new" (uncommitted) value. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323803#comment-15323803 ] Erick Erickson commented on SOLR-5944: -- I had a question come up today that I wanted to ask for posterity. What gets returned between the time we update one of these and a commit occurs? The old value or the new one? I'd assumed the old one but wanted to be sure. And explicitly this _only_ applies to fields for which indexed=false I see, which answers another of the questions that came up. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323784#comment-15323784
]
Hoss Man commented on SOLR-5944:
bq. When update2 (say a partial update) arrives before update1 (say a full
update, on which update2 depends), *then the call for indexing update2 is a
blocking call* (which finishes either after update1 is indexed, or timeout is
reached).
Ahhh... now it makes sense to me. The part I wasn't getting before was that
update2 blocks on the replica until it sees the update1 it is dependent on.
I feel like there is probably a way we could write a more sophisticate "grey
box" type test for this leveraging callbacks in the DebugFilter, but I'm having
trouble working out what that would really look like.
I think the hueristic approach you're taking here is generall fine for now (as
a way to _try_ to run the updates in a given order even though we know there
are no garuntees) but i have a few suggestions to improve things:
* lots more comments in the test code to make it clear that we use multiple
threads because each update may block if it depends on another update
* replace the comments on the sleep calls to make it clear that while we can't
garuntee/trust what order the updates are executed in since multiple threads
are involved, we're trying to bias the thread scheduling to run them in the
order submitted
** (the wording right now seems definitive and makes the code look clearly
suspicious)
* create {{atLeast(3)}} updates instead of just a fixed set of "3" so we
increase our odds of finding potential bugs when more then one update is out of
order.
* loop over multiple (random) permutations of orderings of the updates
** don't worry about wether a given ordering is actually correct, that's a
valid random ordering for the purposes of the test
** a simple comment saying we know it's possible but it doesn't affect any
assumptions/assertions in the test is fine
* for each random permutation, execute it (and check the results) multiple times
** this will help increase the odds that the thread scheduling actaully winds
up running our updates in the order we were hoping for.
* essentially this should be a a micro "stress test" of updates in arbitrary
order
Something like...
{code}
final String ID = "0";
final int numUpdates = atLeast(3);
final int numPermutationTotest = atLeast(5);
for (int p = 0; p < numPermutationTotest; p++) {
del("*:*);
commit();
index("id",ID, ...); // goes to all replicas
commit();
long version = assertExpectedValuesViaRTG(LEADER, ID, ...);
List updates = makeListOfSequentialSimulatedUpdates(ID,
version, numUpdates);
for (UpdateRequest req : updates) {
assertEquals(0, REPLICA_1.requets(req).getStatus());
}
Collections.shuffle(updates, random());
// this method is where you'd comment the hell out of why we use threads for
this,
// and can be re-used in the other place where a threadpool is used...
assertSendUpdatesInThreadsWithDelay(REPLICA_0, updates, 100ms);
for (SolrClient client : NONLEADERS) [
// assert value on replica matches original value + numUpdates
}
}
{code}
As a related matter -- if we are expecting a replica to "block & eventually
time out" when it sees an out of order update, then there should be a white box
test asserting the expected failure situation as well -- something like...
{code}
final String ID = "0";
del("*:*);
commit();
index("id",ID, ...);
UpdateRequest req = simulatedUpdateRequest(version + 1, ID, ...);
Timer timer = new Timer();
timer.start();
SolrServerException e = expectThrows(() -> { REPLICA_0.request(req); });
timer.stop();
assert( /* elapsed time of timer is at least the X that we expect it to block
for */ )
assert(e.getgetHttpStatusMesg().contains("something we expect it to say if the
update was out of order"))
assertEquls(/* whatever we expect in this case */, e.getHttpStatusCode());
{code}
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
>
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323101#comment-15323101 ] Ishan Chattopadhyaya commented on SOLR-5944: When update2 (say a partial update) arrives before update1 (say a full update, on which update2 depends), then the call for indexing update2 is a blocking call (which finishes either after update1 is indexed, or timeout is reached). The intention was to: # shuffle the updates (so that the 3 updates are in one of the 6 possible permutations, one of those permutations being in-order) # send them out in sequence of the shuffle # have them arrive at Solr in the intended order (as intended in steps 1 and 2). However, since an out of order update waits for the dependent update and blocks the call until such a dependent update arrives (or timeout is reached), the intention is to have these calls non-blocking. So, I wanted to send updates out sequentially (deliberately re-ordered, through a shuffle), but asynchronously (so as to keep those calls non-blocking). bq. ...My impression, based on the entirety of that method, was that the intent of the test was to bypass the normal distributed update logic and send carefully crafted "simulated" updates direct to each replica, such that one repliica got the (simulated from leader) updates "in order" and another replica got the (simulated from leader) updates "out of order" That is exactly my intention. bq. if the point was for replica2 to get the (simulated from leader) updates "out of order" then why shuffle them - why not explicitly put them in the "wrong" order? There could be possibly 6 permutations in terms of the mutual ordering of the 3 updates, so I used shuffle instead of choosing a particular "wrong" ordering. Of course, one of those 6 permutations is the "right" order, so that case is not consistent with the name of the test; I can make a fix to exclude that case. bq. if the goal was send them asynchronously, and try to get them to happen as concurrently as possible (as you indicated above in your answer to my question) then what was the point of the "shuffle" ? I think I was trying: (a) asynchronously (so that out of order update doesn't block out the next update request that sends a dependent order), (b) intention was not really to test for race conditions, but just to be concurrent enough so that a dependent update arrives before an out of order update times out. bq. Thread.sleep(10); The point of this was to avoid situations where the shuffled list (and intended order for that testcase) was, say, "update1, update3, update2", but it actually went to the Solr server in the order "update1, update2, update3" due to parallel threads sending the updates at nearly the same time. Do you think this makes sense? I am open to revise this entire logic if you suggest. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323045#comment-15323045
]
Hoss Man commented on SOLR-5944:
I don't understand this comment -- particularly in light of the changes you've
made to the test since...
{quote}
bq. what's the point of using a threadpool and SendUpdateToReplicaTask here?
why not just send the updates in a (randdomly assigned) determinisitc order?
Essentially, I need a way to send three updates to the replica asynchronously.
To achieve the effect of asynchronous updates, I used a threadpool here. Three
updates sent one after the other, each being a blocking call, wouldn't have
simulated the leader -> replica interaction sufficiently.
{quote}
When i posted that particular question it was about
outOfOrderUpdatesIndividualReplicaTest -- the code in question at teh time
looked like this...
{code}
// re-order the updates for replica2
List reorderedUpdates = new ArrayList<>(updates);
Collections.shuffle(reorderedUpdates, random());
for (UpdateRequest update : reorderedUpdates) {
SendUpdateToReplicaTask task = new SendUpdateToReplicaTask(update,
clients.get(1), random());
threadpool.submit(task);
}
{code}
...My impression, based on the entirety of that method, was that the intent of
the test was to bypass the normal distributed update logic and send carefully
crafted "simulated" updates direct to each replica, such that one repliica got
the (simulated from leader) updates "in order" and another replica got the
(simulated from leader) updates "out of order"
* if the point was for replica2 to get the (simulated from leader) updates "out
of order" then why shuffle them - why not explicitly put them in the "wrong"
order?
* if the goal was send them asynchronously, and try to get them to happen as
concurrently as possible (as you indicated above in your answer to my question)
then what was the point of the "shuffle" ?
Looking at the modified version of that code in your latest patch doesn't
really help clarify things for me...
{code}
// re-order the updates for NONLEADER 0
List reorderedUpdates = new ArrayList<>(updates);
Collections.shuffle(reorderedUpdates, random());
List updateResponses = new ArrayList<>();
for (UpdateRequest update : reorderedUpdates) {
AsyncUpdateWithRandomCommit task = new AsyncUpdateWithRandomCommit(update,
NONLEADERS.get(0), seed);
updateResponses.add(threadpool.submit(task));
// send the updates with a slight delay in between so that they arrive in the
intended order
Thread.sleep(10);
}
{code}
In the context of your answer, that it's intentional for the updates to be
async...
* why shuffle them?
* why is there now a {{sleep}} call with an explicit comment "...so that they
arrive in the intended order" ... if there is an "intended" order why would you
want them to be async?
the other SendUpdateToReplicaTask/AsyncUpdateWithRandomCommit usages exhibit
the same behavior of a "sleep" in between {{ threadpool.submit(task); }} calls
with similar comments about wanting to "...ensure requests are sequential..."
hence my question about why threadpools are being used at all.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317318#comment-15317318
]
Hoss Man commented on SOLR-5944:
bq. Removed all non 3x, 4x codec suppression. They need to be suppressed as per
a comment from Mikhail. ...
that comment is over 2 years old, from a time when those codecs existed but did
not support updating doc values.
those codecs no longer exist (on either master or branch_6x) -- even if someone
had na existing index with segments from those codecs, they would not be
supported by any Solr 6.x version because they are more then 1 major version
old -- we only have to worry about Lucene5* codecs and higher.
bq. As of now, both will generate a new version. I think "inc" 0 should be
dropped, and "set" same value should be versioned. I'll check if behaviour in
this patch is at par with regular atomic updates; and if so, will open a
separate issue for this later.
yeah, sorry -- my point was: "whatever the current, non-patched, behavior is
for the version returned from these types of updates, we need to assert that
behavior is true here." -- we should not be changing any semantics here,
absolutely open a distinct issue for that if you think it makes sense as a
future improvement.
bq. I think we can do the same in TestStressInPlaceUpdates, by randomly setting
number of writer threads to 1 sometimes.
Isn't that still a cloud based test with multiple nodes/shards? Even with only
1 writer thread it's going ot be harder to debug then doing more randomized
testing in a single node test (via something like checkReplay as in my previous
suggestion)
bq. I couldn't find a way to do this (check RTG against uncommitted model) for
the TestInPlaceUpdate (now called TestInPlaceUpdatesStandalone in this patch).
This is based on SolrTestCaseJ4.
{{SolrTestCaseJ4.addAndGetVersion(...)}}
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316984#comment-15316984 ] Ishan Chattopadhyaya commented on SOLR-5944: Fixed two bugs: 1. Added a check while writing a document to exclude anything to do with the id field. 2. Added an exception when a "set" or "inc" operation is attempted at a non-existent document. Review comments: bq. * in general, all these tests seem to depend on autoCommit being disabled, and use a config that is setup that way, but don't actaully assert that it's true in case someone changes the configs in the future TODO bq. * TestInPlaceUpdate Renamed this test to TestInPlaceUpdatesStandalone. bq. ** SuppressCodecs should be removed Removed all non 3x, 4x codec suppression. They need to be suppressed as per a comment from Mikhail. https://issues.apache.org/jira/browse/LUCENE-5189?focusedCommentId=13958205=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13958205 bq. ** should at least have class level javadocs explaining what's being tested TODO ** testUpdatingDocValues bq. *** for addAndGetVersion calls where we don't care about the returned version, don't bother assigning it to a variable (distracting) Fixed bq. *** for addAndGetVersion calls where we do care about the returned version, we need check it for every update to that doc... TODO bq. currently version1 is compared to newVersion1 to assert that an update incrememnted the version, but in between those 2 updates are 4 other places where that document was updated -- we have to assert it has the expected value (either the same as before, or new - and if new record it) after all of those addAndGetVersion calls, or we can't be sure where/why/how a bug exists if that existing comparison fails. TODO bq. ideally we should be asserting the version of every doc when we query it right along side the assertion for it's updated "ratings" value TODO bq. *** most of the use of "field(ratings)" can probbaly just be replaced with "ratings" now that DV are returnable -- although it's nice to have both included in the test at least once to demo that both work, but when doing that there should be a comment making it clear why Fixed ** testOnlyPartialUpdatesBetweenCommits *** ditto comment about checking return value from addAndGetVersion bq. *** this also seems like a good place to to test if doing a redundent atomic update (either via set to the same value or via inc=0) returns a new version or not -- should it? As of now, both will generate a new version. I think "inc" 0 should be dropped, and "set" same value should be versioned. I'll check if behaviour in this patch is at par with regular atomic updates; and if so, will open a separate issue for this later. bq. ** DocInfo should be a private static class and have some javadocs Fixed bq. ** based on how testing has gone so far, and the discover of LUCENE-7301 it seems clear that adding even single thread, single node, randomized testing of lots of diff types of add/update calls would be good to add I think we can do the same in TestStressInPlaceUpdates, by randomly setting number of writer threads to 1 sometimes. bq. *** we could refactor/improve the "checkReplay" function I added in the last patch to do more testing of a randomly generated Iterable of "commands" (commit, doc, doc+atomic mutation, etc...) TODO bq. *** and of course: improve checkReplay to verify RTG against hte uncommited model as well I couldn't find a way to do this for the TestInPlaceUpdate (now called TestInPlaceUpdatesStandalone in this patch). This is based on SolrTestCaseJ4. bq. *** testReplayFromFile and getSDdoc should probably go away once we have more structured tests for doing this Fixed bq. ** createMap can be elimianted -- callers can just use SolrTestCaseJ4.map(...) Fixed bq. ** In general the tests in this class should include more queries / sorting against the updated docvalues field after commits to ensure that the updated value is searchable & sortable TODO bq. ** Likewise the test methods in this class should probably have a lot more RTG checks -- with filter queries that constrain against the updated docvalues field, and checks of the expected version field -- to ensure that is all working properly. Couldn't figure out how to do RTGs with this test, but will check RTGs + filter queries in the TestInPlaceUpdatesDistrib test (which was formerly InPlaceUpdateDistribTest) bq. * InPlaceUpdateDistribTest Renamed to TestInPlaceUpdatesDistrib now bq. ** SuppressCodecs should be removed 3x and 4x codec suppressions cannot be removed. bq. ** should at least have class level javadocs explaining what's being tested TODO bq. ** Once LUCENE-7301 is fixed and we can demonstate that this passes reliably all of the time, we should ideally refactor this to subclass SolrCloudTestCase TODO bq. ** in general,
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303367#comment-15303367
]
Hoss Man commented on SOLR-5944:
Ok ... more in depth comments reviewing the latest patch (ignoring some of the
general higher level stuff i've previously commented on).
(So far i've still focused on reviewing the tests, because we should make sure
they're rock solid before any discussion of refacoting/improving/changing the
code)
* in general, all these tests seem to depend on autoCommit being disabled, and
use a config that is setup that way, but don't actaully assert that it's true
in case someone changes the configs in the future
** TestInPlaceUpdate can get direct access to the SolrCore verify that for
certain to
** the distrib tests might be able to use one of hte new cnfig APIs to check
this (i don't know off the top of my head)
*** at a minimum define a String constant for the config file name in
TestInPlaceUpdate and refer to it in the other tests where the same config is
expected with a comment explaining that we're *assuming* it has autoCommit
disabled and that TestInPlaceUpdate will fail if it does not.
* TestInPlaceUpdate
** SuppressCodecs should be removed
** should at least have class level javadocs explaining what's being tested
** testUpdatingDocValues
*** for addAndGetVersion calls where we don't care about the returned version,
don't bother assigning it to a variable (distracting)
*** for addAndGetVersion calls where we do care about the returned version, we
need check it for every update to that doc...
currently version1 is compared to newVersion1 to assert that an update
incrememnted the version, but in between those 2 updates are 4 other places
where that document was updated -- we have to assert it has the expected value
(either the same as before, or new - and if new record it) after all of those
addAndGetVersion calls, or we can't be sure where/why/how a bug exists if that
existing comparison fails.
ideally we should be asserting the version of every doc when we query it
right along side the assertion for it's updated "ratings" value
*** most of the use of "field(ratings)" can probbaly just be replaced with
"ratings" now that DV are returnable -- although it's nice to have both
included in the test at least once to demo that both work, but when doing that
there should be a comment making it clear why
** testOnlyPartialUpdatesBetweenCommits
*** ditto comment about checking return value from addAndGetVersion
*** this also seems like a good place to to test if doing a redundent atomic
update (either via set to the same value or via inc=0) returns a new version or
not -- should it?
** DocInfo should be a private static class and have some javadocs
** based on how testing has gone so far, and the discover of LUCENE-7301 it
seems clear that adding even single thread, single node, randomized testing of
lots of diff types of add/update calls would be good to add
*** we could refactor/improve the "checkReplay" function I added in the last
patch to do more testing of a randomly generated Iterable of "commands"
(commit, doc, doc+atomic mutation, etc...)
*** and of course: improve checkReplay to verify RTG against hte uncommited
model as well
*** testReplayFromFile and getSDdoc should probably go away once we have more
structured tests for doing this
** createMap can be elimianted -- callers can just use SolrTestCaseJ4.map(...)
** In general the tests in this class should include more queries / sorting
against the updated docvalues field after commits to ensure that the updated
value is searchable & sortable
** Likewise the test methods in this class should probably have a lot more RTG
checks -- with filter queries that constrain against the updated docvalues
field, and checks of the expected version field -- to ensure that is all
working properly.
* InPlaceUpdateDistribTest
** SuppressCodecs should be removed
** should at least have class level javadocs explaining what's being tested
** Once LUCENE-7301 is fixed and we can demonstate that this passes reliably
all of the time, we should ideally refactor this to subclass SolrCloudTestCase
** in general, the "pick a random client" logic should be refactored so that
sometimes it randomly picks a CloudSolrClient
** there should almost certianly be some "delete all docs and optimize" cleanup
in between all of these tests
*** easy to do in an @Before method if we refactor to subclass SolrCloudTestCase
** docValuesUpdateTest
*** should randomize numdocs
*** we need to find away to eliminate the hardcoded "Thread.sleep(500);"
calls...
if initially no docs have a rating value, then make the (first) test query
be for {{rating:\[\* TO \*\]}} and execute it in a rety loop until the numFound
matches numDocs.
likewise if we ensure all ratings have a value such that abs(ratings) < X,
then the second
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301985#comment-15301985
]
Ishan Chattopadhyaya commented on SOLR-5944:
bq. (I'm pretty sure) I was able to reproduce the root cause of the randomized
failures in LUCENE-7301.
Thanks Hoss for beating me to it!
{quote}
testReplay5 - still uses "inc" for doc id=0, but uses "set" for every other doc
in the index
this currently fails with an NPE in
AtomicUpdateDocumentMerger.doInPlaceUpdateMerge(AtomicUpdateDocumentMerger.java:283)
{quote}
I think the problem there is that a "set" operation was attempted at a document
that still doesn't exist in the index. I think such an operation works with
atomic updates, but the underlying docValues API doesn't support updates of dv
fields that don't exist yet. I will try to handle this better, instead of
throwing NPE.
I shall work on fixing your review comments regarding the tests, and increase
their scope as you suggest. My idea behind the tests were (and naming could be
improved): TestInPlaceUpdate just tests some basic cases in non-cloud mode,
TestStressInPlaceUpdates tests lots of documents, lots of updates, lots of
threads and cloud mode, InPlaceUpdateDistribTest for some basic
operations/scenarios in cloud (including testing if same document was updated,
or a new one was created). I was thinking that if we can get past the DV
updates flushing issue (LUCENE-7301), we can focus well on improving scope of
tests more. Thanks for your review!
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301210#comment-15301210 ] Hoss Man commented on SOLR-5944: (I'm pretty sure) I was able to reproduce the root cause of the randomized failures in LUCENE-7301. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300541#comment-15300541
]
Hoss Man commented on SOLR-5944:
I'm still very slowly trying to get up to speed on this ... i started out by
reviewing the tests ishan wrote specifically for this issue, but once i
realized there really weren't any pre-existing, non trivial, "distributed
atomic updates" I put that on the backburner to work on SOLR-9159 -- now that
that test is solid and running on master & 6x, it's helped uncover an NPE in
the new code when the latest patch is applied...
{noformat}
[junit4] 2> 20685 ERROR (qtp1751829478-197) [n:127.0.0.1:58791_solr
c:test_col s:shard1 r:core_node4 x:test_col_shard1_replica2]
o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Exception
writing document id 34 to the index; possible analysis error.
[junit4] 2>at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:188)
[junit4] 2>at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
[junit4] 2>at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
[junit4] 2>at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:954)
[junit4] 2>at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1145)
[junit4] 2>at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:729)
[junit4] 2>at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
[junit4] 2>at
org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
[junit4] 2>at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:179)
[junit4] 2>at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:135)
[junit4] 2>at
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:274)
[junit4] 2>at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
[junit4] 2>at
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:239)
[junit4] 2>at
org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:157)
[junit4] 2>at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:186)
[junit4] 2>at
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:108)
[junit4] 2>at
org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
[junit4] 2>at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
[junit4] 2>at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
[junit4] 2>at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
[junit4] 2>at
org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
[junit4] 2>at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
[junit4] 2>at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465)
[junit4] 2>at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
[junit4] 2>at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
[junit4] 2>at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
[junit4] 2>at
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:138)
[junit4] 2>at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
[junit4] 2>at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
[junit4] 2>at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2>at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
[junit4] 2>at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
[junit4] 2>at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2>at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
[junit4] 2>at
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296663#comment-15296663
]
Ishan Chattopadhyaya commented on SOLR-5944:
I ran the test all weekend and found the following failures which reproduce
reliably:
{code}
8CF844B8D2C14DFA
AE673569D5853984
{code}
It seems that the common pattern in these tests is that they fail when one of
the docValues fields is Lucene54 and one of the fields is Memory. Here's an
example (from the latter seed):
{code}
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62):
{title_s=PostingsFormat(name=MockRandom), id=Lucene50(blocksize=128)},
docValues:{val1_i_dvo=DocValuesFormat(name=Direct),
_version_=DocValuesFormat(name=Direct),
val2_l_dvo=DocValuesFormat(name=Memory), ratings=DocValuesFormat(name=Memory),
price=DocValuesFormat(name=Lucene54)}, maxPointsInLeafNode=132,
maxMBSortInHeap=7.882796951749762,
sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=fr-CA,
timezone=Pacific/Pago_Pago
{code}
I found that the same pattern was present in all previously failing tests for
which I had logs.
As a logical next step, I suppressed both "Lucene54" and "Memory" codecs in the
test and ran them. The failing tests passed, and so did lots and lots of other
seeds. However, one of the tests failed after suppressing Lucene54 and Memory:
seed {{F9C1398E563942D5}} (this seed didn't fail before). Surprisingly, for
this failing seed, I don't see the "NOTE" line mentioning per field codecs, but
just the following info:
{code}
NOTE: test params are:
codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST,
chunkSize=24740, maxDocsPerChunk=7, blockSize=236),
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST,
chunkSize=24740, blockSize=236)),
sim=RandomSimilarity(queryNorm=true,coord=no): {}, locale=sv,
timezone=America/Catamarca
{code}
I'm looking into how to make these tests fail in a more simple, reproducible
test, not requiring lots of threads etc.
[~hossman] FYI.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295236#comment-15295236 ] Noble Paul commented on SOLR-5944: -- Awesome > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294456#comment-15294456 ] Steve Rowe commented on SOLR-5944: -- I beasted 2000 iterations of `TestStressInPlaceUpdates` against [https://github.com/chatman/lucene-solr/commit/d61c4fc520111f721468edb236930180bd91d7eb] and there were zero failures. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292921#comment-15292921 ] Ishan Chattopadhyaya commented on SOLR-5944: [~hoss...@fucit.org] pointed me to SOLR-8733, due to which returned versions are probably 0 unless the version is done locally (i.e. the update is sent to the correct shard leader). I found out that the stress test was affected by this issue. I have committed a fix to the stress test so as to workaround this problem by only sending updates to the leader of the shard. After having done that, I cannot reproduce the failures that [~steve_rowe] reported. Here's the corresponding commit: https://github.com/chatman/lucene-solr/commit/d61c4fc520111f721468edb236930180bd91d7eb > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289163#comment-15289163
]
Ishan Chattopadhyaya commented on SOLR-5944:
In the 587 file,
between 30557 (when the expected update was written):
{code}
[junit4] 2> 30557 INFO (qtp2010985731-180) [n:127.0.0.1:37972__m
c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog TLOG:
added id 13(ver=1534607780231512064, prevVersion=1534607779993485312,
prevPtr=2343) to
tlog{file=/tmp/beast-tmp-output/587/J0/temp/solr.cloud.TestStressInPlaceUpdates_FFC46C473EC471E6-001/shard-1-001/cores/collection1/data/tlog/tlog.004
refcount=1} LogPtr(2396) map=1977112331, actual doc=SolrInputDocument(fields:
[id=13, val2_l_dvo=300012, _version_=1534607780231512064, val1_i_dvo=3])
{code}
and the ERROR line at 34456,
{code}
[junit4] 2> 34456 ERROR (READER2) [] o.a.s.c.TestStressInPlaceUpdates
Realtime=true, ERROR, id=13
found={response={numFound=1,start=0,docs=[SolrDocument{id=13,
title_s=[title13], val1_i_dvo=3, val2_l_dvo=36,
_version_=1534607778351415296, ratings=0.0, price=0}]}}
model=[1534607780231512064, 3, 300012]
{code},
there are two suspicious events:
# There was some reordering of updates (34422).
# There was a commit (start: 33306, 33363, 33398; end: 33871, 34209).
I think either of those, or both, could be the reason for the inconsistency.
bq. I don't know anything about this AtomicUpdateDocumentMerger class, but this
logging smells particularly suspicious to me because it is the last time
AtomicUpdateDocumentMerger logs anything about doc #13
The reason this is the last time AUDM logged anything about the doc #13 could
be because soon after that line, the commit happened. And doc #13 was no longer
found in the tlog, and hence there was no "uncommitted doc" with id=13.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288019#comment-15288019 ] Ishan Chattopadhyaya commented on SOLR-5944: Thanks for your analysis, Hoss. I'll take a deeper look as soon as possible. A pattern I have observed with such failures (and these failures are the ones I was referring to in the past) that documents get in trouble immediately after or during a commit (i.e. between the commit start and end) happening in a parallel thread. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287640#comment-15287640
]
Steve Rowe commented on SOLR-5944:
--
Both of those seeds (FFC46C473EC471E6 and 15E180DC7142CBF3) reproduce for me
too (only tried each one once).
A third beasting failure, run 783, does NOT reproduce for me (0 failures out of
4 runs):
{noformat}
ant test -Dtestcase=TestStressInPlaceUpdates -Dtests.method=stressTest
-Dtests.seed=CCB5FA74FA9BB974 -Dtests.slow=true -Dtests.locale=sr
-Dtests.timezone=Africa/Gaborone -Dtests.asserts=true
-Dtests.file.encoding=US-ASCII
{noformat}
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287525#comment-15287525
]
Hoss Man commented on SOLR-5944:
FWIW, I'm testing ishan's latest patch against lucene-solr master and the two
"reproduce" lines from steve's logs (minus the linedocs path) fail 100% of the
time for me on my box - although the specific doc listed in the failure message
varies from run to run, presumably because of the parallel threads? ...
{noformat}
ant test -Dtestcase=TestStressInPlaceUpdates -Dtests.method=stressTest
-Dtests.seed=FFC46C473EC471E6 -Dtests.slow=true -Dtests.locale=sr-ME
-Dtests.timezone=Europe/Riga -Dtests.asserts=true -Dtests.file.encoding=UTF-8
ant test -Dtestcase=TestStressInPlaceUpdates -Dtests.method=stressTest
-Dtests.seed=15E180DC7142CBF3 -Dtests.slow=true -Dtests.locale=pt-BR
-Dtests.timezone=Africa/Juba -Dtests.asserts=true -Dtests.file.encoding=UTF-8
{noformat}
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287052#comment-15287052 ] Ishan Chattopadhyaya commented on SOLR-5944: bq. Yeah, I've stopped using the Reply thing because of this I see.. I'll consider stopping the use of the reply feature now :-) Sorry for the confusion. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286995#comment-15286995 ] Steve Rowe commented on SOLR-5944: -- bq. but using the "Reply" feature so it appears inline. Yeah, I've stopped using the Reply thing because of this - you can't find all the new posts at the bottom if people use this misfeature (as Muir called it). > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286987#comment-15286987 ] Hoss Man commented on SOLR-5944: oh ... wait ... i see now, it was posted *after* my other comments/attachments ... but using the "Reply" feature so it appears inline. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286986#comment-15286986 ] Hoss Man commented on SOLR-5944: bq. https://github.com/chatman/lucene-solr/tree/solr_5944 Noble's last commit is: 4572983839e3943b7dea52a8a2d55aa2b3b5ca3a Ugh ... somehow i completley missed seeing this comment yesterday, I don't know why i couldn't find that branch on github. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285885#comment-15285885 ] Ishan Chattopadhyaya commented on SOLR-5944: Thanks for looking into the issue, Hoss. bq. The comments says the changes were made to a github repo on a "solr_5944" branch, but the linked repo doesn't have a branch with that name https://github.com/chatman/lucene-solr/tree/solr_5944 Noble's last commit is: 4572983839e3943b7dea52a8a2d55aa2b3b5ca3a bq. what is the issue that causes data loss When it happens, my understanding is that some uncommitted in-place updates don't reflect after a commit in a re-opened searcher. bq. can you describe the "rarely" situation? is there a test that demonstrates the problem? TestStressInPlaceUpdates is the affected test. Used to happen once in around 2000 runs, on a 3Ghz 5960X CPU (I'm unable to reproduce, as I mentioned in my previous comment, on dual 2.0Ghz Xeon E5 2658 V3). bq. is this the same (non-reproducible) test failure that Ishan mentioned in in his comment this morning (May16) https://issues.apache.org/jira/browse/SOLR-5944?focusedCommentId=15193478=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15193478 I think Noble is referring to the same. It could be due to some underlying issue with either how Solr uses Lucene or how Lucene is flushing the dv updates during a commit. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284905#comment-15284905 ] Hoss Man commented on SOLR-5944: [~noble.paul] added a comment - 04/Apr/16 02:20 bq. Ishan Chattopadhyaya I'm done with my review. I have done some minor changes to your private branch https://github.com/chatman/lucene-solr and branch solr_5944 . We have an issue that causes some data loss very rarely. if that is fixed, I'm +1 to commit this There are so many questions raised by this comment... * what are the minor changes? This comment was posted long _after_ the most recent patch attached this this issue (on Mar31), implying that whatever the the changes are they didn't make it into any attached patch. The comments says the changes were made to a github repo on a "solr_5944" branch, but the linked repo doesn't have a branch with that name -- it *DOES* have a branch named "SOLR-5944" but the most recent commit on that branch is from Feb -- 2 months before this comment was posted (and _way_ older then the most recent patch attached to this issue) * what is the issue that causes data loss? can you describe the "rarely" situation? is there a test that demonstrates the problem? ... is this the same (non-reproducible) test failure that Ishan mentioned in in his comment this morning (May16) or is there some other bug we need to be worried about? > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284838#comment-15284838 ] Ishan Chattopadhyaya commented on SOLR-5944: Thanks for the forbidden API fixes. I shall incorporate them into the next patch. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284816#comment-15284816
]
Ishan Chattopadhyaya commented on SOLR-5944:
The only remaining issue is a test failure, which used to happen very
intermittently for me with TestStressInPlaceUpdates. Here are the partial logs
(from April):
{code}
[beaster] 2> NOTE: reproduce with: ant test
-Dtestcase=TestStressInPlaceUpdates -Dtests.method=stressTest
-Dtests.seed=3E8431D13FA03373 -Dtests.slow=true -Dtests.locale=sr-Latn
-Dtests.timezone=Antarctica/Syowa -Dtests.asserts=true
-Dtests.file.encoding=US-ASCII
[beaster] [18:34:34.713] ERROR 51.8s | TestStressInPlaceUpdates.stressTest
<<<
[beaster]> Throwable #1:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught
exception in thread: Thread[id=181, name=READER10, state=RUNNABLE,
group=TGRP-TestStressInPlaceUpdates]
[beaster]>at
__randomizedtesting.SeedInfo.seed([3E8431D13FA03373:55E2EE7C0175E789]:0)
[beaster]> Caused by: java.lang.RuntimeException:
java.lang.AssertionError: Vals are: 3, 28, id=4
[beaster]>at
__randomizedtesting.SeedInfo.seed([3E8431D13FA03373]:0)
[beaster]>at
org.apache.solr.cloud.TestStressInPlaceUpdates$3.run(TestStressInPlaceUpdates.java:416)
[beaster]> Caused by: java.lang.AssertionError: Vals are: 3, 28,
id=4
[beaster]>at org.junit.Assert.fail(Assert.java:93)
[beaster]>at org.junit.Assert.assertTrue(Assert.java:43)
[beaster]>at
org.apache.solr.cloud.TestStressInPlaceUpdates$3.run(TestStressInPlaceUpdates.java:390)
{code}
However, I ran a beast of the same test on a 24 core (48 thread) machine, 1000
rounds, 24 at a time, and the beasting passed. Maybe I'll need to try harder to
reproduce it over the week or the coming weekend.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252850#comment-15252850 ] Justin Deoliveira commented on SOLR-5944: - I've been following this patch for a while, and am super excited about recent progress. I just applied the latest patch locally and built with maven and it resulted in some forbidden api failures. I don't know of this helps but here is a minor [patch|https://gist.githubusercontent.com/jdeolive/5b56848603fe5cbac804cd5acb8ebcd2/raw/35b3a3150c652738713618e7d8fcf7bd6cf3e0e6/forbiddenapis.patch] that addresses them. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15223864#comment-15223864 ] Noble Paul commented on SOLR-5944: -- [~ichattopadhyaya] I'm done with my review. I have done some minor changes to your private branch https://github.com/chatman/lucene-solr and branch solr_5944 . We have an issue that causes some data loss very rarely. if that is fixed, I'm +1 to commit this > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220582#comment-15220582 ] Ishan Chattopadhyaya commented on SOLR-5944: Thats a good point, Noble; there isn't a test, I'll add one. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220505#comment-15220505 ] Noble Paul commented on SOLR-5944: -- is there a testcase for partial update as the first operation? instead of a full insert (like set and inc) > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219447#comment-15219447
]
Noble Paul commented on SOLR-5944:
--
There are a lot of places where a toString() is done and then parsing is done.
It much cheaper to do an {{instanceof}} check
look at the following block in
{{AtomicUpdateDocumentMerger.doInPlaceUpdateMerge()}}
{code:java}
if (oldValue instanceof Long) {
result = ((Long) oldValue).longValue() +
Long.parseLong(value.toString());
} else if (oldValue instanceof Float) {
result = ((Float) oldValue).floatValue() +
Float.parseFloat(value.toString());
} else if (oldValue instanceof Double) {
result = ((Double) oldValue).doubleValue() +
Double.parseDouble(value.toString());
} else {
// int, short, byte
result = ((Integer) oldValue).intValue() +
Integer.parseInt(value.toString());
}
{code}
can be optimized as follows
{code:java}
if (oldValue instanceof Long) {
result = (Long) oldValue + (value instanceof Number ?
((Number) value).longValue() : Long.parseLong(value.toString()));
} else if (oldValue instanceof Float) {
result = (Float) oldValue + (value instanceof Number ?
((Number) value).floatValue() : Float.parseFloat(value.toString()));
} else if (oldValue instanceof Double) {
result = (Double) oldValue + (value instanceof Number ?
((Number) value).doubleValue() : Double.parseDouble(value.toString()));
} else {
// int, short, byte
result = (Integer) oldValue + (value instanceof Number ?
((Number) value).intValue() : Integer.parseInt(value.toString()));
}
{code}
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215802#comment-15215802
]
Noble Paul commented on SOLR-5944:
--
In {{DUP.waitForDependentUpdates()}} it is not wise to blindly wait for 10
milliseconds. What if the dependent update came right after it entered the
sleep() . In that case we are unnecessarily waiting while the thread could
proceed immediately . We can use a {{wait()/notify()}} mechanism to avoid that
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15213308#comment-15213308 ] Ishan Chattopadhyaya commented on SOLR-5944: Hi Gopal, sorry to have missed your comment. I think this can land in Solr 6.1 or later, since 6.0 release branch has already been cut. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194359#comment-15194359 ] Gopal Patwa commented on SOLR-5944: --- Great! to see progress and near to completion this feature. can this be part of 6.0 release? > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962099#comment-14962099 ] Ishan Chattopadhyaya commented on SOLR-5944: Had productive discussions around this at Revolution with [~yo...@apache.org] and [~noble.paul]. Here's a list of TODO items I am going to work on: Here's a summary: # Expand the current tests to do more updates, using more threads, something like stress reorder test does. [Yonik] # Before timing out (in a replica) after waiting for dependent update long enough, instead of throwing an error (which triggers LIR), request from replica to leader to pull a range of updates (the missing ones), i.e. from what last version we have to the version we expect. This can be easily done by using (a slightly modified version of) the RTG's getUpdates handler. [Noble] # Since there are changes to the javabin format (for inplace updates), add tests for backcompat. [Noble] # Code cleanup, fix the todos/nocommits etc. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956297#comment-14956297 ] Noble Paul commented on SOLR-5944: -- I guess yonik meant something like CAS. Anyway let's meet up during LSR > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954900#comment-14954900 ] Yonik Seeley commented on SOLR-5944: bq. Also, am I correct in my assumption that the no. of updates processed during this buffering phase will not be more than numUpdatesToKeep? That's a minimum, not a maximum. There really is no maximum. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955031#comment-14955031 ] Yonik Seeley commented on SOLR-5944: Do we have any sort of conditional update yet (or something generic like update-by-script)? That would be great to help test this stuff out (i.e. to make sure that partial updates aren't reordered with respect to each-other). If you're going to be around the conference this week, It would be great to chat about some of this stuff in person... > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955620#comment-14955620 ] Ishan Chattopadhyaya commented on SOLR-5944: bq. Do we have any sort of conditional update yet (or something generic like update-by-script)? That would be great to help test this stuff out (i.e. to make sure that partial updates aren't reordered with respect to each-other). "inc" is supported, I can try writing a suite that indexes bunch of full documents, "set" updates and "inc" updates (perhaps along with \_version\_s for optimistic concurrency) and then test for consistency across replicas. By update-by-script, do you mean something like SOLR-5979 (Alas it is not there yet)? bq. If you're going to be around the conference this week, It would be great to chat about some of this stuff in person... +1. I'm planning to be around on 15th and 16th. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14953060#comment-14953060
]
Ishan Chattopadhyaya commented on SOLR-5944:
I am close to a patch for the above proposal, and shall post it soon.
One place where I am somewhere stuck is in the log buffering/replaying part.
Here's the problem:
When a replica is put into recovery by the leader, it comes back up and tries
to perform a peersync. This seems to be happening in a two phase process:
buffering (where the updates, after being obtained from the leader's tlog, are
played back and written to the replica's tlog but not its index/ulog) and
replaying (where the tlog is replayed and the updates are written to
ulog/index, but not into tlog again). The problem I'm facing is that during
this buffering phase, the inplace updates can't find dependent updates if they
are not in the index, since the updates are not written to ulog in the
buffering phase.
I have two choices at the moment to get around this:
# During a buffering phase, I can keep a separate map of all updates (id to
tlog pointer) to be used during and discarded after the buffering phase. That
map can help resolve inplace updates that follow. (Pro: fast, Con: memory)
# For every inplace update, I traverse back into the tlog and linearly scan for
the required dependent update. (Pro: no memory, Con: Slow / O(n))
At this point, I'm inclined to go for option 1, but I'm wondering if there are
any serious downsides to doing this. Any suggestions, please?
Also, am I correct in my assumption that the no. of updates processed during
this buffering phase will not be more than {{numUpdatesToKeep}}?
In case I sound confused/unclear, please let me know and I'll post the relevant
failing test for this.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952290#comment-14952290
]
Shai Erera commented on SOLR-5944:
--
Thanks Yonik for the clarification. The issue then is that the {{version}}
field is not 'updateable' right? If it was e.g. a numeric DV field itself, then
each update to a document could involve updating both the version field, as
well the numeric DV field in question. And regular doc updates would work fine
as well.
I remember reading about making the version field a numeric DV for that (or
similar) purpose, so apologies if I re-iterate someone else's idea...
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952494#comment-14952494 ] Yonik Seeley commented on SOLR-5944: bq. The issue then is that the version field is not 'updateable' right? Well, that was one issue we identified earlier and decided we would move to docvals for that field. It's a good idea in any case. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
