[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya updated SOLR-5944:
---
Attachment: regular-vs-dv-updates.png
master-vs-5944-regular-updates.png
I did some multithreaded benchmarks on the jira/solr-5944 branch. Here are the
two main experiments I performed:
h2. Regular update vs. In-Place updates on branch
First add 100,000 documents. Each document contains an numeric id field, a
numeric version field, a text field with around 1000 words (generated using
lucene-test-framework's {{TestUtil.randomSimpleString()}}), a stored+indexed
long field (called stored_l) and a non-stored, non-indexed long DV field
(called inplace_dvo_l).
Then, there were 10 iterations of 25,000 updates to each of the two long
fields. That is, 25k updates to stored_l, then 25k to inplace_dvo_l, and repeat
this 10 times. Used a CUSC for sending these updates, using a configurable
thread count.
Repeated this with different values of thread count to control the parallelism
of requests. Recorded and plotted the cumulative times per field:
!regular-vs-dv-updates.png!
h2. Only regular updates: master branch vs. 5944 branch
To evaluate any impact to regular updates, I performed the same experiment as
above, but with the following change: only update the stored_l field in every
iteration. Carried out this experiment on master as well as on jira/solr-5944
branch.
!master-vs-5944-regular-updates.png!
h2. Conclusion
# It seems the in-place updates are much faster than regular updates, esp. when
the document contains text fields. (Hypothesis: speed of in-place updates is
not proportional to document size)
# It seems that there is a very slight, but not significant, slowdown for
regular updates (master vs branch).
h2. Reproducing these results
The solr-upgrade-tests (SOLR-8581) seemed to be easy to extend for these
benchmarks. It takes in a git commit sha, checks out the repository, builds a
package, starts zookeeper and solr, performs the benchmarks, stops and cleans
up.
https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md
For these tests, I used the following commits:
master: ca50e5b61c2d8bfb703169cea2fb0ab20fd24c6b
jira/solr-5944: fcf71e34f20ea74f99933b80d5bd43cd487751f1
For the second experiment, I passed in an additional parameter
{{-onlyRegularUpdates true}}.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: defensive-checks.log.gz,
> demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch,
> DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt, master-vs-5944-regular-updates.png,
> regular-vs-dv-updates.png, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: (was: SOLR-5944.patch) > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updating the patch (from jira/solr-5944 branch). Tested with the PointFields fixes for SOLR-9996 and SOLR-10011, which are both in master now. Unless someone has any suggestions / objections, I plan to commit this (or perhaps squash merge branch jira/solr-5944) to master soon. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Here's the latest patch from the jira/solr-5944 branch. Steve's jenkins has been running all tests on the branch and they seem to pass fine [0]. This is very close now, and barring new issues / review comments / suggestions, this patch can be committed. [0] - http://jenkins.sarowe.net/job/Solr-tests-SOLR-5944/ > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: defensive-checks.log.gz, > demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch, > DUP.patch, hoss.62D328FA1DEA57FD.fail2.txt, hoss.62D328FA1DEA57FD.fail3.txt, > hoss.62D328FA1DEA57FD.fail.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya updated SOLR-5944:
---
Attachment:
demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch
{quote}
that doesn't really answer my question at all: why does it matter if it's a
dynamic field?
{quote}
If a document's dv field is updated for the first time (i.e. that field never
existed before in the document before), then the in-place update succeeds if it
is an explicit field but fails if it is a dynamic field.
It fails with this error:
{code}
Caused by: java.lang.IllegalArgumentException: can only update existing
docvalues fields! field=abc_f_dvo, type=NUMERIC
at
org.apache.lucene.index.IndexWriter.updateDocValues(IndexWriter.java:1715)
at
org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:875)
at
org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:279)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:213)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:168)
... 57 more
{code}
The reason for this is that when a document is created, DV fields for all
explicit fields are created. But when a non-existent dynamic field is attempted
to be updated subsequently, then the underlying DV field doesn't exist.
Attached a patch to demonstrate this. To demonstrate this, I've disabled the
schema.isDynamicField() check and also another check that aborts the in-place
updating for non-existent fields. (Fyi, the latter check is actually erroneous
and needs to be re-written as I mentioned in the comment
https://issues.apache.org/jira/browse/SOLR-5944?focusedCommentId=15729798=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15729798)
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz,
> demo-why-dynamic-fields-cannot-be-inplace-updated-first-time.patch,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-5944:
---
Attachment: SOLR-5944.patch
The only diff between this patch and the previous one is some improvements to
TestInPlaceUpdatesStandalone and some neccessary additions to
schema-inplace-updates.xml to support them...
* replace hackish getFieldValueRTG with a regular SolrClient.getById
simplifying most caller code
* refactor getFieldValueIndex / getDocId
** only usaage of getFieldValueIndex was in getDocId or places that should have
been calling getDocId instead
** refactored logic to use SolrClient.query instead of hackish low level access
* added new randomized test methods leveraging checkReply
** this helped uncover a few minor bugs in checkReplay which i fixed
** in the process I also cleaned up the datatypes used in the existing callers
of checkReply to reduce a lot of String/Number parsing/formatting logic in
checkReply
** this uncovered a ClassCastException when inplace updates are mixed with
non-inplace atomic updates (see below)
*** added testReplay_MixOfInplaceAndNonInPlaceAtomicUpdates to demonstrate this
directly
* new schema assertions warranted by fields added for the above changes
Here's an example of the ClassCastException that shows up in the logs when
running testReplay_MixOfInplaceAndNonInPlaceAtomicUpdates ...
{noformat}
[junit4] 2> 2514 ERROR
(TEST-TestInPlaceUpdatesStandalone.testReplay_MixOfInplaceAndNonInPlaceAtomicUpdates-seed#[70DBFB363B6DA180])
[] o.a.s.h.RequestHandlerBase java.lang.ClassCastException:
org.apache.solr.common.SolrDocument cannot be cast to
org.apache.solr.common.SolrInputDocument
[junit4] 2>at
org.apache.solr.handler.component.RealTimeGetComponent.getInputDocumentFromTlog(RealTimeGetComponent.java:512)
[junit4] 2>at
org.apache.solr.handler.component.RealTimeGetComponent.getInputDocument(RealTimeGetComponent.java:568)
[junit4] 2>at
org.apache.solr.handler.component.RealTimeGetComponent.getInputDocument(RealTimeGetComponent.java:546)
[junit4] 2>at
org.apache.solr.update.processor.DistributedUpdateProcessor.getUpdatedDocument(DistributedUpdateProcessor.java:1424)
[junit4] 2>at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1072)
[junit4] 2>at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:751)
[junit4] 2>at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
[junit4] 2>at
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.handleAdds(JsonLoader.java:492)
[junit4] 2>at
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:139)
[junit4] 2>at
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:115)
[junit4] 2>at
org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:78)
[junit4] 2>at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
[junit4] 2>at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
[junit4] 2>at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:152)
[junit4] 2>at
org.apache.solr.core.SolrCore.execute(SolrCore.java:2228)
[junit4] 2>at
org.apache.solr.servlet.DirectSolrConnection.request(DirectSolrConnection.java:124)
[junit4] 2>at
org.apache.solr.SolrTestCaseJ4.updateJ(SolrTestCaseJ4.java:1173)
[junit4] 2>at
org.apache.solr.SolrTestCaseJ4.addAndGetVersion(SolrTestCaseJ4.java:1319)
[junit4] 2>at
org.apache.solr.update.TestInPlaceUpdatesStandalone.checkReplay(TestInPlaceUpdatesStandalone.java:823)
[junit4] 2>at
org.apache.solr.update.TestInPlaceUpdatesStandalone.testReplay_MixOfInplaceAndNonInPlaceAtomicUpdates(TestInPlaceUpdatesStandalone.java:570)
{noformat}
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch,
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-5944:
---
Attachment: SOLR-5944.patch
Ok -- i've been reviewing Ishan's latest patch.
In a conversation I had with him offline, he mentioned that he hadn't been
doing any further work on this locally since his last update, so w/o any risk
of collision I went ahead and made a bunch of small updates directly...
{panel:title=Small changes I made directly in latest patch}
* misc javadoc tweaks & improvements
* some logging cleanup & template improvements
** Anytime we log anything that includes a complex object (that we expect to be
toString()-ified) is a situation where we should _definitely_ be using
templated log messages to avoid potentially expensive calls to toString() in
the case that the log level results in the method being a No-Op.
** I probably missed some, but i tried to catch as many as i could
* some simple method renames:
** UpdateLog.addUpdateCommandFromTlog -> convertTlogEntryToAddUpdateCommand
*** older name was ambigiuous because of dual meaning of "add", now verb is
clear.
** AtomicUpdateDocumentMerger.getSearcherNonStoredDVs ->
getNonStoredDocValueFieldNamesFromSearcher
* refactored some duplicate code into new helper methods:
** DirectUpdateHandler2.updateDocOrDocValues
* refactor some loops where methods were being called unneccessarily in every
iteration
** DocumentBuilder.toDocument
** AtomicUpdateDocumentMerger.isInPlaceUpdate (IndexSchema.getUniqueKeyField
and IndexWriter.getFieldNames())
* added some comments containing important details/reasoning that had only been
captured in jira comments so far
** AtomicUpdateDocumentMerger.doInPlaceUpdateMerge
* increased usage of expectThrows, beefed up asserts related to expected
Exceptions
* added some nocommit comments regarding some questionable lines of code
(either new questions, or things I've provided feedback on before that keep
slipping through the cracks)
* updated to master/98f75723f3bc6a718f1a7b47a50b820c4fb408f6
{panel}
More in-depth/concrete responses/comments to the previous patch are listed
below -- anything mentioned below that still seems problematic to me should
also have nocommit comments in the code associated with them, but i wanted to
elaborate a bit here in the jira comments.
(The reverse is not neccessarily true however: I added nocommits for some small
things that I didn't bother to add jira comments aboute)
{panel:title=schema.xml}
FWIW: I was initially concerned that change all these _existing_ dynamicFields
from indexed=true to indexed=false might be weakening other tests, but I did a
quick grep for their usage in other test methods (prior to this jira) and the
only impacts i found were in AtomicUpdatesTest, where the only concern was
docValues=true and stored=false. So changing these to indexed=false doesn't
impact that test at all (they are never queried against)
Based on the "dvo" naming, (which i think a reasonable person would read as
"doc values only") making these fields indexed=false seems like a good idea
{panel}
{panel:title=DistributedUpdateProcessor}
* In general, i think there's a gap in the DUP logic related to how the results
of fetchFullUpdateFromLeader are used when replicas never get dependent updates
** see detailed nocommit comments in the body of fetchFullUpdateFromLeader for
details
*** they apply to both call stacks where the method is used, but i wnated ot
write it once in one place
* versionAdd
** {{if (fetchedFromLeader == null)}}
*** I'm a little concerned/confused by the "else" block of this condition
grep code for "nocommit: OUTER IF", "nocommit: OUTER ELSE" & "nocommit:
INNER ELSE"
*** at this point, we've replaced our "in-place" update with a regular update
using the full SolrInputDocument from the leader (by overwritting cmd.solrDoc,
cmd.prevVersion, cmd.setVersion(...))
*** but this if/else is itself wrapped in a bigger {{if
(cmd.isInPlaceUpdate())}} whose else clause does some sanity checking /
processing logic for "// non inplace update, i.e. full document update"
*** shouldn't the logic in that outer else clause also be applied to our "no
longer an in-place" update?
* fetchFullUpdateFromLeader
** this method still calls {{forceUpdateCollection}} ... note shalin's comments
back in August...{quote}
Under no circumstances should we we calling `forceUpdateCollection` in the
indexing code path. It is just too dangerous in the face of high indexing
rates. Instead we should do what the DUP is already doing i.e. calling
getLeaderRetry to figure out the current leader. If the current replica is
partitioned from the leader then we have other mechanisms for taking care of it
and the replica has no business trying to determine this.
{quote}
{panel}
{panel:title=DirectUpdateHandler2}
* I previously asked about LUCENE-7344...{quote}
* so what's
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Added another test, a specific unit test for AUDM's doInPlaceUpdateMerge(). > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Added another patch. The PeerSyncTest was failing, due to fingerprint caching issue. This patch now depends on SOLR-9506's "SOLR-9506-combined-deletion-key.patch" patch. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Some minor changes: 1. Split the UpdateLogTest to smaller tests, more readable. 2. Added a javadoc comment to DUP. 3. Combined the testReplay1, 2, 3 into a single test. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Some minor changes: 1. Split the UpdateLogTest to smaller tests, more readable. 2. Added a javadoc comment to DUP. 3. Combined the testReplay1, 2, 3 into a single test. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch New patch. # Implements the Solution 1 approach from my above comment on the DBQs issue (with a slight difference: instead of pulling the document with the given version from the leader, I am pulling the latest document that there is on the leader). # Fixing many TODO review comments from Hoss. # TestStressInPlaceUpdates, TestInPlaceUpdatesDistrib and TestInPlaceUpdatesStandalone all seem to be passing regularly. # Fixed for precommit failures. [~hossman], can you please review? > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya updated SOLR-5944:
---
Attachment: SOLR-5944.patch
Updated patch, brought up to master. Here are my replies inline (not all of
them, but I'll keep editing this comment to provide all replies).
Ok -- it took a while, but here's my notes after reviewing the latest patch
{panel:title=DistributedUpdateProcessor}
* waitForDependentUpdates
** I know you & shalin went back and forth a bit on the wait call (ie:
wait(100) with max retries vs wait(5000)) but i think the way things settled
out {{bucket.wait(waitTimeout.timeLeft(TimeUnit.MILLISECONDS));}} would be
better then a generic {{wait(5000)}} [Ishan]FIXED[/Ishan]
*** consider the scenerio where: the dependent update is never going to come; a
spurious notify/wake happens during the first "wait" call @ 4950ms; the
lookupVersion call takes 45ms. Now we've only got 5ms left on our original
TimeOut, but we _could_ wind up "wait"ing another full 5s (total of 10s) unless
we get another spurrious notify/wake inthe mean time.
** {{log.info("Fetched the update: " + missingUpdate);}} that's a really good
candidate for templating since the AddUpdateCommand.toString() could be
expensive if log.info winds up being a no-op (ie: {{log.info("Fetched the
update: \{\}", missingUpdate);}}) [Ishan]FIXED[/Ishan]
* fetchMissingUpdateFromLeader
** In response to a previous question you said...{quote}
[FIXED. Initially, I wanted to fetch all missing updates, i.e. from what we
have till what we want. Noble suggested that fetching only one at a time makes
more sense.]
{quote} ... but from what i can tell skimming RTGC.processGetUpdates() it's
still possible that multiple updates will be returned, notably in the case
where: {{// Must return all delete-by-query commands that occur after the first
add requested}}. How is that possibility handled in the code paths that use
fetchMissingUpdateFromLeader?
*** that seems like a scenerio that would be really easy to test for -- similar
to how outOfOrderDeleteUpdatesIndividualReplicaTest works
** {{assert ((List) missingUpdates).size() == 1: "More than 1 update ...}}
*** based on my skimming of the code, an empty list is just as possible, so the
assertion is missleading (ideally it should say how many updates it got, or
maybe toString() the whole List ?)
{panel}
{panel:title=AtomicUpdateDocumentMerger}
* isSupportedFieldForInPlaceUpdate
** javadocs
* getFieldNamesFromIndex
** javadocs
** method name seems VERY missleading considering what it does [Ishan]Changed
it to getSearcherNonStoredDVs[/Ishan]
* isInPlaceUpdate
** javadocs should be clear what hapens to inPlaceUpdatedFields if result is
false (even if answer is "undefined"
** based on usage, wouldn't it be simplier if instead of returning a boolean,
this method just returned a (new) Set of inplace update fields found, and if
the set is empty that means it's not an in place update? [Ishan]FIXED[/Ishan]
** isn't getFieldNamesFromIndex kind of an expensive method to call on every
AddUpdateCommand ?
*** couldn't this list of fields be created by the caller and re-used at least
for the entire request (ie: when adding multiple docs) ? [Ishan]The set
returned is precomputed upon the opening of a searcher. The only cost I see is
to create a new unmodifiableSet every time. I'd prefer to take up this
optimization later, if needed.[/Ishan]
** {{if (indexFields.contains(fieldName) == false &&
schema.isDynamicField(fieldName))}}
*** why does it matter one way or the other if it's a dynamicField?
[Ishan]Changed the logic to check in the IW for presence of field. Added a
comment: "// if dynamic field and this field doesn't exist, DV update can't
work"[/Ishan]
** the special {{DELETED}} sentinal value still isn't being checked against the
return value of {{getInputDocumentFromTlog}} [Ishan]Not using
getInputDocumentFromTlog call anymore[/Ishan]
** this method still seems like it could/should do "cheaper" validation (ie:
not requiring SchemaField object creation, or tlog lookups) first. (Ex: the
set of supported atomic ops are checked after isSupportedFieldForInPlaceUpdate
& a possible read from the tlog). [Ishan]FIXED[/Ishan]
*** My suggested rewrite would be something like...{code}
Set candidateResults = new HashSet<>();
// first pass, check the things that are virtually free,
// and bail out early if anything is obviously not a valid in-place update
for (String fieldName : sdoc.getFieldNames()) {
if (schema.getUniqueKeyField().getName().equals(fieldName)
|| fieldName.equals(DistributedUpdateProcessor.VERSION_FIELD)) {
continue;
}
Object fieldValue = sdoc.getField(fieldName).getValue();
if (! (fieldValue instanceof Map) ) {
// not even an atomic update, definitely not an in-place update
return Collections.emptySet();
}
// else it's a
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: defensive-checks.log.gz Here's a log for the failure that I am looking into. This happened around once in 200 runs of the stress test. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, defensive-checks.log.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updated patch. Fixes the nocommits, test for reordered DBQs (which now can potentially throw a replica into LIR). There are occasional failures with the TestStressInPlaceUpdates test, that are basically due to: "ClusterState says we are the leader, but locally we don't think so. Request came from null". I'm looking into them. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch New patch. # Fixed the DBQ by updated value, as per Yonik's suggestion. (Thus working around LUCENE-7344) # With reordered DBQs, [~shalinmangar] pointed out a scenario whereby a document that needs in-place update cannot be "resurrected" if it has been deleted by an out of order DBQ. TODO: I'm working on the fix and the test for this. # TODO: Fix a few nocommits in this patch. # TODO: In-line reply to Hoss' suggestions. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Attaching a patch that is based on the previous patch, just adding a new test: TestInPlaceUpdatesStandalone#testDVUpdatesWithDBQofUpdatedValue() demonstrating the DBQ issue. Meanwhile, I'm working on another patch incorporating Hoss' and Shalin's review comments, and shall post it soon. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Added tests for the two scenarios described in the previous comment. # TestInPlaceUpdatesDistrib.outOfOrderDeleteUpdatesIndividualReplicaTest() for the scenario where in-place update arrives after a delete. # Added another scenario to TestInPlaceUpdatesDistrib.delayedReorderingFetchesMissingUpdateFromLeaderTest() where a missing update needs to be fetched from leader, even though the document itself has been deleted at the leader. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch bq. it is acquired by indexing threads who only want to read stuff from update log Ah, I see. That makes it clear for me; I've updated the patch now and have done the wait loop as per your suggestion. bq. In your latest patch, acquiring the read lock to call versionAdd is not necessary – it will do that anyway. You can re-acquire it for reading the version after the method call returns. Done, removed. bq. I don't think the case of vinfo.lookupVersion returning a negative value (for deletes) is handled here at all. Indeed, I hadn't handled it! I've now handled it in this new patch. bq. What happens if the document has been deleted already (due to reordering on the replica) when you enter waitForDependentUpdates? i.e. what if re-ordering leads to new_doc (v1) -> del_doc (v3) -> dv_update (v2) on the replica? I think it should now be fixed: v2 update would be dropped since abs(v3) > v2 (earlier I was just checking v3 > v2, which didn't consider negative versions). I'll add a test for this. bq. Similarly, how do we handle the case when the doc has been deleted on the leader when you execute fetchMissingUpdateFromLeader. Does RTG return the requested version even if the doc has been deleted already? I suspect it does but be nice to confirm. I think as long as a particular update for the given version is present at the leader's ulog/tlog, it will be returned. I will add a test for this scenario specifically anyway. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Thanks Shalin, I've fixed it in my updated patch, mostly on the lines of what you suggested in that snippet. Can you please take a look? Btw, I've surrounded only the lookupVersion() calls with the acquiring and releasing of the lock, instead of surrounding the entire wait loop with the acquiring/releasing of the lock: I reasoned that while we are waiting in that wait loop, other threads need to have indexed the update that we're waiting on, and hence I released the lock as soon as it was not needed, only to re-acquire it after 100ms. Does that sound like a valid reason? bq. Secondly, this method can be made more efficient. It currently wakes up every 100ms and reads the new "lastFoundVersion" from the update log or index. This is wasteful. A better way would be to wait for the timeout period directly after calling vinfo.lookupVersion() inside the synchronized block. Since this method enters the wait loop for every in-place update that has arrived out of order at a replica (an event, that I think is frequent under multithreaded load), I don't want every such update to be waiting for the full timeout period (5s here), but instead check back from time to time. In most of the cases, the dependent update would've been written (by another thread) within the first 100ms, after which we can bail out. Do you think that makes sense? > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Another TODO item got missed out. # Refactored calls like entry.get(1) etc. (for entries fetched from the ulog/tlog) to entry.get(UpdateLog.VERSION_IDX). > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updated the patch: # Fixed a bug with mixing atomic and in-place updates. Problem was that after in-place update, the RTGC.getInputDocument() got only the partial document, and hence further atomic updates on it failed. Changed this to return a "resolved" document for use during atomic update. # Added direct unit tests for AUDM.isInPlaceUpdate() at TestInPlaceUpdatesCopyField.java and UpdateLogTest. applyPartialUpdates() at UpdateLogTest.java. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Thanks Shalin. Updated the patch to include a test for the UpdateLog's applyPartialUpdates method. Also, removed a possibly erroneous optimization from DUP that I introduced in my last patch (an if check for lastDependentVersion == -1 inside the synchronization block). > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-5944: Attachment: SOLR-5944.patch Patch updated to master after resolving conflicts with the SOLR-9285 commit. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updated patch: # Refactored the logic to add a previous version to an AddUpdateCommand, moved it to DUP from JavabinLoader/XMLLoader # Updated javadocs to make them more detailed. Added javadocs to some related existing methods in VersionInfo. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya updated SOLR-5944:
---
Attachment: SOLR-5944.patch
New patch fixing all nocommits. Still a few additional tests, which Hoss
mentioned, are TODO. Here's a stab at replying to Hoss' comments (Maybe I'll
keep updating this comment itself as and when I fix some of the TODO items
here):
{panel:title=JettySolrRunner}
* javadocs, javadocs, javadocs [FIXED]
{panel}
{panel:title=XMLLoader + JavabinLoader}
* why is this param checks logic duplicated in these classes? [Not sure what
you mean here, I just set the prevVersion to the cmd here now]
* why not put this in DUP (which already has access to the request params) when
it's doing it's "FROMLEADER" logic? [Since commitWithin and overwrite was being
set here, I thought this is an appropriate place to set the prevVersion to the
cmd]
{panel}
{panel:title=AddUpdateCommand}
* these variables (like all variables) should have javadocs explaining what
they are and what they mean [FIXED]
** people skimming a class shouldn't have to grep the code for a variable name
to understand it's purpose
* having 2 variables here seems like it might be error prone? what does it
mean if {{prevVersion < 0 && isInPlaceUpdate == true}} ? or {{0 < prevVersion
&& isInPlaceUpdate == false}} ? [FIXED: Now just have one variable]
** would it make more sense to use a single {{long prevVersion}} variable and
have a {{public boolean isInPlaceUpdate()}} that simply does {{return (0 <
prevVersion); }} ? [FIXED]
{panel}
{panel:title=TransactionLog}
* javadocs for both the new {{write}} method and the existig {{write}} method
[FIXED]
** explain what "prevPointer" means and note in the 2 arg method what the
effective default "prevPoint" is.
* we should really have some "int" constants for refering to the List indexes
involved in these records, so instead of code like {{entry.get(3)}} sprinkled
in various classes like UpdateLog and PeerSync it can be smething more readable
like {{entry.get(PREV_VERSION_IDX)}} [TODO]
{panel}
{panel:title=UpdateLog}
* javadocs for both the new {{LogPtr}} constructure and the existing
constructor [FIXED]
** explain what "prevPointer" means and note in the 2 arg constructure what the
effective default "prevPoint" is. [FIXED]
* {{add(AddUpdateCommand, boolean)}}
** this new code for doing lookups in {{map}}, {{prevMap}} and {{preMap2}}
seems weird to me (but admitedly i'm not really an expert on UpdateLog in
general and how these maps are used
** what primarily concerns me is what the expected behavior is if the "id"
isn't found in any of these maps -- it looks like prevPointer defaults to "-1"
regardless of whether this is an inplace update ... is that intentional? ... is
it possible there are older records we will miss and need to flag that? [Yes,
this was intentional, and I think it doesn't make any difference. If an "id"
isn't found in any of these maps, it would mean that the previous update was
committed and should be looked up in the index. ]
** ie: do we need to worry about distinguising here between "not an in place
update, therefore prePointer=-1" vs "is an in place update, but we can't find
the prevPointer" ?? [I think we don't need to worry. Upon receiving a
prevPointer=-1 by whoever reads this LogPtr, it should be clear why it was -1:
if the command's {{flags|UpdateLog.UPDATE_INPLACE}} is set, then this command
is an in-place update whose previous update is in the index and not in the
tlog; if that flag is not set, it is not an in-place update at all, and don't
bother about the prevPointer value at all (which is -1 as a dummy value).]
** assuming this code is correct, it might be a little easier to read if it
were refactored into something like:{code}
// nocommit: jdocs
private synchronized long getPrevPointerForUpdate(AddUpdateCommand cmd) {
// note: sync required to ensure maps aren't changed out form under us
if (cmd.isInPlaceUpdate) {
BytesRef indexedId = cmd.getIndexedId();
for (Map currentMap : Arrays.asList(map, prevMap,
prevMap2)) {
LogPtr prevEntry = currentMap.get(indexedId);
if (null != prevEntry) {
return prevEntry.pointer;
}
}
}
return -1; // default when not inplace, or if we can't find a previous entry
}
{code} [FIXED: Refactored into something similar to above]
* {{applyPartialUpdates}}
** it seems like this method would be a really good candidate for some direct
unit testing? [TODO]
*** ie: construct a synthetic UpdateLog, and confirm applyPartialUpdates does
the right thing
** the sync block in this method, and how the resulting {{lookupLogs}} list is
used subsequently, doesn't seem safe to me -- particularly the way
{{getEntryFromTLog}} calls incref/decref on each TransactionLog as it loops
over that list...
*** what prevents some
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch # Added Javadocs, fixed some nocommits # Addressed an issue due to which in-place updating of non-existing DVs was throwing exceptions. For this, it was needed to know which fields have already been added to the index, so that if an update is needed to non-existent DV, then we can resort to a traditional full document atomic update. This check could've been easy if access to IW.globalFieldNumberMap was possible publicly. Instead resorted to checking with the RT searcher's list of DVs, and if field not found there then getting the document from tlog (RTG) and checking if the field exists in that document. # Added some simple tests to PeerSyncTest and TestRecovery for in-place updates. # TODO: A few more tests remain, few nocommits remain (mostly test code related). > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch New patch: # Addressed most of code level comments from Hoss. I think the readability of the code has now improved and also more robust exception handling. (Thanks Hoss). I shall reply inline to all of your suggestions, perhaps along with my next patch. # Nocommits remain, I think these are all related to Javadocs. # Stress test now has DBQ and DBI. It seems to pass 1000 runs successfully. # PeerSync, TestReplay etc. and a few more unit tests remain. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch New patch. # Increasing the timeout of the threadpool termination in TestInPlaceUpdatesDistrib from 2s to 5s, and added assertion message. # Fixing some review comments in the TestStressInPlaceUpdates (more remains to be fixed). > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch New patch. Fixes to TestInPlaceUpdatesStandalone (formerly TestInPlaceUpdate). TODO items in the test are: # Randomize the replay based tests. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch New patch. Fixed all comments in TestInPlaceUpdatesDistrib (formerly InPlaceUpdateDistribTest). > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updated the patch. Fixed some issues from Hoss' comments. Some nocommits are remaining. I'll reply to Hoss' suggestions in-line shortly. Couple of error handling fixes, but mostly changes to test suites. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt, > TestStressInPlaceUpdates.eb044ac71.failures.tar.gz, > hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt, > hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt, > hoss.D768DD9443A98DC.pass.txt > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-5944:
---
Attachment: SOLR-5944.patch
Ugh, testReplay4 in my last patch was a waste of time -- somehoe i screwed
something up with copy/paste.
The point was to try and check if using "set" operations had any different
effect on the test compared to using "inc" operations (like in testReplay3)
This patch fixes that, so that both tests demonstrate the problem with the seed
ishan mentioned (and aren't identical) ... but more interesting then that is
the new testReplay5, testReplay6 & testReplay7 (still using hte same seed)...
* testReplay5 - still uses "inc" for doc id=0, but uses "set" for every other
doc in the index
** this currently fails with an NPE in
{{AtomicUpdateDocumentMerger.doInPlaceUpdateMerge(AtomicUpdateDocumentMerger.java:283)}}
* testReplay6 was a quick hack to work around the NPE in testReplay5 - ensuring
that the first time a doc is added, a regular "add" is done, but everytime
after that "set" is used (except for id=0 where "inc" is still used)
** this test currently passes -- even though it should be effectively the same
as testReplay3 and testReplay4
* testReplay7 is the same as testReplay6 but (like testReplya4) does all the
id=0 updates using "set" as well just to sanity check no diff in behavior
** this also passes
...so based on the fact that testReplay6/7 pass, but testReplay3/4 fail, any
ishan's observations about the setMaxBufferedDocs my suspicion is that the
underlying problem probably has something to do with how frequently docs are
flushed and trying to read back "updated" DV values when other docs were added
with regular DV values i nte hsame field just before/after the DV update.
(looking at TestNumericDocValuesUpdates i don't see many tests that mix calls
to addDocument with calls to updateNumericDocValue -- except testSegmentMerges
& testManyReopensAndFields which update the DV of *every* document added up to
that point)
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-5944:
---
Attachment: SOLR-5944.patch
minor tweaks to ishan's last patch...
* updated to master (some compliation failures due to other jiras getting
commited)
* refactored TestInPlaceUpdate
** testReplay3 & testReplay4 reproduce the previously mentioned failure that
required using the external file.
** you can still refer to external files i nteh same format via
"testReplayFromFile" but only if a sys prop is set...{noformat}
ant test -Dtestcase=TestInPlaceUpdate -Dtests.seed=1D9D6101E3D231FF
-Dtests.verbose=true -Dtests.method=testReplayFromFile
-Dargs="-Dtests.inplace.file=/home/hossman/lucene/dev/operations3.txt"
{noformat}
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya updated SOLR-5944:
---
Attachment: SOLR-5944.patch
Attaching fresh patch from the
https://github.com/chatman/lucene-solr/tree/solr_5944 branch.
I've added a test based on SolrTestCaseJ4, which fails, despite suppressing the
Lucene54 and Memory codecs. I am now trying to reproduce this issue in a
LuceneTestCase test.
https://github.com/chatman/lucene-solr/commit/b80be98d9e1a98b97bee670a2d775f6acc2182c7
Couple things I noticed that this happens when Compressing codec is used, along
with the "crazy" value of max buffered docs is used as per this part from the
randomization in LuceneTestCase.java
{code}
if (r.nextBoolean()) {
if (rarely(r)) {
// crazy value
c.setMaxBufferedDocs(TestUtil.nextInt(r, 2, 15));
} else {
// reasonable value
c.setMaxBufferedDocs(TestUtil.nextInt(r, 16, 1000));
}
}
{code}
However, I can't reproduce the failure based on just those two factors; there
must be something else that I'm unable to pinpoint.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Rowe updated SOLR-5944:
-
Attachment: TestStressInPlaceUpdates.eb044ac71.failures.tar.gz
Beasting found 6 more failures, for a total of 9.
I'm attaching a tar.gz file containing all 9 runs' logs (including those that
are separately attached).
Here are the repro lines (without the linedocs sysprop) for the 6 failing runs
not previously mentioned on this issue:
{noformat}
run 927:
ant test -Dtestcase=TestStressInPlaceUpdates -Dtests.method=stressTest
-Dtests.seed=D936BF4963826AEC -Dtests.slow=true -Dtests.locale=ja
-Dtests.timezone=GMT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
run 1246:
ant test -Dtestcase=TestStressInPlaceUpdates -Dtests.method=stressTest
-Dtests.seed=9717476E3589BED0 -Dtests.slow=true -Dtests.locale=pl
-Dtests.timezone=Europe/Samara -Dtests.asserts=true -Dtests.file.encoding=UTF-8
run 1543:
ant test -Dtestcase=TestStressInPlaceUpdates -Dtests.method=stressTest
-Dtests.seed=9D53690B34363B1C -Dtests.slow=true -Dtests.locale=es-NI
-Dtests.timezone=America/Argentina/San_Juan -Dtests.asserts=true
-Dtests.file.encoding=ISO-8859-1
run 1558:
ant test -Dtestcase=TestStressInPlaceUpdates -Dtests.method=stressTest
-Dtests.seed=655C990D41E11FF7 -Dtests.slow=true -Dtests.locale=en-NZ
-Dtests.timezone=MST -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
run 1604:
ant test -Dtestcase=TestStressInPlaceUpdates -Dtests.method=stressTest
-Dtests.seed=DD4A15BAF279FCB9 -Dtests.slow=true -Dtests.locale=de-AT
-Dtests.timezone=Pacific/Pitcairn -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
run 1854:
ant test -Dtestcase=TestStressInPlaceUpdates -Dtests.method=stressTest
-Dtests.seed=9D68461E2404622 -Dtests.slow=true -Dtests.locale=pl
-Dtests.timezone=Brazil/West -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
{noformat}
5 of the above 6 runs reproduced for me on the first try, but I could not
reproduce run 1246 (seed 9717476E3589BED0) - it succeeded in 4/4 attempts.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.failures.tar.gz,
> hoss.62D328FA1DEA57FD.fail.txt, hoss.62D328FA1DEA57FD.fail2.txt,
> hoss.62D328FA1DEA57FD.fail3.txt, hoss.D768DD9443A98DC.fail.txt,
> hoss.D768DD9443A98DC.pass.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-5944:
---
Attachment: hoss.62D328FA1DEA57FD.fail3.txt
hoss.62D328FA1DEA57FD.fail2.txt
hoss.62D328FA1DEA57FD.fail.txt
hoss.D768DD9443A98DC.fail.txt
hoss.D768DD9443A98DC.pass.txt
I've been reviewing the logs from sarowe's failures -- I won't pretend to
understand half of what i'm looking at here (I'm still not up on most of the
new code) but here's some interesting patterns i've noticed...
* in both failure logs posted, doc "13" was the doc having problems
* the specific docId is probably just a coincidence, but it does mean that the
same egrep command works on both log files to give you the particularly
interesting bits realtive to the failure...{noformat}
egrep add=\\[13\|id=13\|ids=13
TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt >
beast-167.important.txt
egrep add=\\[13\|id=13\|ids=13
TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt >
beast-587.important.txt
{noformat}
* looking first at beast-587.important.txt:
** the ERROR that failed the test was first logged by READER2 @ (timestamp)
34456:{noformat}
34456 ERROR (READER2) [] o.a.s.c.TestStressInPlaceUpdates Realtime=true,
ERROR, id=13 found={response={numFound=1,start=0,docs=[SolrDocument{id=13,
title_s=[title13], val1_i_dvo=3, val2_l_dvo=36,
_version_=1534607778351415296, ratings=0.0, price=0}]}}
model=[1534607780231512064, 3, 300012]
{noformat}
** Working backwards, that expected version 1534607780231512064 was logged by
WRITER10 as being returned to a PARTIAL update @ 31219:{noformat}
31219 INFO (WRITER10) [] o.a.s.c.TestStressInPlaceUpdates PARTIAL: Writing
id=13, val=[3,300012], version=1534607779993485312, Prev
was=[3,39]. Returned version=1534607780231512064
{noformat}
*** WRITER10's logging of this "Returned version=1534607780231512064" came
after core_node1, core_node2, and core_node3 all logged it being written to
their TLOG & reported it via LogUpdateProc:{noformat}
30557 INFO (qtp2010985731-180) [n:127.0.0.1:37972__m c:collection1 s:shard1
r:core_node1 x:collection1] o.a.s.u.UpdateLog TLOG: added id
13(ver=1534607780231512064, prevVersion=1534607779993485312, prevPtr=2343) to
tlog{file=/tmp/beast-tmp-output/587/J0/temp/solr.cloud.TestStressInPlaceUpdates_FFC46C473EC471E6-001/shard-1-001/cores/collection1/data/tlog/tlog.004
refcount=1} LogPtr(2396) map=1977112331, actual doc=SolrInputDocument(fields:
[id=13, val2_l_dvo=300012, _version_=1534607780231512064, val1_i_dvo=3])
30589 INFO (qtp1755078679-232) [n:127.0.0.1:38407__m c:collection1 s:shard1
r:core_node2 x:collection1] o.a.s.u.UpdateLog TLOG: added id
13(ver=1534607780231512064, prevVersion=1534607779993485312, prevPtr=2343) to
tlog{file=/tmp/beast-tmp-output/587/J0/temp/solr.cloud.TestStressInPlaceUpdates_FFC46C473EC471E6-001/shard-2-001/cores/collection1/data/tlog/tlog.002
refcount=1} LogPtr(2396) map=1630836284, actual doc=SolrInputDocument(fields:
[id=13, val2_l_dvo=300012, _version_=1534607780231512064, val1_i_dvo=3])
30589 INFO (qtp1755078679-232) [n:127.0.0.1:38407__m c:collection1 s:shard1
r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]
webapp=/_m path=/update
params={update.distrib=FROMLEADER=http://127.0.0.1:37972/_m/collection1/=1534607779993485312=javabin=2=true}{add=[13
(1534607780231512064)]} 0 0
31216 INFO (qtp2143623462-144) [n:127.0.0.1:58295__m c:collection1 s:shard1
r:core_node3 x:collection1] o.a.s.u.UpdateLog TLOG: added id
13(ver=1534607780231512064, prevVersion=1534607779993485312, prevPtr=2343) to
tlog{file=/tmp/beast-tmp-output/587/J0/temp/solr.cloud.TestStressInPlaceUpdates_FFC46C473EC471E6-001/shard-3-001/cores/collection1/data/tlog/tlog.002
refcount=1} LogPtr(2396) map=1500522809, actual doc=SolrInputDocument(fields:
[id=13, val2_l_dvo=300012, _version_=1534607780231512064, val1_i_dvo=3])
31216 INFO (qtp2143623462-144) [n:127.0.0.1:58295__m c:collection1 s:shard1
r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]
webapp=/_m path=/update
params={update.distrib=FROMLEADER=http://127.0.0.1:37972/_m/collection1/=1534607779993485312=javabin=2=true}{add=[13
(1534607780231512064)]} 0 0
31219 INFO (qtp2010985731-180) [n:127.0.0.1:37972__m c:collection1 s:shard1
r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]
webapp=/_m path=/update params={versions=true=javabin=2}{add=[13
(1534607780231512064)]} 0 662
{noformat}
** but looking *after* the ERROR was first logged @ 34456, we see that before
the test had a chance to shut down all the nodes, there was some suspicious
looking logging from core_node2 regarding updates out of order, that refer to
the expected
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Rowe updated SOLR-5944:
-
Attachment: TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt
TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt
I'm beasting 2000 iterations of {{TestStressInPlaceUpdates}} with Miller's
beast script against https://github.com/chatman/lucene-solr/tree/solr_5944 at
revision eb044ac71 and have so far seen two failures, at iteration 167 and at
iteration 587, the stdout from which I'm attaching here.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> TestStressInPlaceUpdates.eb044ac71.beast-167-failure.stdout.txt,
> TestStressInPlaceUpdates.eb044ac71.beast-587-failure.stdout.txt
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Combined Justin's fixes, Hoss' fixes and Noble's fixes (which were already there), updated to master and committed to the solr_5944 branch (https://github.com/chatman/lucene-solr/tree/solr_5944). Attached the patch for the same here. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated SOLR-5944: --- Attachment: SOLR-5944.patch small update -- the previous patch was breaking QueryEqualityTest because of how it abused ValueSourceParser.addParser (SOLR-9119) to add a ValueSource parser to the static list. If this special ValueSource parser was critical to the test, it could have been registered in the solrconfig.xml used by the test, but it didn't appear to serve any purpose beyond what the DocIdAugmenterFactory already provided, so i replaced the existing usages with that transformer (if anything the old code could have suffered from false positives since the valuesource was returning the segment docId, but the transformer returns the id used by the top level SolrIndexSearcher( > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated SOLR-5944: --- Attachment: SOLR-5944.patch this is the most recent patch (attached by Noble on Mar31) updated to master (some hunk offset shifts and one trivial conflict) with Justin's precommit fixes added in. (precommit & the new test classes pass, still running fulltests to ensure no obvious problems, have not actually reviewed anything personally) > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noble Paul updated SOLR-5944:
-
Attachment: SOLR-5944.patch
fixed logging.
* It's not OK to cache the values of {{log.isDebugEnabled()}} etc. These can
chnange in between
* if a string is constructed, always do conditional logging
* if a template is used upto two variables eg: {{log.info("v1:{}, v2:{}", v1,v2
)}} , it is OK not to do conditional logging, because no new objects are created
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Thanks Noble, I've added your suggestion and your DUP patch to the full patch. Also did some cleanup, javadocs, removed some unnecessary logging. The corresponding change to the patch is here: https://github.com/chatman/lucene-solr/commit/f203d770dbdcb36edf472544902bc62bbbc28794 > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-5944: - Attachment: DUP.patch > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-5944: - Attachment: (was: DUP.patch) > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Thanks Noble, I added your changes to the full patch. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noble Paul updated SOLR-5944:
-
Attachment: DUP.patch
use a {{wait()/notify()}} instead of {{sleep()}}
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: DUP.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updating the patch to master, since parts of this patch are now committed as part of SOLR-8831 and SOLR-8865. Would be nice if someone could please review. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updated the patch to trunk with the following: 1. Some bug fixes. 2. Stress test. 3. Some cleanups. 4. Fixed nocommits. [~noble.paul], [~yo...@apache.org], could you please review? Thanks. There is one very very rare situation whereby after a commit the recently reopened searcher doesn't see a dv update which was supposed to be in this newly opened searcher. There maybe some issue in having the updates flushed properly at the Lucene level; I have so far been unable to reproduce this during one off plain Lucene tests or Solr tests, but it is exposed only during the Stress tests (once in 2000 runs or so). I'll continue to investigate this, perhaps by writing a similar stress test for plain Lucene level DV updates. Apart from that, I think this patch is feature complete. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updating the patch to latest master. # Added stress test, TestStressInPlaceUpdates, similar to TestStressReorder # Has lots and lots of unnecessary logging info, which I left there as I tried to reproduce a bug (LUCENE-7025) due ot which the stress test is failing. # Lots of nocommits due to formatting/refactoring/unnecessary logging etc. Currently, in the stress test, RTG works, but committed model / main searcher based search seems to be inconsistent, possibly due to the LUCENE-7025 issue or something to do with that. Seems to me that whatever is being written through an IndexWriter is not exactly what I search via searcher at the time when parallel commits are happening. I am still investigating why that could be. It seems the problem of reordered updates is solved (in terms of correctness). Also seems the RTG is working properly. I am currently looking into the searcher inconsistency while commit is happening. Once I get past that, I'll do a serious cleanup of the code and the logging messages I am printing out. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updating to trunk. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updating patch, depends on SOLR-8220. If waiting for pending/reordered dependent update fails, it tries to get the updates from the leader now (as per point 2 in above comment). Fixed a few nocommits. Still need to add the extensive stress tests, and maybe some more refactoring needed. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[
https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ishan Chattopadhyaya updated SOLR-5944:
---
Attachment: SOLR-5944.patch
Adding patch for the above approach.
# Added a test that times out a reordered update, when it exceeds 1s (currently
hardcoded), and puts the replica in LIR and subsequently performs a peer sync.
# Used a {{bufferingMap}} in UpdateLog, and populated it at
{{DistributedUpdateProcessor}} at a time when ulog is in buffering state.
TODOs:
# Remove the nocommits, and some of the TODOs.
# Add more tests.
> Support updates of numeric DocValues
>
>
> Key: SOLR-5944
> URL: https://issues.apache.org/jira/browse/SOLR-5944
> Project: Solr
> Issue Type: New Feature
>Reporter: Ishan Chattopadhyaya
>Assignee: Shalin Shekhar Mangar
> Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch,
> SOLR-5944.patch, SOLR-5944.patch
>
>
> LUCENE-5189 introduced support for updates to numeric docvalues. It would be
> really nice to have Solr support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Made some progress on this. * Reordered updates working. * Added test for reordered updates. (TODO: some more needed) * Fixed some issues with RealTimeGet of updates from my previous patch. Need to clean up the nocommits, add more tests. Also, need to deal with multivalued docvalues fields: my initial thought is to not support in-place partial updates to multivalued docvalues fields. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch WIP patch, updated to trunk. The logic to deal with reorders in a replica is still TODO. Also, another TODO: this doesn't deal with multivalued docvalues yet. > Support updates of numeric DocValues > > > Key: SOLR-5944 > URL: https://issues.apache.org/jira/browse/SOLR-5944 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, > SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch > > > LUCENE-5189 introduced support for updates to numeric docvalues. It would be > really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Here's a WIP strawman patch. * Using _version_ as docValue field. * Transaction log items have a previous pointer: to point from a partial update document to its previous version of the document in the tlog. * Realtime get works for partial updates. * TODO: Adding reordered updates in a follower replica doesn't work yet. (many of the supporting pieces to do this are there). * TODO: Lots of nocommits, which would require some careful treatment of docvalue types or need some refactoring. * TODO: Tests. Support updates of numeric DocValues Key: SOLR-5944 URL: https://issues.apache.org/jira/browse/SOLR-5944 Project: Solr Issue Type: New Feature Reporter: Ishan Chattopadhyaya Assignee: Shalin Shekhar Mangar Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch LUCENE-5189 introduced support for updates to numeric docvalues. It would be really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Bringing the last patch up to trunk, so that further work can be done on this issue. This still suffers from the potential consistency issue in replicas if updates are reordered, as Yonik mentioned. Support updates of numeric DocValues Key: SOLR-5944 URL: https://issues.apache.org/jira/browse/SOLR-5944 Project: Solr Issue Type: New Feature Reporter: Ishan Chattopadhyaya Assignee: Shalin Shekhar Mangar Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch LUCENE-5189 introduced support for updates to numeric docvalues. It would be really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updated the patch with the following changes: 1. Removed the script based updates 2. Added support for incrementing docvalues based on uncommitted documents (from transaction log) 3. Earlier, atomic updates were not carrying forward the non-stored, non-indexed docvalues to the new document. Added a fix for this. 4. Added tests for 3 and 4. Support updates of numeric DocValues Key: SOLR-5944 URL: https://issues.apache.org/jira/browse/SOLR-5944 Project: Solr Issue Type: New Feature Reporter: Ishan Chattopadhyaya Assignee: Shalin Shekhar Mangar Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch LUCENE-5189 introduced support for updates to numeric docvalues. It would be really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Added support for the inc update operator (increment the existing docValue by given amount). Support updates of numeric DocValues Key: SOLR-5944 URL: https://issues.apache.org/jira/browse/SOLR-5944 Project: Solr Issue Type: New Feature Reporter: Ishan Chattopadhyaya Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch LUCENE-5189 introduced support for updates to numeric docvalues. It would be really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updated the patch, added support for updating a docValue using a javascript expression which binds the old docValue, thus allowing users to use simple functions to compute the updated docValue. Support updates of numeric DocValues Key: SOLR-5944 URL: https://issues.apache.org/jira/browse/SOLR-5944 Project: Solr Issue Type: New Feature Reporter: Ishan Chattopadhyaya Assignee: Shalin Shekhar Mangar Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch LUCENE-5189 introduced support for updates to numeric docvalues. It would be really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updated the patch to add a test for update of numeric docValues in a distributed setup (1 shard, many replicas). Support updates of numeric DocValues Key: SOLR-5944 URL: https://issues.apache.org/jira/browse/SOLR-5944 Project: Solr Issue Type: New Feature Reporter: Ishan Chattopadhyaya Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch LUCENE-5189 introduced support for updates to numeric docvalues. It would be really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updated the patch to cover cases for replication of the in-place update. However, the patch still doesn't have tests for replication; working on it. Manually verified: after in-place updating a document in leader, the same document in a replica reflects the in-place update. TODO: 1) Write more tests 2) Support update of non-numeric fields using binaryDocValues Support updates of numeric DocValues Key: SOLR-5944 URL: https://issues.apache.org/jira/browse/SOLR-5944 Project: Solr Issue Type: New Feature Reporter: Ishan Chattopadhyaya Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch LUCENE-5189 introduced support for updates to numeric docvalues. It would be really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updating patch to trunk. Support updates of numeric DocValues Key: SOLR-5944 URL: https://issues.apache.org/jira/browse/SOLR-5944 Project: Solr Issue Type: New Feature Reporter: Ishan Chattopadhyaya Attachments: SOLR-5944.patch, SOLR-5944.patch LUCENE-5189 introduced support for updates to numeric docvalues. It would be really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Updating the patch, the test passes now! Thanks to [~shalinmangar] for pointing out that I wasn't suppressing the Lucene40, 41, .. 45 codecs in the test. I shall start working on addressing other issues with the patch. Support updates of numeric DocValues Key: SOLR-5944 URL: https://issues.apache.org/jira/browse/SOLR-5944 Project: Solr Issue Type: New Feature Reporter: Ishan Chattopadhyaya Attachments: SOLR-5944.patch, SOLR-5944.patch, SOLR-5944.patch LUCENE-5189 introduced support for updates to numeric docvalues. It would be really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5944) Support updates of numeric DocValues
[ https://issues.apache.org/jira/browse/SOLR-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-5944: --- Attachment: SOLR-5944.patch Here's an early patch where I'm just attempting to modify DirectUpdateHandler2 to do an in-place update to a document for which the only fields to update are stored=false, indexed=false, docValues=true. However, I'm stuck with the unit test for this. Whenever I run the unit test (added in the patch), about 40% of the times it passes, 40% of the time it fails (the default docvalues are used, not the updated ones) and 20% of the time I'm hitting the lucene3x codecs issue (as noted by Mikhail [0] in LUCENE-5189), even though I've explicitly suppressed lucene3x. When Solr runs as a server, I manually test that I'm able to update the NDVs via the update request handler's REST endpoint and updated values are immediately reflected for sorting queries. I am wondering if there is some fundamental issue with the approach taken in the patch, or if there's some issue with the test setup (or both). Help and feedback are most welcome! If I can get the tests to pass, I can begin to address various other corner cases and issues with replication etc. [0] - https://issues.apache.org/jira/browse/LUCENE-5189?focusedCommentId=13958205page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13958205 Support updates of numeric DocValues Key: SOLR-5944 URL: https://issues.apache.org/jira/browse/SOLR-5944 Project: Solr Issue Type: New Feature Reporter: Ishan Chattopadhyaya Attachments: SOLR-5944.patch LUCENE-5189 introduced support for updates to numeric docvalues. It would be really nice to have Solr support this. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
