[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451681#comment-16451681
]
ASF subversion and git services commented on SOLR-12065:
Commit 8894db1a727dff5a52444f9b4a5838995a8f7513 in lucene-solr's branch
refs/heads/branch_7_3 from [~varun_saxena]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8894db1 ]
SOLR-12065: A successful restore collection should mark the shard state as
active and not buffering
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Fix For: 7.4, master (8.0), 7.3.1
>
> Attachments: 12065.patch, 12605UTLogs.txt.zip, SOLR-12065.patch,
> SOLR-12065.patch, SOLR-12065.patch, SOLR-12065.patch, logs_and_metrics.zip,
> restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435972#comment-16435972
]
ASF subversion and git services commented on SOLR-12065:
Commit b25ae6779c9ad875eabd8b3e2a72b7108e740a33 in lucene-solr's branch
refs/heads/branch_7x from [~varun_saxena]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b25ae67 ]
SOLR-12065: A successful restore collection should mark the shard state as
active and not buffering
(cherry picked from commit 7a57ca8)
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Fix For: 7.4
>
> Attachments: 12065.patch, 12605UTLogs.txt.zip, SOLR-12065.patch,
> SOLR-12065.patch, SOLR-12065.patch, SOLR-12065.patch, logs_and_metrics.zip,
> restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435747#comment-16435747
]
ASF subversion and git services commented on SOLR-12065:
Commit 7a57ca8c0d10ceb23cad6fe9bc3538314ce6b6ce in lucene-solr's branch
refs/heads/master from [~varun_saxena]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7a57ca8c0 ]
SOLR-12065: A successful restore collection should mark the shard state as
active and not buffering
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: 12065.patch, 12605UTLogs.txt.zip, SOLR-12065.patch,
> SOLR-12065.patch, SOLR-12065.patch, SOLR-12065.patch, logs_and_metrics.zip,
> restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434857#comment-16434857
]
Varun Thacker commented on SOLR-12065:
--
Uploaded a new patch with a CHANGES entry . Running tests and precommit on this
patch before committing it
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: 12065.patch, 12605UTLogs.txt.zip, SOLR-12065.patch,
> SOLR-12065.patch, SOLR-12065.patch, SOLR-12065.patch, logs_and_metrics.zip,
> restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434833#comment-16434833
]
Varun Thacker commented on SOLR-12065:
--
{quote}With the latest patch , the HDFS backup test fails quite regularly on my
machine.
{quote}
Looking more closely at the logs, there were 3 replicas for the shard . And the
replica that got queried was a pull replica. I think I'll revert to querying
the leaders explicitly and counting the total
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: 12065.patch, 12605UTLogs.txt.zip, SOLR-12065.patch,
> SOLR-12065.patch, logs_and_metrics.zip, restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434763#comment-16434763
]
Varun Thacker commented on SOLR-12065:
--
{code:java}
for (Slice shard : restoreCollection.getSlices()) {
ocmh.waitForNewShard(restoreCollectionName, shard.getName());
}{code}
I don't think we need to explicitly do this in RestoreCmd. We call
createcollection command which internally wait's for the shard to become
active. So we are good here.
I removed this logging line
{code:java}
int numberOfActiveShards = restoreCollection.getActiveSlices().size();
log.info("Number of activeShards: " + numberOfActiveShards);{code}
Also in the test case I cleaned up how we are counting docs before and after
indexing
With the latest patch , the HDFS backup test fails quite regularly on my
machine.
here's what's happening:
- we add docs after the restore is complete
- call commit
- query to assert doc count. Now if the query hits a non-leader replica and
open searcher hasn't been called the replica gives the old count
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: 12065.patch, 12605UTLogs.txt.zip, SOLR-12065.patch,
> logs_and_metrics.zip, restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426778#comment-16426778
]
Rohit commented on SOLR-12065:
--
1. In AbstractCloudBackupRestoreTestCase we added a new method
indexNewDocsToCollection . Can't we reuse indexDocs ? We can make changes to
that method so that it's generally more reusable for your test
- Done
2. Maybe \{[getDocCountInCollection}} can be something like this since we are
counting the number of docs in a collection , so hitting any underlying node
will be fine
- Reused the existing getShardToDocCountMap function
3. RestoreCmd has a unused import. This will make {{ant precommit}} fail
* Removed the un-used import and verified with ant precommit
* [exec] Crawl/parse...
[exec]
[exec] Verify...
[echo] Checking for malformed docs...
precommit:
BUILD SUCCESSFUL
Total time: 16 minutes 30 seconds
4. Should both of the log.info lines added to RestoreCmd be debug level? Also
can we use parameterized logging ?
- Changed at one place and used parameterised logging
5. After the REQUESTAPPLYUPDATES command is sent , shouldn't we validate the
response? So something like this
{code:java}
ocmh.processResponses(new NamedList(), shardHandler, true,
"REQUESTAPPLYUPDATES calls did not succeed", asyncId, requestMap);{code}
* Done, added it to the RestoreCmd source.
Verified the build on Master and state in tlog.state for test_restore
collection using the admin/metrics API on two node Solr cluster
{{SEARCHER.searcher.searcherName":"Searcher@70ed14d[test_restore_shard1_replica_n47]
main",}}
{{ "SEARCHER.searcher.warmupTime":0,}}
{{..}}
{{ "TLOG.replay.remaining.bytes":0,}}
{{ "TLOG.replay.remaining.logs":0,}}
{{
"TLOG.state":3,}}{{"SEARCHER.searcher.registeredAt":"2018-04-05T11:24:20.667Z",}}
{{
"SEARCHER.searcher.searcherName":"Searcher@475f6bda[test_restore_shard2_replica_n45]
main",}}
{{ "TLOG.replay.remaining.bytes":104,}}
{{ "TLOG.replay.remaining.logs":1,}}
{{ "TLOG.state":3,}}
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: 12065.patch, 12605UTLogs.txt.zip, SOLR-12065.patch,
> logs_and_metrics.zip, restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420787#comment-16420787
]
Varun Thacker commented on SOLR-12065:
--
Lastly, if you upload the patch as SOLR-12065.patch , you can then hit the
submit patch button and the automated QA system will be able to validate your
patch as well. That's the general convention we follow for patch naming formats.
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: 12065.patch, 12605UTLogs.txt.zip, logs_and_metrics.zip,
> restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420785#comment-16420785
]
Varun Thacker commented on SOLR-12065:
--
Hi Rohit,
Thanks for the patch. This looks great!
Some feedback from the patch:
1. In AbstractCloudBackupRestoreTestCase we added a new method
indexNewDocsToCollection . Can't we reuse indexDocs ? We can make changes to
that method so that it's generally more reusable for your test
Also in general we don't need to do \{{Random random = new Random(docsSeed);}}
. We can do \{{Random r = random();}} because the test setup already has a
static method to get random.
2. Maybe \{[getDocCountInCollection}} can be something like this since we are
counting the number of docs in a collection , so hitting any underlying node
will be fine
{code}
private long getDocCountInCollection(String collectionName) throws
SolrServerException, IOException {
return cluster.getSolrClient().query(collectionName, new
SolrQuery("*:*")).getResults().getNumFound();
}
{code}
3. RestoreCmd has a unused import. This will make \{{ant precommit}} fail
4. Should both of the log.info lines added to RestoreCmd be debug level? Also
can we use parameterized logging ?
5. After the REQUESTAPPLYUPDATES command is sent , shouldn't we validate the
response? So something like this
{code}
ocmh.processResponses(new NamedList(), shardHandler, true,
"REQUESTAPPLYUPDATES calls did not succeed", asyncId, requestMap);
{code}
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: 12065.patch, 12605UTLogs.txt.zip, logs_and_metrics.zip,
> restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419749#comment-16419749
]
Mikhail Khludnev commented on SOLR-12065:
-
The patch seems great. Thanks, [~rohitcse]. [~varunthacker], would you mind to
skim through? If you approve, I'll proceed with commit.
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: 12065.patch, 12605UTLogs.txt.zip, logs_and_metrics.zip,
> restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417843#comment-16417843
]
Rohit commented on SOLR-12065:
--
Changed flow in RestoreCmd.java and improved the backup and restore test case:
# Post restoration index new docs to the restored collection
# Verify that the new documents are searchable and the original doc count +
newly indexed doc count matches are correct
Additional steps performed to test and
verify:[^12605UTLogs.txt.zip][^logs_and_metrics.zip]
1.
[http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=2=2=2=2]
2. curl http://127.0.0.1:8983/solr/test_backup/update?commit=true -H
'Content-type:application/json' -d '[ \{"id" : "1"}]'
3.
[http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/Rohit/Desktop/backup]
4.
[http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/Rohit/Desktop/backup=test_restore]
5. curl http://127.0.0.1:8983/solr/test_restore/update?commit=true -H
'Content-type:application/json' -d '[ \{"id" : "2"}]'
a) Verify doc count on all replicas for each shard.
"response":\{"numFound":2,"start":0,"maxScore":1.0,"docs":[ { "id":"2"}, \{
"id":"1"}]
b) Verify from Solr logs (attached)
c) Verify from tlog.state in the admin/metrics api response for node 8983 and
7574 (attached)
Attaching the patch and logs to the JIRA.
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: 12605UTLogs.txt.zip, logs_and_metrics.zip,
> restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403520#comment-16403520
]
Amrit Sarkar commented on SOLR-12065:
-
oh damn. [~mkhludnev], thanks for pointing this out.
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403486#comment-16403486
]
Mikhail Khludnev commented on SOLR-12065:
-
[~sarkaramr...@gmail.com], not that [^SOLR-12071.patch] jira.
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-12071.patch, restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390870#comment-16390870
]
Varun Thacker commented on SOLR-12065:
--
I spent some time looking into this but my initial assessment is we need to
replay the tlogs which will then mark it as active
{code:java}
requestMap = new HashMap<>();
for (Slice slice : restoreCollection.getSlices()) {
Replica oneReplica = slice.getReplicas().iterator().next();
log.info("Applying buffered updates on : " + oneReplica.getName());
ModifiableSolrParams params = new ModifiableSolrParams();
params.set(CoreAdminParams.ACTION,
CoreAdminParams.CoreAdminAction.REQUESTAPPLYUPDATES.toString());
params.set(CoreAdminParams.NAME, oneReplica.getCoreName());
//ocmh.sliceCmd(clusterState, params, null, slice, shardHandler, asyncId,
requestMap);
ocmh.sendShardRequest(oneReplica.getNodeName(), params, shardHandler, asyncId,
requestMap);
}{code}
Something like this around Line 284 to RestoreCmd.java
I'll try spending some time tomorrow and working on this
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12065) Restore replica always in buffering state
[
https://issues.apache.org/jira/browse/SOLR-12065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390397#comment-16390397
]
Varun Thacker commented on SOLR-12065:
--
I've attached a log file from the local run:
What's interesting is that Line 83 says that the collection is restored
successfully ( while the shard state is in "construction" ) while only in Line
101 that the shard state becomes active
This looks like a race condition as it should first become active and then
complete the task.
[https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/cloud/api/collections/RestoreCmd.java#L280-L292]
tries to mark it as active , refresh the cluster-state and then proceed which
seems broken as it's not really refreshed yet and hence is seeing a stale state
> Restore replica always in buffering state
> -
>
> Key: SOLR-12065
> URL: https://issues.apache.org/jira/browse/SOLR-12065
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: restore_snippet.log
>
>
> Steps to reproduce:
>
> -
> [http://localhost:8983/solr/admin/collections?action=CREATE=test_backup=1=1]
> - curl [http://127.0.0.1:8983/solr/test_backup/update?commit=true] -H
> 'Content-type:application/json' -d '
> [ \{"id" : "1"}
> ]'
> -
> [http://localhost:8983/solr/admin/collections?action=BACKUP=test_backup=test_backup=/Users/varunthacker/backups]
> -
> [http://localhost:8983/solr/admin/collections?action=RESTORE=test_backup=/Users/varunthacker/backups=test_restore]
> * curl [http://127.0.0.1:8983/solr/test_restore/update?commit=true] -H
> 'Content-type:application/json' -d '
> [
> {"id" : "2"}
> ]'
> * Snippet when you try adding a document
> {code:java}
> INFO - 2018-03-07 22:48:11.555; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.DistributedUpdateProcessor; Ignoring commit
> while not ACTIVE - state: BUFFERING replay: false
> INFO - 2018-03-07 22:48:11.556; [c:test_restore s:shard1 r:core_node22
> x:test_restore_shard1_replica_n21]
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
> [test_restore_shard1_replica_n21] webapp=/solr path=/update
> params={commit=true}{add=[2 (1594320896973078528)],commit=} 0 4{code}
> * If you see "TLOG.state" from [http://localhost:8983/solr/admin/metrics]
> it's always 1 (BUFFERING)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
