Hello,
I just tried opening a Jira ticket for an issue I was seeing, but after filling
out all the info and hitting create it didn't work. Now any time I click create
I get an error message saying "The Jira server could not be contacted. This may
be a temporary glitch or the server may be down.". I also tried this on 3 other
computers, but they all hit the same issue. Once they try to create the bug
they are permanently blocked from bug creation. I'm not sure what to do at this
point, so this email seems to be my last chance to submit this bug. I would
really appreciate if someone could either create this bug, or if there is a
known issue with Jira right now, then let me know what that issue is and how I
should proceed.
Bug Info
Summary: Incremental backup attempts fail after a shard split operation has
completed
Component: Backup/Restore
Affects Version: 8.9
Description:
I have been attempting to use the incremental backup API on Solr 8.9.0, but
while testing in our product we would occasionally get into a state where all
subsequent backup attempts would fail. After some triage we found that it was
happening to any collection which had undergone a shard split operation. If we
did a backup, completed a shard split operation, then attempted another backup,
the second backup would fail with a FileNotFound exception relating to the
backup id of the second backup as the error message.
Steps to reproduce:
* Create a new collection with no associated backups
* Run a backup for this collection
*
/admin/collections?action=BACKUP&name=myBackupName&collection=myCollectionName&location=/path/to/my/shared/drive
* Run a shard split operation
* /admin/collections?action=SPLITSHARD&collection=name&shard=shardID
* Attempt another backup
Expected Outcome:
* If this operation is being blocked intentionally, then I would expect an
informative error message explaining why it failed. Otherwise I would expect
the backup to complete successfully.
Actual Outcome:
* The backup operation fails with a NoSuchFileException.
NOTE: In the below exception message the number in the file which isn’t found
(in this case zk_backup_1) relates to the backup attempt which is currently
being attempted.
{
"responseHeader":{
"status":500,
"QTime":54},
"failure":{
"MYIPADDRESS:31018_solr":"org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException:Error
from server at null: Error handling 'BACKUPCORE' action"},
"Operation backup caused
exception:":"java.nio.file.NoSuchFileException:java.nio.file.NoSuchFileException:
/opt/hci/solrBackups/reproCollectionBackup/reproCollection/zk_backup_1",
"exception":{
"msg":"/opt/hci/solrBackups/reproCollectionBackup/reproCollection/zk_backup_1",
"rspCode":-1},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"/opt/hci/solrBackups/reproCollectionBackup/reproCollection/zk_backup_1",
"trace":"org.apache.solr.common.SolrException:
/opt/hci/solrBackups/reproCollectionBackup/reproCollection/zk_backup_1\n\tat
org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:65)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:301)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:257)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:216)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:836)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:800)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:545)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:357)\n\tat
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)\n\tat
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)\n\tat
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:516)\n\tat
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)\n\tat
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)\n\tat
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)\n\tat
java.lang.Thread.run(Thread.java:748)\n",
"code":500}}
I tried a few different workaround attempts, but after going through these
steps I wasn’t able to run another backup for the collection.
Workaround attempt 1:
* Use the API to delete the backup
* Used the API to purge unused backup files
* Restarted Solr
* Attempted another backup
* Encountered the same failure
Workaround attempt 2:
* Deleted all files in my Solr backup mount location
* Restarted Solr
* Attempted another backup
* Encountered the same failure
Thanks for your time,
Jordan Diehl
