[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[ https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852425#comment-16852425 ] Varun Thacker edited comment on SOLR-9952 at 5/30/19 11:40 PM: --- Here is what I tried on Solr 8.1.1 and the error I got - Added this to my solr.xml {code:java} s3a://my-bucket/solr_811_backups s3a://my-bucket/solr_811_backups {code} Manually added [https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.375] and [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.2.0] to the classpath Ran these commands {code:java} ./bin/solr start -c -p 1234 ./bin/solr create -c test curl "http://localhost:1234/solr/admin/collections?action=BACKUP=testS3A=test=S3"{code} Got this error {code:java} { "responseHeader":{ "status":500, "QTime":1211}, "error":{ "metadata":[ "error-class","org.apache.solr.common.SolrException", "root-error-class","org.apache.solr.common.SolrException"], "msg":"specified location s3a://my-bucket/solr_811_backups does not exist.", "trace":"org.apache.solr.common.SolrException: specified location s3a://bucket/solr_811_backups does not exist.\n\tat org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$33(CollectionsHandler.java:1064)\n\tat org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1270)\n\tat org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:263)\n\tat org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:251)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat java.lang.Thread.run(Thread.java:748)\n", "code":500}} {code} EDIT : I was able to get it to work when I ran this command before the backup - {code:java} aws s3api put-object --bucket my-bucket --key solr_811_backups{code} Output of the backup {code:java} aws s3 ls s3:/my-bucket/solr_811_backups/testS3A/ PRE snapshot.shard1/
[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[ https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852425#comment-16852425 ] Varun Thacker edited comment on SOLR-9952 at 5/30/19 11:39 PM: --- Here is what I tried on Solr 8.1.1 and the error I got - Added this to my solr.xml {code:java} s3a://my-bucket/solr_811_backups s3a://my-bucket/solr_811_backups {code} Manually added [https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.375] and [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.2.0] to the classpath Ran these commands {code:java} ./bin/solr start -c -p 1234 ./bin/solr create -c test curl "http://localhost:1234/solr/admin/collections?action=BACKUP=testS3A=test=S3"{code} Got this error {code:java} { "responseHeader":{ "status":500, "QTime":1211}, "error":{ "metadata":[ "error-class","org.apache.solr.common.SolrException", "root-error-class","org.apache.solr.common.SolrException"], "msg":"specified location s3a://my-bucket/solr_811_backups does not exist.", "trace":"org.apache.solr.common.SolrException: specified location s3a://bucket/solr_811_backups does not exist.\n\tat org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$33(CollectionsHandler.java:1064)\n\tat org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1270)\n\tat org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:263)\n\tat org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:251)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat java.lang.Thread.run(Thread.java:748)\n", "code":500}}{code} Related code - [https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L1063 ] EDIT : I was able to get it to work when I ran this command before the backup - {code:java} aws s3api put-object --bucket my-bucket --key
[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[ https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852425#comment-16852425 ] Varun Thacker edited comment on SOLR-9952 at 5/30/19 11:39 PM: --- Here is what I tried on Solr 8.1.1 and the error I got - Added this to my solr.xml {code:java} s3a://my-bucket/solr_811_backups s3a://my-bucket/solr_811_backups {code} Manually added [https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.375] and [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.2.0] to the classpath Ran these commands {code:java} ./bin/solr start -c -p 1234 ./bin/solr create -c test curl "http://localhost:1234/solr/admin/collections?action=BACKUP=testS3A=test=S3"{code} Got this error {code:java} { "responseHeader":{ "status":500, "QTime":1211}, "error":{ "metadata":[ "error-class","org.apache.solr.common.SolrException", "root-error-class","org.apache.solr.common.SolrException"], "msg":"specified location s3a://my-bucket/solr_811_backups does not exist.", "trace":"org.apache.solr.common.SolrException: specified location s3a://bucket/solr_811_backups does not exist.\n\tat org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$33(CollectionsHandler.java:1064)\n\tat org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1270)\n\tat org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:263)\n\tat org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:251)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat java.lang.Thread.run(Thread.java:748)\n", "code":500}}{code} Related code - [https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L1063 EDIT : I was able to get it to work when I ran this command before the backup - {code:java} aws s3api put-object --bucket my-bucket --key solr_811_backups{code}
[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[ https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822363#comment-15822363 ] Hrishikesh Gadre edited comment on SOLR-9952 at 1/13/17 9:21 PM: - Ok let me take my statement back. The problem is that we are *reusing* system properties in the solr.xml. If we use absolute params, we should be OK. {noformat} ${solr.hdfs.home:} ${solr.hdfs.confdir:} ${solr.hdfs.security.kerberos.enabled:false} ${solr.hdfs.security.kerberos.keytabfile:} ${solr.hdfs.security.kerberos.principal:} ${solr.hdfs.permissions.umask-mode:000} {noformat} was (Author: hgadre): Ok let me take my statement back. The problem is that we are *reusing* system properties in the solr.xml. May be if we use absolute params, we should be OK. {noformat} ${solr.hdfs.home:} ${solr.hdfs.confdir:} ${solr.hdfs.security.kerberos.enabled:false} ${solr.hdfs.security.kerberos.keytabfile:} ${solr.hdfs.security.kerberos.principal:} ${solr.hdfs.permissions.umask-mode:000} {noformat} > S3BackupRepository > -- > > Key: SOLR-9952 > URL: https://issues.apache.org/jira/browse/SOLR-9952 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mikhail Khludnev > Attachments: Running Solr on S3.pdf > > > I'd like to have a backup repository implementation allows to snapshot to AWS > S3 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[ https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822124#comment-15822124 ] Trey Cahill edited comment on SOLR-9952 at 1/13/17 6:32 PM: I've done work to run Solr on S3 via HDFS as [~risdenk] described (he also suggested it). See the attachment "Running Solr on S3.pdf". I tested the backup/restore using the HDFS backup repository (just back up and restore thought). was (Author: cahilltr): I've done work to run Solr on S3 via HDFS as [~risdenk] described (he also suggested it). See the attachment "Running Solr on S3". I tested the backup/restore using the HDFS backup repository (just back up and restore thought). > S3BackupRepository > -- > > Key: SOLR-9952 > URL: https://issues.apache.org/jira/browse/SOLR-9952 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mikhail Khludnev > Attachments: Running Solr on S3.pdf > > > I'd like to have a backup repository implementation allows to snapshot to AWS > S3 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[ https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822124#comment-15822124 ] Trey Cahill edited comment on SOLR-9952 at 1/13/17 6:32 PM: I've done work to run Solr on S3 via HDFS as [~risdenk] described (he also suggested it). See the attachment "Running Solr on S3.pdf". I tested the backup/restore using the HDFS backup repository (just back up and restore though). was (Author: cahilltr): I've done work to run Solr on S3 via HDFS as [~risdenk] described (he also suggested it). See the attachment "Running Solr on S3.pdf". I tested the backup/restore using the HDFS backup repository (just back up and restore thought). > S3BackupRepository > -- > > Key: SOLR-9952 > URL: https://issues.apache.org/jira/browse/SOLR-9952 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mikhail Khludnev > Attachments: Running Solr on S3.pdf > > > I'd like to have a backup repository implementation allows to snapshot to AWS > S3 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org