[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[
https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852425#comment-16852425
]
Varun Thacker edited comment on SOLR-9952 at 5/30/19 11:40 PM:
---
Here is what I tried on Solr 8.1.1 and the error I got -
Added this to my solr.xml
{code:java}
s3a://my-bucket/solr_811_backups
s3a://my-bucket/solr_811_backups
{code}
Manually added
[https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.375]
and [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.2.0] to
the classpath
Ran these commands
{code:java}
./bin/solr start -c -p 1234
./bin/solr create -c test
curl
"http://localhost:1234/solr/admin/collections?action=BACKUP=testS3A=test=S3"{code}
Got this error
{code:java}
{
"responseHeader":{
"status":500,
"QTime":1211},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"specified location s3a://my-bucket/solr_811_backups does not exist.",
"trace":"org.apache.solr.common.SolrException: specified location
s3a://bucket/solr_811_backups does not exist.\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$33(CollectionsHandler.java:1064)\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1270)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:263)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:251)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat
java.lang.Thread.run(Thread.java:748)\n", "code":500}} {code}
EDIT : I was able to get it to work when I ran this command before the backup -
{code:java}
aws s3api put-object --bucket my-bucket --key solr_811_backups{code}
Output of the backup
{code:java}
aws s3 ls s3:/my-bucket/solr_811_backups/testS3A/
PRE snapshot.shard1/
[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[
https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852425#comment-16852425
]
Varun Thacker edited comment on SOLR-9952 at 5/30/19 11:39 PM:
---
Here is what I tried on Solr 8.1.1 and the error I got -
Added this to my solr.xml
{code:java}
s3a://my-bucket/solr_811_backups
s3a://my-bucket/solr_811_backups
{code}
Manually added
[https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.375]
and [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.2.0] to
the classpath
Ran these commands
{code:java}
./bin/solr start -c -p 1234
./bin/solr create -c test
curl
"http://localhost:1234/solr/admin/collections?action=BACKUP=testS3A=test=S3"{code}
Got this error
{code:java}
{
"responseHeader":{
"status":500,
"QTime":1211},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"specified location s3a://my-bucket/solr_811_backups does not exist.",
"trace":"org.apache.solr.common.SolrException: specified location
s3a://bucket/solr_811_backups does not exist.\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$33(CollectionsHandler.java:1064)\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1270)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:263)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:251)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat
java.lang.Thread.run(Thread.java:748)\n", "code":500}}{code}
Related code -
[https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L1063
]
EDIT : I was able to get it to work when I ran this command before the backup -
{code:java}
aws s3api put-object --bucket my-bucket --key
[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[
https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852425#comment-16852425
]
Varun Thacker edited comment on SOLR-9952 at 5/30/19 11:39 PM:
---
Here is what I tried on Solr 8.1.1 and the error I got -
Added this to my solr.xml
{code:java}
s3a://my-bucket/solr_811_backups
s3a://my-bucket/solr_811_backups
{code}
Manually added
[https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.375]
and [https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.2.0] to
the classpath
Ran these commands
{code:java}
./bin/solr start -c -p 1234
./bin/solr create -c test
curl
"http://localhost:1234/solr/admin/collections?action=BACKUP=testS3A=test=S3"{code}
Got this error
{code:java}
{
"responseHeader":{
"status":500,
"QTime":1211},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"specified location s3a://my-bucket/solr_811_backups does not exist.",
"trace":"org.apache.solr.common.SolrException: specified location
s3a://bucket/solr_811_backups does not exist.\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$33(CollectionsHandler.java:1064)\n\tat
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1270)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:263)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:251)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:796)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:762)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:522)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat
java.lang.Thread.run(Thread.java:748)\n", "code":500}}{code}
Related code -
[https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L1063
EDIT : I was able to get it to work when I ran this command before the backup -
{code:java}
aws s3api put-object --bucket my-bucket --key solr_811_backups{code}
[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[
https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822363#comment-15822363
]
Hrishikesh Gadre edited comment on SOLR-9952 at 1/13/17 9:21 PM:
-
Ok let me take my statement back. The problem is that we are *reusing* system
properties in the solr.xml. If we use absolute params, we should be OK.
{noformat}
${solr.hdfs.home:}
${solr.hdfs.confdir:}
${solr.hdfs.security.kerberos.enabled:false}
${solr.hdfs.security.kerberos.keytabfile:}
${solr.hdfs.security.kerberos.principal:}
${solr.hdfs.permissions.umask-mode:000}
{noformat}
was (Author: hgadre):
Ok let me take my statement back. The problem is that we are *reusing* system
properties in the solr.xml. May be if we use absolute params, we should be OK.
{noformat}
${solr.hdfs.home:}
${solr.hdfs.confdir:}
${solr.hdfs.security.kerberos.enabled:false}
${solr.hdfs.security.kerberos.keytabfile:}
${solr.hdfs.security.kerberos.principal:}
${solr.hdfs.permissions.umask-mode:000}
{noformat}
> S3BackupRepository
> --
>
> Key: SOLR-9952
> URL: https://issues.apache.org/jira/browse/SOLR-9952
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Mikhail Khludnev
> Attachments: Running Solr on S3.pdf
>
>
> I'd like to have a backup repository implementation allows to snapshot to AWS
> S3
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[ https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822124#comment-15822124 ] Trey Cahill edited comment on SOLR-9952 at 1/13/17 6:32 PM: I've done work to run Solr on S3 via HDFS as [~risdenk] described (he also suggested it). See the attachment "Running Solr on S3.pdf". I tested the backup/restore using the HDFS backup repository (just back up and restore thought). was (Author: cahilltr): I've done work to run Solr on S3 via HDFS as [~risdenk] described (he also suggested it). See the attachment "Running Solr on S3". I tested the backup/restore using the HDFS backup repository (just back up and restore thought). > S3BackupRepository > -- > > Key: SOLR-9952 > URL: https://issues.apache.org/jira/browse/SOLR-9952 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mikhail Khludnev > Attachments: Running Solr on S3.pdf > > > I'd like to have a backup repository implementation allows to snapshot to AWS > S3 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9952) S3BackupRepository
[ https://issues.apache.org/jira/browse/SOLR-9952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822124#comment-15822124 ] Trey Cahill edited comment on SOLR-9952 at 1/13/17 6:32 PM: I've done work to run Solr on S3 via HDFS as [~risdenk] described (he also suggested it). See the attachment "Running Solr on S3.pdf". I tested the backup/restore using the HDFS backup repository (just back up and restore though). was (Author: cahilltr): I've done work to run Solr on S3 via HDFS as [~risdenk] described (he also suggested it). See the attachment "Running Solr on S3.pdf". I tested the backup/restore using the HDFS backup repository (just back up and restore thought). > S3BackupRepository > -- > > Key: SOLR-9952 > URL: https://issues.apache.org/jira/browse/SOLR-9952 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mikhail Khludnev > Attachments: Running Solr on S3.pdf > > > I'd like to have a backup repository implementation allows to snapshot to AWS > S3 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
