Perfect. Thanks. David
> Le 28 janv. 2015 à 03:55, Amos S <[email protected]> a écrit : > > I opened an issue for AWS plugin project on github, I hope this is what you > were referring to. Here is the issue: > https://github.com/elasticsearch/elasticsearch-cloud-aws/issues/167 > About the "type missing" error - it turned out to be my mistake in trying to > copy the output of the "GET" verbatim to the input of the PUT, it turned out > that I had to peel off a couple of "{}"'s. Once I did that, the order of the > "type" attribute in relation to the rest didn't matter and the PUT succeeded. > See the update I gave in a previous message. > I ended up making a new copy of the bucket. I'm now trying to restore it to > the test node. > Thanks for your help. > > --Amos > >> On Wednesday, 28 January 2015 13:18:24 UTC+11, David Pilato wrote: >> Could you open an issue in AWS plugin project (and may be in azure and gce) >> to support verify option as well? >> >> BTW, I think we should try to support have type after settings or to clearly >> document it needs to be on the first line. Could you open an issue for this >> in elasticsearch? >> >> Coming back to your issue, I'm afraid you need to wait for a next cloud >> plugin release or patch it yourself (and send a PR) or make your repo >> writable. >> >> Best >> >> -- >> David ;-) >> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs >> >>> Le 27 janv. 2015 à 23:30, Amos S <[email protected]> a écrit : >>> >>> OK, following previous responses by you about the "type is missing" error, >>> I corrected the JSON payload I send to the PUT and got another error: >>> >>> $ curl -XPUT 'http://localhost:9200/_snapshot/amos0' -d '{ >>> "type":"s3", >>> "settings": { >>> "region": "ap-southeast-1", >>> "bucket": "prod-es-backup", >>> "base_path": "elasticsearch/dev/snapshots0", >>> "verify": "false" >>> } >>> }' >>> {"error":"RepositoryVerificationException[[amos0] path >>> [elasticsearch][dev][snapshots0] is not accessible on master node]; nested: >>> IOException[Unable to upload object >>> elasticsearch/dev/snapshots0/tests-ng9f5N6tTm6HZhrGF9s8aQ-master]; nested: >>> AmazonS3Exception[Access Denied (Service: Amazon S3; Status Code: 403; >>> Error Code: AccessDenied; Request ID: 7F3DE23BB617FFF4)]; ","status":500} >>> >>> I think this confirms your suspicion that the repo creation process tries >>> to verify the repository by uploading a test object onto it and also that >>> it ignores the "verify: false" setting. >>> >>> I can either allow this role to write only to this specific prefix or just >>> make a copy of the bucket and allow access to the copy. I'll try the later. >>> >>> Thanks, >>> >>> --Amos >>> >>>> On Wednesday, 28 January 2015 09:20:25 UTC+11, Amos S wrote: >>>> Thanks David, >>>> >>>> It seems that the "verify: false" setting is specific to the "fs" type and >>>> not recognised by the "s3" type. >>>> I tried it anyway and got the same worrying "type is missing" error: >>>> >>>> $ curl -XPUT 'http://localhost:9200/_snapshot/amos0' -d '{ >>>> "s3dev0": { >>>> "settings": { >>>> "base_path": "elasticsearch/dev/snapshots0", >>>> "bucket": "prod-es-backup", >>>> "region": "ap-southeast-1", >>>> "verify": "false" >>>> }, >>>> "type": "s3" >>>> } >>>> } >>>> ' >>>> {"error":"ActionRequestValidationException[Validation Failed: 1: type is >>>> missing;]","status":400} >>>> >>>> I think I first have to address the "type is missing" issue. I suspect >>>> ElasticSearch doesn't recognise the "s3" type. >>>> >>>>> On Tuesday, 27 January 2015 17:13:53 UTC+11, David Pilato wrote: >>>>> Could you try to set verify to false? >>>>> >>>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_repositories >>>>> >>>>> Not sure if it works but would love to know. >>>>> >>>>> -- >>>>> David ;-) >>>>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs >>>>> >>>>>> Le 27 janv. 2015 à 07:09, Amos S <[email protected]> a écrit : >>>>>> >>>>>> Thanks David, >>>>>> >>>>>> That would explain it. >>>>>> >>>>>> Is there a way to skip the validation? >>>>>> >>>>>>> On Tuesday, 27 January 2015 16:52:11 UTC+11, David Pilato wrote: >>>>>>> IIRC when you create a repository we first try to validate it by >>>>>>> writing a sample file in it. >>>>>>> >>>>>>> As you set it to read only, I guess it could be the cause. >>>>>>> >>>>>>> -- >>>>>>> David ;-) >>>>>>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs >>>>>>> >>>>>>>> Le 27 janv. 2015 à 06:20, Amos S <[email protected]> a écrit : >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> For some investigation work, I'm trying to restore specific indices >>>>>>>> from our production ES cluster to a single one-off node. >>>>>>>> >>>>>>>> We run a cluster of ES 1.4.2 on EC2, the data is stored locally on >>>>>>>> each EC2 instance with snapshots stored on an S3 bucket. >>>>>>>> >>>>>>>> I've setup a one-off EC2 instance and am trying to restore a single >>>>>>>> index from snapshot into that new instance. >>>>>>>> >>>>>>>> The instance has its own cluster name and node name, and I've setup a >>>>>>>> read-only S3 role for it so it doesn't accidentally overwrite our >>>>>>>> backup. >>>>>>>> >>>>>>>> Trying to follow instructions I found in various locations on the web, >>>>>>>> I think the next step for me is to configure the S3 snapshot bucket as >>>>>>>> a repository on the new instance, is that correct? >>>>>>>> >>>>>>>> So I did the following to find the S3 snapshot repository >>>>>>>> configuration in the production environment: >>>>>>>> >>>>>>>> $ curl -XGET http://production-cluster:9200/_snapshot/ | python >>>>>>>> -mjson.tool >>>>>>>> { >>>>>>>> "s3prod0": { >>>>>>>> "settings": { >>>>>>>> "base_path": "elasticsearch/prod/snapshots0", >>>>>>>> "bucket": "prod-es-backup", >>>>>>>> "region": "ap-southeast-1" >>>>>>>> }, >>>>>>>> "type": "s3" >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> I then tried to feed this into my new node's configuration: >>>>>>>> >>>>>>>> $ curl -XPUT 'http://localhost:9200/_snapshot/amos0' -d ' >>>>>>>> { >>>>>>>> "type": "s3", >>>>>>>> "s3prod0": { >>>>>>>> "settings": { >>>>>>>> "base_path": "elasticsearch/prod/snapshots0", >>>>>>>> "bucket": "prod-es-backup", >>>>>>>> "region": "ap-southeast-1" >>>>>>>> } >>>>>>>> } >>>>>>>> } >>>>>>>> ' >>>>>>>> {"error":"RepositoryException[[amos0] failed to create repository]; >>>>>>>> nested: CreationException[Guice creation errors:\n\n1) Error injecting >>>>>>>> constructor, org.elasticsearch.repositories.RepositoryException: >>>>>>>> [amos0] No bucket defined for s3 gateway\n at >>>>>>>> org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown >>>>>>>> Source)\n while locating >>>>>>>> org.elasticsearch.repositories.s3.S3Repository\n while locating >>>>>>>> org.elasticsearch.repositories.Repository\n\n1 error]; nested: >>>>>>>> RepositoryException[[amos0] No bucket defined for s3 gateway]; >>>>>>>> ","status":500} >>>>>>>> >>>>>>>> What am I missing here? >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> --Amos >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "elasticsearch" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>>> an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elasticsearch/e3576b32-02c2-4e05-92de-2a1e5de285d2%40googlegroups.com. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/facc1f6f-1626-47be-bd18-28d02a76380e%40googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/40cc5b7f-5d3b-4abf-9473-9a77f8bdf215%40googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/68e2bc5c-3417-4116-a4bc-f8fb7f9a178e%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2E39B7A3-B983-4879-89D6-0DC1C4F8ACAE%40pilato.fr. For more options, visit https://groups.google.com/d/optout.
