I put a copy of the camlistore binaries and GPG key in the s3 bucket so that I had everything needed to restore my backups in one place. Removing them from the bucket fixed the issue, but I still wanted to investigate.
I found that the s3 driver starts listing files from "" (instead of say "sha-000000000000000000"), so when it asked for the first 1000 files only 987 were blobs. For some reason that caused the sync to hang after the first batch (I suspect something assumed that asking for 1000 files and getting back <1000 meant you have all the files). > On Nov 3, 2016, at 10:00 AM, Mathieu Lonjaret <[email protected]> > wrote: > > What do you mean by "non-blob file"? > > > On 2 November 2016 at 05:56, Daniel Heath <[email protected]> wrote: >> It is indeed the presence of non-blob files in the bucket. >> >> In pkg/blobserver/s3/enumerate.go line 76 there's a comment asking whether >> we should error out if unexpected files are encountered. >> >> I assume somewhere there's an assumption that if you ask for 1000 items and >> get back less than 1000, you've read all of them; I suspect that's a better >> place to fix the issue. >> >> - Daniel >> >> >> >> On Wednesday, November 2, 2016 at 12:07:06 PM UTC+11, Daniel Heath wrote: >>> >>> That's been running for a few days and will take some time yet ( 30+gb on >>> a 0.2mb/s uplink ). >>> >>> The blobs being uploaded are already present in S3, and have been for some >>> time. >>> >>> Separately, when I run: >>> >>> ./camtool sync --verbose --src http://localhost:3179/sto-s3/ --dest >>> http://localhost:3179/bs >>> >>> I get: >>> >>> 2016/10/30 21:51:14 At source blob >>> sha1-00003e60f806230ce742737809021a427867ebf1 (1 blobs, 65682 bytes) >>> 2016/10/30 21:51:14 Total blobs: 987, 57103774 bytes >>> >>> I have 13 'other' files in that bucket which makes me think it's only >>> finding the first 1000 blobs in the s3 bucket and not synching any after >>> that. >>> >>> On Thursday, October 20, 2016 at 3:43:22 AM UTC+11, mpl wrote: >>>> >>>> Hi, >>>> >>>> To rule out out of order indexing errors, could you please try just >>>> syncing from one bs to another, instead of using --all ? >>>> >>>> To help with that, you could define the targets in your client config >>>> file, e.g.: >>>> >>>> "servers": { >>>> "src": { >>>> "server": "http://ec2server:3179/bs/";, >>>> "auth": "userpass:foo:bar" >>>> }, >>>> "dest": { >>>> "server": "http://localhost:3179/bs/";, >>>> "auth": "userpass:foo:bar" >>>> } >>>> }, >>>> >>>> (/bs-packed/ instead of /bs/ if using blobpacked) >>>> >>>> And then >>>> >>>> camtool sync -src src -dest dest >>>> >>>> If that goes well, you can then start your dest server with a -reindex >>>> and see if things look ok. >>>> >>>> On 8 October 2016 at 12:28, Daniel Heath <[email protected]> wrote: >>>>> So taking one failure as an example: >>>>> >>>>> error fetching public key blob >>>>> sha1-65682c91f928bcf81741ccc8ab4b0efac33a986e: blob was not fully >>>>> indexed >>>>> because of a missing dependency >>>>> >>>>> sha1-65682c91f928bcf81741ccc8ab4b0efac33a986e is a public key blob; the >>>>> camlistore version that created it was: 2016-08-22-0367de7 >>>>> >>>>> Could syncing old -> new affect it? >>>>> >>>>> From the web interface for the blob: >>>>> >>>>> Blob content >>>>> >>>>> No data >>>>> >>>>> Indexer metadata >>>>> >>>>> { >>>>> "blobRef": "sha1-65682c91f928bcf81741ccc8ab4b0efac33a986e", >>>>> "size": 449 >>>>> } >>>>> >>>>> Mutation claims >>>>> >>>>> { >>>>> "claims": null >>>>> } >>>>> >>>>> Referenced by >>>>> No references >>>>> >>>>> >>>>> >>>>> >>>>> On Oct 8, 2016, at 2:00 PM, Daniel Heath <[email protected]> wrote: >>>>> >>>>> >>>>> On Oct 8, 2016, at 1:44 AM, Mathieu Lonjaret <[email protected]> >>>>> wrote: >>>>> >>>>> On 7 October 2016 at 07:58, Daniel Heath <[email protected]> wrote: >>>>> >>>>> I have a camlistore instance at home which syncs to an s3 target for >>>>> backups. >>>>> >>>>> I decided to try out the backups and make sure they still work. >>>>> >>>>> On a new VPS, I installed camlistore and setup the server config: >>>>> { >>>>> "listen": ":3179", >>>>> "identity": "00ABA6E4", >>>>> "identitySecretRing": >>>>> "/home/ec2-user/.config/camlistore/identity-secring.gpg", >>>>> "blobPath": "/home/ec2-user/var/camlistore/blobs", >>>>> "packRelated": true, >>>>> "levelDB": "/home/ec2-user/var/camlistore/index.leveldb", >>>>> "auth": "<redacted>", >>>>> "s3": "<redacted>", >>>>> "dbNames": null >>>>> } >>>>> >>>>> Then I ran: >>>>> >>>>> ./camtool sync --all >>>>> >>>>> >>>>> Let me make sure I understand the config above, regarding what you're >>>>> trying to do. >>>>> You start with an empty /home/ec2-user/var/camlistore/blobs , "s3" is >>>>> configured to where your backup blobs are, and you're expecting >>>>> camtool sync to fill "/home/ec2-user/var/camlistore/blobs" as a >>>>> destination, from s3 as the source, is that it? >>>>> >>>>> >>>>> Yes, that's correct. >>>>> >>>>> ... >>>>> Destination needs blob: [sha1-000e795be15c30a04b8f51b8e92e62b8bd064e1d; >>>>> 1424 >>>>> bytes] >>>>> 2016/10/07 05:47:34 Upload of >>>>> sha1-000e795be15c30a04b8f51b8e92e62b8bd064e1d >>>>> to destination blobserver failed: Server didn't receive blob. >>>>> Destination needs blob: [sha1-0010c726455975f0bd4cd7da20b9defc73932dd7; >>>>> 740 >>>>> bytes] >>>>> ... >>>>> Error: sync all failed: 2 errors during sync >>>>> >>>>> >>>>> Just to be sure, have you checked that the problem isn't related to >>>>> https://github.com/camlistore/camlistore/issues/681 ? i.e. have you >>>>> tried setting "packRelated": false in the new VPS configuraton? >>>>> >>>>> >>>>> I had not; wiping the server and retrying with packRelated: false >>>>> resulted >>>>> in the same error. >>>>> >>>>> Over in the window running camlistored, I get: >>>>> >>>>> ./camlistored >>>>> ... >>>>> ... >>>>> ... >>>>> 2016/10/07 05:47:34 error fetching public key blob >>>>> sha1-65682c91f928bcf81741ccc8ab4b0efac33a986e: blob was not fully >>>>> indexed >>>>> because of a missing dependency >>>>> ... >>>>> >>>>> >>>>> At this point, I stopped my home camlistore server, copied the leveldb >>>>> directory and blobs onto the VPS, pointed the VPS instance of >>>>> camlistore at >>>>> the copy and restarted it. >>>>> >>>>> This resulted in a completely blank 'no results found' UI. >>>>> >>>>> Running camtool sync --all again just gave me hundreds of pages of >>>>> 'error >>>>> fetching <sha>: file does not exist' and 'Destination needs blob: >>>>> <sha>'. >>>>> >>>>> I am running the latest release from github: >>>>> [ec2-user@ip-172-31-39-15 camlistore]$ ./camtool -version >>>>> ./camtool version: 7b78c50007 >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups >>>>> "Camlistore" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an >>>>> email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Camlistore" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/camlistore/_eCFdGJFluA/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Camlistore" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/camlistore/_eCFdGJFluA/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups >>>>> "Camlistore" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an >>>>> email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Camlistore" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to a topic in the Google > Groups "Camlistore" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/camlistore/_eCFdGJFluA/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Camlistore" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
