What do you mean by "non-blob file"?
On 2 November 2016 at 05:56, Daniel Heath <[email protected]> wrote: > It is indeed the presence of non-blob files in the bucket. > > In pkg/blobserver/s3/enumerate.go line 76 there's a comment asking whether > we should error out if unexpected files are encountered. > > I assume somewhere there's an assumption that if you ask for 1000 items and > get back less than 1000, you've read all of them; I suspect that's a better > place to fix the issue. > > - Daniel > > > > On Wednesday, November 2, 2016 at 12:07:06 PM UTC+11, Daniel Heath wrote: >> >> That's been running for a few days and will take some time yet ( 30+gb on >> a 0.2mb/s uplink ). >> >> The blobs being uploaded are already present in S3, and have been for some >> time. >> >> Separately, when I run: >> >> ./camtool sync --verbose --src http://localhost:3179/sto-s3/ --dest >> http://localhost:3179/bs >> >> I get: >> >> 2016/10/30 21:51:14 At source blob >> sha1-00003e60f806230ce742737809021a427867ebf1 (1 blobs, 65682 bytes) >> 2016/10/30 21:51:14 Total blobs: 987, 57103774 bytes >> >> I have 13 'other' files in that bucket which makes me think it's only >> finding the first 1000 blobs in the s3 bucket and not synching any after >> that. >> >> On Thursday, October 20, 2016 at 3:43:22 AM UTC+11, mpl wrote: >>> >>> Hi, >>> >>> To rule out out of order indexing errors, could you please try just >>> syncing from one bs to another, instead of using --all ? >>> >>> To help with that, you could define the targets in your client config >>> file, e.g.: >>> >>> "servers": { >>> "src": { >>> "server": "http://ec2server:3179/bs/";, >>> "auth": "userpass:foo:bar" >>> }, >>> "dest": { >>> "server": "http://localhost:3179/bs/";, >>> "auth": "userpass:foo:bar" >>> } >>> }, >>> >>> (/bs-packed/ instead of /bs/ if using blobpacked) >>> >>> And then >>> >>> camtool sync -src src -dest dest >>> >>> If that goes well, you can then start your dest server with a -reindex >>> and see if things look ok. >>> >>> On 8 October 2016 at 12:28, Daniel Heath <[email protected]> wrote: >>> > So taking one failure as an example: >>> > >>> > error fetching public key blob >>> > sha1-65682c91f928bcf81741ccc8ab4b0efac33a986e: blob was not fully >>> > indexed >>> > because of a missing dependency >>> > >>> > sha1-65682c91f928bcf81741ccc8ab4b0efac33a986e is a public key blob; the >>> > camlistore version that created it was: 2016-08-22-0367de7 >>> > >>> > Could syncing old -> new affect it? >>> > >>> > From the web interface for the blob: >>> > >>> > Blob content >>> > >>> > No data >>> > >>> > Indexer metadata >>> > >>> > { >>> > "blobRef": "sha1-65682c91f928bcf81741ccc8ab4b0efac33a986e", >>> > "size": 449 >>> > } >>> > >>> > Mutation claims >>> > >>> > { >>> > "claims": null >>> > } >>> > >>> > Referenced by >>> > No references >>> > >>> > >>> > >>> > >>> > On Oct 8, 2016, at 2:00 PM, Daniel Heath <[email protected]> wrote: >>> > >>> > >>> > On Oct 8, 2016, at 1:44 AM, Mathieu Lonjaret <[email protected]> >>> > wrote: >>> > >>> > On 7 October 2016 at 07:58, Daniel Heath <[email protected]> wrote: >>> > >>> > I have a camlistore instance at home which syncs to an s3 target for >>> > backups. >>> > >>> > I decided to try out the backups and make sure they still work. >>> > >>> > On a new VPS, I installed camlistore and setup the server config: >>> > { >>> > "listen": ":3179", >>> > "identity": "00ABA6E4", >>> > "identitySecretRing": >>> > "/home/ec2-user/.config/camlistore/identity-secring.gpg", >>> > "blobPath": "/home/ec2-user/var/camlistore/blobs", >>> > "packRelated": true, >>> > "levelDB": "/home/ec2-user/var/camlistore/index.leveldb", >>> > "auth": "<redacted>", >>> > "s3": "<redacted>", >>> > "dbNames": null >>> > } >>> > >>> > Then I ran: >>> > >>> > ./camtool sync --all >>> > >>> > >>> > Let me make sure I understand the config above, regarding what you're >>> > trying to do. >>> > You start with an empty /home/ec2-user/var/camlistore/blobs , "s3" is >>> > configured to where your backup blobs are, and you're expecting >>> > camtool sync to fill "/home/ec2-user/var/camlistore/blobs" as a >>> > destination, from s3 as the source, is that it? >>> > >>> > >>> > Yes, that's correct. >>> > >>> > ... >>> > Destination needs blob: [sha1-000e795be15c30a04b8f51b8e92e62b8bd064e1d; >>> > 1424 >>> > bytes] >>> > 2016/10/07 05:47:34 Upload of >>> > sha1-000e795be15c30a04b8f51b8e92e62b8bd064e1d >>> > to destination blobserver failed: Server didn't receive blob. >>> > Destination needs blob: [sha1-0010c726455975f0bd4cd7da20b9defc73932dd7; >>> > 740 >>> > bytes] >>> > ... >>> > Error: sync all failed: 2 errors during sync >>> > >>> > >>> > Just to be sure, have you checked that the problem isn't related to >>> > https://github.com/camlistore/camlistore/issues/681 ? i.e. have you >>> > tried setting "packRelated": false in the new VPS configuraton? >>> > >>> > >>> > I had not; wiping the server and retrying with packRelated: false >>> > resulted >>> > in the same error. >>> > >>> > Over in the window running camlistored, I get: >>> > >>> > ./camlistored >>> > ... >>> > ... >>> > ... >>> > 2016/10/07 05:47:34 error fetching public key blob >>> > sha1-65682c91f928bcf81741ccc8ab4b0efac33a986e: blob was not fully >>> > indexed >>> > because of a missing dependency >>> > ... >>> > >>> > >>> > At this point, I stopped my home camlistore server, copied the leveldb >>> > directory and blobs onto the VPS, pointed the VPS instance of >>> > camlistore at >>> > the copy and restarted it. >>> > >>> > This resulted in a completely blank 'no results found' UI. >>> > >>> > Running camtool sync --all again just gave me hundreds of pages of >>> > 'error >>> > fetching <sha>: file does not exist' and 'Destination needs blob: >>> > <sha>'. >>> > >>> > I am running the latest release from github: >>> > [ec2-user@ip-172-31-39-15 camlistore]$ ./camtool -version >>> > ./camtool version: 7b78c50007 >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> > Groups >>> > "Camlistore" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> > an >>> > email to [email protected]. >>> > For more options, visit https://groups.google.com/d/optout. >>> > >>> > >>> > -- >>> > You received this message because you are subscribed to a topic in the >>> > Google Groups "Camlistore" group. >>> > To unsubscribe from this topic, visit >>> > https://groups.google.com/d/topic/camlistore/_eCFdGJFluA/unsubscribe. >>> > To unsubscribe from this group and all its topics, send an email to >>> > [email protected]. >>> > For more options, visit https://groups.google.com/d/optout. >>> > >>> > >>> > >>> > -- >>> > You received this message because you are subscribed to a topic in the >>> > Google Groups "Camlistore" group. >>> > To unsubscribe from this topic, visit >>> > https://groups.google.com/d/topic/camlistore/_eCFdGJFluA/unsubscribe. >>> > To unsubscribe from this group and all its topics, send an email to >>> > [email protected]. >>> > For more options, visit https://groups.google.com/d/optout. >>> > >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> > Groups >>> > "Camlistore" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> > an >>> > email to [email protected]. >>> > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Camlistore" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Camlistore" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
