That's been running for a few days and will take some time yet ( 30+gb on a 0.2mb/s uplink ).
The blobs being uploaded are already present in S3, and have been for some time. Separately, when I run: ./camtool sync --verbose --src http://localhost:3179/sto-s3/ --dest http://localhost:3179/bs I get: 2016/10/30 21:51:14 At source blob sha1-00003e60f806230ce742737809021a427867ebf1 (1 blobs, 65682 bytes) 2016/10/30 21:51:14 Total blobs: 987, 57103774 bytes I have 13 'other' files in that bucket which makes me think it's only finding the first 1000 blobs in the s3 bucket and not synching any after that. On Thursday, October 20, 2016 at 3:43:22 AM UTC+11, mpl wrote: > > Hi, > > To rule out out of order indexing errors, could you please try just > syncing from one bs to another, instead of using --all ? > > To help with that, you could define the targets in your client config > file, e.g.: > > "servers": { > "src": { > "server": "http://ec2server:3179/bs/";, > "auth": "userpass:foo:bar" > }, > "dest": { > "server": "http://localhost:3179/bs/";, > "auth": "userpass:foo:bar" > } > }, > > (/bs-packed/ instead of /bs/ if using blobpacked) > > And then > > camtool sync -src src -dest dest > > If that goes well, you can then start your dest server with a -reindex > and see if things look ok. > > On 8 October 2016 at 12:28, Daniel Heath <[email protected]> wrote: > > So taking one failure as an example: > > > > error fetching public key blob > > sha1-65682c91f928bcf81741ccc8ab4b0efac33a986e: blob was not fully > indexed > > because of a missing dependency > > > > sha1-65682c91f928bcf81741ccc8ab4b0efac33a986e is a public key blob; the > > camlistore version that created it was: 2016-08-22-0367de7 > > > > Could syncing old -> new affect it? > > > > From the web interface for the blob: > > > > Blob content > > > > No data > > > > Indexer metadata > > > > { > > "blobRef": "sha1-65682c91f928bcf81741ccc8ab4b0efac33a986e", > > "size": 449 > > } > > > > Mutation claims > > > > { > > "claims": null > > } > > > > Referenced by > > No references > > > > > > > > > > On Oct 8, 2016, at 2:00 PM, Daniel Heath <[email protected]> wrote: > > > > > > On Oct 8, 2016, at 1:44 AM, Mathieu Lonjaret <[email protected] > <javascript:>> > > wrote: > > > > On 7 October 2016 at 07:58, Daniel Heath <[email protected] > <javascript:>> wrote: > > > > I have a camlistore instance at home which syncs to an s3 target for > > backups. > > > > I decided to try out the backups and make sure they still work. > > > > On a new VPS, I installed camlistore and setup the server config: > > { > > "listen": ":3179", > > "identity": "00ABA6E4", > > "identitySecretRing": > > "/home/ec2-user/.config/camlistore/identity-secring.gpg", > > "blobPath": "/home/ec2-user/var/camlistore/blobs", > > "packRelated": true, > > "levelDB": "/home/ec2-user/var/camlistore/index.leveldb", > > "auth": "<redacted>", > > "s3": "<redacted>", > > "dbNames": null > > } > > > > Then I ran: > > > > ./camtool sync --all > > > > > > Let me make sure I understand the config above, regarding what you're > > trying to do. > > You start with an empty /home/ec2-user/var/camlistore/blobs , "s3" is > > configured to where your backup blobs are, and you're expecting > > camtool sync to fill "/home/ec2-user/var/camlistore/blobs" as a > > destination, from s3 as the source, is that it? > > > > > > Yes, that's correct. > > > > ... > > Destination needs blob: [sha1-000e795be15c30a04b8f51b8e92e62b8bd064e1d; > 1424 > > bytes] > > 2016/10/07 05:47:34 Upload of > sha1-000e795be15c30a04b8f51b8e92e62b8bd064e1d > > to destination blobserver failed: Server didn't receive blob. > > Destination needs blob: [sha1-0010c726455975f0bd4cd7da20b9defc73932dd7; > 740 > > bytes] > > ... > > Error: sync all failed: 2 errors during sync > > > > > > Just to be sure, have you checked that the problem isn't related to > > https://github.com/camlistore/camlistore/issues/681 ? i.e. have you > > tried setting "packRelated": false in the new VPS configuraton? > > > > > > I had not; wiping the server and retrying with packRelated: false > resulted > > in the same error. > > > > Over in the window running camlistored, I get: > > > > ./camlistored > > ... > > ... > > ... > > 2016/10/07 05:47:34 error fetching public key blob > > sha1-65682c91f928bcf81741ccc8ab4b0efac33a986e: blob was not fully > indexed > > because of a missing dependency > > ... > > > > > > At this point, I stopped my home camlistore server, copied the leveldb > > directory and blobs onto the VPS, pointed the VPS instance of camlistore > at > > the copy and restarted it. > > > > This resulted in a completely blank 'no results found' UI. > > > > Running camtool sync --all again just gave me hundreds of pages of > 'error > > fetching <sha>: file does not exist' and 'Destination needs blob: > <sha>'. > > > > I am running the latest release from github: > > [ec2-user@ip-172-31-39-15 camlistore]$ ./camtool -version > > ./camtool version: 7b78c50007 > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Camlistore" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "Camlistore" group. > > To unsubscribe from this topic, visit > > https://groups.google.com/d/topic/camlistore/_eCFdGJFluA/unsubscribe. > > To unsubscribe from this group and all its topics, send an email to > > [email protected] <javascript:>. > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "Camlistore" group. > > To unsubscribe from this topic, visit > > https://groups.google.com/d/topic/camlistore/_eCFdGJFluA/unsubscribe. > > To unsubscribe from this group and all its topics, send an email to > > [email protected] <javascript:>. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Camlistore" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Camlistore" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
