Re: Cloud Backup
I think Shachar is missing one point about S3 and similar amazon services. You are assuming that amazon created infrastructure specifically for S3. S3 and other products is amazon renting off it's over-capacity, as such it *pays* for amazon to have a very reliable and stable infrastructure because it's for their total business and amazon.com being down probably costs them in the order of millions per second. That is why for amazon it pays to make big investments in infrastructure and renting space there is probably mainly offsetting the cost of building very robust infrastructure... Now they probably invest in extra infrastructure specifically for S3 and similar products... Other then that no one in their right mind making even 5 nines SLA claims will accepts responsibility for downtime you suffer as a result of problems on your side, I the company commit that my systems will be available and reachable from the Internet, if your ISP has a problem you can't blame me (that's why I built datacenters in multiple locations all over the world and you can fail-over instantly and transparently to any of them). I don't work for amazon, but this is what it looks like from my end. Regards, Eliyahu - אליהו 2013/5/18 Ghiora Drori ghioradr...@gmail.com: Shachar Shemesh enjoys being rude and wrong. I suggest he install new fuses. On Fri, May 17, 2013 at 10:05 AM, Shachar Shemesh shac...@shemesh.biz wrote: On 17/05/13 11:43, Oleg Goldshmidt wrote: Shachar Shemesh shac...@shemesh.biz writes: On 17/05/13 10:13, Ghiora Drori wrote: https://aws.amazon.com/glacier/#highlights Quote: Amazon Glacier is designed to provide average annual durability of 99.9% If this is not good enough for you too bad. When you see someone, anyone, saying such a thing, run. As fast and as far as you can. This level of assurance is called nine nines(henceforth 9*9). It amounts to one thousandth of a second of downtime a year. I think you are misreading the claim, Shachar. It is not about availablity, it is about durability. I read it as a measure of the probability that your data will not be lost before a year passes. You are right that this is not about availability. The previous response was my fuse jumping because of the pure ludiciority of people claiming 9*9 availability. After reading the actual text, however, it is not clear what it is about. It is possible that this means that they will lose (on average) ten bits per Terabyte per year. If that is the case, honestly, this does not sound very good. Assuming they have several exabytes of customers data, this means that they have several actual cases of customer data lose all the time. Not a particularly good track record. Or, and this is the more likely scenario, they are talking out of their asses, and put the number in because it sounds impressive. Omer Zak wrote: IMO, the quote does not promise a nine nines assurance. It only says that Amazon Glacier WAS DESIGNED to provide this kind of assurance. See my previous comment for why this is equally ludicrous. Shachar Disclaimer: I have never used the service and th above is my common sense and reading comprehension take on what is written in the above website. -- Mark Twain - If you don't read the newspaper, you're uninformed. If you read the newspaper, you're mis-informed. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
Hi, Please read the doc's regarding S3 and Glacier again. The best way is to backup Linux under Amazon is to S3. This way you have immediate access to the recently backed up stuff. Once the backups are in S3 you can tell Amazon to move it to Glacier based on the names of the files in S3 and* time constraints; *lets say after two weeks when it becomes less likely you will need it. Using the S3 Amazon console, you can also specify when to delete the backups from glacier automatically a very nice feature. The cost in glacier is very low. You only pay high fees for restoring from glacier to S3 (From where you can easily recover to Linux) if you break the rules. Read the rules below: https://aws.amazon.com/glacier/pricing Quote Glacier is designed with the expectation that retrievals are infrequent and unusual, and data will be stored for extended periods of time. You can retrieve up to 5% of your average monthly storage (pro-rated daily) for free each month. If you choose to retrieve more than this amount of data in a month, you are charged a retrieval fee starting at $0.01 per gigabyte. Learn more. In addition, there is a pro-rated charge of $0.03 per gigabyte for items deleted prior to 90 days. There is an article by some one who has comprehension problems on the Internet saying it will cost a fortune to restore, and people seem to be quoting it a lot. I have done restores and it is simply not so. I backup a few 100GB's this way every night. I have done restorations. As to reliability: (This is effectively a contract): https://aws.amazon.com/glacier/#highlights Quote: Amazon Glacier is designed to provide average annual durability of 99.9% If this is not good enough for you too bad. ( I do not work for Amazon) Thanks Ghiora On Fri, May 17, 2013 at 12:04 AM, Nadav Har'El n...@math.technion.ac.ilwrote: On Thu, May 16, 2013, Steve Litt wrote about Re: Cloud Backup: If anyone was following this thread, I'll give you the latest news. rsync.net just announced (see https://news.ycombinator.com/item?id=5638295) that during May, they sell 50 GB of quota for $60 a year. I switched to this deal, and it What happens in June? I was looking at http://www.rsync.net/resources/faq.html#13 , and that listed the price before quantity discounts at $0.80/gigabyte_month, so with my 30 GB, I'd be paying $24/month. I like the idea of using rsync or sftp to upload and download my files, but $24/month is pretty steep. They promised the 10 cent per gigabyte per month will stay forever, if you just *enroll* in May. Until this month, I've been paying them just 40 cents/gigabyte/month and thought I was getting a good deal (50% discount), because I'm a free software author :-) -- Nadav Har'El| Thursday, May 16 2013, 8 Sivan 5773 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |When you handle yourself, use your head; http://nadav.harel.org.il |when you handle others, use your heart. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il -- Mark Twain - If you don't *read* the newspaper, you're *uninformed*. If you *read* the newspaper, you're *mis*-*informed*. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
On 17/05/13 10:13, Ghiora Drori wrote: As to reliability: (This is effectively a contract): No, it isn't (see below). https://aws.amazon.com/glacier/#highlights Quote: Amazon Glacier is designed to provide average annual durability of 99.9% If this is not good enough for you too bad. When you see someone, anyone, saying such a thing, run. As fast and as far as you can. This level of assurance is called nine nines(henceforth 9*9). It amounts to one thousandth of a second of downtime a year. Amazon is talking out of their asses in offering it. First, even if their service is 100% reliable, you will not get 9*9 of service. You home internet connection is not that reliable. The fiber connecting Israel to the world is not that reliable. The BGP protocol that is meant to keep the internet alive should a link go down is not that reliable. No matter what Amazon are doing, nine nines is not the SLA you will be getting. Now, you might claim that that is not Amazon's fault. THEY are providing 9*9, and it is the rest of the internet that is not reliable enough. This claim is bullshit. They are not. No single server can provide 9*9. Servers fail. Hard disks fail. Memory fails. NICs fail. Network switches fail. In order to provide a 9*9 SLA, you must be able to detect each and every one of those failures + provide an alternative path *in less than 1 millisecond*, plus assure that only one such failure happens in a year for every customer. It is not impossible to build such a system, but it will not be affordable. The very fact that Amazon is affordable means that they are not providing 9*9, nor anything even close. Just to give you a taste of how expensive such a system might be, take head of the following interesting fact. I just ran a ping between two computers connected via a crossed ethernet cable over a 1Gb/s link. The average ping time was 0.431ms. In other words, just the round-trip time (including kernel wakeup and related activities) between two computers connected over a 3 meter cable is half the time you have at your disposal to react to a downtime *per year*. At this rate, you cannot afford to ping a second time in the hope that the machine was just slightly busy, or that the packet was lost. If you do not get a reply within half a millisecond, you must act. You only have half a millisecond to set up the actual diversion. What about further away computers? From my home, pinging a server located at the server farm of the same ISP I'm connected to takes 17ms. This means I cannot react to a server downtime in less time than half that no matter what. If the server is down, it will take me no less than 8ms to even find out about it. That is, by the time I find out about the server down, I am already violating my SLA by a factor of 8. The only way to have redundancy is to be on the same segment and use specialized low-latency equipment. Since the ISP's link itself might go down, and since BGP is nowhere fast enough to recover, *the only way to provide a 9*9 service is to build a duplicate of the internet in order to do so*. I think we can all agree that Amazon did not do that, or their service would have been, by several orders of magnitude, more expensive than it is. However, supposing that money was no object, would that work? The answer is no. The reason the answer is no is that external factors were not taken into account. A 9*9 SLA means that the chances of a problem are less than 1:10^11. The chances of a Reichter 8+ earthquake, tsunami, volcano eruption or meteorite striking are all higher than that. TLDR version: The SLA is not a contractual question. Especially when counting nines, it is a technological infrastructure question. Amazon is not providing the nine nines it seems to be promising, and is therefor lying on its SLA. ( I do not work for Amazon) I do not work for Amazon either. I did use to run a service that was a (very humble) competitor to this one (in which we did not offer SLA for service availability at all, only for the actual data). I currently work for Akamai, for which Amazon is a competitor (though not this particular service). It should be clear that I do not speak on behalf of my employer. All opinions are my own, and only my own. Shachar ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
SLA nine ninths (was: Re: Cloud Backup)
IMO, the quote does not promise a nine nines assurance. It only says that Amazon Glacier WAS DESIGNED to provide this kind of assurance. On Fri, 2013-05-17 at 11:04 +0300, Shachar Shemesh wrote: On 17/05/13 10:13, Ghiora Drori wrote: As to reliability: (This is effectively a contract): No, it isn't (see below). https://aws.amazon.com/glacier/#highlights Quote: Amazon Glacier is designed to provide average annual durability of 99.9% If this is not good enough for you too bad. When you see someone, anyone, saying such a thing, run. As fast and as far as you can. This level of assurance is called nine nines(henceforth 9*9). It amounts to one thousandth of a second of downtime a year. Amazon is talking out of their asses in offering it. -- Your liberty to swing your fist ends just where my nose begins. Your freedom of expression ends where my freedom of expression begins. Your freedom of religion ends where my rights for equality and accessibility begin. My own blog is at http://www.zak.co.il/tddpirate/ My opinions, as expressed in this E-mail message, are mine alone. They do not represent the official policy of any organization with which I may be affiliated in any way. WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
Shachar Shemesh shac...@shemesh.biz writes: On 17/05/13 10:13, Ghiora Drori wrote: https://aws.amazon.com/glacier/#highlights Quote: Amazon Glacier is designed to provide average annual durability of 99.9% If this is not good enough for you too bad. When you see someone, anyone, saying such a thing, run. As fast and as far as you can. This level of assurance is called nine nines(henceforth 9*9). It amounts to one thousandth of a second of downtime a year. I think you are misreading the claim, Shachar. It is not about availablity, it is about durability. I read it as a measure of the probability that your data will not be lost before a year passes. Disclaimer: I have never used the service and th above is my common sense and reading comprehension take on what is written in the above website. -- Oleg Goldshmidt | p...@goldshmidt.org ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
On 17/05/13 11:43, Oleg Goldshmidt wrote: Shachar Shemesh shac...@shemesh.biz writes: On 17/05/13 10:13, Ghiora Drori wrote: https://aws.amazon.com/glacier/#highlights Quote: Amazon Glacier is designed to provide average annual durability of 99.9% If this is not good enough for you too bad. When you see someone, anyone, saying such a thing, run. As fast and as far as you can. This level of assurance is called nine nines(henceforth 9*9). It amounts to one thousandth of a second of downtime a year. I think you are misreading the claim, Shachar. It is not about availablity, it is about durability. I read it as a measure of the probability that your data will not be lost before a year passes. You are right that this is not about availability. The previous response was my fuse jumping because of the pure ludiciority of people claiming 9*9 availability. After reading the actual text, however, it is not clear what it is about. It is possible that this means that they will lose (on average) ten bits per Terabyte per year. If that is the case, honestly, this does not sound very good. Assuming they have several exabytes of customers data, this means that they have several actual cases of customer data lose all the time. Not a particularly good track record. Or, and this is the more likely scenario, they are talking out of their asses, and put the number in because it sounds impressive. Omer Zak wrote: IMO, the quote does not promise a nine nines assurance. It only says that Amazon Glacier WAS DESIGNED to provide this kind of assurance. See my previous comment for why this is equally ludicrous. Shachar Disclaimer: I have never used the service and th above is my common sense and reading comprehension take on what is written in the above website. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
Shachar Shemesh enjoys being rude and wrong. I suggest he install new fuses. On Fri, May 17, 2013 at 10:05 AM, Shachar Shemesh shac...@shemesh.bizwrote: On 17/05/13 11:43, Oleg Goldshmidt wrote: Shachar Shemesh shac...@shemesh.biz shac...@shemesh.biz writes: On 17/05/13 10:13, Ghiora Drori wrote: https://aws.amazon.com/glacier/#highlights Quote: Amazon Glacier is designed to provide average annual durability of 99.9% If this is not good enough for you too bad. When you see someone, anyone, saying such a thing, run. As fast and as far as you can. This level of assurance is called nine nines(henceforth 9*9). It amounts to one thousandth of a second of downtime a year. I think you are misreading the claim, Shachar. It is not about availablity, it is about durability. I read it as a measure of the probability that your data will not be lost before a year passes. You are right that this is not about availability. The previous response was my fuse jumping because of the pure ludiciority of people claiming 9*9 availability. After reading the actual text, however, it is not clear what it is about. It is possible that this means that they will lose (on average) ten bits per Terabyte per year. If that is the case, honestly, this does not sound very good. Assuming they have several exabytes of customers data, this means that they have several actual cases of customer data lose all the time. Not a particularly good track record. Or, and this is the more likely scenario, they are talking out of their asses, and put the number in because it sounds impressive. Omer Zak wrote: IMO, the quote does not promise a nine nines assurance. It only says that Amazon Glacier WAS DESIGNED to provide this kind of assurance. See my previous comment for why this is equally ludicrous. Shachar Disclaimer: I have never used the service and th above is my common sense and reading comprehension take on what is written in the above website. -- Mark Twain - If you don't *read* the newspaper, you're *uninformed*. If you *read* the newspaper, you're *mis*-*informed*. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
Nadav, Amazon has a special service made just for this: http://aws.amazon.com/glacier/pricing/ The trick with Glacier is that the data is stored _offline_. That means two things: 1) It is actually more failure-redundant than EBS, S3, or rsync.net 2) You get your data about an hour _after_ you request it. Data transfer in is free and data transfer out has a free tier per month. Data storage is the absolute cheapest on the internet. Again I stress, this is the _most_failure_redundant_ service that exists. It is specifically designed for long-term backup and archiving. Furthermore, I am very happy with Amazon's cloud offerings. I have had very good experience with their support teams. On Sat, Feb 23, 2013 at 6:15 PM, Nadav Har'El n...@math.technion.ac.il wrote: Hi, I'm looking for a cloud backup solution for Linux, where I'll be able to use rsync, sftp (and similar utilities) to a remote server to back up by files, and when needed, look at individual files (e.g., using sshfs) or restore all my files. I am *not* looking for a solution based on special purpose (and usually, closed source) utilities or daemons that attempt to decide for me what to back up and when - I want to be of full control of this process. For the last 3 years, I've been using the services of rsync.net, and they're doing exactly what I want. However, the storage price I pay them is 40 cents per gigabyte per month, is 4 times that of Amazon's, so I think there must be a cheaper solution. One thing I've been thinking - wouldn't it be fairly easy to store my files on Amazon's S3 or even more simply EBS, and then run rsync server on a micro instance on EC2? Sounds like a cheap, convenient backup solution for Linux diehards like myself, and I wonder if anyone has done this before and then I won't need to code this myself? Thanks, Nadav. -- Nadav Har'El| Saturday, Feb 23 2013, 13 Adar 5773 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |Creativity consists of coming up with http://nadav.harel.org.il |many ideas, not just that one great idea. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il -- Dotan Cohen http://gibberish.co.il http://what-is-what.com ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
On Thu, 16 May 2013 07:50:36 +0300 Nadav Har'El n...@math.technion.ac.il wrote: On Sat, Feb 23, 2013, Nadav Har'El wrote about Cloud Backup: Hi, I'm looking for a cloud backup solution for Linux, where I'll be able to use rsync, sftp (and similar utilities) to a remote server to back up by files, and when needed, look at individual files (e.g., using sshfs) or restore all my files. ... For the last 3 years, I've been using the services of rsync.net, and they're doing exactly what I want. However, the storage price I pay them is 40 cents per gigabyte per month, is 4 times that of Amazon's, so I think there must be a cheaper solution. If anyone was following this thread, I'll give you the latest news. rsync.net just announced (see https://news.ycombinator.com/item?id=5638295) that during May, they sell 50 GB of quota for $60 a year. I switched to this deal, and it gives me 4 times the quota (!) I've been getting up to now, at a price of just just 10 cents per gigabyte per month. Now it will be much easier for me to backup my Linux system without thinking twice about the quota, and I don't need to code anything on Amazon. What happens in June? I was looking at http://www.rsync.net/resources/faq.html#13 , and that listed the price before quantity discounts at $0.80/gigabyte_month, so with my 30 GB, I'd be paying $24/month. I like the idea of using rsync or sftp to upload and download my files, but $24/month is pretty steep. Thanks SteveT ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
On Thu, May 16, 2013, Dotan Cohen wrote about Re: Cloud Backup: Nadav, Amazon has a special service made just for this: http://aws.amazon.com/glacier/pricing/ Well, Glacier indeed appears to be great service, but it's not very convenient to back up a Linux machine with it. I was looking for something natural and convenient like rsync, hoping that I wouldn't need to write my own backup software. I have some very interesting thoughts on how I would write a backup software that worked directly on S3 (or Glacier) without requiring any other paid service (EC2, EBS, database, etc.), but I am not aware of any such backup software on Linux already existing. If anyone wants to write one, I'll be happy to give you my ideas. (or, has anybody know of an existing tool like that?) Another problem with Glacier compared to S3 is its ridiculously complex pricing model. Can anybody tell me what it would cost me to download a 50 GB backup over, say, two days? My calculation comes out to $7.5, but their pricing algorithm is so complex, that it could be $750 and I wouldn't know ;-) The trick with Glacier is that the data is stored _offline_. That means two things: 1) It is actually more failure-redundant than EBS, S3, or rsync.net Why have you concluded that Glacier is more redundant or safer than, for example, S3? And what does being offline help? The data is probably stored on hard disks, just like on S3, and it's obviously connected to the Internet (otherwise how can you get it), so what makes it safer? 2) You get your data about an hour _after_ you request it. Data transfer in is free and data transfer out has a free tier per month. Data storage is the absolute cheapest on the internet. Indeed :-) -- Nadav Har'El| Thursday, May 16 2013, 8 Sivan 5773 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |The person who knows how to laugh at http://nadav.harel.org.il |himself will never cease to be amused. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
On Thu, May 16, 2013, Steve Litt wrote about Re: Cloud Backup: If anyone was following this thread, I'll give you the latest news. rsync.net just announced (see https://news.ycombinator.com/item?id=5638295) that during May, they sell 50 GB of quota for $60 a year. I switched to this deal, and it What happens in June? I was looking at http://www.rsync.net/resources/faq.html#13 , and that listed the price before quantity discounts at $0.80/gigabyte_month, so with my 30 GB, I'd be paying $24/month. I like the idea of using rsync or sftp to upload and download my files, but $24/month is pretty steep. They promised the 10 cent per gigabyte per month will stay forever, if you just *enroll* in May. Until this month, I've been paying them just 40 cents/gigabyte/month and thought I was getting a good deal (50% discount), because I'm a free software author :-) -- Nadav Har'El| Thursday, May 16 2013, 8 Sivan 5773 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |When you handle yourself, use your head; http://nadav.harel.org.il |when you handle others, use your heart. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
On Sat, Feb 23, 2013, Nadav Har'El wrote about Cloud Backup: Hi, I'm looking for a cloud backup solution for Linux, where I'll be able to use rsync, sftp (and similar utilities) to a remote server to back up by files, and when needed, look at individual files (e.g., using sshfs) or restore all my files. ... For the last 3 years, I've been using the services of rsync.net, and they're doing exactly what I want. However, the storage price I pay them is 40 cents per gigabyte per month, is 4 times that of Amazon's, so I think there must be a cheaper solution. If anyone was following this thread, I'll give you the latest news. rsync.net just announced (see https://news.ycombinator.com/item?id=5638295) that during May, they sell 50 GB of quota for $60 a year. I switched to this deal, and it gives me 4 times the quota (!) I've been getting up to now, at a price of just just 10 cents per gigabyte per month. Now it will be much easier for me to backup my Linux system without thinking twice about the quota, and I don't need to code anything on Amazon. -- Nadav Har'El| Thursday, May 16 2013, 7 Sivan 5773 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |Life can only be understood backwards but http://nadav.harel.org.il |it must be lived forwards. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
Hi, Few things: - You can use S3, but then the rsync could be problematic, since there is no rsync server on the other side. - Amazon EBS is nice, but the Micro instance to use it with EBSis free based on your usage. I used the free Micro instance as a slave DNS and after 2 months I had to pay for it since my free usage has been somehow finished. - You can use several scripts that you can find on Google to rsync with Dropbox. Thanks, Hetz 2013/2/23 Nadav Har'El n...@math.technion.ac.il Hi, I'm looking for a cloud backup solution for Linux, where I'll be able to use rsync, sftp (and similar utilities) to a remote server to back up by files, and when needed, look at individual files (e.g., using sshfs) or restore all my files. I am *not* looking for a solution based on special purpose (and usually, closed source) utilities or daemons that attempt to decide for me what to back up and when - I want to be of full control of this process. For the last 3 years, I've been using the services of rsync.net, and they're doing exactly what I want. However, the storage price I pay them is 40 cents per gigabyte per month, is 4 times that of Amazon's, so I think there must be a cheaper solution. One thing I've been thinking - wouldn't it be fairly easy to store my files on Amazon's S3 or even more simply EBS, and then run rsync server on a micro instance on EC2? Sounds like a cheap, convenient backup solution for Linux diehards like myself, and I wonder if anyone has done this before and then I won't need to code this myself? Thanks, Nadav. -- Nadav Har'El| Saturday, Feb 23 2013, 13 Adar 5773 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |Creativity consists of coming up with http://nadav.harel.org.il |many ideas, not just that one great idea. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
On Sat, Feb 23, 2013, Hetz Ben Hamo wrote about Re: Cloud Backup: - You can use S3, but then the rsync could be problematic, since there is no rsync server on the other side. This is the part I'd need to code - run some sort of server process on EC2. I agree that it would be easier to do this with EBS, so I don't think I'll actually want to use S3 here (on an unrelated note, I do have a pretty good plan on how to do incremental backup to S3 without any server-side software, i.e., no code running on EC2 at all, but doing this will be more complicated than assuming I can run code on EC2). - Amazon EBS is nice, but the Micro instance to use it with EBSis free based on your usage. I used the free Micro instance as a slave DNS and after 2 months I had to pay for it since my free usage has been somehow finished. The difference here is that while DNS has to be always on - so it costs you 2 cents an hour for the whole month - my backup isn't always on, I can only turn it on when I want to back up, and then it will cost me 2 cents for a full hour (you can't pay for less than an hour). If I run it once every day and assuming the increments will rarely take more than an hour to send, this amounts to 60 cents a month, which is perfectly acceptable. - You can use several scripts that you can find on Google to rsync with Dropbox. I wonder how this can work without rsync support on the server side, but thanks, I'll look. -- Nadav Har'El| Saturday, Feb 23 2013, 14 Adar 5773 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |Lottery: A tax on people who are bad at http://nadav.harel.org.il |math. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
On Sat, Feb 23, 2013 at 6:33 PM, Nadav Har'El n...@math.technion.ac.ilwrote: On Sat, Feb 23, 2013, Hetz Ben Hamo wrote about Re: Cloud Backup: - You can use S3, but then the rsync could be problematic, since there is no rsync server on the other side. This is the part I'd need to code - run some sort of server process on EC2. I agree that it would be easier to do this with EBS, so I don't think I'll actually want to use S3 here (on an unrelated note, I do have a pretty good plan on how to do incremental backup to S3 without any server-side software, i.e., no code running on EC2 at all, but doing this will be more complicated than assuming I can run code on EC2). - Amazon EBS is nice, but the Micro instance to use it with EBSis free based on your usage. I used the free Micro instance as a slave DNS and after 2 months I had to pay for it since my free usage has been somehow finished. The difference here is that while DNS has to be always on - so it costs you 2 cents an hour for the whole month - my backup isn't always on, I can only turn it on when I want to back up, and then it will cost me 2 cents for a full hour (you can't pay for less than an hour). If I run it Google lets you pay for a minimal time of 15 minutes. ProfitBricks charges by 3 minute chunks. once every day and assuming the increments will rarely take more than an hour to send, this amounts to 60 cents a month, which is perfectly acceptable. You are neglecting the most important cost: of data transfer. See To Cloud Or Not To Cloud? Musings On Costs and Viability. by Yao Chen , Radu Sion http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.174.8806 - You can use several scripts that you can find on Google to rsync with Dropbox. I wonder how this can work without rsync support on the server side, but thanks, I'll look. -- Nadav Har'El| Saturday, Feb 23 2013, 14 Adar 5773 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |Lottery: A tax on people who are bad at http://nadav.harel.org.il |math. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il -- Orna Agmon Ben-Yehuda. http://ladypine.org ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
On Sat, Feb 23, 2013, Orna Agmon Ben-Yehuda wrote about Re: Cloud Backup: Google lets you pay for a minimal time of 15 minutes. ProfitBricks charges by 3 minute chunks. Thanks, good to know. However, for my purposes, 60 cents (one hour a day) vs 15 cents (15 minutes a day) are not big differentiator. You are neglecting the most important cost: of data transfer. See To Cloud Or Not To Cloud? Musings On Costs and Viability. This would definitely *not* be the most important cost for me. I'm doing incremental backup never deleting anything (rsync --backup with a new --backup-dir every time) on rsync.net for 3 years, and so far my total is 15 GB. This means that over 3 years I sent to them roughly 15 GB, and requested a tiny fraction of that. In Amazon, transer into EC2 is actually free, but even if it weren't, at their most expensive rate of 12 cents per gigabyte, transferring those 15 gigabytes would cost me 0.12*15 = $1.80 for the 3 years doesn't sound much compared to the $0.09*15 = $1.60 I'll pay every month for storing the data. Restoring the *whole* thing will also cost me $1.80 - again not much of considering the use case (if I lose all my data, $1.8 is negligable) and not much compared to the usual monthly cost. The situation would be different, of course, for people who generate gigabytes of new data every month. I'm not one of these people. Thanks, Nadav. -- Nadav Har'El| Saturday, Feb 23 2013, 14 Adar 5773 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |Evening news begins with 'Good evening', http://nadav.harel.org.il |and then proceeds to say why it isn't. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
For this purpose I am using an old desktop computer equipped with big enough disk and a cron job to automatically fetch backups from remote servers when it is switched on. Yes, it store data in a fixed location, but given that it is only meant to backups, I can live with it for now. By the way, the same computer also has a mysql instance that is used to extract backups and rebuild databases locally, which is good or testing that the backups actually works. While Dropbox required some propriety software to sync, I've heard that Google Drive has better API. On Sat, Feb 23, 2013 at 6:15 PM, Nadav Har'El n...@math.technion.ac.ilwrote: Hi, I'm looking for a cloud backup solution for Linux, where I'll be able to use rsync, sftp (and similar utilities) to a remote server to back up by files, and when needed, look at individual files (e.g., using sshfs) or restore all my files. I am *not* looking for a solution based on special purpose (and usually, closed source) utilities or daemons that attempt to decide for me what to back up and when - I want to be of full control of this process. For the last 3 years, I've been using the services of rsync.net, and they're doing exactly what I want. However, the storage price I pay them is 40 cents per gigabyte per month, is 4 times that of Amazon's, so I think there must be a cheaper solution. One thing I've been thinking - wouldn't it be fairly easy to store my files on Amazon's S3 or even more simply EBS, and then run rsync server on a micro instance on EC2? Sounds like a cheap, convenient backup solution for Linux diehards like myself, and I wonder if anyone has done this before and then I won't need to code this myself? Thanks, Nadav. -- Nadav Har'El| Saturday, Feb 23 2013, 13 Adar 5773 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |Creativity consists of coming up with http://nadav.harel.org.il |many ideas, not just that one great idea. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il -- Tomer Cohen http://tomercohen.com ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cloud Backup
On Sat, Feb 23, 2013 at 6:15 PM, Nadav Har'El n...@math.technion.ac.ilwrote: Hi, I'm looking for a cloud backup solution for Linux, where I'll be able to use rsync, sftp (and similar utilities) to a remote server to back up by files, and when needed, look at individual files (e.g., using sshfs) or restore all my files. I am *not* looking for a solution based on special purpose (and usually, closed source) utilities or daemons that attempt to decide for me what to back up and when - I want to be of full control of this process. For the last 3 years, I've been using the services of rsync.net, and they're doing exactly what I want. However, the storage price I pay them is 40 cents per gigabyte per month, is 4 times that of Amazon's, so I think there must be a cheaper solution. One thing I've been thinking - wouldn't it be fairly easy to store my files on Amazon's S3 or even more simply EBS, and then run rsync server on a micro instance on EC2? Sounds like a cheap, convenient backup solution for Linux diehards like myself, and I wonder if anyone has done this before and then I won't need to code this myself? There's http://s3rsync.com - they allow you to use rsync and they act as a layer to S3 Not sure if their pricing model (especially 'to be consumed within X days from the prepayment) fits you, however. and obviously the data goes through them... but that didn't disturb you in rsync.net, so I am suggesting it. HTH, -- Shimi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
