Re: [CentOS] Amazon S3FS Automounter of sorts
On 12/04/2010 17:44, Matt Keating wrote: I found it works well with FTPing into the server and uploading to the mounted bucket. Reason its like this is that there are lots of different people who upload throughout our company. It was much easier giving out the FTP details, which were totally under our control (Username/Pass,Firewall,etc), rather than giving out the S3 logins. Sounds like a good reason :) I hope you are aware that everything you put on your s3 is publicly available if someone knows your bucket name. Yes, I am aware of that - its all being served on the net anyway. If I remove the Pubic read only, will the files still be accessible via cloudfront? I hope not. And a little test confirms this. If you are serving it out anyway that is fine. I just had a client that had all his backup files publicly readable, because of this type of configuration error. I tried mounting it like you, but just ran into too many problems, especially if you access files from many machines. What issues did you run into? As I haven't had any problems as of yet. If it is a one way transfer it is fine. But if you modify files etc caching issues where horrible. Files overwritten etc ... But if you are just pushing stuff onto a server it should work. For the backup I have used s3tools too. I have a little script that looks at what is in the bucket and what is in the local folder and then syncs them up. But I suppose that is what the fuse file system does :) For your auto-mount script. Can't you mount it when someone logs on over ftp. And then if no one is logged on any more unmount it. Cheers Didi -- Hoffmann Geerd-Dietger http://contact.ribalba.de ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Amazon S3FS Automounter of sorts
From: Geerd-Dietger Hoffmann d...@ribalba.de Reply-To: CentOS mailing list centos@centos.org Date: Tue, 13 Apr 2010 13:11:59 +0100 To: CentOS mailing list centos@centos.org Subject: Re: [CentOS] Amazon S3FS Automounter of sorts On 12/04/2010 17:44, Matt Keating wrote: I found it works well with FTPing into the server and uploading to the mounted bucket. Reason its like this is that there are lots of different people who upload throughout our company. It was much easier giving out the FTP details, which were totally under our control (Username/Pass,Firewall,etc), rather than giving out the S3 logins. Sounds like a good reason :) I hope you are aware that everything you put on your s3 is publicly available if someone knows your bucket name. Yes, I am aware of that - its all being served on the net anyway. If I remove the Pubic read only, will the files still be accessible via cloudfront? I hope not. And a little test confirms this. If you are serving it out anyway that is fine. I just had a client that had all his backup files publicly readable, because of this type of configuration error. I tried mounting it like you, but just ran into too many problems, especially if you access files from many machines. What issues did you run into? As I haven't had any problems as of yet. If it is a one way transfer it is fine. But if you modify files etc caching issues where horrible. Files overwritten etc ... But if you are just pushing stuff onto a server it should work. Thanks for reply, We never need to modify things, so once they are live - they stay that way. So I guess my situation is ok then. For the backup I have used s3tools too. I have a little script that looks at what is in the bucket and what is in the local folder and then syncs them up. But I suppose that is what the fuse file system does :) For keeping things in sync, I have a cron job that downloads the list of files plus hashes from amazon and stores them in a db. I compare that list to the latest list downloaded and put the new files into a new table. I have another script running that checks the 'download' table and downoads the files in there, once completed puts that file into the main list table. It works fine - no intention on changing that. For your auto-mount script. Can't you mount it when someone logs on over ftp. And then if no one is logged on any more unmount it. Cheers Didi Never thought about writing my own automounter, will give it a go. Thanks for the input. MattNOTE: The information in this email is confidential and may be legally privileged, unless stated to the contrary. If you are not the intended recipient, you must not read, use or disseminate that information. Any opinions or comments are personal to the writer and do not represent the official view of Dennis Publishing Ltd. If you have received this email and are not a named addressee, please contact itdirec...@dennis.co.uk immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purpose, or disclose its contents to any other person. Although this email and any attachments are believed to be free of any virus, or other defects, it is the responsibility of the recipient to ensure that they are virus free and no responsibility is accepted by Dennis Publishing Ltd for any loss or damage arising from the receipt or use thereof. Company registered in England No. 1138891 Registered office: 30, Cleveland Street, London, W1T 4JD ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Amazon S3FS Automounter of sorts
Hi, I¹ve just started using Amazon S3 for storage and have used s3fs to mount the buckets on the file system. 1. I was wondering if anyone has a better method even though this meets my current needs. 2. if there is an automounter that could be tweaked to work with s3fs. The entries in my fstab are rather different to the normal disk based ones, s3fs#BUCKETNAME /mnt/s3/BUCKETNAME fuse allow_other,default_acl=public-read,noauto 0 0 ,so I wasn¹t sure if it is possible. My reasoning to want an automounter is I¹m not sure on the amount of extra calls made to the S3, so would prefer it not be connected if its not needed. Thanks, Matt NOTE: The information in this email is confidential and may be legally privileged, unless stated to the contrary. If you are not the intended recipient, you must not read, use or disseminate that information. Any opinions or comments are personal to the writer and do not represent the official view of Dennis Publishing Ltd. If you have received this email and are not a named addressee, please contact itdirec...@dennis.co.uk immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purpose, or disclose its contents to any other person. Although this email and any attachments are believed to be free of any virus, or other defects, it is the responsibility of the recipient to ensure that they are virus free and no responsibility is accepted by Dennis Publishing Ltd for any loss or damage arising from the receipt or use thereof. Company registered in England No. 1138891 Registered office: 30, Cleveland Street, London, W1T 4JD ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Amazon S3FS Automounter of sorts
Hey On Mon, Apr 12, 2010 at 5:24 PM, Matt Keating matt_keat...@dennis.co.uk wrote: I’ve just started using Amazon S3 for storage and have used s3fs to mount the buckets on the file system. What type of storage are you using it for. Web-app, Map/Reduce stuff, File-backup, etc ... I was wondering if anyone has a better method – even though this meets my current needs. Why do you want to use s3 like a file system? s3 does not have the properties of a posix system so you are bound be get some minor errors and problems. if there is an automounter that could be tweaked to work with s3fs. The entries in my fstab are rather different to the normal disk based ones, s3fs#BUCKETNAME /mnt/s3/BUCKETNAME fuse allow_other,default_acl=public-read,noauto 0 0 ,so I wasn’t sure if it is possible. I hope you are aware that everything you put on your s3 is publicly available if someone knows your bucket name. I use s3tools to transfer data between s3 and a folder on my machine. I tried mounting it like you, but just ran into too many problems, especially if you access files from many machines. Maybe a little more information on your use-case :) Cheers Didi -- Hoffmann Geerd-Dietger http://contact.ribalba.de ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Amazon S3FS Automounter of sorts
Thanks for the quick reply, What type of storage are you using it for. Web-app, Map/Reduce stuff, File-backup, etc ... Currently using it for serving files through cloudfront - so purely as a storage space for CDN delivery. Why do you want to use s3 like a file system? s3 does not have the properties of a posix system so you are bound be get some minor errors and problems. I found it works well with FTPing into the server and uploading to the mounted bucket. Reason its like this is that there are lots of different people who upload throughout our company. It was much easier giving out the FTP details, which were totally under our control (Username/Pass,Firewall,etc), rather than giving out the S3 logins. I hope you are aware that everything you put on your s3 is publicly available if someone knows your bucket name. Yes, I am aware of that - its all being served on the net anyway. If I remove the Pubic read only, will the files still be accessible via cloudfront? I use s3tools to transfer data between s3 and a folder on my machine. I tried mounting it like you, but just ran into too many problems, especially if you access files from many machines. What issues did you run into? As I haven't had any problems as of yet. MattNOTE: The information in this email is confidential and may be legally privileged, unless stated to the contrary. If you are not the intended recipient, you must not read, use or disseminate that information. Any opinions or comments are personal to the writer and do not represent the official view of Dennis Publishing Ltd. If you have received this email and are not a named addressee, please contact itdirec...@dennis.co.uk immediately by reply email and then delete this message from your system. Please do not copy it or use it for any purpose, or disclose its contents to any other person. Although this email and any attachments are believed to be free of any virus, or other defects, it is the responsibility of the recipient to ensure that they are virus free and no responsibility is accepted by Dennis Publishing Ltd for any loss or damage arising from the receipt or use thereof. Company registered in England No. 1138891 Registered office: 30, Cleveland Street, London, W1T 4JD ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
