Re: [ceph-users] ceph on peta scale
So is there any other alternative for over the WAN deployment .. I hava use case to connect two Swedish unversities (few hundreds km apart) . Target is that user from univ A can write to cluster to univ B and can read the data from other users . /Zee On Tue, Jan 13, 2015 at 7:41 AM, Robert van Leeuwen robert.vanleeu...@spilgames.com wrote: however for geographic distributed datacentres specially when network flactuate how to handle that as i read it seems CEPH need big pipe of network Ceph isn't really suited for WAN-style distribution. Some users have high-enough and consistent-enough bandwidth (with low enough latency) to do it, but otherwise you probably want to use Ceph within the data centers and layer something else on top of it. Indeed. Ceph is not aware of WAN links. So reads and writes will be done remotely even if there is a copy locally. Bandwidth might not be much of an issue but latency certainly will be. Although bandwidth during a rebalance of data might also be problematic... Cheers, Robert van Leeuwen -- Regards Zeeshan Ali Shah System Administrator - PDC HPC PhD researcher (IT security) Kungliga Tekniska Hogskolan +46 8 790 9115 http://www.pdc.kth.se/members/zashah ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph on peta scale
Gregory Farnum greg@... writes: Ceph isn't really suited for WAN-style distribution. Some users have high-enough and consistent-enough bandwidth (with low enough latency) to do it, but otherwise you probably want to use Ceph within the data centers and layer something else on top of it. -Greg cockroachDB might be what you are looking for? http://cockroachdb.org/ hth, James ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph on peta scale
Thanks James, I will look into it Zeeshan On Tue, Jan 13, 2015 at 2:00 PM, James wirel...@tampabay.rr.com wrote: Gregory Farnum greg@... writes: Ceph isn't really suited for WAN-style distribution. Some users have high-enough and consistent-enough bandwidth (with low enough latency) to do it, but otherwise you probably want to use Ceph within the data centers and layer something else on top of it. -Greg cockroachDB might be what you are looking for? http://cockroachdb.org/ hth, James ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Regards Zeeshan Ali Shah System Administrator - PDC HPC PhD researcher (IT security) Kungliga Tekniska Hogskolan +46 8 790 9115 http://www.pdc.kth.se/members/zashah ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph on peta scale
So is there any other alternative for over the WAN deployment .. I hava use case to connect two Swedish unversities (few hundreds km apart) . Target is that user from univ A can write to cluster to univ B and can read the data from other users . You could have a look at OpenStack Swift: it has geo-replication. Swift is an object store so it is not mountable like a block device (same functionality/api as the Rados gateway). Depending on the expected usage this could pose a problem... Another thing to take into account is that Swift is eventually consistent instead of strongly consistent. I think Ceph is working on something similar for the Rados gateway. Cheers, Robert van Leeuwen ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph on peta scale
On Mon, Jan 12, 2015 at 3:55 AM, Zeeshan Ali Shah zas...@pdc.kth.se wrote: Thanks Greg, No i am more into large scale RADOS system not filesystem . however for geographic distributed datacentres specially when network flactuate how to handle that as i read it seems CEPH need big pipe of network Ceph isn't really suited for WAN-style distribution. Some users have high-enough and consistent-enough bandwidth (with low enough latency) to do it, but otherwise you probably want to use Ceph within the data centers and layer something else on top of it. -Greg /Zee On Fri, Jan 9, 2015 at 7:15 PM, Gregory Farnum g...@gregs42.com wrote: On Thu, Jan 8, 2015 at 5:46 AM, Zeeshan Ali Shah zas...@pdc.kth.se wrote: I just finished configuring ceph up to 100 TB with openstack ... Since we are also using Lustre in our HPC machines , just wondering what is the bottle neck in ceph going on Peta Scale like Lustre . any idea ? or someone tried it If you're talking about people building a petabyte Ceph system, there are *many* who run clusters of that size. If you're talking about the Ceph filesystem as a replacement for Lustre at that scale, the concern is less about the raw amount of data and more about the resiliency of the current code base at that size...but if you want to try it out and tell us what problems you run into we will love you forever. ;) (The scalable file system use case is what actually spawned the Ceph project, so in theory there shouldn't be any serious scaling bottlenecks. In practice it will depend on what kind of metadata throughput you need because the multi-MDS stuff is improving but still less stable.) -Greg -- Regards Zeeshan Ali Shah System Administrator - PDC HPC PhD researcher (IT security) Kungliga Tekniska Hogskolan +46 8 790 9115 http://www.pdc.kth.se/members/zashah ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph on peta scale
Thanks Greg, No i am more into large scale RADOS system not filesystem . however for geographic distributed datacentres specially when network flactuate how to handle that as i read it seems CEPH need big pipe of network /Zee On Fri, Jan 9, 2015 at 7:15 PM, Gregory Farnum g...@gregs42.com wrote: On Thu, Jan 8, 2015 at 5:46 AM, Zeeshan Ali Shah zas...@pdc.kth.se wrote: I just finished configuring ceph up to 100 TB with openstack ... Since we are also using Lustre in our HPC machines , just wondering what is the bottle neck in ceph going on Peta Scale like Lustre . any idea ? or someone tried it If you're talking about people building a petabyte Ceph system, there are *many* who run clusters of that size. If you're talking about the Ceph filesystem as a replacement for Lustre at that scale, the concern is less about the raw amount of data and more about the resiliency of the current code base at that size...but if you want to try it out and tell us what problems you run into we will love you forever. ;) (The scalable file system use case is what actually spawned the Ceph project, so in theory there shouldn't be any serious scaling bottlenecks. In practice it will depend on what kind of metadata throughput you need because the multi-MDS stuff is improving but still less stable.) -Greg -- Regards Zeeshan Ali Shah System Administrator - PDC HPC PhD researcher (IT security) Kungliga Tekniska Hogskolan +46 8 790 9115 http://www.pdc.kth.se/members/zashah ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph on peta scale
however for geographic distributed datacentres specially when network flactuate how to handle that as i read it seems CEPH need big pipe of network Ceph isn't really suited for WAN-style distribution. Some users have high-enough and consistent-enough bandwidth (with low enough latency) to do it, but otherwise you probably want to use Ceph within the data centers and layer something else on top of it. Indeed. Ceph is not aware of WAN links. So reads and writes will be done remotely even if there is a copy locally. Bandwidth might not be much of an issue but latency certainly will be. Although bandwidth during a rebalance of data might also be problematic... Cheers, Robert van Leeuwen ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph on peta scale
On Thu, Jan 8, 2015 at 5:46 AM, Zeeshan Ali Shah zas...@pdc.kth.se wrote: I just finished configuring ceph up to 100 TB with openstack ... Since we are also using Lustre in our HPC machines , just wondering what is the bottle neck in ceph going on Peta Scale like Lustre . any idea ? or someone tried it If you're talking about people building a petabyte Ceph system, there are *many* who run clusters of that size. If you're talking about the Ceph filesystem as a replacement for Lustre at that scale, the concern is less about the raw amount of data and more about the resiliency of the current code base at that size...but if you want to try it out and tell us what problems you run into we will love you forever. ;) (The scalable file system use case is what actually spawned the Ceph project, so in theory there shouldn't be any serious scaling bottlenecks. In practice it will depend on what kind of metadata throughput you need because the multi-MDS stuff is improving but still less stable.) -Greg ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
