Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-24 Thread Pete Zaitcev
On Thu, 18 Jul 2013 12:31:02 -0500 Chuck Thier cth...@gmail.com wrote: I'm with Chmouel though. It seems to me that EC policy should be chosen by the provider and not the client. For public storage clouds, I don't think you can make the assumption that all users/clients will understand the

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-22 Thread Luse, Paul E
and flexible definitions. Thx Paul From: David Hadas [mailto:david.ha...@gmail.com] Sent: Monday, July 22, 2013 9:35 AM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] [Swift] erasure codes, digging deeper Hi, In Portland, we discussed a somewhat related issue of having multiple

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-22 Thread John Dickinson
On Jul 22, 2013, at 9:34 AM, David Hadas david.ha...@gmail.com wrote: Hi, In Portland, we discussed a somewhat related issue of having multiple replication levels in one Swift cluster. It may be that a provider would not wish to expose the use of EC or the level of replication used.

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread Chmouel Boudjnah
On Thu, Jul 18, 2013 at 12:42 AM, John Dickinson m...@not.mn wrote: * Erasure codes (vs replicas) will be set on a per-container basis I was wondering if there was any reasons why it couldn't be as per-account basis as this would allow an operator to have different type of an account and

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread John Dickinson
Check out the slides I linked. The plan is to enable an EC policy that is then set on a container. A cluster may have a replication policy and one or more EC policies. Then the user will be able to choose the policy for a particular container. --John On Jul 18, 2013, at 2:50 AM, Chmouel

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread Clay Gerrard
I'm not sure if I agree with the giving justifications (harder to bill, confusing to users), but I *do* think it could simplify some of the implementation (since users can't create accounts withe the wrong storage policy willy nilly). I think eventually some use cases may want mixed accounts

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread Chuck Thier
I think you are missing the point. What I'm talking about is who chooses what data is EC and what is not. The point that I am trying to make is that the operators of swift clusters should decide what data is EC, not the clients/users. How the data is stored should be totally transparent to the

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread Christian Schwede
A solution to this might be to set the default policy as a configuration setting in the proxy. If you want a replicated swift cluster just allow this policy in the proxy and set it to default. The same for EC cluster, just set the allowed policy to EC. If you want both (and let your users decide

Re: [openstack-dev] [Swift] erasure codes, digging deeper

2013-07-18 Thread John Dickinson
Yes, and I'd imagine that the normal default would be for replicated data. Moving the granularity from a container to account-based, as Chmouel and Chuck said, is interesting too. --John On Jul 18, 2013, at 11:32 AM, Christian Schwede i...@cschwede.de wrote: A solution to this might be to

[openstack-dev] [Swift] erasure codes, digging deeper

2013-07-17 Thread John Dickinson
Last week we wrote a blog post about introducting erasure codes into Swift. Today, I'm happy to share more technical details around this feature. We've posted an overview of our design and some of the tradeoffs in the feature at