Re: [openstack-dev] [Openstack] [Swift] Erasure coding reconstructor doesn't work
Correct, it by design. Swift doesn’t expect people to delete things “under the covers”. When the auditor finds a corrupted file, it’s the one that quantities it and knows that it also needs to invalidate the hashes.pkl file. This mechanism is there to minimize extra ‘stuff’ going on both at the node and on the cluster when it comes to making sure there is durability in the system. Wrt why the replication code seems to work if you delete just a .data (again, you shouldn’t do this as files don’t just disappear, the intention is that the auditor is in charge here) is because of some code in the replicator that I didn’t ‘mimic’ in the reconstructor and it doesn’t look like clay did either when he worked on it. Not really sure it was there – forces a listing every 10 passes for some reason. Clay? (see do_listdir in update() in the replciator) Thx Paul From: Changbin Liu [mailto:changbin@gmail.com] Sent: Wednesday, July 22, 2015 12:24 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Openstack] [Swift] Erasure coding reconstructor doesn't work Thanks, Paul and Clay. By deleted one data fragment I meant I rm only the data file. I did not delete the hashes.pkl file in the outer directory. I tried it again. This time deleting both the data file and the hashes.pkl file. The reconstructor is able to restore the data file correctly. But now I wonder: is it by design that EC does not handle an accidental deletion of just the data file? Deleting both data file and hashes.pkl file is more like a deliberately-created failure case instead of a normal one. To me Swift EC repairing seems different from the triple-replication mode, where you delete any data file copy, it will be restored. Thanks Changbin On Tue, Jul 21, 2015 at 5:28 PM, Luse, Paul E paul.e.l...@intel.commailto:paul.e.l...@intel.com wrote: I was about to ask that very same thing and, at the same time, if you can indicate if you’ve seen errors in any logs and if so please provide those as well. I’m hoping you just didn’t delete the hashes.pkl file though ☺ -Paul From: Clay Gerrard [mailto:clay.gerr...@gmail.commailto:clay.gerr...@gmail.com] Sent: Tuesday, July 21, 2015 2:22 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Openstack] [Swift] Erasure coding reconstructor doesn't work How did you deleted one data fragment? Like replication the EC consistency engine uses some sub directory hashing to accelerate replication requests in a consistent system - so if you just rm a file down in an hashdir somewhere you also need to delete the hashes.pkl up in the part dir (or call the invalidate_hash method like PUT, DELETE, POST, and quarantine do) Every so often someone discusses the idea of having the auditor invalidate a hash after long enough or take some action on empty hashdirs (mind the races!) - but its really only an issue when someone delete's something by hand so we normally manage to get distracted with other things. -Clay On Tue, Jul 21, 2015 at 1:38 PM, Changbin Liu changbin@gmail.commailto:changbin@gmail.com wrote: Folks, To test the latest feature of Swift erasure coding, I followed this document (http://docs.openstack.org/developer/swift/overview_erasure_code.html) to deploy a simple cluster. I used Swift 2.3.0. I am glad that operations like object PUT/GET/DELETE worked fine. I can see that objects were correctly encoded/uploaded and downloaded at proxy and object servers. However, I noticed that swift-object-reconstructor seemed don't work as expected. Here is my setup: my cluster has three object servers, and I use this policy: [storage-policy:1] policy_type = erasure_coding name = jerasure-rs-vand-2-1 ec_type = jerasure_rs_vand ec_num_data_fragments = 2 ec_num_parity_fragments = 1 ec_object_segment_size = 1048576 After I uploaded one object, I verified that: there was one data fragment on each of two object servers, and one parity fragment on the third object server. However, when I deleted one data fragment, no matter how long I waited, it never got repaired, i.e., the deleted data fragment was never regenerated by the swift-object-reconstructor process. My question: is swift-object-reconstructor supposed to be NOT WORKING given the current implementation status? Or, is there any configuration I missed in setting up swift-object-reconstructor? Thanks Changbin __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ
Re: [openstack-dev] [Openstack] [Swift] Erasure coding reconstructor doesn't work
I was about to ask that very same thing and, at the same time, if you can indicate if you’ve seen errors in any logs and if so please provide those as well. I’m hoping you just didn’t delete the hashes.pkl file though ☺ -Paul From: Clay Gerrard [mailto:clay.gerr...@gmail.com] Sent: Tuesday, July 21, 2015 2:22 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Openstack] [Swift] Erasure coding reconstructor doesn't work How did you deleted one data fragment? Like replication the EC consistency engine uses some sub directory hashing to accelerate replication requests in a consistent system - so if you just rm a file down in an hashdir somewhere you also need to delete the hashes.pkl up in the part dir (or call the invalidate_hash method like PUT, DELETE, POST, and quarantine do) Every so often someone discusses the idea of having the auditor invalidate a hash after long enough or take some action on empty hashdirs (mind the races!) - but its really only an issue when someone delete's something by hand so we normally manage to get distracted with other things. -Clay On Tue, Jul 21, 2015 at 1:38 PM, Changbin Liu changbin@gmail.commailto:changbin@gmail.com wrote: Folks, To test the latest feature of Swift erasure coding, I followed this document (http://docs.openstack.org/developer/swift/overview_erasure_code.html) to deploy a simple cluster. I used Swift 2.3.0. I am glad that operations like object PUT/GET/DELETE worked fine. I can see that objects were correctly encoded/uploaded and downloaded at proxy and object servers. However, I noticed that swift-object-reconstructor seemed don't work as expected. Here is my setup: my cluster has three object servers, and I use this policy: [storage-policy:1] policy_type = erasure_coding name = jerasure-rs-vand-2-1 ec_type = jerasure_rs_vand ec_num_data_fragments = 2 ec_num_parity_fragments = 1 ec_object_segment_size = 1048576 After I uploaded one object, I verified that: there was one data fragment on each of two object servers, and one parity fragment on the third object server. However, when I deleted one data fragment, no matter how long I waited, it never got repaired, i.e., the deleted data fragment was never regenerated by the swift-object-reconstructor process. My question: is swift-object-reconstructor supposed to be NOT WORKING given the current implementation status? Or, is there any configuration I missed in setting up swift-object-reconstructor? Thanks Changbin __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Swift] storage policies merge plan
Hi Chmouel, Storage policies can be fully tested and exercised on an SAIO; the docs that were recently merged to the feature/ec branch also include a second example policy using 2x replication to show this. No reason why devstack can’t be used but I’d propose it doesn’t make anything easier and/or more testable. Thx Paul From: Chmouel Boudjnah [mailto:chmo...@enovance.com] Sent: Tuesday, May 27, 2014 2:08 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Swift] storage policies merge plan On Tue, May 27, 2014 at 10:02 AM, Hua ZZ Zhang zhu...@cn.ibm.commailto:zhu...@cn.ibm.com wrote: Do it make sense to support storage policy work in devstack so that it can be more easily tested? -Edward Zhang I don't think storage policy on one VM (which has other OpenStack services) like usually setup for devstack is very practical, but I may be wrong. Chmouel. [Inactive hide details for John Dickinson ---2014-05-24 上午 06:27:13---John Dickinson m...@not.mn]John Dickinson ---2014-05-24 上午 06:27:13---John Dickinson m...@not.mnmailto:m...@not.mn John Dickinson m...@not.mnmailto:m...@not.mn 2014-05-24 上午 06:27 Please respond to OpenStack Development Mailing List \(not for usage questions\) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org To OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org, openstack-operat...@lists.openstack.orgmailto:openstack-operat...@lists.openstack.org cc Subject [openstack-dev] [Swift] storage policies merge plan We've been working for a long time on the feature/ec branch in the swift repo. It's now done and needs to be merged into master to be generally available. Here's how the integration is going to work: 1) The feature/ec branch will be refactored into a series of dependent reviewable patches 2) The patch chain will be proposed to master, and master will enter a freeze until the storage policy patches land 3) The first patch in the chain will be marked as -2 to plug the chain 4) The Swift community will review and approve all patches in the chain. 5) When all patches in the chain are approved, the first -2 will be removed and the whole chain will be sent to the CI system There are two things that I'll ask of you during this time. First, please commit time to reviewing the storage policy patches. Second, please do not deploy a version of Swift that is midway through the storage policy patch chain. I don't expect it to break anything, but it's a complicating factor best to be avoided. I will send out another email when the patch chain has been proposed to master and to announce the freeze. --John (See attached file: signature.asc)___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Swift] new core members
Congrats guys!! -Original Message- From: John Dickinson [mailto:m...@not.mn] Sent: Thursday, March 27, 2014 1:34 PM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [Swift] new core members I'm pleased to announce that Alistair Coles and Christian Schwede have both joined the Swift core team. They have both been very active in the Swift community, contributing both code and reviews. Both Alistair and Christian work with large-scale production Swift clusters, and I'm happy to have them on the core team. Alistair and Christian, thanks for your work in the community and for taking on more responsibility. I'm glad we'll be able to continue to work together on Swift. --John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] question about storage policies for Swift
future writes to that container will operate only on devices assorted with the policy specified. For example if you wanted to create a container that maps to the turkey policy: o curl -v -X PUT -H 'X-Auth-Token:AUTH_tk13909e8beb0e4b5faafc5c9c7d219c6a' -H X-Storage-Policy: turkey http://127.0.0.1:8080/v1/AUTH_test/myCont/ -Object operations (API) do not change, the fact that you're using a container created with a specific policy is all that's needed -Note that you can't change the policy of a container once set - you'll get an error if you try) but you can move object from one container to another where the container can, of course, have different policies -You will find, if you look, that the directory hierarchy on the object nodes is slightly different, instead of just an objects directory, you will see objects, objects-1, and so on. Hopefully that answers your question and helps you, and anyone else, get started if you want an early look Thanks! Paul From: Gil Vernik [mailto:g...@il.ibm.com] Sent: Saturday, February 8, 2014 11:29 PM To: Luse, Paul E Subject: question about storage policies for Swift Dear Paul Luse, I am trying to better understand your new additions to Swift about storage policies. I started with your recently merged patch - Storage Policy / DiskFile Interface Support . Unfortunately I find it a bit hard to understand or to find correct place where is the definition of storage policy itself. Can you please, explain to me in few sentences, what is storage policy looks like? Some real examples? I see that storage_policy.py loads policies from swift.conf and each policy has index and name, but I don't understand how policies looks like, since i didn't find swift.conf and I didn't find examples in documentation. Thanking you in advance, Gil Vernik. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Swift] meeting time updated
What a corporate thing to do :) Good call though John -Paul -Original Message- From: John Dickinson [mailto:m...@not.mn] Sent: Thursday, February 6, 2014 9:16 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [Swift] meeting time updated Historically, the Swift team meetings have been every other week. In order to keep better track of things (and hopefully to get more specific attention on languishing reviews), we're moving to a weekly meeting schedule. New meeting time: every Wednesday at 1900UTC in #openstack-meeting The meeting agenda is tracked at https://wiki.openstack.org/wiki/Meetings/Swift --John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Swift] question on Application class concurrency; paste.app_factory mechanism
Thanks for the reply Pete. Don't quite follow you so will continue to do some reading and experimenting and see if I can't come up some additional questions... Thx Paul -Original Message- From: Pete Zaitcev [mailto:zait...@redhat.com] Sent: Thursday, July 25, 2013 9:51 AM To: OpenStack Development Mailing List Cc: Luse, Paul E Subject: Re: [openstack-dev] [Swift] question on Application class concurrency; paste.app_factory mechanism On Wed, 24 Jul 2013 02:31:45 + Luse, Paul E paul.e.l...@intel.com wrote: I was thinking that each connection would get its own instance thus it would be sage to store connection-transient information there but I was surprised by my quick test. Yeah, you have it tracked in __call__, typically hanging off controller. -- Pete ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Swift] erasure codes, digging deeper
I think most folks are on the same page wrt EC being consider a level or storage policy as we've been discussing in other forums. I saw the previous note on account versus container and was actually thinking it made more sense to me at least to enable billing per container as opposed to trying to shoehorn attribute of a set of data (like a storage policy) to and account so I like where you're going w/the suggestion. I'm not even sure we need to answer the detailed question of what defines these levels, or storage policies, only that the design allows for their efficient use and flexible definitions. Thx Paul From: David Hadas [mailto:david.ha...@gmail.com] Sent: Monday, July 22, 2013 9:35 AM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] [Swift] erasure codes, digging deeper Hi, In Portland, we discussed a somewhat related issue of having multiple replication levels in one Swift cluster. It may be that a provider would not wish to expose the use of EC or the level of replication used. For example a provider may offer a predefined set of services such as Gold, Silver and Bronze, Aluminum which a user can choose from. The provider may decide how each level is implemented (As a silly example: Gold is 4 way replication, Silver is a 3 way replication, Bronze is EC, Aluminum is single replica without EC). Does it make sense to consider EC as an implementation of a certain service level (the same as for example the choice of the number of replicas)? Now we are back to the question of what is the right unit in which we define this 'level of service' (Gold, Silver...). Should the level of service be defined when the account is created or should we allow a user to state: I would like to have a container with Gold to keep my work, Bronze to keep my family pictures and videos and Aluminum to keep a copy of all my music files. If we choose to enable container service levels, we need to enable billing per level of service such that a single account could be billed for the actual use it has done per each level of service. Maybe we even need to have all statistics gathered to be grouped by service level. I am not sure if we can escape that even with account service levels. DH On Thu, Jul 18, 2013 at 9:37 PM, John Dickinson m...@not.mnmailto:m...@not.mn wrote: Yes, and I'd imagine that the normal default would be for replicated data. Moving the granularity from a container to account-based, as Chmouel and Chuck said, is interesting too. --John On Jul 18, 2013, at 11:32 AM, Christian Schwede i...@cschwede.demailto:i...@cschwede.de wrote: A solution to this might be to set the default policy as a configuration setting in the proxy. If you want a replicated swift cluster just allow this policy in the proxy and set it to default. The same for EC cluster, just set the allowed policy to EC. If you want both (and let your users decide which policy to use) simply configure a list of allowed policies with the first one in the list as the default policy in case they don't set a policy during container creation. Am 18.07.13 20:15, schrieb Chuck Thier: I think you are missing the point. What I'm talking about is who chooses what data is EC and what is not. The point that I am trying to make is that the operators of swift clusters should decide what data is EC, not the clients/users. How the data is stored should be totally transparent to the user. Now if we want to down the road offer user defined classes of storage (like how S3 does reduced redundancy), I'm cool with that, just that it should be orthogonal to the implementation of EC. -- Chuck On Thu, Jul 18, 2013 at 12:57 PM, John Dickinson m...@not.mnmailto:m...@not.mn mailto:m...@not.mnmailto:m...@not.mn wrote: Are you talking about the parameters for EC or the fact that something is erasure coded vs replicated? For the first, that's exactly what we're thinking: a deployer sets up one (or more) policies and calls them Alice, Bob, or whatever, and then the API client can set that on a particular container. This allows users who know what they are doing (ie those who know the tradeoffs and their data characteristics) to make good choices. It also allows deployers who want to have an automatic policy to set one up to migrate data. For example, a deployer may choose to run a migrator process that moved certain data from replicated to EC containers over time (and drops a manifest file in the replicated tier to point to the EC data so that the URL still works). Like existing features in Swift (eg large objects), this gives users the ability to flexibly store their data with a nice interface yet still have the ability to get at some of the pokey bits underneath. --John On Jul 18, 2013, at 10:31 AM, Chuck Thier cth...@gmail.commailto:cth...@gmail.com mailto:cth...@gmail.commailto:cth...@gmail.com wrote: I'm with
