[openstack-dev] [Swift] PTL candidacy
I'd be honored to continue to serve as your PTL for OpenStack Swift. ## Looking back at the Newton cycle During the Newton cycle, the Swift community has delivered at-rest encryption. This feature is the culmination of a more than year of work, and it enables Swift to be deployed in places where it was previously not possible to be used. The other big thing that's been going on during the Newton cycle is our work on a Golang implementation of the object server and replication engine. This change not only solved critical issues for existing deployments, but it also enables a lot of the future work that needs to happen in the years to come (more on that later!). Golang in Swift is a great OpenStack community success story. An existing Swift deployer (Rackspace) was seeing issues in their environment. One developer there started playing with Golang and reimplemented a small part of Swift. The results were good--very good --and the rest of the development team there started focusing on this Golang implementation. The developers shared these results with the rest of the community, and the Golang code started being developed in a feature branch of Swift, where others started playing with this code as well. Rackspace has been running this in production with great success, and as a community we're bringing into "mainline" Swift so that everyone can benefit. It's great to see community members play with new ideas, bring those to the rest of the community, and see them adopted to solve problems for everyone. As most people in the Swift community know, and many in the wider OpenStack community have seen, there has been a very large debate on the use of Golang in OpenStack projects. Much of my time in this cycle has been spent working with the TC on this conversation, and I expect it to continue into the Ocata cycle as well. I'm not satisfied with the current decision by the TC, and I'll keep pushing for including Golang inside of OpenStack. ## Looking forward to Ocata and Beyond Beyond grand debates about programming languages, there's a ton of stuff going on in Swift right now. Here's a partial list of things people are working on: - Golang object server and replication - Transparent container sharding - Increase ring partition power - Global erasure codes - Policy migrations - Symlinks - Improvements to encryption functionality - Automatic tiering - Composite rings Many of these have been in progress for a while, and I fully expect several to be finished within the Ocata cycle. Many of these features are related to bigger and bigger clusters. It's tremendously exciting to see where Swift is being used today, but we're in no way "done". There is a lot more for us to do to continue to solve storage problems for users. Looking much further into the future, there's several things I'd like us to work towards. - Small file optimization - Dealing with larger drives and more dense storage servers - Using NV memory The reason I know we'll get this stuff done and continue to make Swift the best open source object storage system is because of our community. As the Swift PTL, I feel it's my duty to enable the community to solve these problems by making every community member more productive. This is my focus. --John signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [swift] PTL candidacy
I am submitting my candidacy for Swift PTL for the Newton cycle. ### Community growth During the Mitaka cycle, we've seen the community grow from about 440 contributors to 531 today. That's a 20% growth in the last six months. This growth rate is huge, and I'm really happy to see all the new people coming into the community. New contributors bring new ideas, new use cases, and fresh eyes on old problems. This is the hallmark of a healthy community. New contributors reflect the broader picture of what's going on with Swift, too. When we see new contributors from companies who are already actively participating in the community, it shows that they are using Swift more. When new companies join the community for the first time, it shows new places where Swift is being used to solve real-world storage problems. Both situations are exciting. Swift is growing. More people are using it to store more data and solve more storage problems. As I've said many times before, my vision for Swift is that it will be used by everyone, every day, even if people don't realize it. Community growth is one way we can see that this is happening today. ### Tracking the community One thing I've been working on over the last year is tracking our community and finding ways to understand it and help it continue to grow. You can see some of this work in the graphs below. http://d.not.mn/total_contribs.png http://d.not.mn/active_contribs.png I've been working on a few more interesting graphs and metrics too. I've shared one of these before: visualizing individual contributor activity over time. You can see the results at http://d.not.mn/contrib_activity.png. This is a rather large graph, but it's fairly simple to read. The x-axis is time (from Swift's initial release to today). Each line on the y-axis is a person's activity in Swift. Blue boxes are for authoring a patch; green boxes are for a review. This graph has been invaluable in helping understand who is contributing to the project and what activity active contributors are engaged in. One thing I've learned is that there is a class of contributor who has been involved for a long time, but is only active infrequently. Most of these people are operators who are normally running production clusters but occasionally need to submit a patch or review upstream. Another thing I've worked on is a way to find out what the community as a whole thinks is important. We can start to find a lot of this info from information we already have. For example, we can get a list of every patch that every contributor has starred to see if there is any commonality between them. The patches that are more often starred are likely to be more important, from the community perspective. I've taken that basic idea, along with some further parsing of information from gerrit, and created a review dashboard. It's going to keep changing, but you can see its current incarnation at http://not.mn/swift/swift_community_dashboard.html (and linked in the #openstack-swift channel topic). So far, I've seen this dashboard result in a dramatic decrease in the number of unreviewed patches, and as the dashboard improves, I expect it to lead to a similar improvement in review times for important patches. ### Current struggles Tracking the community (both with metrics like above and from simply talking to people) shines a light on common problems in the community. Two of the most-often raised issues are that of long patch review times and review prioritization. We've been improving both of these with several tools in the past, and we're currently in a much better place than we've been in the past. As PTL, I feel it's my duty to enable the community to solve these problems. I will continue to improve the tools we have and create new tools as necessary so that we know what's going on, what to work on, and make every contributor effective. ### Goals for newton Looking forward to the Newton cycle, I want to see three things happen. 1. Increased contributor growth 2. Progress on significant code efforts, including crypto, improved global clusters, and improved performance 3. Better tools and info for community prioritization and community tracking The community is well-positioned to meet these goals, and I will be honored to continue to serve you as PTL for OpenStack Swift. --John signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Swift] PTL Candidacy
I'm announcing my candidacy for OpenStack Swift PTL for the Liberty cycle. Swift was originally written to provide large-scale, multi-user, highly-available, durable object storage. Why do we need that? Because the way people produce and consume data is changing, and, therefore, the requirements that applications put on storage infrastructure is also changing. When Swift was released way back in the Austin release of OpenStack (5 years ago!), it already met these goals. And during the last 5 years, we've seen that base functionality improve and important new functionality added. In the last two years alone, we've added global cluster support, storage policies, and we're just now finishing up erasure code support. These kind of things are crucial for enabling Swift to be the storage system used for all unstructured data. As I've said many times before, my vision for Swift is that everyone, everywhere will use it every day, even if they don't realize it. We're well on our way there, with Swift being used today from everything from backups, to games, CDNs, financial institutions, websites, mobile apps, and beyond. During the next six months (the OpenStack Liberty cycle), we've got some important work to do. We'll be finishing up the erasure code work, we're working on on-disk encryption, and we've got a lot of work to do to improve existing areas of the code. For example, I'd like to see improvements in object server and replication performance, and I'd like to see fruit come from the ongoing discussions about large container support. These sorts of things are important to ensure Swift's success for the next five years of it's life. Longer term (beyond the Liberty cycle), there are some other things I think are pretty important. First, and this is one of the more exciting things going on in Swift right now, many different media vendors are coming to the Swift community asking how they can ensure that Swift natively supports their media. There's a company that has written a tape library connector for Swift and is open-sourcing it. Hard drive vendors are developing new technology (like SMR drives) and new protocols (like Kinetic), and Swift needs to work with these natively. And on the flash side, I want to see Swift directly support the nuances of flash so that we can be well-positioned for any use case that people need. In addition to native media support, I want to ensure that we are supportive of the application developers who are using Swift. This means encouraging and promoting work on SDKs and ensuring that application developers use Swift as their first choice for cloud storage. An important part of the PTL role is supporting the developer community. As PTL I want to continue working on making sure the tools we have as a community are the best for the job we need to do--the best ways to track what's going on, coordinate with each other, and get stuff done. It's been a privilege to work with the incredible Swift contributors, and I'm proud to be a part of Swift. I'd be honored to continue as your OpenStack Swift PTL. Thank you, John signature.asc Description: Message signed with OpenPGP using GPGMail __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Swift] PTL Candidacy
confirmed On Tue, Apr 7, 2015 at 2:53 PM, John Dickinson m...@not.mn wrote: I'm announcing my candidacy for OpenStack Swift PTL for the Liberty cycle. Swift was originally written to provide large-scale, multi-user, highly-available, durable object storage. Why do we need that? Because the way people produce and consume data is changing, and, therefore, the requirements that applications put on storage infrastructure is also changing. When Swift was released way back in the Austin release of OpenStack (5 years ago!), it already met these goals. And during the last 5 years, we've seen that base functionality improve and important new functionality added. In the last two years alone, we've added global cluster support, storage policies, and we're just now finishing up erasure code support. These kind of things are crucial for enabling Swift to be the storage system used for all unstructured data. As I've said many times before, my vision for Swift is that everyone, everywhere will use it every day, even if they don't realize it. We're well on our way there, with Swift being used today from everything from backups, to games, CDNs, financial institutions, websites, mobile apps, and beyond. During the next six months (the OpenStack Liberty cycle), we've got some important work to do. We'll be finishing up the erasure code work, we're working on on-disk encryption, and we've got a lot of work to do to improve existing areas of the code. For example, I'd like to see improvements in object server and replication performance, and I'd like to see fruit come from the ongoing discussions about large container support. These sorts of things are important to ensure Swift's success for the next five years of it's life. Longer term (beyond the Liberty cycle), there are some other things I think are pretty important. First, and this is one of the more exciting things going on in Swift right now, many different media vendors are coming to the Swift community asking how they can ensure that Swift natively supports their media. There's a company that has written a tape library connector for Swift and is open-sourcing it. Hard drive vendors are developing new technology (like SMR drives) and new protocols (like Kinetic), and Swift needs to work with these natively. And on the flash side, I want to see Swift directly support the nuances of flash so that we can be well-positioned for any use case that people need. In addition to native media support, I want to ensure that we are supportive of the application developers who are using Swift. This means encouraging and promoting work on SDKs and ensuring that application developers use Swift as their first choice for cloud storage. An important part of the PTL role is supporting the developer community. As PTL I want to continue working on making sure the tools we have as a community are the best for the job we need to do--the best ways to track what's going on, coordinate with each other, and get stuff done. It's been a privilege to work with the incredible Swift contributors, and I'm proud to be a part of Swift. I'd be honored to continue as your OpenStack Swift PTL. Thank you, John __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Elizabeth Krumbach Joseph || Lyz || pleia2 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Swift] PTL candidacy
John, You are awesome! There are very few PTL who answer queries like you. Thanks! On Fri, Sep 26, 2014 at 1:46 AM, Tristan Cacqueray tristan.cacque...@enovance.com wrote: confirmed On 25/09/14 02:50 PM, John Dickinson wrote: I'm announcing my candidacy for Swift PTL. I've been involved with Swift specifically and OpenStack in general since the beginning. I'd like to continue to serve in the role as Swift PTL. In my last candidacy email[1], I talked about several things I wanted to focus on in Swift. 1) Storage policies. This is done, and we're currently building on it to implement erasure code storage in Swift. 2) Focus on performance and efficiency. This is an ongoing thing that is never done, but we have made improvements here, and there are some other interesting things in-progress right now (like zero-copy data paths). 3) Better QA. We've added a third-party test cluster to the CI system, but I'd like to improve this further, for example by adding our internal integration tests (probe tests) to our QA pipeline. 4) Better community efficiency. Again, we've made some small improvements here, but we have a ways to go yet. Our review backlog is large, and it takes a while for patches to land. We need to continue to improve community efficiency on these metrics. Overall, I want to ensure that Swift continues to provide a stable and robust object storage engine. Focusing on the areas listed above will help us do that. We'll continue to build functionality that allows applications to rely on Swift to take over hard problems of storage so that apps can focus on adding their value without worrying about storage. My vision for Swift is that everyone will use it every day, even if they don't realize it. Together we can make it happen. --John [1] http://lists.openstack.org/pipermail/openstack-dev/2014-March/031450.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Swift] PTL candidacy
I'm announcing my candidacy for Swift PTL. I've been involved with Swift specifically and OpenStack in general since the beginning. I'd like to continue to serve in the role as Swift PTL. In my last candidacy email[1], I talked about several things I wanted to focus on in Swift. 1) Storage policies. This is done, and we're currently building on it to implement erasure code storage in Swift. 2) Focus on performance and efficiency. This is an ongoing thing that is never done, but we have made improvements here, and there are some other interesting things in-progress right now (like zero-copy data paths). 3) Better QA. We've added a third-party test cluster to the CI system, but I'd like to improve this further, for example by adding our internal integration tests (probe tests) to our QA pipeline. 4) Better community efficiency. Again, we've made some small improvements here, but we have a ways to go yet. Our review backlog is large, and it takes a while for patches to land. We need to continue to improve community efficiency on these metrics. Overall, I want to ensure that Swift continues to provide a stable and robust object storage engine. Focusing on the areas listed above will help us do that. We'll continue to build functionality that allows applications to rely on Swift to take over hard problems of storage so that apps can focus on adding their value without worrying about storage. My vision for Swift is that everyone will use it every day, even if they don't realize it. Together we can make it happen. --John [1] http://lists.openstack.org/pipermail/openstack-dev/2014-March/031450.html signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Swift] PTL candidacy
confirmed On 25/09/14 02:50 PM, John Dickinson wrote: I'm announcing my candidacy for Swift PTL. I've been involved with Swift specifically and OpenStack in general since the beginning. I'd like to continue to serve in the role as Swift PTL. In my last candidacy email[1], I talked about several things I wanted to focus on in Swift. 1) Storage policies. This is done, and we're currently building on it to implement erasure code storage in Swift. 2) Focus on performance and efficiency. This is an ongoing thing that is never done, but we have made improvements here, and there are some other interesting things in-progress right now (like zero-copy data paths). 3) Better QA. We've added a third-party test cluster to the CI system, but I'd like to improve this further, for example by adding our internal integration tests (probe tests) to our QA pipeline. 4) Better community efficiency. Again, we've made some small improvements here, but we have a ways to go yet. Our review backlog is large, and it takes a while for patches to land. We need to continue to improve community efficiency on these metrics. Overall, I want to ensure that Swift continues to provide a stable and robust object storage engine. Focusing on the areas listed above will help us do that. We'll continue to build functionality that allows applications to rely on Swift to take over hard problems of storage so that apps can focus on adding their value without worrying about storage. My vision for Swift is that everyone will use it every day, even if they don't realize it. Together we can make it happen. --John [1] http://lists.openstack.org/pipermail/openstack-dev/2014-March/031450.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Swift] PTL candidacy
I'm announcing my candidacy for Swift PTL. I've been involved with Swift specifically and OpenStack in general since the beginning. I'd like to continue to serve in the role as Swift PTL. Swift has grown quite a bit over the last 4 years. In this past year, we've added major new features refactored significant areas of the code to improve efficiency and extensibility. We've added support for global clusters. We've significantly refactored replication to be more efficient. We've cleaned up the volume interface to make it much simpler to extend. Swift is a great storage engine, powering some of the world's largest storage clouds. Let's keep making it better. Going forward, I'd like to address four things in Swift in the next year: 1) Finish storage policies, including erasure code support. In my opinion, this is the biggest feature in Swift since it was open-sourced, and I'm really excited by the opportunities it enables. I sent an email earlier this month about our current plan on getting storage policies finished up: http://lists.openstack.org/pipermail/openstack-dev/2014-March/030937.html 2) Focus on performance and efficiency rather than on a feature train. We've started on several things here, including the ssync replication improvements and some profiling middleware. I'd also like to see improvement in replication bandwidth efficiency (especially with global clusters), time-to-first-byte latency improvement, better support of very dense storage, and support higher concurrency with less resources. 3) Better QA. Swift has always been a very stable system. We need to ensure that it remains stable, especially as new feature go in and other parts of the codebase change. Examples here include better functional test coverage, testing against real clusters, more end-to-end testing of workflows, running probetests automatically against submitted changes, and tracking performance metrics against patches. 4) Better community efficiency. As the community has grown, we need to get better at offering feedback channels from production deployments, especially from non-developers. We need to get better at reducing the patch review time and encouraging newer developers to jump in and offer patches. These are the things that I want to focus on as PTL in the next 6 to 12 months. My vision for Swift is that everyone will use it every day, even if they don't realize it. Together we can make it happen. --John signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Swift] PTL Candidacy
I would like to nominate myself for PTL of Swift. I've been involved in OpenStack Swift since it started, and I'd like to share a few of the thins in-progress and where I want to see Swift go. Swift has always been a world-class storage system, proven at scale and production-ready from day one. In the past few years Swift has been deployed in public and private storage clouds all over the world, and it is in use at the largest companies in the world. My goal for Swift is that everyone will use Swift, every day, even if they don't realize it. And taking a look at where Swift is being used today, we're well on our way to that goal. We'll continue to move towards Swift being everywhere as Swift grows to solve more real-world use cases. Right now, there is work being done in Swift that will give deployers a very high degree of flexibility in how they can store data. We're working on implementing storage policies in Swift. These storage policies give deployers the ability to choose: (a) what subset of hardware the data lives on (b) how the data is stored across that hardware (c) how communication with an actual storage volume happens. Supporting (a) allows for storage tiers and isolated storage hardware. Supporting (b) allows for different replication or non-replication schemes. Supporting (c) allows for specific optimizations for particular filesystems or storage hardware. Combined, it's even feasable to have a Swift cluster take advantage of other storage systems as a storage policy (imagine an S3 storage policy). As PTL, I want to help coordinate this work and see it to completion. Many people from many different companies are working on it, in addition to the normal day-to-day activity in Swift. I'm excited by the future of Swift, and would be honored to continue to serve as Swift PTL. --John signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Swift] PTL Candidacy
Good plan! On Wed, Sep 25, 2013 at 6:40 AM, John Dickinson m...@not.mn wrote: I would like to nominate myself for PTL of Swift. I've been involved in OpenStack Swift since it started, and I'd like to share a few of the thins in-progress and where I want to see Swift go. Swift has always been a world-class storage system, proven at scale and production-ready from day one. In the past few years Swift has been deployed in public and private storage clouds all over the world, and it is in use at the largest companies in the world. My goal for Swift is that everyone will use Swift, every day, even if they don't realize it. And taking a look at where Swift is being used today, we're well on our way to that goal. We'll continue to move towards Swift being everywhere as Swift grows to solve more real-world use cases. Right now, there is work being done in Swift that will give deployers a very high degree of flexibility in how they can store data. We're working on implementing storage policies in Swift. These storage policies give deployers the ability to choose: (a) what subset of hardware the data lives on (b) how the data is stored across that hardware (c) how communication with an actual storage volume happens. Supporting (a) allows for storage tiers and isolated storage hardware. Supporting (b) allows for different replication or non-replication schemes. Supporting (c) allows for specific optimizations for particular filesystems or storage hardware. Combined, it's even feasable to have a Swift cluster take advantage of other storage systems as a storage policy (imagine an S3 storage policy). As PTL, I want to help coordinate this work and see it to completion. Many people from many different companies are working on it, in addition to the normal day-to-day activity in Swift. I'm excited by the future of Swift, and would be honored to continue to serve as Swift PTL. --John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Gareth *Cloud Computing, OpenStack, Fitness, Basketball* *OpenStack contributor* *Company: UnitedStack http://www.ustack.com* *My promise: if you find any spelling or grammar mistakes in my email from Mar 1 2013, notify me * *and I'll donate $1 or ￥1 to an open organization you specify.* ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
