Re: [SC-L] BSIMM-V Article in Application Development Times
For anyone interested in this topic and working in appsec and/or dev, there’s a survey by the trusted software alliance which touches on some of these questions here: https://www.surveymonkey.com/s/Developers_and_AppSec On Jan 7, 2014, at 8:07 PM, Christian Heinrich christian.heinr...@cmlh.id.au wrote: Stephen, On Sat, Jan 4, 2014 at 8:12 PM, Stephen de Vries step...@continuumsecurity.net wrote: Leaving the definition of agile aside for the moment, doesn’t the fact that the BSIMM measures organisation wide activities but not individual dev teams mean that we could be drawing inaccurate conclusions from the data? E.g. if an organisation says it is doing Arch reviews, code reviews and sec testing, it doesn’t necessarily mean that every team is doing all of those activities, so it may give the BSIMM reader a false impression of the use of those activities in the real world. In addition to knowing which activities are practiced organisation wide, it would also be valuable to know which activities work well on a per-team or per-project basis. My reading of the Roles section of BSIMM-V.pdf is that the people interviewed for the BSIMM sample are: 1. Executive Leadership (or CISO, VP of Risk, CSO, etc) 2. Everyone else within the Software Security Group (SSG) What you are asking to be included is what is referred to as the Satellite within BSIMM-V.pdf and I believe this may also require the inclusion of http://cmmiinstitute.com/cmmi-solutions/cmmi-for-development/ too (why not :) ). The issue with this is that it would invalidate the statistics from the prior five BSIMM releases due to the inclusion of new questions and in additional these new statistics were not gathered over time either hence the improvements measured over time within BSIMM would be invalid too due tot he new dataset. Furthermore, Gary, Sammy and Brian have limited time to interview all 67 BSIMM participating firms. However, I would be interested to know the BSIMM Advisory Board i.e. http://bsimm.com/community/ view on this is and if it would be possible to undertake this additional sampling within their own BSIMM participating firm to determine if there is additional value would be gained for BSIMM? However, I suspect that an objective measurement would be too hard to quantify due to internal politics of each BSIMM participating firm but I could be wrong. ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates ___
Re: [SC-L] BSIMM-V Article in Application Development Times
Stephen, On Sat, Jan 4, 2014 at 8:12 PM, Stephen de Vries step...@continuumsecurity.net wrote: Leaving the definition of agile aside for the moment, doesn’t the fact that the BSIMM measures organisation wide activities but not individual dev teams mean that we could be drawing inaccurate conclusions from the data? E.g. if an organisation says it is doing Arch reviews, code reviews and sec testing, it doesn’t necessarily mean that every team is doing all of those activities, so it may give the BSIMM reader a false impression of the use of those activities in the real world. In addition to knowing which activities are practiced organisation wide, it would also be valuable to know which activities work well on a per-team or per-project basis. My reading of the Roles section of BSIMM-V.pdf is that the people interviewed for the BSIMM sample are: 1. Executive Leadership (or CISO, VP of Risk, CSO, etc) 2. Everyone else within the Software Security Group (SSG) What you are asking to be included is what is referred to as the Satellite within BSIMM-V.pdf and I believe this may also require the inclusion of http://cmmiinstitute.com/cmmi-solutions/cmmi-for-development/ too (why not :) ). The issue with this is that it would invalidate the statistics from the prior five BSIMM releases due to the inclusion of new questions and in additional these new statistics were not gathered over time either hence the improvements measured over time within BSIMM would be invalid too due tot he new dataset. Furthermore, Gary, Sammy and Brian have limited time to interview all 67 BSIMM participating firms. However, I would be interested to know the BSIMM Advisory Board i.e. http://bsimm.com/community/ view on this is and if it would be possible to undertake this additional sampling within their own BSIMM participating firm to determine if there is additional value would be gained for BSIMM? However, I suspect that an objective measurement would be too hard to quantify due to internal politics of each BSIMM participating firm but I could be wrong. -- Regards, Christian Heinrich http://cmlh.id.au/contact ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates ___
Re: [SC-L] BSIMM-V Article in Application Development Times
Hi Sammy, Antti, On 20 Dec 2013, at 17:29, Sammy Migues smig...@cigital.com wrote: Also, in nearly all cases, it would be very hard to characterize an entire firm or even an entire business unit in larger firms as Agile or not. Many larger firms use Agile for only a small percentage of projects Leaving the definition of agile aside for the moment, doesn’t the fact that the BSIMM measures organisation wide activities but not individual dev teams mean that we could be drawing inaccurate conclusions from the data? E.g. if an organisation says it is doing Arch reviews, code reviews and sec testing, it doesn’t necessarily mean that every team is doing all of those activities, so it may give the BSIMM reader a false impression of the use of those activities in the real world. In addition to knowing which activities are practiced organisation wide, it would also be valuable to know which activities work well on a per-team or per-project basis. On 17 Dec 2013, at 22:01, Antti Vähä-Sipilä a...@iki.fi wrote: Moreover, I think this sort of split would be largely arbitrary. Especially for large companies, it's often not straightforward to classify them as agile or non-agile. Many companies also have mixed-mode dev shops with waterfall product management bolted on top of an agile dev team, or an agile dev team throwing code over the wall to a traditional ops team, or a mix of agile and non-agile teams working side by side. Agree that the split between agile and not-agile would be arbitrary at the organisation wide level. But deciding on an arbitrary line, or better yet an arbitrary scale of agility on a per-project level shouldn’t be too difficult. If we need to start somewhere, then I think borrowing from devops couldn’t hurt, where they measure agility by: - frequency of code deployments - lead time from code deploy to running in production In addition, I don't think you can measure agility through purely measuring cadence. The point of being agile is to be able to respond to change, and not all companies _need_ to be reinventing their product daily like a budding startup with an existential crisis. Although continuous integration would probably help the majority of companies, on the product management (i.e., backlog management) side, it depends on your customers and industry whether more is indeed better. With the BSIMM’s objective of just describing activities it wouldn’t be necessary to promote agile or agile security practices. But it would be interesting to know that if an organisation happens to have chosen agile or continuous delivery as their software dev methodology, then how are they integrating security into that process? The burning questions I have regarding agile and continuous delivery and security are: - What mixture of the BSIMM activities work well in a continuous delivery style environment? - As you move from less-agile to more-agile, which activities tend to fall away and which are more emphasised? - How are the security specialist and time heavy activities like attack models, sec arch review and pentesting performed when new code is pushed to production daily? The BSIMM seems to be the only place where this type of data exists or could be captured- so would be nice to be able to extract this data from it; or include these types of questions in future versions. The devops survey(*) is another potential, but as yet they don’t capture security specific activities. * http://itrevolution.com/the-science-behind-the-2013-puppet-labs-devops-survey-of-practice/ regards, Stephen ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates ___
Re: [SC-L] BSIMM-V Article in Application Development Times
Hi Stephen, I agree that would be interesting. While we have data at the firm level for all BSIMM participants, and at the BU level for many BSIMM participants, we don't formally capture data on development methodology (as opposed to software security activities) for each development team (which may number well into the double digits for many BSIMM participants). Also, in nearly all cases, it would be very hard to characterize an entire firm or even an entire business unit in larger firms as Agile or not. Many larger firms use Agile for only a small percentage of projects (e.g., for mobile or cloud things, if they're a traditional waterfall shop and are just evolving into new technology stacks). Even those firms who do Agile often do it in different ways across different development teams, even in the same business unit. The teams with very large applications or critical applications that go through more testing might do 3-4 week sprints while others do 2-week sprints. However, they might be using exactly the same process, so I'm not sure the frequency of deployment would work as the measure of agility. As for writing Agile rather than Agile above, firms and teams who call themselves Agile mean many different things with that word. I've run into some teams who feel very agile in their quarterly development cycles and at least one that scrums its way through various parts of their waterfall process. Cheers, --Sammy. -Original Message- From: SC-L [mailto:sc-l-boun...@securecoding.org] On Behalf Of Stephen de Vries Sent: Tuesday, December 17, 2013 5:21 AM To: Gary McGraw Cc: Secure Code Mailing List Subject: Re: [SC-L] BSIMM-V Article in Application Development Times On 13 Dec 2013, at 22:51, Gary McGraw g...@cigital.com wrote: From time to time we talk about getting to the dev community here. This article is at least in the right publication! Read it and pass it on: http://adtmag.com/blogs/watersworks/2013/12/bsimm-v-released.aspx Hi Gary, In the current BSIMM-V dataset is it possible to narrow the data down to only organisations practising Agile dev? I think it would be interesting to see which BSIMM activities are popular with agile houses, and which not. Ideally, it would be nice to not only differentiate between Agile and non-agile, but different degrees of agile based on the length of iterations and/or the frequency of deployments. E.g. less-agile = 3 month iterations and multi-month deploys, more-agile = continuous delivery with multiple deploys per day. regards, Stephen de Vries http://www.continuumsecurity.net Twitter: @stephendv ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates ___ ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates ___
Re: [SC-L] BSIMM-V Article in Application Development Times
In the current BSIMM-V dataset is it possible to narrow the data down to only organisations practising Agile dev? I think it would be interesting to see which BSIMM activities are popular with agile houses, and which not. One of the reasons not to do this is that publishing data that would be split into too many or too small pools would potentially allow someone to reverse-engineer the exact results of some of the participating companies. Aggregate data provides a level of anonymity. Moreover, I think this sort of split would be largely arbitrary. Especially for large companies, it's often not straightforward to classify them as agile or non-agile. Many companies also have mixed-mode dev shops with waterfall product management bolted on top of an agile dev team, or an agile dev team throwing code over the wall to a traditional ops team, or a mix of agile and non-agile teams working side by side. Now, some observed activities clearly are purely development activities, and some would not make any sense at all as dev team activities. How would you classify the results if the company had agile dev teams but waterfall product management? Ideally, it would be nice to not only differentiate between Agile and non-agile, but different degrees of agile based on the length of iterations and/or the frequency of deployments. E.g. less-agile = 3 month iterations and multi-month deploys, more-agile = continuous delivery with multiple deploys per day. Even in purely agile shops, not everyone has a concept of an iteration (kanban is a continuous flow of tasks - which is often how maintenance of legacy software would be done), and deploying means different things for different industries (think embedded systems that have no update channel). In addition, I don't think you can measure agility through purely measuring cadence. The point of being agile is to be able to respond to change, and not all companies _need_ to be reinventing their product daily like a budding startup with an existential crisis. Although continuous integration would probably help the majority of companies, on the product management (i.e., backlog management) side, it depends on your customers and industry whether more is indeed better. - Antti ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates ___
[SC-L] BSIMM-V Article in Application Development Times
hi sc-l, From time to time we talk about getting to the dev community here. This article is at least in the right publication! Read it and pass it on: http://adtmag.com/blogs/watersworks/2013/12/bsimm-v-released.aspx Salubrious solstice! One week and one day to go. gem ___ Secure Coding mailing list (SC-L) SC-L@securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates ___
