Re: [Development] Code Review of Build File Changes
Hi Qt Community, I am writing to first thank those who participated in our study as interviewees for their time and insightful ideas, and second to remind you that we are still looking for new participants. Our talks with the interviewees have helped us improve our study and shed light on exciting aspects of build code maintenance and build change review. We want to hear more. If you are interested in the study, please sign up through the link below so that we can have a chat at your earliest convenience. Sing up link: https://forms.gle/Yec6ZvCemNiLktwr5 Best, Mattie [https://lh4.googleusercontent.com/VtYRv_CP5yBDwLHNZl_kkAC7wxQwI49a2GF_THbwBKGZD2m1kY0WtoTRNYG9nVC-iv-NPk4-i1w=w1200-h630-p]<https://forms.gle/Yec6ZvCemNiLktwr5> Code Review of Build File Changes, Interview Registration<https://forms.gle/Yec6ZvCemNiLktwr5> forms.gle From: Development on behalf of Mattie Nejati Sent: Friday, July 22, 2022 9:39 AM To: development@qt-project.org Subject: Re: [Development] Code Review of Build File Changes Hi again! Thank you all for your interest in the discussion. Reading through your responses gave us a better understanding of how the code review process works in your project and shaped some future research ideas. I have replied to your emails individually, but I think it’ll be helpful to share a summary of the conversations I had with everyone on the list. As I previously shared, we observed that changes to build files are 2 to 4 times less likely to receive comments from reviewers compared to changes in source code or test files. This observation comes from analyzing over 7 million changes to files across projects. The comparison was made in different settings, including when the change affects only one, two, or all three types of files (source/test/build code). We also found that when build changes do receive comments from reviewers, these comments are more likely to discuss potential bugs in the code when compared to source/test code. This is an observation we have made after closely analyzing 500 sampled comments (in the context of the build change and the discussion that follows the comment). We aim to understand how build changes are reviewed, meaning we are looking for practices and policies for build change review that might be different from reviewing other code changes. We also want to know if there are any challenges that complicate the build change review process. Our goal is to understand the development teams’ perception of these questions rather than to find a correct answer. As we are controlling for your level of experience with build systems, your participation will be a great contribution no matter how much experience you have with build code review. So, if you find this discussion interesting, please do not hesitate to sign up using the link below. Sign up link: https://forms.gle/Yec6ZvCemNiLktwr5 Best, Mattie From: Development on behalf of Mattie Nejati Sent: Wednesday, July 20, 2022 2:29 PM To: development@qt-project.org Subject: [Development] Code Review of Build File Changes Hi Developers, I’m Mattie, a Ph.D. student at the University of Waterloo and I’ve been studying the code review process of build files in Qt. For example, I’ve found that changes to build files are 2 to 4 times less likely to be discussed during code review than changes to source code or test files. As the next step in our study, we are recruiting developers for a 30-minute interview to help us shed light on: 1. How build changes are reviewed? 2. What are the challenges faced when reviewing build changes? 3. Why is build change review different? Since the Qt review is featured in our analysis, we would appreciate it if you consider participating as an interviewee. If you are interested, please click the signup link below and follow the instructions. Once the study is concluded, we will share our findings with the Qt community first. Signup link: https://forms.gle/Yec6ZvCemNiLktwr5 Best, Mattie ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] Code Review of Build File Changes
Hi again! Thank you all for your interest in the discussion. Reading through your responses gave us a better understanding of how the code review process works in your project and shaped some future research ideas. I have replied to your emails individually, but I think it’ll be helpful to share a summary of the conversations I had with everyone on the list. As I previously shared, we observed that changes to build files are 2 to 4 times less likely to receive comments from reviewers compared to changes in source code or test files. This observation comes from analyzing over 7 million changes to files across projects. The comparison was made in different settings, including when the change affects only one, two, or all three types of files (source/test/build code). We also found that when build changes do receive comments from reviewers, these comments are more likely to discuss potential bugs in the code when compared to source/test code. This is an observation we have made after closely analyzing 500 sampled comments (in the context of the build change and the discussion that follows the comment). We aim to understand how build changes are reviewed, meaning we are looking for practices and policies for build change review that might be different from reviewing other code changes. We also want to know if there are any challenges that complicate the build change review process. Our goal is to understand the development teams’ perception of these questions rather than to find a correct answer. As we are controlling for your level of experience with build systems, your participation will be a great contribution no matter how much experience you have with build code review. So, if you find this discussion interesting, please do not hesitate to sign up using the link below. Sign up link: https://forms.gle/Yec6ZvCemNiLktwr5 Best, Mattie From: Development on behalf of Mattie Nejati Sent: Wednesday, July 20, 2022 2:29 PM To: development@qt-project.org Subject: [Development] Code Review of Build File Changes Hi Developers, I’m Mattie, a Ph.D. student at the University of Waterloo and I’ve been studying the code review process of build files in Qt. For example, I’ve found that changes to build files are 2 to 4 times less likely to be discussed during code review than changes to source code or test files. As the next step in our study, we are recruiting developers for a 30-minute interview to help us shed light on: 1. How build changes are reviewed? 2. What are the challenges faced when reviewing build changes? 3. Why is build change review different? Since the Qt review is featured in our analysis, we would appreciate it if you consider participating as an interviewee. If you are interested, please click the signup link below and follow the instructions. Once the study is concluded, we will share our findings with the Qt community first. Signup link: https://forms.gle/Yec6ZvCemNiLktwr5 Best, Mattie ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] Code Review of Build File Changes
On Thursday, 21 July 2022 00:54:28 PDT Volker Hilsheimer wrote: > These would be observable, quantifiable measurements. And if one of the > reason for those measurements is “there are very few people who can review > build system changes”, then that’s a qualitative statement, getting to > which seems to be why Mattie is asking for people to sign up. Assuming the hypothesis is true, that might be one of the reasons. I personally only make changes after having talked to Alexnadru on IRC, so by the time he reviews it, it's already what he wanted. If he's not available, I ask in #cmake on Slack. My argument is that I don't think the hypothesis is true or at least it's not proven. Just because there was no discussion on Gerrit does not mean there was no discussion. -- Thiago Macieira - thiago.macieira (AT) intel.com Cloud Software Architect - Intel DCAI Cloud Engineering ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] Code Review of Build File Changes
On Wednesday, 20 July 2022 11:29:57 PDT Mattie Nejati wrote: >> I’m Mattie, a Ph.D. student at the University of Waterloo and I’ve >> been studying the code review process of build files in Qt. For >> example, I’ve found that changes to build files are 2 to 4 times less >> likely to be discussed during code review than changes to source code >> or test files. Thiago Macieira (21 July 2022 00:30) replied > I dispute the hypothesis. You haven't proven it, and you're now > proceeding to find out why something is before ascertaining that it is > so. You also need to control for complexity of changes. I will hazard a guess that a significant subset of changes to build files are simple changes such as adding a new source file to a list of source files; a second significant subset will be adding a new sub-directory to some project, with a fairly straightforward build config that's a copy of some other sub-directory with trivial details changed. It is no surprise that these get no comment in review. So you need to distinguish trivial changes to build configuration (to be compared with trivial changes to source code, of which there are also several categories) from Real Work on the build system, which defines new CMake functions or reworks existing CMake functions to fix bugs or extend their capabilities. These last are then comparable to the Real Work changes to source code (that, if they touch the build system at all, typically do so in trivial ways). In short, when comparing how likely different categories of change are to be discussed in review, the first question to ask is how complex the change is; and you'll need to control for that when comparing changes in different categories. In particular, a commit that mixes changes of different categories may be complex in one but trivial in the others. > There are two people who can review build system changes. You should > find other code areas that only have one or two people who can ever > review them (or, worse, zero) and there's a similar symptom there. and you'll likely find those two people do most of the non-trivial change to the build system, and review each other's work, at which point Volker's observation that they share an office may be relevant. In contrast, the two people who do most (and review each other's) non-trivial changes to date-time code sit on opposite sides of the Atlantic, so you'll see in review discussions the build team would have had across their desk. Eddy. ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] Code Review of Build File Changes
> On 21 Jul 2022, at 00:30, Thiago Macieira wrote: > > On Wednesday, 20 July 2022 11:29:57 PDT Mattie Nejati wrote: >> I’m Mattie, a Ph.D. student at the University of Waterloo and I’ve been >> studying the code review process of build files in Qt. For example, I’ve >> found that changes to build files are 2 to 4 times less likely to be >> discussed during code review than changes to source code or test files. > > I dispute the hypothesis. You haven't proven it, and you're now proceeding to > find out why something is before ascertaining that it is so. > > There are two people who can review build system changes. You should find > other > code areas that only have one or two people who can ever review them (or, > worse, zero) and there's a similar symptom there. > > I also recommend you see how documentation-only changes are reviewed, and > ditto for unit-test-only changes. It would perhaps be interesting to know how you measure the “likelihood for a change to be discussed". Is it the average number of comments given to a change? Or the number of patch sets it takes before a change gets approved and merged? These would be observable, quantifiable measurements. And if one of the reason for those measurements is “there are very few people who can review build system changes”, then that’s a qualitative statement, getting to which seems to be why Mattie is asking for people to sign up. Although if that would be the reason, I’d rather expect reviews to take longer, rather than the discussion to be less. But given that the main contributors to the build system share an office location, perhaps we learn that e.g. pair programming and face to face discussions reduce the amount of time it takes to get changes through review. Volker ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] Code Review of Build File Changes
On Wednesday, 20 July 2022 11:29:57 PDT Mattie Nejati wrote: > I’m Mattie, a Ph.D. student at the University of Waterloo and I’ve been > studying the code review process of build files in Qt. For example, I’ve > found that changes to build files are 2 to 4 times less likely to be > discussed during code review than changes to source code or test files. I dispute the hypothesis. You haven't proven it, and you're now proceeding to find out why something is before ascertaining that it is so. There are two people who can review build system changes. You should find other code areas that only have one or two people who can ever review them (or, worse, zero) and there's a similar symptom there. I also recommend you see how documentation-only changes are reviewed, and ditto for unit-test-only changes. -- Thiago Macieira - thiago.macieira (AT) intel.com Cloud Software Architect - Intel DCAI Cloud Engineering ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
Re: [Development] Code Review of Build File Changes
Note that the form no longer requires participants to use a Google account. Best, Mattie From: Development on behalf of Mattie Nejati Sent: Wednesday, July 20, 2022 2:29 PM To: development@qt-project.org Subject: [Development] Code Review of Build File Changes Hi Developers, I’m Mattie, a Ph.D. student at the University of Waterloo and I’ve been studying the code review process of build files in Qt. For example, I’ve found that changes to build files are 2 to 4 times less likely to be discussed during code review than changes to source code or test files. As the next step in our study, we are recruiting developers for a 30-minute interview to help us shed light on: 1. How build changes are reviewed? 2. What are the challenges faced when reviewing build changes? 3. Why is build change review different? Since the Qt review is featured in our analysis, we would appreciate it if you consider participating as an interviewee. If you are interested, please click the signup link below and follow the instructions. Once the study is concluded, we will share our findings with the Qt community first. Signup link: https://forms.gle/Yec6ZvCemNiLktwr5 Best, Mattie ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
[Development] Code Review of Build File Changes
Hi Developers, I’m Mattie, a Ph.D. student at the University of Waterloo and I’ve been studying the code review process of build files in Qt. For example, I’ve found that changes to build files are 2 to 4 times less likely to be discussed during code review than changes to source code or test files. As the next step in our study, we are recruiting developers for a 30-minute interview to help us shed light on: 1. How build changes are reviewed? 2. What are the challenges faced when reviewing build changes? 3. Why is build change review different? Since the Qt review is featured in our analysis, we would appreciate it if you consider participating as an interviewee. If you are interested, please click the signup link below and follow the instructions. Once the study is concluded, we will share our findings with the Qt community first. Signup link: https://forms.gle/Yec6ZvCemNiLktwr5 Best, Mattie ___ Development mailing list Development@qt-project.org https://lists.qt-project.org/listinfo/development
