sjwiesman commented on a change in pull request #478: URL: https://github.com/apache/flink-web/pull/478#discussion_r740478608
########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. Review comment: ```suggestion We looked at the origin of these emotions and analyzed what went well and what could be improved. We also incorporated some feedback gathered from the community. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. Review comment: ```suggestion It has now been a month since the community released [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) into the wild. We had a comprehensive look at the enhancements, additions, and fixups in the release announcement blog post, and now we will look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + +## Things enjoyed + +The implementation of some features, such as [buffer debloating](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/network_mem_tuning/#the-buffer-debloating-mechanism) and [fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/finegrained_resource/), went really smoothly. Though a few issues are now popping up. We enjoyed the moments while they last. :smiling_face_with_tear: + +We also said goodbye to some components. It always feels refreshing to remove code and reduce complexity. :slightly_smiling_face: + + +# What we want to achieve through process changes + +## Transparency - let the community participate + +When approaching a release (usually a couple of weeks after the previous release has been done) we set up bi-weekly meetings for the community to discuss any issues regarding the release. The usefulness of those meetings varied a lot and so we started to [track the efforts](https://cwiki.apache.org/confluence/display/FLINK/1.14+Release) in the Apache Flink Confluence wiki. + +We came up with a system to label the current states of each feature: "independent", "won’t make it", "very unlikely", "will make it", "done", and "done done". We introduced the "done done" state since we were lacking a shared understanding of the definition of done. To qualify for "done done", the feature is manually tested by someone who has not been involved in the implementation of the specific effort and there exists comprehensive documentation that enables users to use the feature. + +After each meeting, we provided updates on the mailing list and created a corresponding burn down chart. Those efforts have been perceived positively although they might still require some improvements. + +The meeting used to only be for those who have been driving the main efforts, but we opened it up to the whole community for this release. Nobody ended up joining :smiling_face_with_tear: but we will continue to make the meetings open to everyone. + + +## Stability - reduce building and testing pain + +At one point as we were coming close to the feature freeze, the stability of the master branch became quite unstable. Although we have encountered this issue in the past, building and testing Flink under such conditions was not ideal. Let's not count how often Kafka integrations tests have failed. Review comment: I don't like the Kafka comment, I understand it's a joke it doesn't come across well. We can certainly talk about why Kafka is difficult to test against but that feel separate from this. ```suggestion At one point, as we were coming close to the feature freeze, the stability of the master branch became quite unstable. Although we have encountered this issue in the past, building and testing Flink under such conditions was not ideal. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + +## Things enjoyed + +The implementation of some features, such as [buffer debloating](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/network_mem_tuning/#the-buffer-debloating-mechanism) and [fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/finegrained_resource/), went really smoothly. Though a few issues are now popping up. We enjoyed the moments while they last. :smiling_face_with_tear: + +We also said goodbye to some components. It always feels refreshing to remove code and reduce complexity. :slightly_smiling_face: + + +# What we want to achieve through process changes + +## Transparency - let the community participate + +When approaching a release (usually a couple of weeks after the previous release has been done) we set up bi-weekly meetings for the community to discuss any issues regarding the release. The usefulness of those meetings varied a lot and so we started to [track the efforts](https://cwiki.apache.org/confluence/display/FLINK/1.14+Release) in the Apache Flink Confluence wiki. + +We came up with a system to label the current states of each feature: "independent", "won’t make it", "very unlikely", "will make it", "done", and "done done". We introduced the "done done" state since we were lacking a shared understanding of the definition of done. To qualify for "done done", the feature is manually tested by someone who has not been involved in the implementation of the specific effort and there exists comprehensive documentation that enables users to use the feature. + +After each meeting, we provided updates on the mailing list and created a corresponding burn down chart. Those efforts have been perceived positively although they might still require some improvements. + +The meeting used to only be for those who have been driving the main efforts, but we opened it up to the whole community for this release. Nobody ended up joining :smiling_face_with_tear: but we will continue to make the meetings open to everyone. + + +## Stability - reduce building and testing pain + +At one point as we were coming close to the feature freeze, the stability of the master branch became quite unstable. Although we have encountered this issue in the past, building and testing Flink under such conditions was not ideal. Let's not count how often Kafka integrations tests have failed. + +As a result, we tried to focus on reducing stability issues and the release managers have tried to organize and manage this effort. In future development cycles, the whole community needs to focus on the stability of the master branch. There are already improvements in the making and they will hopefully enhance the experience of contributing significantly. + +## Documentation - make it user-friendly + +Coming back to Apache traditions, most of the documentation (if any) was still being pushed after the feature freeze. As mentioned before, documentation is required to achieve the level of "done done". Going forward, we will keep more of an eye on pushing documentation earlier in the development process. Apache Flink is an amazing piece of software that can solve so many problems, but we can do so much more in improving the user experience and introducing it to a wider audience. + +## API consistency - a timeless, joyful experience + +The issue of API consistency was not caused by the 1.14 release, but popped up during the development cycle nevertheless, including a bigger discussion on the mailing list. While we tried to be transparent about the [stability guarantees of an API](https://cwiki.apache.org/confluence/display/FLINK/Stability+Annotations) (there are no guarantees across major versions), this was not made very clear or easy to find. Since many users rely on PublicEvolving APIs (due to a lack of Public API additions), this resulted in problems for downstream projects. + +Moving forward, we will document more clearly what the guarantees are and introduce a process for promoting PublicEvolving APIs. This might involve generating a report on any removed/modified PublicEvolving APIs during the release cycle so that downstream projects can prepare for the changes. + +# Some noteworthy items + +The first iteration for the buffer debloat feature was done in a Hackathon. :nerd_face: + +Our [Apache Flink 1.14 Release wiki page](https://cwiki.apache.org/confluence/display/FLINK/1.14+Release) has 167 historic versions. For comparison, [FlIP 147](https://cwiki.apache.org/confluence/display/FLINK/FLIP-147%3A+Support+Checkpoints+After+Tasks+Finished) (one of the most active FLIPs) has just 76. Review comment: ```suggestion Our [Apache Flink 1.14 Release wiki page](https://cwiki.apache.org/confluence/display/FLINK/1.14+Release) has 167 historic versions. For comparison, [FLIP 147](https://cwiki.apache.org/confluence/display/FLINK/FLIP-147%3A+Support+Checkpoints+After+Tasks+Finished) (one of the most active FLIPs) has just 76. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + +## Things enjoyed + +The implementation of some features, such as [buffer debloating](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/network_mem_tuning/#the-buffer-debloating-mechanism) and [fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/finegrained_resource/), went really smoothly. Though a few issues are now popping up. We enjoyed the moments while they last. :smiling_face_with_tear: + +We also said goodbye to some components. It always feels refreshing to remove code and reduce complexity. :slightly_smiling_face: + + +# What we want to achieve through process changes + +## Transparency - let the community participate + +When approaching a release (usually a couple of weeks after the previous release has been done) we set up bi-weekly meetings for the community to discuss any issues regarding the release. The usefulness of those meetings varied a lot and so we started to [track the efforts](https://cwiki.apache.org/confluence/display/FLINK/1.14+Release) in the Apache Flink Confluence wiki. + +We came up with a system to label the current states of each feature: "independent", "won’t make it", "very unlikely", "will make it", "done", and "done done". We introduced the "done done" state since we were lacking a shared understanding of the definition of done. To qualify for "done done", the feature is manually tested by someone who has not been involved in the implementation of the specific effort and there exists comprehensive documentation that enables users to use the feature. + +After each meeting, we provided updates on the mailing list and created a corresponding burn down chart. Those efforts have been perceived positively although they might still require some improvements. + +The meeting used to only be for those who have been driving the main efforts, but we opened it up to the whole community for this release. Nobody ended up joining :smiling_face_with_tear: but we will continue to make the meetings open to everyone. + + +## Stability - reduce building and testing pain + +At one point as we were coming close to the feature freeze, the stability of the master branch became quite unstable. Although we have encountered this issue in the past, building and testing Flink under such conditions was not ideal. Let's not count how often Kafka integrations tests have failed. + +As a result, we tried to focus on reducing stability issues and the release managers have tried to organize and manage this effort. In future development cycles, the whole community needs to focus on the stability of the master branch. There are already improvements in the making and they will hopefully enhance the experience of contributing significantly. + +## Documentation - make it user-friendly + +Coming back to Apache traditions, most of the documentation (if any) was still being pushed after the feature freeze. As mentioned before, documentation is required to achieve the level of "done done". Going forward, we will keep more of an eye on pushing documentation earlier in the development process. Apache Flink is an amazing piece of software that can solve so many problems, but we can do so much more in improving the user experience and introducing it to a wider audience. + +## API consistency - a timeless, joyful experience + +The issue of API consistency was not caused by the 1.14 release, but popped up during the development cycle nevertheless, including a bigger discussion on the mailing list. While we tried to be transparent about the [stability guarantees of an API](https://cwiki.apache.org/confluence/display/FLINK/Stability+Annotations) (there are no guarantees across major versions), this was not made very clear or easy to find. Since many users rely on PublicEvolving APIs (due to a lack of Public API additions), this resulted in problems for downstream projects. + +Moving forward, we will document more clearly what the guarantees are and introduce a process for promoting PublicEvolving APIs. This might involve generating a report on any removed/modified PublicEvolving APIs during the release cycle so that downstream projects can prepare for the changes. + +# Some noteworthy items + +The first iteration for the buffer debloat feature was done in a Hackathon. :nerd_face: Review comment: ```suggestion The first iteration for the buffer debloat feature was done in a Hackathon. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + +## Things enjoyed + +The implementation of some features, such as [buffer debloating](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/network_mem_tuning/#the-buffer-debloating-mechanism) and [fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/finegrained_resource/), went really smoothly. Though a few issues are now popping up. We enjoyed the moments while they last. :smiling_face_with_tear: + +We also said goodbye to some components. It always feels refreshing to remove code and reduce complexity. :slightly_smiling_face: Review comment: ```suggestion We also said goodbye to some components, the old table planner and integrated Mesos support. As any developer will tell you, there's nothing better than deleting old code and reducing complexity. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + +## Things enjoyed + +The implementation of some features, such as [buffer debloating](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/network_mem_tuning/#the-buffer-debloating-mechanism) and [fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/finegrained_resource/), went really smoothly. Though a few issues are now popping up. We enjoyed the moments while they last. :smiling_face_with_tear: Review comment: ```suggestion The implementation of some features, such as [buffer debloating](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/network_mem_tuning/#the-buffer-debloating-mechanism) and [fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/finegrained_resource/), went smoothly. Though a few issues are now popping up as people begin using them in production, it is satisfying to see an engineering effort go according to plan. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + +## Things enjoyed + +The implementation of some features, such as [buffer debloating](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/network_mem_tuning/#the-buffer-debloating-mechanism) and [fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/finegrained_resource/), went really smoothly. Though a few issues are now popping up. We enjoyed the moments while they last. :smiling_face_with_tear: + +We also said goodbye to some components. It always feels refreshing to remove code and reduce complexity. :slightly_smiling_face: + + +# What we want to achieve through process changes + +## Transparency - let the community participate + +When approaching a release (usually a couple of weeks after the previous release has been done) we set up bi-weekly meetings for the community to discuss any issues regarding the release. The usefulness of those meetings varied a lot and so we started to [track the efforts](https://cwiki.apache.org/confluence/display/FLINK/1.14+Release) in the Apache Flink Confluence wiki. + +We came up with a system to label the current states of each feature: "independent", "won’t make it", "very unlikely", "will make it", "done", and "done done". We introduced the "done done" state since we were lacking a shared understanding of the definition of done. To qualify for "done done", the feature is manually tested by someone who has not been involved in the implementation of the specific effort and there exists comprehensive documentation that enables users to use the feature. Review comment: ```suggestion We came up with a system to label the current states of each feature: “independent”, “won’t make it”, “very unlikely”, “will make it”, “done”, and “done done”. We introduced the “done done” state since we lacked a shared understanding of the definition of done. To qualify for “done done”, the feature is manually tested by someone not involved in the implementation. Additionally, there must exist comprehensive documentation that enables users to use the feature. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + +## Things enjoyed + +The implementation of some features, such as [buffer debloating](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/network_mem_tuning/#the-buffer-debloating-mechanism) and [fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/finegrained_resource/), went really smoothly. Though a few issues are now popping up. We enjoyed the moments while they last. :smiling_face_with_tear: + +We also said goodbye to some components. It always feels refreshing to remove code and reduce complexity. :slightly_smiling_face: + + +# What we want to achieve through process changes + +## Transparency - let the community participate + +When approaching a release (usually a couple of weeks after the previous release has been done) we set up bi-weekly meetings for the community to discuss any issues regarding the release. The usefulness of those meetings varied a lot and so we started to [track the efforts](https://cwiki.apache.org/confluence/display/FLINK/1.14+Release) in the Apache Flink Confluence wiki. + +We came up with a system to label the current states of each feature: "independent", "won’t make it", "very unlikely", "will make it", "done", and "done done". We introduced the "done done" state since we were lacking a shared understanding of the definition of done. To qualify for "done done", the feature is manually tested by someone who has not been involved in the implementation of the specific effort and there exists comprehensive documentation that enables users to use the feature. + +After each meeting, we provided updates on the mailing list and created a corresponding burn down chart. Those efforts have been perceived positively although they might still require some improvements. Review comment: ```suggestion After each meeting, we provided updates on the mailing list and created a corresponding burn-down chart. Those efforts have well been received by our contributors, although they might still require some improvements. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + +## Things enjoyed + +The implementation of some features, such as [buffer debloating](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/network_mem_tuning/#the-buffer-debloating-mechanism) and [fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/finegrained_resource/), went really smoothly. Though a few issues are now popping up. We enjoyed the moments while they last. :smiling_face_with_tear: + +We also said goodbye to some components. It always feels refreshing to remove code and reduce complexity. :slightly_smiling_face: + + +# What we want to achieve through process changes + +## Transparency - let the community participate + +When approaching a release (usually a couple of weeks after the previous release has been done) we set up bi-weekly meetings for the community to discuss any issues regarding the release. The usefulness of those meetings varied a lot and so we started to [track the efforts](https://cwiki.apache.org/confluence/display/FLINK/1.14+Release) in the Apache Flink Confluence wiki. Review comment: ```suggestion When approaching a release, usually a couple of weeks after the previous release has been done, we set up bi-weekly meetings for the community to discuss any issues regarding the release. The usefulness of those meetings varied a lot, and so we started to [track the efforts](https://cwiki.apache.org/confluence/display/FLINK/1.14+Release) in the Apache Flink Confluence wiki. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + +## Things enjoyed + +The implementation of some features, such as [buffer debloating](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/network_mem_tuning/#the-buffer-debloating-mechanism) and [fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/finegrained_resource/), went really smoothly. Though a few issues are now popping up. We enjoyed the moments while they last. :smiling_face_with_tear: + +We also said goodbye to some components. It always feels refreshing to remove code and reduce complexity. :slightly_smiling_face: + + +# What we want to achieve through process changes + +## Transparency - let the community participate + +When approaching a release (usually a couple of weeks after the previous release has been done) we set up bi-weekly meetings for the community to discuss any issues regarding the release. The usefulness of those meetings varied a lot and so we started to [track the efforts](https://cwiki.apache.org/confluence/display/FLINK/1.14+Release) in the Apache Flink Confluence wiki. + +We came up with a system to label the current states of each feature: "independent", "won’t make it", "very unlikely", "will make it", "done", and "done done". We introduced the "done done" state since we were lacking a shared understanding of the definition of done. To qualify for "done done", the feature is manually tested by someone who has not been involved in the implementation of the specific effort and there exists comprehensive documentation that enables users to use the feature. + +After each meeting, we provided updates on the mailing list and created a corresponding burn down chart. Those efforts have been perceived positively although they might still require some improvements. + +The meeting used to only be for those who have been driving the main efforts, but we opened it up to the whole community for this release. Nobody ended up joining :smiling_face_with_tear: but we will continue to make the meetings open to everyone. + + +## Stability - reduce building and testing pain + +At one point as we were coming close to the feature freeze, the stability of the master branch became quite unstable. Although we have encountered this issue in the past, building and testing Flink under such conditions was not ideal. Let's not count how often Kafka integrations tests have failed. + +As a result, we tried to focus on reducing stability issues and the release managers have tried to organize and manage this effort. In future development cycles, the whole community needs to focus on the stability of the master branch. There are already improvements in the making and they will hopefully enhance the experience of contributing significantly. Review comment: ```suggestion As a result, we focused on reducing stability issues, and the release managers have tried to organize and manage this effort. In future development cycles, the whole community needs to focus on the stability of the master branch. There are already improvements in the making, and they will hopefully enhance the experience of contributing significantly. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + Review comment: This feels like 3 unrelated sentences smashed together. How about something like this: ```suggestion No release is perfect, and the community is constantly looking to improve. Apache Flink has active contributors from around the globe, many of whom do not speak English as a first language. The community is still ironing out processes for delivering high-quality documentation and blog posts from a content perspective. We have added contribution guidelines for writing and have contributors focusing on this component. Each Flink release is built with the help of hundreds of contributors, each working on different parts of the project. Changes to one module may affect others in ways that are not always obvious. To maintain quality, the community supports an expansive test suite. Invariably some tests are found to be flakey. Whenever we discover a test issue, the community opens a blocker issue that we must resolve before the next release. In practice, this leads to contributors triaging most test instabilities towards the end of each release cycle. From now on, we want to be more mindful of these failures and prioritize them when discovered. Finally, the community pushed the planned feature freeze for 1.14 by two weeks. Two weeks is an improvement from previous release cycles, but we hope to continue improving this metric 1.15. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + +## Things enjoyed + +The implementation of some features, such as [buffer debloating](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/network_mem_tuning/#the-buffer-debloating-mechanism) and [fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/finegrained_resource/), went really smoothly. Though a few issues are now popping up. We enjoyed the moments while they last. :smiling_face_with_tear: + +We also said goodbye to some components. It always feels refreshing to remove code and reduce complexity. :slightly_smiling_face: + + +# What we want to achieve through process changes + +## Transparency - let the community participate + +When approaching a release (usually a couple of weeks after the previous release has been done) we set up bi-weekly meetings for the community to discuss any issues regarding the release. The usefulness of those meetings varied a lot and so we started to [track the efforts](https://cwiki.apache.org/confluence/display/FLINK/1.14+Release) in the Apache Flink Confluence wiki. + +We came up with a system to label the current states of each feature: "independent", "won’t make it", "very unlikely", "will make it", "done", and "done done". We introduced the "done done" state since we were lacking a shared understanding of the definition of done. To qualify for "done done", the feature is manually tested by someone who has not been involved in the implementation of the specific effort and there exists comprehensive documentation that enables users to use the feature. + +After each meeting, we provided updates on the mailing list and created a corresponding burn down chart. Those efforts have been perceived positively although they might still require some improvements. + +The meeting used to only be for those who have been driving the main efforts, but we opened it up to the whole community for this release. Nobody ended up joining :smiling_face_with_tear: but we will continue to make the meetings open to everyone. Review comment: ```suggestion The meeting used to only be for those driving the primary efforts, but we opened it up to the whole community for this release. While nobody ended up joining, we will continue to make the meetings open to everyone. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + Review comment: I know see I've added some overlap to what's written below, so happy to change this but I think it needs more than what was already there. ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + +## Things enjoyed + +The implementation of some features, such as [buffer debloating](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/network_mem_tuning/#the-buffer-debloating-mechanism) and [fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/finegrained_resource/), went really smoothly. Though a few issues are now popping up. We enjoyed the moments while they last. :smiling_face_with_tear: + +We also said goodbye to some components. It always feels refreshing to remove code and reduce complexity. :slightly_smiling_face: + + +# What we want to achieve through process changes + +## Transparency - let the community participate + +When approaching a release (usually a couple of weeks after the previous release has been done) we set up bi-weekly meetings for the community to discuss any issues regarding the release. The usefulness of those meetings varied a lot and so we started to [track the efforts](https://cwiki.apache.org/confluence/display/FLINK/1.14+Release) in the Apache Flink Confluence wiki. + +We came up with a system to label the current states of each feature: "independent", "won’t make it", "very unlikely", "will make it", "done", and "done done". We introduced the "done done" state since we were lacking a shared understanding of the definition of done. To qualify for "done done", the feature is manually tested by someone who has not been involved in the implementation of the specific effort and there exists comprehensive documentation that enables users to use the feature. + +After each meeting, we provided updates on the mailing list and created a corresponding burn down chart. Those efforts have been perceived positively although they might still require some improvements. + +The meeting used to only be for those who have been driving the main efforts, but we opened it up to the whole community for this release. Nobody ended up joining :smiling_face_with_tear: but we will continue to make the meetings open to everyone. + + +## Stability - reduce building and testing pain + +At one point as we were coming close to the feature freeze, the stability of the master branch became quite unstable. Although we have encountered this issue in the past, building and testing Flink under such conditions was not ideal. Let's not count how often Kafka integrations tests have failed. Review comment: I don't like the Kafka comment, I understand it's a joke it but it doesn't come across well. We can certainly talk about why Kafka causes us so much pain but that feel separate from this. ```suggestion At one point, as we were coming close to the feature freeze, the stability of the master branch became quite unstable. Although we have encountered this issue in the past, building and testing Flink under such conditions was not ideal. ``` ########## File path: _posts/2021-11-03-flink-backward.md ########## @@ -0,0 +1,79 @@ +--- +layout: post +title: "Flink Backward - The Apache Flink Retrospective" +date: 2021-11-03 00:00:00 +authors: +- joemoe: + name: "Johannes Moser" +excerpt: A look back at the development cycle for Flink 1.14 +--- + +It has now been a month since [Apache Flink 1.14](https://flink.apache.org/downloads.html#apache-flink-1140) has been released into the wild. :partying_face: We had a comprehensive look at the enhancements, additions, and fixups in the [release announcement blog post](https://flink.apache.org/news/2021/09/29/release-1.14.0.html) and now we will have a look at the development cycle from a different angle. Based on feedback collected from contributors involved in this release, we will explore the experiences and processes behind it all. + +{% toc %} + +# A retrospective on the release cycle + +From the team, we collected emotions that have been attributed to points in time of the 1.14 release cycle: + +<center> +<img src="{{site.baseurl}}/img/blog/2021-11-03-flink-backward/1.14-weather.png" width="70%"/> +</center> + +The overall sentiment seems to be quite good. A ship crushed a robot two times, someone felt sick towards the end, an octopus causing negative emotions appeared in June... + +We looked at the origin of these emotions and came up with an analysis of what went well and what could be improved. We also incorporated some feedback gathered from the community. + +## Problems faced + +From a content perspective, the community is still ironing out processes around documentation and blog posts. There have been a lot of test instabilities and issues with some of the efforts to fix them. We also had to push the feature freeze by two weeks, which might actually be considered early and not actually be living up to the Apache Flink release tradition. :p + Review comment: I now see I've added some overlap to what's written below, so happy to change this but I think it needs more than what was already there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
