Dear community, this week's update contains a few pointers to upcoming changes in the development process, two recent proposals around metrics as well as smaller discussions on the dev mailing list. Still processing the backlog of topics & discussions by including a few of them each week.
As always, please feel free to add additional updates and news to this thread! Flink Development =============== * [development-process] Back in February the Flink community discussed changes to its *code contribution process* (implemented in Jira) [1]. In the future only committers will be able to assign contributors to tickets hopefully moving discussions to an earlier stage in the contribution process. The new process has now been fully documented and published on the Flink Website [2]. As far as I know, the next step is changing the Jira permissions accordingly. * [development-process] Stephan has recently started a conversation about adopting a c*ode style and quality guide* for contributions to Apache Flink. It covers Flink-specific aspects like PR naming, but also general Java best practices and lessons-learned in the context of a complex system like Flink. The feedback was quite positive and Robert is now going to split the guide into subsections and add it to the Flink website. [3] * [releases] The last currently known blocker for* Flink 1.8.1* has been resolved and Jincheng is in the process of creating a first release candidate. [4] * [releases] The feature freeze for *Flink 1.9.0 *has been set to the originally proposed date, 28th of June [5,6]. * [metrics] In May Hwanju proposed to significantly enhance the *availability metrics* of Flink and classify downtimes based on the root cause of the failure (mainly between system and user code). [8] This will make it easier for service providers to monitor the SLAs of a Flink-based service. The discussion thread has by now been split into two design documents for exception classification [9] and more fine-grained execution tracking [10]. * [metrics] Becket proposed to standardize the metrics provided by connectors. It defines a set of *standard metrics*, which each source or sink should expose (e.g. numBytesIn, pendingRecord). As far as I can tell, the FLIP is still being voted on with positive feedback so far [11]. * Dyana brought up the possibility to include the *Kinesis Connector* module in the official Apache Flink releases as these are now available under Apache 2.0 license. The corresponding contribution is pending. [12] * Xingcam started a discussion of whether to deprecate, fix or replace the split/select methods of the DataStream API. They were marked deprecated in Flink 1.8.0, but the current discussion has not come to a conclusion yet on whether to really remove them going forward. [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-A-more-restrictive-JIRA-workflow-td27344.html [2] https://flink.apache.org/contributing/contribute-code.html [3] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Adopting-a-Code-Style-and-Quality-Guide-td29524.html#a29532 [4] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Flink-1-8-1-td29154.html [5] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-9-0-td28701.html [6] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Feature-freeze-for-Apache-Flink-1-9-0-release-td29751.html [7] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Proposal-for-Flink-job-execution-availability-metrics-impovement-td28882.html [8] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-33-Standardize-connector-metrics-td26869.html [9] https://docs.google.com/document/d/1pcHg9F3GoDDeVD5GIIo2wO67Hmjgy0-hRDeuFnrMgT4/edit#heading=h.v3ssap5tvszp [10] https://docs.google.com/document/d/1oLF3w1wYyr8vqoFoQZhw1QxTofmAtlD8IF694oPLjNI/edit#heading=h.9zul23uwx8c6 [11] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-33-Standardize-connector-metrics-td29527.html [12] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Updating-Kinesis-Connector-to-latest-Apache-licensed-libs-td29559.html#a29566 [13] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/About-Deprecating-split-select-for-DataStream-API-td29617.html Notable Bugs =========== * [FLINK-12871] [1.7.2] [ 1.8.0] The documentation how to setup SSL for Flink's REST API and intra-cluster communication was erroneous. Fixed in [1.7.3] [1.8.1]. [14] * [FLINK-12912] [1.6.4] [1.7.2] [ 1.8.0] With the configuration parameter "task.checkpoint.alignment.max-size" users can limit the data buffered during the alignment phase of a checkpoint. Once reached, the checkpoint is aborted. Currently, this check does not work correctly when one checkpoint subsumes another one. Unresolved. [15] [14] https://issues.apache.org/jira/browse/FLINK-12871 [15] https://issues.apache.org/jira/browse/FLINK-12912 Events, Blog Posts, Misc ==================== * This week there were a couple of Flink-related talks at Berlin Buzzwords and Beam Summit Europe. Talks from Berlin Buzzwords are already available online [16]. Not sure if there are recordings of Beam Summit, but definitely saw a lot of slide sets being shared on Twitter. * Liu Biao did an online training with about 400 participants on metrics in Flink [16]. * There are a couple of meetups coming up in the next weeks: * 2019/06/24: Cloud Native Meetup in *Aarhus* with a Flink talk by Lasse Nedergard (TrackUnit) [18] * 2019/06/26: *Bay Area* Apache Flink Meetup with talks by Zendesk, Parag Kesar and Ben Liu (Pinterest) and Ken Krugler (Scale Unlimited) [19] * 2019/07/01: *Paris *Apache Beam Meetup with a Flink talk by myself (Ververica) [20] * 2019/07/05: Apache Flink Meetup *Munich* with talks by Steffen Hausmann (AWS) and Michel David (Ryte) [21] [16] https://berlinbuzzwords.de/19/sessions [17] https://zh.ververica.com/developers/flink-training-course2 [18] https://flink.apache.org/2019/06/05/flink-network-stack.html [19] https://www.meetup.com/Cloud-Native-Aarhus/events/261346897/ [20] https://www.meetup.com/Bay-Area-Apache-Flink-Meetup/events/262216929 [21] https://www.meetup.com/Paris-Apache-Beam-Meetup/events/261775884/ [22] https://www.meetup.com/Apache-Flink-Meetup-Munich/events/261282757/ Cheers, Konstantin (@snntrable) -- Konstantin Knauf | Solutions Architect +49 160 91394525 -- Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Data Artisans GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
