Dear community, happy to share this week's community update with news about Flink 1.10 & 1.9.1, two FLIPs for better programmatic job and cluster control, improvements to the web user interface and a bit more.
Flink Development ============== * [releases] Jark has started a discussion on releasing a first patch release for Flink 1.9. Looking at the discussion I expect a first release candidate soonish. [1] * [releases] Yu Li has shared a small progress report on Flink 1.10, which gives a very good overview of ongoing FLIPs including their status. [2] * [ui] Yadong has started a discussion on contributing improvements to the Flink Web UI. It includes a variety of pretty cool changes such as better overview over the cluster resources, flame graphs, additional backpressure monitoring and much more. Best you check it out yourself. [3] * [python] FLIP-58 aims to support stateless Python UDFs in the Table API. Wei has shared a design document to support the usage of Python dependencies inside such UDFs and is looking for feedback. [4] * [sql] There a currently three ongoing votes for previously covered FLIPs in the SQL ecosystem (FLIP-66 (Time-Attribute in SQL DDL), FLIP-57 (Function Catalog), FLIP-68 (Modular Plugins)). [5,6,7] * [client] A couple of weeks ago Zili had started a discussion on how to improve Flink's API for job submission and job and cluster management. As a result of this discussion Kostas has now shared a design document to refactor the (Stream)ExecutionEnvironments. The basic idea is to move the responsibility of job submission out of the ExecutionEnvironment into Executors. This would result in only one ExecutionEnvironment per API in the future, and one Executor per environment (YARN, Standalone, ...). The ExecutionEnvironment would then choose the correct Executor based on its configuration. [8] * [client] The second FLIP that originated from this discussion was also shared by Zili this week and proposes to expose JobClient, ClusterClient and ClusterDescriptor to users to programmatically control Flink Jobs and Clusters during their runtime. [9] [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Flink-1-9-1-tp33343.html [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Progress-of-Apache-Flink-1-10-1-tp33570.html [3] https://docs.google.com/document/d/1tIa8yN2prWWKJI_fa1u0t6h1r6RJpp56m48pXEyh6iI/edit# [4] https://docs.google.com/document/d/1vq5J3TSyhscQXbpRhz-Yd3KCX62PBJeC_a_h3amUvJ4/edit#heading=h.lvy7nudjmhjd [5] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-66-Support-Time-Attribute-in-SQL-DDL-2-tp33513.html [6] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-57-Rework-FunctionCatalog-tp33373.html [7] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-68-Extend-Core-Table-System-with-Modular-Plugins-tp33372.html [8] https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission [9] https://cwiki.apache.org/confluence/display/FLINK/FLIP-74:+Flink+JobClient+API Notable Bugs ========== [FLINK-14145] [1.9.0] When preferring checkpoints (as opposed to savepoints) for recovery, the wrong checkpoint could be chosen for recovery in certain situations. Fixed for 1.9.1. [10] [FLINK-13708] [1.9.0] When calling "execute" twice on the same table environment, transformations from the first execution will also be part of the second execution. PR pending, might get into 1.9.1. [11] [10] https://issues.apache.org/jira/browse/FLINK-14145 [11] https://issues.apache.org/jira/browse/FLINK-13708 Events, Blog Posts, Misc =================== * Flink Forward Europe is getting close, 7th - 9th of October. Keynotes by AWS, Cloudera, Cloud Foundry Foundation, Google and Ververica. [12] * Upcoming Meetups * Tomorrow * Dean Shaw* and *Max McKittrick* will talk about click stream analysis at scale at Capital One at the next dataCouncil.ai NYC Data Engineering meetup. [13] [12] https://europe-2019.flink-forward.org/ [13] https://www.meetup.com/DataCouncil-AI-NYC-Data-Engineering-Science/events/264748638/ Cheers, Konstantin -- Konstantin Knauf | Solutions Architect +49 160 91394525 Follow us @VervericaData Ververica <https://www.ververica.com/> -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Tony) Cheng
