Dear community, happy to share this week's community update featuring a set of exciting proposals from very different areas of Apache Flink, e.g.
* Initial GPU Support * Flink Development Blog * "LIKE" in "CREATE TABLE" and a bit more. Enjoy. Flink Development ============== * [deployment] In FLIP-108, Yangze Guo is proposing to add support for *GPU* resource management to Flink. In a first step, users would be able to configure the number of GPUs a Taskmanager should request from the resource manager (YARN, Kubernetes). The number of available GPUs would then be made available to operators (e.g. via the RuntimeContext). [1] * [docker] Since Flink 1.10.0 the library/fink Docker repository is maintained by Apache Flink project. Now, Andrey is proposing to *consolidate the different Dockerfiles* and Docker utilities spread over the documentation. Basically, all Dockerfiles would be based on library/flink and different utilities would be provided to customize/extend/configure library/flink are proposed. [2] * [state management] Arvid has posted an update on FLIP-76 (*Unaligned Checkpoints*) and would like to proceed to a vote soon. [3] * [datastream] Tison proposes to change the *dependency structure* around flink-streaming-java [4] * [development process] Arvid has proposed to start a *"Flink Development Blog"*, which covers recent & ongoing development work in Apache Flink and is primarily targeted to the community of Apache Flink contributors. The proposal has received a lot of positive feedback from other committers and contributors. It looks like we will start with a dedicated section in Flink's Wiki [5]. From there posts could then also be promoted to the Flink Blog Post, if there are suited for a larger audience. [6] * [development process] Jark Wu started a discussion to disable Github's *"Squash and Merge" *for Apache Flink mostly as it results in lost author information of the original commits and can mess up email addresses. The discussion revealed that there are also good reasons to keep it, so in the end only committer guide has been updated. [7,8] * [sql] Dawid proposes to support *"LIKE" in "CREATE TABLE"* statement in Flink SQL to make it easier to slightly adjust/extend existing tables. This becomes more relevant as Flink is adding support for more external catalogs (SchemaRegistry, HiveCatalog, JDBCCatalog, ...). [9] * [sql, hive] Jingsong proposes to provide "convenience *fat JARs*" for the* Flink Hive* connector. These JARs called "flink-connector-x.x" would come in different versions and contain all the required dependencies to integrate Flink with Hive. [10] [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-108-Add-GPU-support-in-Flink-tp38286.html [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-111-Docker-image-unification-tp38444.html [3] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-76-Unaligned-checkpoints-tp33651.html [4] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/PROPOSAL-Reverse-the-dependency-from-flink-streaming-java-to-flink-client-tp38469.html [5] https://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+Home [6] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-dev-blog-tp38316p38344.html [7] https://cwiki.apache.org/confluence/display/FLINK/General+Information+for+Committers [8] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Disable-Squash-and-merge-button-for-Flink-repository-on-GitHub-tp38497.html [9] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-110-Support-LIKE-clause-in-CREATE-TABLE-tp38378p38421.html [10] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Introduce-flink-connector-hive-xx-modules-tp38440.html Notable Bugs ========== * [FLINK-16414] [1.10.0] You can not use a UDTF or UDAF without overriding getResultType(). Fixed for 1.10.1. [11] [11] https://issues.apache.org/jira/browse/FLINK-16414 flink-packages.org ============= In the last months, I have covered the contribution of the Pulsar connector to Apache Flink a couple of times. Currently, it is paused waiting for the implementation of FLIP-27, the new source interface. Nevertheless, there has already been a Pulsar Flink Connector maintained by StreamNative for quite some time. You can check it out on flink-packages.org. [12] [12] https://flink-packages.org/packages/pulsar-flink-connector Events, Blog Posts, Misc =================== * Upcoming Meetups * On March 12th, Prateep Kumar will host an online event comparing Kafka Streams and Apache Flink [13]. * On April 22, Ververica will host the next Apache Flink meetup in Berlin. [14] * Cloudera is hosting a couple of "Future of Data" events on stream processing with Apache Flink in * Vienna (March 4th, full-day workshop) [15] * Zurich (March 10th, full-day workshop) [16] * New Jersey (May 5th, meetup) [17] [13] https://www.meetup.com/apache-flink-aws-kinesis-hyd-india/events/268930388/ [14] https://www.meetup.com/Apache-Flink-Meetup/events/269005339/ [15] https://www.meetup.com/futureofdata-vienna/events/268418974/ [16] https://www.meetup.com/futureofdata-zurich/events/268423809/ [17] https://www.meetup.com/futureofdata-princeton/events/268830725/ Cheers, Konstantin (@snntrable) -- Konstantin Knauf | Head of Product +49 160 91394525 Follow us @VervericaData Ververica <https://www.ververica.com/> -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Tony) Cheng
