Hi community,
It has been more than 6 weeks since the previous announcement and as we are
approaching the expected feature freeze we would like to share the Flink
1.11 status update with you.
Initially we were aiming for the feature freeze to happen in late April
(now), however it was recently proposed to be postponed by a couple of
weeks to mid May. [0]
A lot of people in the community are working hard to complete promised
features and there is good progress. We even have managed to already
complete a couple of features. We have updated the features list from the
previous announcement and we have highlighted features that are already
done and also the features that are no longer aimed for Flink 1.11 release
and will be most likely postponed to a later date.
Your release managers,
Zhijiang & Piotr Nowojski
Features already done and ready for Flink 1.11
-
PyFlink
-
FLIP-96: Add Python ML API [54]
-
FLINK-14500: Fully support all kinds of Python UDF [55]
-
Runtime
-
FLIP-67: Support for cluster partitions [20]
-
FLIP-92: Add N-Ary input stream operator in Flink [24]
-
[FLINK-10742] Let Netty use Flink's buffers on downstream side [28]
-
[FLINK-15911][FLINK-15154] Support Flink work over NAT [39]
-
[FLINK-15672] Switch to Log4j2 by default [34]
Features not targeted for Flink 1.11 anymore:
-
SQL / Table
-
FLIP-91 Introduce SQL client gateway and provide JDBC driver [4]
-
FLIP-107: Reading table columns from different parts of source
records [7]
-
ML / Connectors
-
FLIP-72: Pulsar source / sink / catalog [49]
-
Update ML Pipeline API interface to better support Flink ML lib
algorithms
-
PyFlink
-
Support running python UDF in docker workers
-
Runtime
-
[FLINK-15786] Use the separated classloader to load connectors’ jar
[37]
-
Calculate required shuffle memory before allocating slots
-
State Backend:
-
Support getCustomizedState in KeyedStateStore [47]
Features still in progress for Flink 1.11:
-
SQL / Table
-
FLIP-65: New type inference for Table API UDFs [2]
-
FLIP-84: Improve TableEnv’s interface [3]
-
FLIP-93: Introduce JDBC catalog and Postgres catalog [5]
-
FLIP-105: Support to interpret and emit changelog in Flink SQL [6]
-
[FLINK-14807] Add Table#collect API for fetching data [8]
-
Support query and table hints
-
ML / Connectors
-
FLIP-27: New source API [9]
-
[FLINK-15670] Wrap a source/sink pair to persist intermediate result
for subgraph failure recovery [10]
-
PyFlink
-
FLIP-106, FLIP-114: Expand the usage scope of Python UDF [12][50]
-
FLIP-112: Debugging and monitoring of Python UDF [11]
-
FLIP-97, FLIP-120
<https://docs.google.com/document/d/1rUZHxS7rguLi4oJNEAu6xcRJcW7ldIxAdewoIxoO5w8/edit#heading=h.ghlv7e457i4>:
Integration with most popular Python libraries (Pandas) [51][52]
-
FLIP-121 Performance improvements of Python UDF [53]
-
Web UI
-
FLIP-98: Better back pressure detection [13]
-
FLIP-99: Make max exception configurable [14]
-
FLIP-100: Add attempt information [15]
-
FLIP-102: Add more metrics to TaskManager [16]
-
FLIP-103: Better TM/JM log display [17]
-
[FLINK-14816] Add thread dump feature for TaskManager [18]
-
Runtime
-
FLIP-56: Support for dynamic slots on the TaskExecutor [19]
-
FLIP-76: Unaligned checkpoints [21]
-
FLIP-83: Flink e2e performance testing framework [22]
-
FLIP-85: Support cluster deploy mode [23]
-
FLIP-108: Add GPU to the resource management (specifically for UDTF &
UDF) [25]
-
FLIP-111: Consolidate docker images [26]
-
FLIP-116: Unified memory configuration for JobManager [56]
-
[FLINK-9407] ORC format for StreamingFileSink [27]
-
[FLINK-10934] Support per-job mode for Kubernetes integration [29]
-
[FLINK-11395] Avro writer for StreamingFileSink [30]
-
[FLINK-11427] Protobuf parquet writer for StreamingFileSink [31]
-
[FLINK-11499] Extend StreamingFileSink BulkFormats to support
arbitrary roll policies [32]
-
[FLINK-14106] Make SlotManager pluggable [33]
-
[FLINK-15674] Consolidate Java and Scala type extraction stack [35]
-
[FLINK-15679] Improve Flink’s ID system [36]
-
[FLINK-15788] Various Kubernetes improvements [38]
-
[FLINK-16408] Bind user code class loader to lifetime of a slot [40]
-
[FLINK-16428] Network memory management for backpressure [41]
-
[FLINK-16430] Pipelined region scheduling [42]
-
[FLINK-16605] Specify upper bound for number of allocated
TaskManagers [57]
-
State Backend:
-
[FLINK-5763] Make savepoint self-contained / relocatable [43]
-
[FLINK-8871] Complete checkpoint cancellation messages [44]
-
[FLINK-12692] Support disk spilling in HeapKeyedStateBackend [45]
-
[FLINK-15012] Cleanup of leftover files in HDFS/OSS/S3 [46]
-
Enable local recovery by default
-
[FLINK-15532] Enable strict capacity limit for memory usage for
RocksDB [48]
-
Other:
-
FLIP-42: Restructure documentation (partially in Flink 1.12) [1]
[0]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Exact-feature-freeze-date-td40624.html
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-65%3A+New+type+inference+for+Table+API+UDFs
[3]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878
[4]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
[5]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-93%3A+JDBC+catalog+and+Postgres+catalog
[6]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL
[7]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Reading+table+columns+from+different+parts+of+source+records
[8] https://issues.apache.org/jira/browse/FLINK-14807
[9]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
[10] https://issues.apache.org/jira/browse/FLINK-15670
[11]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF
[12]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
[13]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
[14]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable
[15]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
[16]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager
[17]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
[18] https://issues.apache.org/jira/browse/FLINK-14816
[19]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation
[20]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle
[21]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints
[22]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework
[23]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode
[24]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-92%3A+Add+N-Ary+Stream+Operator+in+Flink
[25]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink
[26]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification
[27] https://issues.apache.org/jira/browse/FLINK-9407
[28] https://issues.apache.org/jira/browse/FLINK-10742
[29] https://issues.apache.org/jira/browse/FLINK-10934
[30] https://issues.apache.org/jira/browse/FLINK-11395
[31] https://issues.apache.org/jira/browse/FLINK-11427
[32] https://issues.apache.org/jira/browse/FLINK-11499
[33] https://issues.apache.org/jira/browse/FLINK-14106
[34] https://issues.apache.org/jira/browse/FLINK-15672
[35] https://issues.apache.org/jira/browse/FLINK-15674
[36] https://issues.apache.org/jira/browse/FLINK-15679
[37] https://issues.apache.org/jira/browse/FLINK-15786
[38] https://issues.apache.org/jira/browse/FLINK-15788
[39] https://issues.apache.org/jira/browse/FLINK-15911
[39] https://issues.apache.org/jira/browse/FLINK-15154
[40] https://issues.apache.org/jira/browse/FLINK-16408
[41] https://issues.apache.org/jira/browse/FLINK-16428
[42] https://issues.apache.org/jira/browse/FLINK-16430
[43] https://issues.apache.org/jira/browse/FLINK-5763
[44] https://issues.apache.org/jira/browse/FLINK-8871
[45] https://issues.apache.org/jira/browse/FLINK-12692
[46] https://issues.apache.org/jira/browse/FLINK-15012
[47]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-customize-state-in-customized-KeyedStateBackend-td32771.html
[48] https://issues.apache.org/jira/browse/FLINK-15532
[49]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector
[50]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-114%3A+Support+Python+UDF+in+SQL+Client
[51]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-97%3A+Support+Scalar+Vectorized+Python+UDF+in+PyFlink
[52]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-120%3A+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
[53]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-121%3A+Support+Cython+Optimizing+Python+User+Defined+Function
[54]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-96%3A+Support+Python+ML+Pipeline+API
[55] https://issues.apache.org/jira/browse/FLINK-14500
[56]
https://cwiki.apache.org/confluence/display/FLINK/FLIP+116%3A+Unified+Memory+Configuration+for+Job+Managers
[57] https://issues.apache.org/jira/browse/FLINK-16605
