Thanks Gary for kicking off this discussion. Really appreciate that you and Yu offer to help to manage 1.10 release.
+1 for Gary and Yu as release managers. Thanks, Zhu Zhu Dian Fu <dian0511...@gmail.com> 于2019年9月7日周六 下午12:26写道： > Hi Gary, > > Thanks for kicking off the release schedule of 1.10. +1 for you and Yu Li > as the release manager. > > The feature freeze/release time sounds reasonable. > > Thanks, > Dian > > > 在 2019年9月7日，上午11:30，Jark Wu <imj...@gmail.com> 写道： > > > > Thanks Gary for kicking off the discussion for 1.10 release. > > > > +1 for Gary and Yu as release managers. Thank you for you effort. > > > > Best, > > Jark > > > > > >> 在 2019年9月7日，00:52，zhijiang <wangzhijiang...@aliyun.com.INVALID> 写道： > >> > >> Hi Gary, > >> > >> Thanks for kicking off the features for next release 1.10. I am very > supportive of you and Yu Li to be the relaese managers. > >> > >> Just mention another two improvements which want to be covered in > FLINK-1.10 and I already confirmed with Piotr to reach an agreement before. > >> > >> 1. Data serialize and copy only once for broadcast partition [1]: It > would improve the throughput performance greatly in broadcast mode and was > actually proposed in Flink-1.8. Most of works already done before and only > left the last critical jira/PR. It will not take much efforts to make it > ready. > >> > >> 2. Let Netty use Flink's buffers directly in credit-based mode [2] : It > could avoid memory copy from netty stack to flink managed network buffer. > The obvious benefit is decreasing the direct memory overhead greatly in > large-scale jobs. I also heard of some user cases encounter direct OOM > caused by netty memory overhead. Actually this improvment was proposed by > nico in FLINK-1.7 and always no time to focus then. Yun Gao already > submitted a PR half an year ago but have not been reviewed yet. I could > help review the deign and PR codes to make it ready. > >> > >> And you could make these two items as lowest priority if possible. > >> > >> [1] https://issues.apache.org/jira/browse/FLINK-10745 > >> [2] https://issues.apache.org/jira/browse/FLINK-10742 > >> > >> Best, > >> Zhijiang > >> ------------------------------------------------------------------ > >> From:Gary Yao <g...@apache.org> > >> Send Time:2019年9月6日(星期五) 17:06 > >> To:dev <dev@flink.apache.org> > >> Cc:carp84 <car...@gmail.com> > >> Subject:[DISCUSS] Features for Apache Flink 1.10 > >> > >> Hi community, > >> > >> Since Apache Flink 1.9.0 has been released more than 2 weeks ago, I > want to > >> start kicking off the discussion about what we want to achieve for the > 1.10 > >> release. > >> > >> Based on discussions with various people as well as observations from > >> mailing > >> list threads, Yu Li and I have compiled a list of features that we deem > >> important to be included in the next release. Note that the features > >> presented > >> here are not meant to be exhaustive. As always, I am sure that there > will be > >> other contributions that will make it into the next release. This email > >> thread > >> is merely to kick off a discussion, and to give users and contributors > an > >> understanding where the focus of the next release lies. If there is > anything > >> we have missed that somebody is working on, please reply to this thread. > >> > >> > >> ** Proposed features and focus > >> > >> Following the contribution of Blink to Apache Flink, the community > released > >> a > >> preview of the Blink SQL Query Processor, which offers better SQL > coverage > >> and > >> improved performance for batch queries, in Flink 1.9.0. However, the > >> integration of the Blink query processor is not fully completed yet as > there > >> are still pending tasks, such as implementing full TPC-DS support. With > the > >> next Flink release, we aim at finishing the Blink integration. > >> > >> Furthermore, there are several ongoing work threads addressing > long-standing > >> issues reported by users, such as improving checkpointing under > >> backpressure, > >> and limiting RocksDBs native memory usage, which can be especially > >> problematic > >> in containerized Flink deployments. > >> > >> Notable features surrounding Flink’s ecosystem that are planned for the > next > >> release include active Kubernetes support (i.e., enabling Flink’s > >> ResourceManager to launch new pods), improved Hive integration, Java 11 > >> support, and new algorithms for the Flink ML library. > >> > >> Below I have included the list of features that we compiled ordered by > >> priority – some of which already have ongoing mailing list threads, > JIRAs, > >> or > >> FLIPs. > >> > >> - Improving Flink’s build system & CI [1] [2] > >> - Support Java 11 [3] > >> - Table API improvements > >> - Configuration Evolution [4] [5] > >> - Finish type system: Expression Re-design [6] and UDF refactor > >> - Streaming DDL: Time attribute (watermark) and Changelog support > >> - Full SQL partition support for both batch & streaming [7] > >> - New Java Expression DSL [8] > >> - SQL CLI with DDL and DML support > >> - Hive compatibility completion (DDL/UDF) to support full Hive > integration > >> - Partition/Function/View support > >> - Remaining Blink planner/runtime merge > >> - Support all TPC-DS queries [9] > >> - Finer grained resource management > >> - Unified TaskExecutor Memory Configuration [10] > >> - Fine Grained Operator Resource Management [11] > >> - Dynamic Slots Allocation [12] > >> - Finish scheduler re-architecture [13] > >> - Allows implementing more sophisticated scheduling strategies such as > >> better batch scheduler or speculative execution. > >> - New DataStream Source Interface [14] > >> - A new source connector architecture to unify the implementation of > >> source connectors and make it simpler to implement custom source > connectors. > >> - Add more source/system metrics > >> - For better flink job monitoring and facilitate customized solutions > >> like auto-scaling. > >> - Executor Interface / Client API [15] > >> - Allow Flink downstream projects to easier and better monitor and > >> control flink jobs. > >> - Interactive Programming [16] > >> - Allow users to cache the intermediate results in Table API for later > >> usage to avoid redundant computation when a Flink application contains > >> multiple jobs. > >> - Python User Defined Function [17] > >> - Support native user-defined functions in Flink Python, including > >> UDF/UDAF/UDTF in Table API and Python-Java mixed UDF. > >> - Spillable heap backend [18] > >> - A new state backend supporting automatic data spill and load when > >> memory exhausted/regained. > >> - RocksDB backend memory control [19] > >> - Prevent excessive memory usage from RocksDB, especially in container > >> environment. > >> - Unaligned checkpoints [20] > >> - Resolve the checkpoint timeout issue under backpressure. > >> - Separate framework and user class loader in per-job mode > >> - Active Kubernetes Integration [21] > >> - Allow ResourceManager talking to Kubernetes to launch new pods > >> similar to Flink's Yarn/Mesos integration > >> - ML pipeline/library > >> - Aims at delivering several core algorithms, including Logistic > >> Regression, Native Bayes, Random Forest, KMeans, etc. > >> - Add vertex subtask log url on WebUI [22] > >> > >> > >> ** Suggested release timeline > >> > >> Based on our usual time-based release schedule [23], and considering > that > >> several events, such as Flink Forward Europe and Asia, are overlapping > with > >> the current release cycle, we should aim at releasing 1.10 around the > >> beginning of January 2020. To give the community enough testing time, I > >> propose the feature freeze to be at the end of November. We should > announce > >> an > >> exact date later in the release cycle. > >> > >> Lastly, I would like to use the opportunity to propose Yu Li and myself > as > >> release managers for the upcoming release. > >> > >> What do you think? > >> > >> > >> Best, > >> Gary > >> > >> [1] > >> > https://lists.apache.org/thread.html/775447a187410727f5ba6f9cefd6406c58ca5cc5c580aecf30cf213e@%3Cdev.flink.apache.org%3E > >> [2] > >> > https://lists.apache.org/thread.html/b90aa518fcabce94f8e1de4132f46120fae613db6e95a2705f1bd1ea@%3Cdev.flink.apache.org%3E > >> [3] https://issues.apache.org/jira/browse/FLINK-10725 > >> [4] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-54%3A+Evolve+ConfigOption+and+Configuration > >> [5] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-59%3A+Enable+execution+configuration+from+Configuration+object > >> [6] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design > >> [7] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > >> [8] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-55%3A+Introduction+of+a+Table+API+Java+Expression+DSL > >> [9] https://issues.apache.org/jira/browse/FLINK-11491 > >> [10] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > >> [11] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management > >> [12] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation > >> [13] https://issues.apache.org/jira/browse/FLINK-10429 > >> [14] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > >> [15] > >> > https://lists.apache.org/thread.html/498dd3e0277681cda356029582c1490299ae01df912e15942e11ae8e@%3Cdev.flink.apache.org%3E > >> [16] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink > >> [17] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table > >> [18] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend > >> [19] https://issues.apache.org/jira/browse/FLINK-7289 > >> [20] > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Checkpointing-under-backpressure-td31616.html > >> [21] > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Best-practice-to-run-flink-on-kubernetes-td31532.html > >> [22] https://issues.apache.org/jira/browse/FLINK-13894 > >> [23] > https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases > >> > > > >
