Looking at feature list, I don't see an item for complete the data
type
support. Specifically, high precision timestamp is needed to Hive
integration, as it's so common. Missing it would damage the
completeness of
our Hive effort.
Thanks,
Xuefu
On Sat, Sep 7, 2019 at 7:06 PM Xintong Song <tonysong...@gmail.com>
wrote:
Thanks Gray and Yu for compiling the feature list and kicking off
this
discussion.
+1 for Gary and Yu being the release managers for Flink 1.10.
Thank you~
Xintong Song
On Sat, Sep 7, 2019 at 4:58 PM Till Rohrmann <trohrm...@apache.org>
wrote:
Thanks for compiling the list of 1.10 efforts for the community
Gary. I
think this helps a lot to better understand what the community is
currently
working on.
Thanks for volunteering as the release managers for the next major
release. +1 for Gary and Yu being the RMs for Flink 1.10.
Cheers,
Till
On Sat, Sep 7, 2019 at 7:26 AM Zhu Zhu <reed...@gmail.com> wrote:
Thanks Gary for kicking off this discussion.
Really appreciate that you and Yu offer to help to manage 1.10
release.
+1 for Gary and Yu as release managers.
Thanks,
Zhu Zhu
Dian Fu <dian0511...@gmail.com> 于2019年9月7日周六
下午12:26写道：
Hi Gary,
Thanks for kicking off the release schedule of 1.10. +1 for you
and
Yu
Li
as the release manager.
The feature freeze/release time sounds reasonable.
Thanks,
Dian
在 2019年9月7日，上午11:30，Jark Wu <imj...@gmail.com>
写道：
Thanks Gary for kicking off the discussion for 1.10 release.
+1 for Gary and Yu as release managers. Thank you for you
effort.
Best,
Jark
在 2019年9月7日，00:52，zhijiang
<wangzhijiang...@aliyun.com.INVALID>
写道：
Hi Gary,
Thanks for kicking off the features for next release 1.10. I
am
very
supportive of you and Yu Li to be the relaese managers.
Just mention another two improvements which want to be covered
in
FLINK-1.10 and I already confirmed with Piotr to reach an
agreement
before.
1. Data serialize and copy only once for broadcast partition
[1]:
It
would improve the throughput performance greatly in broadcast
mode
and
was
actually proposed in Flink-1.8. Most of works already done before
and
only
left the last critical jira/PR. It will not take much efforts to
make
it
ready.
2. Let Netty use Flink's buffers directly in credit-based mode
[2] :
It
could avoid memory copy from netty stack to flink managed network
buffer.
The obvious benefit is decreasing the direct memory overhead
greatly
in
large-scale jobs. I also heard of some user cases encounter
direct
OOM
caused by netty memory overhead. Actually this improvment was
proposed
by
nico in FLINK-1.7 and always no time to focus then. Yun Gao
already
submitted a PR half an year ago but have not been reviewed yet. I
could
help review the deign and PR codes to make it ready.
And you could make these two items as lowest priority if
possible.
[1] https://issues.apache.org/jira/browse/FLINK-10745
[2] https://issues.apache.org/jira/browse/FLINK-10742
Best,
Zhijiang
------------------------------------------------------------------
From:Gary Yao <g...@apache.org>
Send Time:2019年9月6日(星期五) 17:06
To:dev <dev@flink.apache.org>
Cc:carp84 <car...@gmail.com>
Subject:[DISCUSS] Features for Apache Flink 1.10
Hi community,
Since Apache Flink 1.9.0 has been released more than 2 weeks
ago,
I
want to
start kicking off the discussion about what we want to achieve
for
the
1.10
release.
Based on discussions with various people as well as
observations
from
mailing
list threads, Yu Li and I have compiled a list of features that
we
deem
important to be included in the next release. Note that the
features
presented
here are not meant to be exhaustive. As always, I am sure that
there
will be
other contributions that will make it into the next release.
This
email
thread
is merely to kick off a discussion, and to give users and
contributors
an
understanding where the focus of the next release lies. If
there
is
anything
we have missed that somebody is working on, please reply to
this
thread.
** Proposed features and focus
Following the contribution of Blink to Apache Flink, the
community
released
a
preview of the Blink SQL Query Processor, which offers better
SQL
coverage
and
improved performance for batch queries, in Flink 1.9.0.
However,
the
integration of the Blink query processor is not fully completed
yet
as
there
are still pending tasks, such as implementing full TPC-DS
support.
With
the
next Flink release, we aim at finishing the Blink integration.
Furthermore, there are several ongoing work threads addressing
long-standing
issues reported by users, such as improving checkpointing under
backpressure,
and limiting RocksDBs native memory usage, which can be
especially
problematic
in containerized Flink deployments.
Notable features surrounding Flink’s ecosystem that are
planned
for
the
next
release include active Kubernetes support (i.e., enabling
Flink’s
ResourceManager to launch new pods), improved Hive integration,
Java
11
support, and new algorithms for the Flink ML library.
Below I have included the list of features that we compiled
ordered
by
priority – some of which already have ongoing mailing list
threads,
JIRAs,
or
FLIPs.
- Improving Flink’s build system & CI [1] [2]
- Support Java 11 [3]
- Table API improvements
- Configuration Evolution [4] [5]
- Finish type system: Expression Re-design [6] and UDF
refactor
- Streaming DDL: Time attribute (watermark) and Changelog
support
- Full SQL partition support for both batch & streaming [7]
- New Java Expression DSL [8]
- SQL CLI with DDL and DML support
- Hive compatibility completion (DDL/UDF) to support full Hive
integration
- Partition/Function/View support
- Remaining Blink planner/runtime merge
- Support all TPC-DS queries [9]
- Finer grained resource management
- Unified TaskExecutor Memory Configuration [10]
- Fine Grained Operator Resource Management [11]
- Dynamic Slots Allocation [12]
- Finish scheduler re-architecture [13]
- Allows implementing more sophisticated scheduling
strategies
such
as
better batch scheduler or speculative execution.
- New DataStream Source Interface [14]
- A new source connector architecture to unify the
implementation
of
source connectors and make it simpler to implement custom
source
connectors.
- Add more source/system metrics
- For better flink job monitoring and facilitate customized
solutions
like auto-scaling.
- Executor Interface / Client API [15]
- Allow Flink downstream projects to easier and better
monitor
and
control flink jobs.
- Interactive Programming [16]
- Allow users to cache the intermediate results in Table API
for
later
usage to avoid redundant computation when a Flink application
contains
multiple jobs.
- Python User Defined Function [17]
- Support native user-defined functions in Flink Python,
including
UDF/UDAF/UDTF in Table API and Python-Java mixed UDF.
- Spillable heap backend [18]
- A new state backend supporting automatic data spill and
load
when
memory exhausted/regained.
- RocksDB backend memory control [19]
- Prevent excessive memory usage from RocksDB, especially in
container
environment.
- Unaligned checkpoints [20]
- Resolve the checkpoint timeout issue under backpressure.
- Separate framework and user class loader in per-job mode
- Active Kubernetes Integration [21]
- Allow ResourceManager talking to Kubernetes to launch new
pods
similar to Flink's Yarn/Mesos integration
- ML pipeline/library
- Aims at delivering several core algorithms, including
Logistic
Regression, Native Bayes, Random Forest, KMeans, etc.
- Add vertex subtask log url on WebUI [22]
** Suggested release timeline
Based on our usual time-based release schedule [23], and
considering
that
several events, such as Flink Forward Europe and Asia, are
overlapping
with
the current release cycle, we should aim at releasing 1.10
around
the
beginning of January 2020. To give the community enough testing
time,
I
propose the feature freeze to be at the end of November. We
should
announce
an
exact date later in the release cycle.
Lastly, I would like to use the opportunity to propose Yu Li
and
myself
as
release managers for the upcoming release.
What do you think?
Best,
Gary
[1]
https://lists.apache.org/thread.html/775447a187410727f5ba6f9cefd6406c58ca5cc5c580aecf30cf213e@%3Cdev.flink.apache.org%3E
[2]
https://lists.apache.org/thread.html/b90aa518fcabce94f8e1de4132f46120fae613db6e95a2705f1bd1ea@%3Cdev.flink.apache.org%3E
[3] https://issues.apache.org/jira/browse/FLINK-10725
[4]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-54%3A+Evolve+ConfigOption+and+Configuration
[5]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-59%3A+Enable+execution+configuration+from+Configuration+object
[6]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design
[7]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support
[8]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-55%3A+Introduction+of+a+Table+API+Java+Expression+DSL
[9] https://issues.apache.org/jira/browse/FLINK-11491
[10]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
[11]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
[12]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation
[13] https://issues.apache.org/jira/browse/FLINK-10429
[14]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
[15]
https://lists.apache.org/thread.html/498dd3e0277681cda356029582c1490299ae01df912e15942e11ae8e@%3Cdev.flink.apache.org%3E
[16]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
[17]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
[18]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend
[19] https://issues.apache.org/jira/browse/FLINK-7289
[20]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Checkpointing-under-backpressure-td31616.html
[21]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Best-practice-to-run-flink-on-kubernetes-td31532.html
[22] https://issues.apache.org/jira/browse/FLINK-13894
[23]
https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases
--
Xuefu Zhang
"In Honey We Trust!"
