Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

[Peter Huang]

Congratulations Klou!



Best Regards
Peter Huang

On Sat, Sep 7, 2019 at 5:03 PM vino yang  wrote:

> Congratulations Klou!
>
> Best,
> Vino
>
> zhijiang  于2019年9月7日周六 上午12:22写道：
>
> > Congratulations Klou!
> >
> > Best,
> > Zhijiang
> > --
> > From:Zhu Zhu 
> > Send Time:2019年9月6日(星期五) 17:19
> > To:dev 
> > Subject:Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC
> >
> > Congratulations Kostas!
> >
> > Thanks,
> > Zhu Zhu
> >
> > Yu Li  于2019年9月6日周五 下午10:49写道：
> >
> > > Congratulations Klou!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Fri, 6 Sep 2019 at 22:43, Forward Xu 
> wrote:
> > >
> > > > Congratulations Kloudas!
> > > >
> > > >
> > > > Best,
> > > >
> > > > Forward
> > > >
> > > > Dawid Wysakowicz  于2019年9月6日周五 下午10:36写道：
> > > >
> > > > > Congratulations Klou!
> > > > >
> > > > > Best,
> > > > >
> > > > > Dawid
> > > > >
> > > > > On 06/09/2019 14:55, Fabian Hueske wrote:
> > > > > > Hi everyone,
> > > > > >
> > > > > > I'm very happy to announce that Kostas Kloudas is joining the
> Flink
> > > > PMC.
> > > > > > Kostas is contributing to Flink for many years and puts lots of
> > > effort
> > > > in
> > > > > > helping our users and growing the Flink community.
> > > > > >
> > > > > > Please join me in congratulating Kostas!
> > > > > >
> > > > > > Cheers,
> > > > > > Fabian
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> >
>

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

[vino yang]

Congratulations Klou!

Best,
Vino

zhijiang  于2019年9月7日周六 上午12:22写道：

> Congratulations Klou!
>
> Best,
> Zhijiang
> --
> From:Zhu Zhu 
> Send Time:2019年9月6日(星期五) 17:19
> To:dev 
> Subject:Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC
>
> Congratulations Kostas!
>
> Thanks,
> Zhu Zhu
>
> Yu Li  于2019年9月6日周五 下午10:49写道：
>
> > Congratulations Klou!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Fri, 6 Sep 2019 at 22:43, Forward Xu  wrote:
> >
> > > Congratulations Kloudas!
> > >
> > >
> > > Best,
> > >
> > > Forward
> > >
> > > Dawid Wysakowicz  于2019年9月6日周五 下午10:36写道：
> > >
> > > > Congratulations Klou!
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > >
> > > > On 06/09/2019 14:55, Fabian Hueske wrote:
> > > > > Hi everyone,
> > > > >
> > > > > I'm very happy to announce that Kostas Kloudas is joining the Flink
> > > PMC.
> > > > > Kostas is contributing to Flink for many years and puts lots of
> > effort
> > > in
> > > > > helping our users and growing the Flink community.
> > > > >
> > > > > Please join me in congratulating Kostas!
> > > > >
> > > > > Cheers,
> > > > > Fabian
> > > > >
> > > >
> > > >
> > >
> >
>
>

[jira] [Created] (FLINK-14002) FlinkKafkaProducer constructor that takes KafkaSerializationSchema shouldnt take default topic

[Gyula Fora (Jira)]

Gyula Fora created FLINK-14002:
--

 Summary: FlinkKafkaProducer constructor that takes 
KafkaSerializationSchema shouldnt take default topic
 Key: FLINK-14002
 URL: https://issues.apache.org/jira/browse/FLINK-14002
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Reporter: Gyula Fora


When the KafkaSerializationSchema is used the user has the to provide the topic 
always when they create the ProducerRecord.

The defaultTopic specified in the constructor (and enforced not to be null) 
will always be ignored, this is very misleading.

We should depracate these constructors and create new ones without defaultTopic.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (FLINK-14001) add a API to enable user to get the operator code generated by Flink engine

[Dezhi Cai (Jira)]

Dezhi Cai created FLINK-14001:
-

 Summary: add a API to enable user to get the operator code 
generated by Flink engine
 Key: FLINK-14001
 URL: https://issues.apache.org/jira/browse/FLINK-14001
 Project: Flink
  Issue Type: Wish
  Components: Runtime / Task
Affects Versions: 1.9.0
Reporter: Dezhi Cai
 Fix For: 1.9.1


In Spark, we can invoke  Dataset method "queryExecution.debug.codegen()" to get 
the code produced by Catalyst.  If we add a similar API to enable user to get 
the code generated by 
Flink engine, this would be a very nice way to provide more verbose feedback 
for debugging.
 
reference link : 
[https://medium.com/virtuslab/spark-sql-under-the-hood-part-i-26077f85ebf0]  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [DISCUSS] Features for Apache Flink 1.10

[Xintong Song]

Thanks Gray and Yu for compiling the feature list and kicking off this
discussion.

+1 for Gary and Yu being the release managers for Flink 1.10.

Thank you~

Xintong Song



On Sat, Sep 7, 2019 at 4:58 PM Till Rohrmann  wrote:

> Thanks for compiling the list of 1.10 efforts for the community Gary. I
> think this helps a lot to better understand what the community is currently
> working on.
>
> Thanks for volunteering as the release managers for the next major
> release. +1 for Gary and Yu being the RMs for Flink 1.10.
>
> Cheers,
> Till
>
> On Sat, Sep 7, 2019 at 7:26 AM Zhu Zhu  wrote:
>
> > Thanks Gary for kicking off this discussion.
> > Really appreciate that you and Yu offer to help to manage 1.10 release.
> >
> > +1 for Gary and Yu as release managers.
> >
> > Thanks,
> > Zhu Zhu
> >
> > Dian Fu  于2019年9月7日周六 下午12:26写道：
> >
> > > Hi Gary,
> > >
> > > Thanks for kicking off the release schedule of 1.10. +1 for you and Yu
> Li
> > > as the release manager.
> > >
> > > The feature freeze/release time sounds reasonable.
> > >
> > > Thanks,
> > > Dian
> > >
> > > > 在 2019年9月7日，上午11:30，Jark Wu  写道：
> > > >
> > > > Thanks Gary for kicking off the discussion for 1.10 release.
> > > >
> > > > +1 for Gary and Yu as release managers. Thank you for you effort.
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > >
> > > >> 在 2019年9月7日，00:52，zhijiang  写道：
> > > >>
> > > >> Hi Gary,
> > > >>
> > > >> Thanks for kicking off the features for next release 1.10.  I am
> very
> > > supportive of you and Yu Li to be the relaese managers.
> > > >>
> > > >> Just mention another two improvements which want to be covered in
> > > FLINK-1.10 and I already confirmed with Piotr to reach an agreement
> > before.
> > > >>
> > > >> 1. Data serialize and copy only once for broadcast partition [1]: It
> > > would improve the throughput performance greatly in broadcast mode and
> > was
> > > actually proposed in Flink-1.8. Most of works already done before and
> > only
> > > left the last critical jira/PR. It will not take much efforts to make
> it
> > > ready.
> > > >>
> > > >> 2. Let Netty use Flink's buffers directly in credit-based mode [2] :
> > It
> > > could avoid memory copy from netty stack to flink managed network
> buffer.
> > > The obvious benefit is decreasing the direct memory overhead greatly in
> > > large-scale jobs. I also heard of some user cases encounter direct OOM
> > > caused by netty memory overhead. Actually this improvment was proposed
> by
> > > nico in FLINK-1.7 and always no time to focus then. Yun Gao already
> > > submitted a PR half an year ago but have not been reviewed yet. I could
> > > help review the deign and PR codes to make it ready.
> > > >>
> > > >> And you could make these two items as lowest priority if possible.
> > > >>
> > > >> [1] https://issues.apache.org/jira/browse/FLINK-10745
> > > >> [2] https://issues.apache.org/jira/browse/FLINK-10742
> > > >>
> > > >> Best,
> > > >> Zhijiang
> > > >> --
> > > >> From:Gary Yao 
> > > >> Send Time:2019年9月6日(星期五) 17:06
> > > >> To:dev 
> > > >> Cc:carp84 
> > > >> Subject:[DISCUSS] Features for Apache Flink 1.10
> > > >>
> > > >> Hi community,
> > > >>
> > > >> Since Apache Flink 1.9.0 has been released more than 2 weeks ago, I
> > > want to
> > > >> start kicking off the discussion about what we want to achieve for
> the
> > > 1.10
> > > >> release.
> > > >>
> > > >> Based on discussions with various people as well as observations
> from
> > > >> mailing
> > > >> list threads, Yu Li and I have compiled a list of features that we
> > deem
> > > >> important to be included in the next release. Note that the features
> > > >> presented
> > > >> here are not meant to be exhaustive. As always, I am sure that there
> > > will be
> > > >> other contributions that will make it into the next release. This
> > email
> > > >> thread
> > > >> is merely to kick off a discussion, and to give users and
> contributors
> > > an
> > > >> understanding where the focus of the next release lies. If there is
> > > anything
> > > >> we have missed that somebody is working on, please reply to this
> > thread.
> > > >>
> > > >>
> > > >> ** Proposed features and focus
> > > >>
> > > >> Following the contribution of Blink to Apache Flink, the community
> > > released
> > > >> a
> > > >> preview of the Blink SQL Query Processor, which offers better SQL
> > > coverage
> > > >> and
> > > >> improved performance for batch queries, in Flink 1.9.0. However, the
> > > >> integration of the Blink query processor is not fully completed yet
> as
> > > there
> > > >> are still pending tasks, such as implementing full TPC-DS support.
> > With
> > > the
> > > >> next Flink release, we aim at finishing the Blink integration.
> > > >>
> > > >> Furthermore, there are several ongoing work threads addressing
> > > long-standing
> > > >> issues reported by users, such as improving checkpointing under
> > > >> backpressure,
> > > 

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

[Xintong Song]

Congratulations, Klou~

Thank you~

Xintong Song



On Sat, Sep 7, 2019 at 2:44 PM Kurt Young  wrote:

> Congratulations Klou!
>
> Best,
> Kurt
>
>
> On Sat, Sep 7, 2019 at 2:37 PM ying  wrote:
>
> > Congratulations Kostas!
> >
> > On Fri, Sep 6, 2019 at 11:21 PM Gary Yao  wrote:
> >
> > > Congratulations Klou!
> > >
> > > On Sat, Sep 7, 2019 at 6:21 AM Thomas Weise  wrote:
> > >
> > > > Congratulations!
> > > >
> > > >
> > > > On Fri, Sep 6, 2019 at 9:22 AM zhijiang  > > > .invalid>
> > > > wrote:
> > > >
> > > > > Congratulations Klou!
> > > > >
> > > > > Best,
> > > > > Zhijiang
> > > > > --
> > > > > From:Zhu Zhu 
> > > > > Send Time:2019年9月6日(星期五) 17:19
> > > > > To:dev 
> > > > > Subject:Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC
> > > > >
> > > > > Congratulations Kostas!
> > > > >
> > > > > Thanks,
> > > > > Zhu Zhu
> > > > >
> > > > > Yu Li  于2019年9月6日周五 下午10:49写道：
> > > > >
> > > > > > Congratulations Klou!
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > >
> > > > > > On Fri, 6 Sep 2019 at 22:43, Forward Xu 
> > > > wrote:
> > > > > >
> > > > > > > Congratulations Kloudas!
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Forward
> > > > > > >
> > > > > > > Dawid Wysakowicz  于2019年9月6日周五
> > 下午10:36写道：
> > > > > > >
> > > > > > > > Congratulations Klou!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Dawid
> > > > > > > >
> > > > > > > > On 06/09/2019 14:55, Fabian Hueske wrote:
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > I'm very happy to announce that Kostas Kloudas is joining
> the
> > > > Flink
> > > > > > > PMC.
> > > > > > > > > Kostas is contributing to Flink for many years and puts
> lots
> > of
> > > > > > effort
> > > > > > > in
> > > > > > > > > helping our users and growing the Flink community.
> > > > > > > > >
> > > > > > > > > Please join me in congratulating Kostas!
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Fabian
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Supporting multiple Flink versions vs. tech debt

[David Morávek]

sorry, wrong mailing list, I wanted to send this to the beam one.

Sorry for the confusion.
D.

On Sat, Sep 7, 2019 at 12:32 PM David Morávek  wrote:

> Hello,
>
> we currently have an opened PR for Flink 1.9
> , which greatly improves the
> runner for batch use-case. In case the PR gets merged, we would be
> supporting 5 latest major versions of Flink, which obviously come with high
> maintenance price and makes future development harder (there are already a
> sub-optimal parts due to compatibility with previous versions). Thomas and
> Max expressed needs for addressing the issue with the current release.
>
> Let's break down possible solution for the problem.
>
> *1) Current solution*
>
> Currently we maintain separate build for each version. The project
> structure looks as follows:
>
> *flink/*
> +
> *1.5/*
>+ *src/** # implementation of classes that differ between versions*
>- build.gradle
> + *1.6/*
>+ build.gradle #* the version is backward compatible, so it can reuse
> "overrides" from 1.5*
> + *1.7/*
>+ build.gradle #* the version is backward compatible, so it can reuse
> "overrides" from 1.5*
> + *1.8/*
>+ *src/ **# implementation of classes that differ between versions*
>- build.gradle
> + *1.9/*
>+ *src/ **# implementation of classes that differ between versions*
>- build.gradle
> + *src/*
> * # common source, shared among runner versions*
> - flink_runner.gradle
> * # included by  each /build.gradle*
>
> The problem with this structure is, that we always need to copy all of the
> version specific classes between backward incompatible versions, which
> results in *duplicate files* (we can not simply override a single file,
> because it wouldn't compile due to duplicate classes).
>
> *2) Symlink duplicates*
>
> Maybe we can simply symlink duplicates between versions and only override
> the files that need to be changed?
>
> *3) Adjusting the gradle build*
>
> Currently a version build looks something like this (this one is for 1.7.x
> version):
>
> project.ext {
>   // Set the version of all Flink-related dependencies here.
>   flink_version = '1.7.2'
>   // Main source directory and Flink version specific code.
>   main_source_dirs = ["$basePath/src/main/java", "../1.5/src/main/java"]
>   test_source_dirs = ["$basePath/src/test/java", "../1.5/src/test/java"]
>   main_resources_dirs = ["$basePath/src/main/resources"]
>   test_resources_dirs = ["$basePath/src/test/resources"]
>   archives_base_name = 'beam-runners-flink-1.7'
> }
>
> // Load the main build script which contains all build logic.
> apply from: "$basePath/flink_runner.gradle"
>
> It basically says, take the common source and append version specific
> implementations from 1.5 version. Let's say we want to override a single
> file for 1.8. We need to copy everything from 1.5/src and the build file
> would look as follows:
>
> /* All properties required for loading the Flink build script */
> project.ext {
>   // Set the version of all Flink-related dependencies here.
>   flink_version = '1.8.0'
>   // Main source directory and Flink version specific code.
>   main_source_dirs = ["$basePath/src/main/java", "./src/main/java"]
>   test_source_dirs = ["$basePath/src/test/java", "./src/test/java"]
>   main_resources_dirs = ["$basePath/src/main/resources"]
>   test_resources_dirs = ["$basePath/src/test/resources"]
>   archives_base_name = 'beam-runners-flink-1.8'
> }
>
> // Load the main build script which contains all build logic.
> apply from: "$basePath/flink_runner.gradle"
>
> For simplicity, let's only focus on *main_source_dirs*. What we really want
> to do is to tell the build, to use everything from 1.5 and override a
> single class (e.g. CoderTypeSerializer).
>
> def copyOverrides = tasks.register('copyOverrides', Copy) {
>   it.from '../1.5/src/', './src'
>   it.into "${project.buildDir}/flink-overrides/src"
>   it.duplicatesStrategy DuplicatesStrategy.INCLUDE // The last duplicate
> file 'wins'.
> }
>
> compileJava.dependsOn copyOverrides
>
> projext.ext {
>   main_source_dirs = ["$basePath/src/main/java",
> "${project.buildDir}/flink-overrides/src/main/java"]
> }
>
> This would copy all overrides into build directory, and it case of
> duplicate it picks the latest one. Than the build would simple compile
> classes from the newly created java files in build directory.
>
> *4) Maintaining last 3 major versions only*
>
> I recall that Flink community only supports 3 latest major versions
>  (please correct me if I'm
> mistaken). I suggest the the* Beam would do the same*. There is already
> an opened BEAM-7962  that
> suggest dropping 1.5 & 1.6 versions. Maybe this would allow us to keep the
> current structure with bearable amount of technical debt?
>
> Personally I'm in favor of *4)* combined with *3)*.
>
> What do you think? Do you have any other suggestions how to 

[DISCUSS] Supporting multiple Flink versions vs. tech debt

[David Morávek]

Hello,

we currently have an opened PR for Flink 1.9
, which greatly improves the
runner for batch use-case. In case the PR gets merged, we would be
supporting 5 latest major versions of Flink, which obviously come with high
maintenance price and makes future development harder (there are already a
sub-optimal parts due to compatibility with previous versions). Thomas and
Max expressed needs for addressing the issue with the current release.

Let's break down possible solution for the problem.

*1) Current solution*

Currently we maintain separate build for each version. The project
structure looks as follows:

*flink/*
+
*1.5/*
   + *src/** # implementation of classes that differ between versions*
   - build.gradle
+ *1.6/*
   + build.gradle #* the version is backward compatible, so it can reuse
"overrides" from 1.5*
+ *1.7/*
   + build.gradle #* the version is backward compatible, so it can reuse
"overrides" from 1.5*
+ *1.8/*
   + *src/ **# implementation of classes that differ between versions*
   - build.gradle
+ *1.9/*
   + *src/ **# implementation of classes that differ between versions*
   - build.gradle
+ *src/*
* # common source, shared among runner versions*
- flink_runner.gradle
* # included by  each /build.gradle*

The problem with this structure is, that we always need to copy all of the
version specific classes between backward incompatible versions, which
results in *duplicate files* (we can not simply override a single file,
because it wouldn't compile due to duplicate classes).

*2) Symlink duplicates*

Maybe we can simply symlink duplicates between versions and only override
the files that need to be changed?

*3) Adjusting the gradle build*

Currently a version build looks something like this (this one is for 1.7.x
version):

project.ext {
  // Set the version of all Flink-related dependencies here.
  flink_version = '1.7.2'
  // Main source directory and Flink version specific code.
  main_source_dirs = ["$basePath/src/main/java", "../1.5/src/main/java"]
  test_source_dirs = ["$basePath/src/test/java", "../1.5/src/test/java"]
  main_resources_dirs = ["$basePath/src/main/resources"]
  test_resources_dirs = ["$basePath/src/test/resources"]
  archives_base_name = 'beam-runners-flink-1.7'
}

// Load the main build script which contains all build logic.
apply from: "$basePath/flink_runner.gradle"

It basically says, take the common source and append version specific
implementations from 1.5 version. Let's say we want to override a single
file for 1.8. We need to copy everything from 1.5/src and the build file
would look as follows:

/* All properties required for loading the Flink build script */
project.ext {
  // Set the version of all Flink-related dependencies here.
  flink_version = '1.8.0'
  // Main source directory and Flink version specific code.
  main_source_dirs = ["$basePath/src/main/java", "./src/main/java"]
  test_source_dirs = ["$basePath/src/test/java", "./src/test/java"]
  main_resources_dirs = ["$basePath/src/main/resources"]
  test_resources_dirs = ["$basePath/src/test/resources"]
  archives_base_name = 'beam-runners-flink-1.8'
}

// Load the main build script which contains all build logic.
apply from: "$basePath/flink_runner.gradle"

For simplicity, let's only focus on *main_source_dirs*. What we really want
to do is to tell the build, to use everything from 1.5 and override a
single class (e.g. CoderTypeSerializer).

def copyOverrides = tasks.register('copyOverrides', Copy) {
  it.from '../1.5/src/', './src'
  it.into "${project.buildDir}/flink-overrides/src"
  it.duplicatesStrategy DuplicatesStrategy.INCLUDE // The last duplicate
file 'wins'.
}

compileJava.dependsOn copyOverrides

projext.ext {
  main_source_dirs = ["$basePath/src/main/java",
"${project.buildDir}/flink-overrides/src/main/java"]
}

This would copy all overrides into build directory, and it case of
duplicate it picks the latest one. Than the build would simple compile
classes from the newly created java files in build directory.

*4) Maintaining last 3 major versions only*

I recall that Flink community only supports 3 latest major versions
 (please correct me if I'm
mistaken). I suggest the the* Beam would do the same*. There is already an
opened BEAM-7962  that
suggest dropping 1.5 & 1.6 versions. Maybe this would allow us to keep the
current structure with bearable amount of technical debt?

Personally I'm in favor of *4)* combined with *3)*.

What do you think? Do you have any other suggestions how to solve this?

Thanks,
D.

Re: [DISCUSS] Features for Apache Flink 1.10

[Till Rohrmann]

Thanks for compiling the list of 1.10 efforts for the community Gary. I
think this helps a lot to better understand what the community is currently
working on.

Thanks for volunteering as the release managers for the next major
release. +1 for Gary and Yu being the RMs for Flink 1.10.

Cheers,
Till

On Sat, Sep 7, 2019 at 7:26 AM Zhu Zhu  wrote:

> Thanks Gary for kicking off this discussion.
> Really appreciate that you and Yu offer to help to manage 1.10 release.
>
> +1 for Gary and Yu as release managers.
>
> Thanks,
> Zhu Zhu
>
> Dian Fu  于2019年9月7日周六 下午12:26写道：
>
> > Hi Gary,
> >
> > Thanks for kicking off the release schedule of 1.10. +1 for you and Yu Li
> > as the release manager.
> >
> > The feature freeze/release time sounds reasonable.
> >
> > Thanks,
> > Dian
> >
> > > 在 2019年9月7日，上午11:30，Jark Wu  写道：
> > >
> > > Thanks Gary for kicking off the discussion for 1.10 release.
> > >
> > > +1 for Gary and Yu as release managers. Thank you for you effort.
> > >
> > > Best,
> > > Jark
> > >
> > >
> > >> 在 2019年9月7日，00:52，zhijiang  写道：
> > >>
> > >> Hi Gary,
> > >>
> > >> Thanks for kicking off the features for next release 1.10.  I am very
> > supportive of you and Yu Li to be the relaese managers.
> > >>
> > >> Just mention another two improvements which want to be covered in
> > FLINK-1.10 and I already confirmed with Piotr to reach an agreement
> before.
> > >>
> > >> 1. Data serialize and copy only once for broadcast partition [1]: It
> > would improve the throughput performance greatly in broadcast mode and
> was
> > actually proposed in Flink-1.8. Most of works already done before and
> only
> > left the last critical jira/PR. It will not take much efforts to make it
> > ready.
> > >>
> > >> 2. Let Netty use Flink's buffers directly in credit-based mode [2] :
> It
> > could avoid memory copy from netty stack to flink managed network buffer.
> > The obvious benefit is decreasing the direct memory overhead greatly in
> > large-scale jobs. I also heard of some user cases encounter direct OOM
> > caused by netty memory overhead. Actually this improvment was proposed by
> > nico in FLINK-1.7 and always no time to focus then. Yun Gao already
> > submitted a PR half an year ago but have not been reviewed yet. I could
> > help review the deign and PR codes to make it ready.
> > >>
> > >> And you could make these two items as lowest priority if possible.
> > >>
> > >> [1] https://issues.apache.org/jira/browse/FLINK-10745
> > >> [2] https://issues.apache.org/jira/browse/FLINK-10742
> > >>
> > >> Best,
> > >> Zhijiang
> > >> --
> > >> From:Gary Yao 
> > >> Send Time:2019年9月6日(星期五) 17:06
> > >> To:dev 
> > >> Cc:carp84 
> > >> Subject:[DISCUSS] Features for Apache Flink 1.10
> > >>
> > >> Hi community,
> > >>
> > >> Since Apache Flink 1.9.0 has been released more than 2 weeks ago, I
> > want to
> > >> start kicking off the discussion about what we want to achieve for the
> > 1.10
> > >> release.
> > >>
> > >> Based on discussions with various people as well as observations from
> > >> mailing
> > >> list threads, Yu Li and I have compiled a list of features that we
> deem
> > >> important to be included in the next release. Note that the features
> > >> presented
> > >> here are not meant to be exhaustive. As always, I am sure that there
> > will be
> > >> other contributions that will make it into the next release. This
> email
> > >> thread
> > >> is merely to kick off a discussion, and to give users and contributors
> > an
> > >> understanding where the focus of the next release lies. If there is
> > anything
> > >> we have missed that somebody is working on, please reply to this
> thread.
> > >>
> > >>
> > >> ** Proposed features and focus
> > >>
> > >> Following the contribution of Blink to Apache Flink, the community
> > released
> > >> a
> > >> preview of the Blink SQL Query Processor, which offers better SQL
> > coverage
> > >> and
> > >> improved performance for batch queries, in Flink 1.9.0. However, the
> > >> integration of the Blink query processor is not fully completed yet as
> > there
> > >> are still pending tasks, such as implementing full TPC-DS support.
> With
> > the
> > >> next Flink release, we aim at finishing the Blink integration.
> > >>
> > >> Furthermore, there are several ongoing work threads addressing
> > long-standing
> > >> issues reported by users, such as improving checkpointing under
> > >> backpressure,
> > >> and limiting RocksDBs native memory usage, which can be especially
> > >> problematic
> > >> in containerized Flink deployments.
> > >>
> > >> Notable features surrounding Flink’s ecosystem that are planned for
> the
> > next
> > >> release include active Kubernetes support (i.e., enabling Flink’s
> > >> ResourceManager to launch new pods), improved Hive integration, Java
> 11
> > >> support, and new algorithms for the Flink ML library.
> > >>
> > >> Below I have included the list of features that we 

[jira] [Created] (FLINK-14000) Remove legacy ProcessShutDownThread

[Zhu Zhu (Jira)]

Zhu Zhu created FLINK-14000:
---

 Summary: Remove legacy ProcessShutDownThread
 Key: FLINK-14000
 URL: https://issues.apache.org/jira/browse/FLINK-14000
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Affects Versions: 1.10.0
Reporter: Zhu Zhu


The class _org.apache.flink.runtime.util.ProcessShutDownThread_ is not 
referenced by any other classes/configs/scripts any more.

I think it's dead code and we can remove it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [VOTE] FLIP-53: Fine Grained Operator Resource Management

[Kurt Young]

+1 for FLIP-53.

I would like to raise one minor concern regarding to implementing
request absolute amount of memory case. Currently, it will be
translated to a memory fraction during compile, and translate back
to absolute value during execution. There is a risk that the user might
get less than he requested due to floating point number problems.

Best,
Kurt


On Fri, Sep 6, 2019 at 10:13 PM Andrey Zagrebin 
wrote:

> Thanks for starting the vote @Xintong
>
> +1 for the FLIP-53
>
> Best,
> Andrey
>
> On Fri, Sep 6, 2019 at 3:53 PM Till Rohrmann  wrote:
>
> > Hi Xintong,
> >
> > thanks for starting this vote. The proposal looks good and, hence, +1 for
> > it.
> >
> > One comment I have is concerning the first implementation step. I would
> > suggest to not add the flag allSourcesInSamePipelinedRegion to the
> > ExecutionConfig because the ExecutionConfig is public API. Ideally we
> keep
> > this flag internal and don't expose it to the user.
> >
> > Cheers,
> > Till
> >
> > On Fri, Sep 6, 2019 at 1:47 PM Zhu Zhu  wrote:
> >
> > > Thanks Xintong for proposing this better resource management.
> > > This helps a lot to users who want to better manage the job resources.
> > And
> > > would be even more useful if in the future we can have auto-tuning
> > > mechanism for jobs.
> > >
> > > +1 (non-binding)
> > >
> > > Thanks,
> > > Zhu Zhu
> > >
> > > Xintong Song  于2019年9月6日周五 上午11:17写道：
> > >
> > > > Hi all,
> > > >
> > > > I would like to start the voting process for FLIP-53 [1], which is
> > > > discussed and reached consensus in this thread [2].
> > > >
> > > > This voting will be open for at least 72 hours (excluding weekends).
> > I'll
> > > > try to close it Sep. 11, 04:00 UTC, unless there is an objection or
> not
> > > > enough votes.
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
> > > >
> > > > [2]
> > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-53-Fine-Grained-Resource-Management-td31831.html
> > > >
> > >
> >
>

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

[Kurt Young]

Congratulations Klou!

Best,
Kurt


On Sat, Sep 7, 2019 at 2:37 PM ying  wrote:

> Congratulations Kostas!
>
> On Fri, Sep 6, 2019 at 11:21 PM Gary Yao  wrote:
>
> > Congratulations Klou!
> >
> > On Sat, Sep 7, 2019 at 6:21 AM Thomas Weise  wrote:
> >
> > > Congratulations!
> > >
> > >
> > > On Fri, Sep 6, 2019 at 9:22 AM zhijiang  > > .invalid>
> > > wrote:
> > >
> > > > Congratulations Klou!
> > > >
> > > > Best,
> > > > Zhijiang
> > > > --
> > > > From:Zhu Zhu 
> > > > Send Time:2019年9月6日(星期五) 17:19
> > > > To:dev 
> > > > Subject:Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC
> > > >
> > > > Congratulations Kostas!
> > > >
> > > > Thanks,
> > > > Zhu Zhu
> > > >
> > > > Yu Li  于2019年9月6日周五 下午10:49写道：
> > > >
> > > > > Congratulations Klou!
> > > > >
> > > > > Best Regards,
> > > > > Yu
> > > > >
> > > > >
> > > > > On Fri, 6 Sep 2019 at 22:43, Forward Xu 
> > > wrote:
> > > > >
> > > > > > Congratulations Kloudas!
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Forward
> > > > > >
> > > > > > Dawid Wysakowicz  于2019年9月6日周五
> 下午10:36写道：
> > > > > >
> > > > > > > Congratulations Klou!
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Dawid
> > > > > > >
> > > > > > > On 06/09/2019 14:55, Fabian Hueske wrote:
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I'm very happy to announce that Kostas Kloudas is joining the
> > > Flink
> > > > > > PMC.
> > > > > > > > Kostas is contributing to Flink for many years and puts lots
> of
> > > > > effort
> > > > > > in
> > > > > > > > helping our users and growing the Flink community.
> > > > > > > >
> > > > > > > > Please join me in congratulating Kostas!
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Fabian
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

[ying]

Congratulations Kostas!

On Fri, Sep 6, 2019 at 11:21 PM Gary Yao  wrote:

> Congratulations Klou!
>
> On Sat, Sep 7, 2019 at 6:21 AM Thomas Weise  wrote:
>
> > Congratulations!
> >
> >
> > On Fri, Sep 6, 2019 at 9:22 AM zhijiang  > .invalid>
> > wrote:
> >
> > > Congratulations Klou!
> > >
> > > Best,
> > > Zhijiang
> > > --
> > > From:Zhu Zhu 
> > > Send Time:2019年9月6日(星期五) 17:19
> > > To:dev 
> > > Subject:Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC
> > >
> > > Congratulations Kostas!
> > >
> > > Thanks,
> > > Zhu Zhu
> > >
> > > Yu Li  于2019年9月6日周五 下午10:49写道：
> > >
> > > > Congratulations Klou!
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Fri, 6 Sep 2019 at 22:43, Forward Xu 
> > wrote:
> > > >
> > > > > Congratulations Kloudas!
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > Forward
> > > > >
> > > > > Dawid Wysakowicz  于2019年9月6日周五 下午10:36写道：
> > > > >
> > > > > > Congratulations Klou!
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Dawid
> > > > > >
> > > > > > On 06/09/2019 14:55, Fabian Hueske wrote:
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I'm very happy to announce that Kostas Kloudas is joining the
> > Flink
> > > > > PMC.
> > > > > > > Kostas is contributing to Flink for many years and puts lots of
> > > > effort
> > > > > in
> > > > > > > helping our users and growing the Flink community.
> > > > > > >
> > > > > > > Please join me in congratulating Kostas!
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Fabian
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> >
>

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

[Gary Yao]

Congratulations Klou!

On Sat, Sep 7, 2019 at 6:21 AM Thomas Weise  wrote:

> Congratulations!
>
>
> On Fri, Sep 6, 2019 at 9:22 AM zhijiang  .invalid>
> wrote:
>
> > Congratulations Klou!
> >
> > Best,
> > Zhijiang
> > --
> > From:Zhu Zhu 
> > Send Time:2019年9月6日(星期五) 17:19
> > To:dev 
> > Subject:Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC
> >
> > Congratulations Kostas!
> >
> > Thanks,
> > Zhu Zhu
> >
> > Yu Li  于2019年9月6日周五 下午10:49写道：
> >
> > > Congratulations Klou!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Fri, 6 Sep 2019 at 22:43, Forward Xu 
> wrote:
> > >
> > > > Congratulations Kloudas!
> > > >
> > > >
> > > > Best,
> > > >
> > > > Forward
> > > >
> > > > Dawid Wysakowicz  于2019年9月6日周五 下午10:36写道：
> > > >
> > > > > Congratulations Klou!
> > > > >
> > > > > Best,
> > > > >
> > > > > Dawid
> > > > >
> > > > > On 06/09/2019 14:55, Fabian Hueske wrote:
> > > > > > Hi everyone,
> > > > > >
> > > > > > I'm very happy to announce that Kostas Kloudas is joining the
> Flink
> > > > PMC.
> > > > > > Kostas is contributing to Flink for many years and puts lots of
> > > effort
> > > > in
> > > > > > helping our users and growing the Flink community.
> > > > > >
> > > > > > Please join me in congratulating Kostas!
> > > > > >
> > > > > > Cheers,
> > > > > > Fabian
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> >
>