Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

2020-02-23 Thread Dawid Wysakowicz 
Congratulations Jingsong!

Best,

Dawid

On 24/02/2020 08:12, zhenya Sun wrote:
> Congratulations！！！
> | |
> zhenya Sun
> |
> |
> toke...@126.com
> |
> 签名由网易邮箱大师定制
>
>
> On 02/24/2020 14:35，Yu Li wrote：
> Congratulations Jingsong! Well deserved.
>
> Best Regards,
> Yu
>
>
> On Mon, 24 Feb 2020 at 14:10, Congxian Qiu  wrote:
>
> Congratulations Jingsong!
>
> Best,
> Congxian
>
>
> jincheng sun  于2020年2月24日周一 下午1:38写道：
>
> Congratulations Jingsong!
>
> Best,
> Jincheng
>
>
> Zhu Zhu  于2020年2月24日周一 上午11:55写道：
>
> Congratulations Jingsong!
>
> Thanks,
> Zhu Zhu
>
> Fabian Hueske  于2020年2月22日周六 上午1:30写道：
>
> Congrats Jingsong!
>
> Cheers, Fabian
>
> Am Fr., 21. Feb. 2020 um 17:49 Uhr schrieb Rong Rong <
> walter...@gmail.com>:
>
> Congratulations Jingsong!!
>
> Cheers,
> Rong
>
> On Fri, Feb 21, 2020 at 8:45 AM Bowen Li  wrote:
>
> Congrats, Jingsong!
>
> On Fri, Feb 21, 2020 at 7:28 AM Till Rohrmann 
> wrote:
>
> Congratulations Jingsong!
>
> Cheers,
> Till
>
> On Fri, Feb 21, 2020 at 4:03 PM Yun Gao 
> wrote:
>
> Congratulations Jingsong!
>
> Best,
> Yun
>
> --
> From:Jingsong Li 
> Send Time:2020 Feb. 21 (Fri.) 21:42
> To:Hequn Cheng 
> Cc:Yang Wang ; Zhijiang <
> wangzhijiang...@aliyun.com>; Zhenghua Gao ;
> godfrey
> he ; dev ; user <
> u...@flink.apache.org>
> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>
> Thanks everyone~
>
> It's my pleasure to be part of the community. I hope I can make a
> better
> contribution in future.
>
> Best,
> Jingsong Lee
>
> On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng 
> wrote:
> Congratulations Jingsong! Well deserved.
>
> Best,
> Hequn
>
> On Fri, Feb 21, 2020 at 2:42 PM Yang Wang 
> wrote:
> Congratulations！Jingsong. Well deserved.
>
>
> Best,
> Yang
>
> Zhijiang  于2020年2月21日周五 下午1:18写道：
> Congrats Jingsong! Welcome on board!
>
> Best,
> Zhijiang
>
> --
> From:Zhenghua Gao 
> Send Time:2020 Feb. 21 (Fri.) 12:49
> To:godfrey he 
> Cc:dev ; user 
> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>
> Congrats Jingsong！
>
>
> *Best Regards,*
> *Zhenghua Gao*
>
>
> On Fri, Feb 21, 2020 at 11:59 AM godfrey he 
> wrote:
> Congrats Jingsong! Well deserved.
>
> Best,
> godfrey
>
> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
> Congratulations！Jingsong. You deserve it
>
> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
> Congrats Jingsong!
>
> On Fri, 21 Feb 2020 at 11:41, Dian Fu 
> wrote:
>
> Congrats Jingsong!
>
> 在 2020年2月21日，上午11:39，Jark Wu  写道：
>
> Congratulations Jingsong! Well deserved.
>
> Best,
> Jark
>
> On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
>
> Congratulations! Jingsong
>
>
> Best,
> Dan Zou
>
>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
>
> --
> Best, Jingsong Lee
>
>
>
>
>
>



signature.asc
Description: OpenPGP digital signature

Re: [VOTE] FLIP-103: Better TM/JM Log Display

2020-02-23 Thread Zhu Zhu 
+1 (binding)

It can be very helpful to check GC state when GC logging is enabled.

Thanks,
Zhu Zhu

zhenya Sun  于2020年2月24日周一 下午3:51写道：

> 1
>
>
> able-sun
>
>
> On 02/24/2020 15:45，jincheng sun wrote：
> +1(binding) Thank you for driving this Yadong!
>
> Best,
> Jincheng
>
> Yangze Guo  于2020年2月21日周五 下午3:08写道：
>
> +1 (non-binding)
>
> It will make debugging more convenient and hence improve usability.
>
> Best,
> Yangze Guo
>
> On Fri, Feb 21, 2020 at 2:50 PM Yang Wang  wrote:
>
> Hi Jark,
>
> The GC log is not enabled by default. If the users want, he/she should
> configure it by the following config option.
> env.java.opts.jobmanager: -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> -Xloggc:/opt/flink/log/jobmanager-gc.log -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=512M
>
> Best,
> Yang
>
> Jark Wu  于2020年2月21日周五 下午2:19写道：
>
> +1 (binding)
>
> A minor question: I didn't find the GC log in the POC web. Is it only
> generated when some configuration is enabled?
>
> Best,
> Jark
>
> On Fri, 21 Feb 2020 at 11:40, Kurt Young  wrote:
>
> +1 (binding)
>
>
> On Fri, Feb 21, 2020 at 11:25 AM lining jing 
> wrote:
>
> +1 (non-binding)
> It lists all log files, the user could see the GC log.
>
> Xintong Song  于2020年2月21日周五 上午10:44写道：
>
> +1 (non-binding)
>
> I like the ideas of having a list of all log files, and make them
> downloadable.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Feb 21, 2020 at 10:24 AM Yang Wang <
> danrtsey...@gmail.com>
> wrote:
>
> +1 non-binding
>
> The log files list are shown in the new design. I like the
> idea.
> Since
> the
> user defined logs(e.g. my-log.file)
> could be found here now.
>
>
> Best,
> Yang
>
> Yadong Xie  于2020年2月20日周四 下午5:56写道：
>
> Hi all
>
> I want to start the vote for FLIP-103, which proposes to
> provide
> better
> user log display UI in task manager and job manager.
>
> To help everyone better understand the proposal, we spent
> some
> efforts
> on
> making an online POC
>
> Task manager:
>
> previous web:
>
>
>
>
>
>
>
>
> http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/logs
> POC web:
>
>
>
>
>
>
>
>
> http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/log
>
> Job manager:
>
> previous web: http://101.132.122.69:8081/#/job-manager/logs
> POC web: http://101.132.122.69:8081/web/#/job-manager/log
>
>
> The vote will last for at least 72 hours, following the
> consensus
> voting
> process.
>
> FLIP wiki:
>
>
>
>
>
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
>
> Discussion thread:
>
>
>
>
>
>
>
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>
>
>
>
>
>
>
>

Re: [VOTE] FLIP-103: Better TM/JM Log Display

2020-02-23 Thread zhenya Sun 
1


able-sun


On 02/24/2020 15:45，jincheng sun wrote：
+1(binding) Thank you for driving this Yadong!

Best,
Jincheng

Yangze Guo  于2020年2月21日周五 下午3:08写道：

+1 (non-binding)

It will make debugging more convenient and hence improve usability.

Best,
Yangze Guo

On Fri, Feb 21, 2020 at 2:50 PM Yang Wang  wrote:

Hi Jark,

The GC log is not enabled by default. If the users want, he/she should
configure it by the following config option.
env.java.opts.jobmanager: -XX:+PrintGCDetails -XX:+PrintGCDateStamps
-Xloggc:/opt/flink/log/jobmanager-gc.log -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=512M

Best,
Yang

Jark Wu  于2020年2月21日周五 下午2:19写道：

+1 (binding)

A minor question: I didn't find the GC log in the POC web. Is it only
generated when some configuration is enabled?

Best,
Jark

On Fri, 21 Feb 2020 at 11:40, Kurt Young  wrote:

+1 (binding)


On Fri, Feb 21, 2020 at 11:25 AM lining jing 
wrote:

+1 (non-binding)
It lists all log files, the user could see the GC log.

Xintong Song  于2020年2月21日周五 上午10:44写道：

+1 (non-binding)

I like the ideas of having a list of all log files, and make them
downloadable.

Thank you~

Xintong Song



On Fri, Feb 21, 2020 at 10:24 AM Yang Wang <
danrtsey...@gmail.com>
wrote:

+1 non-binding

The log files list are shown in the new design. I like the
idea.
Since
the
user defined logs(e.g. my-log.file)
could be found here now.


Best,
Yang

Yadong Xie  于2020年2月20日周四 下午5:56写道：

Hi all

I want to start the vote for FLIP-103, which proposes to
provide
better
user log display UI in task manager and job manager.

To help everyone better understand the proposal, we spent
some
efforts
on
making an online POC

Task manager:

previous web:







http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/logs
POC web:







http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/log

Job manager:

previous web: http://101.132.122.69:8081/#/job-manager/logs
POC web: http://101.132.122.69:8081/web/#/job-manager/log


The vote will last for at least 72 hours, following the
consensus
voting
process.

FLIP wiki:






https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143

Discussion thread:







http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html

Thanks,

Yadong

Re: [VOTE] FLIP-103: Better TM/JM Log Display

2020-02-23 Thread jincheng sun 
+1(binding) Thank you for driving this Yadong!

Best,
Jincheng

Yangze Guo  于2020年2月21日周五 下午3:08写道：

> +1 (non-binding)
>
> It will make debugging more convenient and hence improve usability.
>
> Best,
> Yangze Guo
>
> On Fri, Feb 21, 2020 at 2:50 PM Yang Wang  wrote:
> >
> > Hi Jark,
> >
> > The GC log is not enabled by default. If the users want, he/she should
> > configure it by the following config option.
> > env.java.opts.jobmanager: -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > -Xloggc:/opt/flink/log/jobmanager-gc.log -XX:+UseGCLogFileRotation
> > -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=512M
> >
> > Best,
> > Yang
> >
> > Jark Wu  于2020年2月21日周五 下午2:19写道：
> >
> > > +1 (binding)
> > >
> > > A minor question: I didn't find the GC log in the POC web. Is it only
> > > generated when some configuration is enabled?
> > >
> > > Best,
> > > Jark
> > >
> > > On Fri, 21 Feb 2020 at 11:40, Kurt Young  wrote:
> > >
> > > > +1 (binding)
> > > >
> > > >
> > > > On Fri, Feb 21, 2020 at 11:25 AM lining jing 
> > > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > > It lists all log files, the user could see the GC log.
> > > > >
> > > > > Xintong Song  于2020年2月21日周五 上午10:44写道：
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > I like the ideas of having a list of all log files, and make them
> > > > > > downloadable.
> > > > > >
> > > > > > Thank you~
> > > > > >
> > > > > > Xintong Song
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 21, 2020 at 10:24 AM Yang Wang <
> danrtsey...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > +1 non-binding
> > > > > > >
> > > > > > > The log files list are shown in the new design. I like the
> idea.
> > > > Since
> > > > > > the
> > > > > > > user defined logs(e.g. my-log.file)
> > > > > > > could be found here now.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Yang
> > > > > > >
> > > > > > > Yadong Xie  于2020年2月20日周四 下午5:56写道：
> > > > > > >
> > > > > > > > Hi all
> > > > > > > >
> > > > > > > > I want to start the vote for FLIP-103, which proposes to
> provide
> > > > > better
> > > > > > > > user log display UI in task manager and job manager.
> > > > > > > >
> > > > > > > > To help everyone better understand the proposal, we spent
> some
> > > > > efforts
> > > > > > on
> > > > > > > > making an online POC
> > > > > > > >
> > > > > > > > Task manager:
> > > > > > > >
> > > > > > > > previous web:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> http://101.132.122.69:8081/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/logs
> > > > > > > > POC web:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> http://101.132.122.69:8081/web/#/task-manager/6df6c5f37b2bff125dbc3a7388128559/log
> > > > > > > >
> > > > > > > > Job manager:
> > > > > > > >
> > > > > > > > previous web: http://101.132.122.69:8081/#/job-manager/logs
> > > > > > > > POC web: http://101.132.122.69:8081/web/#/job-manager/log
> > > > > > > >
> > > > > > > >
> > > > > > > > The vote will last for at least 72 hours, following the
> consensus
> > > > > > voting
> > > > > > > > process.
> > > > > > > >
> > > > > > > > FLIP wiki:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143
> > > > > > > >
> > > > > > > > Discussion thread:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Yadong
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [VOTE] FLIP-104: Add More Metrics to Jobmanager

2020-02-23 Thread Yadong Xie 
Hi Jark
thanks for your suggestion

> I think we can use different color to distinguish the memory usage (from
green to red?).

It is a good idea, but what is the boundary between red and green? giving a
magic number boundary may mislead the users. any suggestions?

> Besides, I think we should add an unit on the "Garbage Collection" ->
"Time", it's hard to know what the value mean. Would be better to display
the value like "10ms", "5ns".

I will add the unit later, thanks for your advice.


Xintong Song  于2020年2月21日周五 下午6:02写道：

> FYI, there's an effort planned for 1.11 to improve the memory configuration
> of the Flink master process, similar to FLIP-49 but definitely less
> complexity.
>
> I would not consider the memory configuration improvement as a blocker for
> this effort. As far as I can see, there's nothing in conflict. Just after
> the memory configuration improvement, we might be able to present more
> information on the JM metrics page, which are tightly corresponding to the
> configuration options, like what we planned for the TM metrics page in
> FLIP-102. Therefore, it might make sense to proceed this FLIP afterwards.
>
> I'm neutral on this, and would leave the call to Yandong and Lining.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, Feb 21, 2020 at 2:47 PM Jark Wu  wrote:
>
> > Thanks Yadong,
> >
> > I think we can use different color to distinguish the memory usage (from
> > green to red?).
> > Besides, I think we should add an unit on the "Garbage Collection" ->
> > "Time", it's hard to know what the value mean.
> > Would be better to display the value like "10ms", "5ns".
> >
> > Best,
> > Jark
> >
> > On Thu, 20 Feb 2020 at 17:58, Yadong Xie  wrote:
> >
> > > Hi all
> > >
> > > I want to start the vote for FLIP-104, which proposes to add more
> metrics
> > > to job manager.
> > >
> > > To help everyone better understand the proposal, we spent some efforts
> on
> > > making an online POC
> > >
> > > previous web: http://101.132.122.69:8081/#/job-manager/config
> > > POC web: http://101.132.122.69:8081/web/#/job-manager/metrics
> > >
> > >
> > > The vote will last for at least 72 hours, following the consensus
> voting
> > > process.
> > >
> > > FLIP wiki:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-104%3A+Add+More+Metrics+to+Jobmanager
> > >
> > > Discussion thread:
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > >
> > > Thanks,
> > >
> > > Yadong
> > >
> >
>

[jira] [Created] (FLINK-16254) Support -p/--parallelism option for StatefulFunctionsClusterEntryPoint

2020-02-23 Thread Tzu-Li (Gordon) Tai (Jira) 
Tzu-Li (Gordon) Tai created FLINK-16254:
---

 Summary: Support -p/--parallelism option for 
StatefulFunctionsClusterEntryPoint
 Key: FLINK-16254
 URL: https://issues.apache.org/jira/browse/FLINK-16254
 Project: Flink
  Issue Type: New Feature
  Components: Stateful Functions
Affects Versions: statefun-1.1
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai


Currently the only way for users to specify parallelism > 1 for Stateful 
Functions applications is to provide a value for {{parallelism.default}} via 
{{flink-conf.yaml}}.

That is not so nice to use, as users would essentially need to rebuild the 
Stateful Functions application image just to change the parallelism.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

2020-02-23 Thread zhenya Sun 
Congratulations！！！
| |
zhenya Sun
|
|
toke...@126.com
|
签名由网易邮箱大师定制


On 02/24/2020 14:35，Yu Li wrote：
Congratulations Jingsong! Well deserved.

Best Regards,
Yu


On Mon, 24 Feb 2020 at 14:10, Congxian Qiu  wrote:

Congratulations Jingsong!

Best,
Congxian


jincheng sun  于2020年2月24日周一 下午1:38写道：

Congratulations Jingsong!

Best,
Jincheng


Zhu Zhu  于2020年2月24日周一 上午11:55写道：

Congratulations Jingsong!

Thanks,
Zhu Zhu

Fabian Hueske  于2020年2月22日周六 上午1:30写道：

Congrats Jingsong!

Cheers, Fabian

Am Fr., 21. Feb. 2020 um 17:49 Uhr schrieb Rong Rong <
walter...@gmail.com>:

Congratulations Jingsong!!

Cheers,
Rong

On Fri, Feb 21, 2020 at 8:45 AM Bowen Li  wrote:

Congrats, Jingsong!

On Fri, Feb 21, 2020 at 7:28 AM Till Rohrmann 
wrote:

Congratulations Jingsong!

Best,
Yun

--
From:Jingsong Li 
Send Time:2020 Feb. 21 (Fri.) 21:42
To:Hequn Cheng 
Cc:Yang Wang ; Zhijiang <
wangzhijiang...@aliyun.com>; Zhenghua Gao ;
godfrey
he ; dev ; user <
u...@flink.apache.org>
Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Thanks everyone~

It's my pleasure to be part of the community. I hope I can make a
better
contribution in future.

Best,
Jingsong Lee

On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng 
wrote:
Congratulations Jingsong! Well deserved.

Best,
Hequn

On Fri, Feb 21, 2020 at 2:42 PM Yang Wang 
wrote:
Congratulations！Jingsong. Well deserved.


Best,
Yang

Zhijiang  于2020年2月21日周五 下午1:18写道：
Congrats Jingsong! Welcome on board!

Best,
Zhijiang

--
From:Zhenghua Gao 
Send Time:2020 Feb. 21 (Fri.) 12:49
To:godfrey he 
Cc:dev ; user 
Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congrats Jingsong！


*Best Regards,*
*Zhenghua Gao*


On Fri, Feb 21, 2020 at 11:59 AM godfrey he 
wrote:
Congrats Jingsong! Well deserved.

Best,
godfrey

Jeff Zhang  于2020年2月21日周五 上午11:49写道：
Congratulations！Jingsong. You deserve it

wenlong.lwl  于2020年2月21日周五 上午11:43写道：
Congrats Jingsong!

On Fri, 21 Feb 2020 at 11:41, Dian Fu 
wrote:

Congrats Jingsong!

在 2020年2月21日，上午11:39，Jark Wu  写道：

Congratulations Jingsong! Well deserved.

Best,
Jark

On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:

Congratulations! Jingsong


Best,
Dan Zou





--
Best Regards

Jeff Zhang



--
Best, Jingsong Lee

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

2020-02-23 Thread Yu Li 
Congratulations Jingsong! Well deserved.

Best Regards,
Yu


On Mon, 24 Feb 2020 at 14:10, Congxian Qiu  wrote:

> Congratulations Jingsong!
>
> Best,
> Congxian
>
>
> jincheng sun  于2020年2月24日周一 下午1:38写道：
>
>> Congratulations Jingsong!
>>
>> Best,
>> Jincheng
>>
>>
>> Zhu Zhu  于2020年2月24日周一 上午11:55写道：
>>
>>> Congratulations Jingsong!
>>>
>>> Thanks,
>>> Zhu Zhu
>>>
>>> Fabian Hueske  于2020年2月22日周六 上午1:30写道：
>>>
 Congrats Jingsong!

 Cheers, Fabian

 Am Fr., 21. Feb. 2020 um 17:49 Uhr schrieb Rong Rong <
 walter...@gmail.com>:

 > Congratulations Jingsong!!
 >
 > Cheers,
 > Rong
 >
 > On Fri, Feb 21, 2020 at 8:45 AM Bowen Li  wrote:
 >
 > > Congrats, Jingsong!
 > >
 > > On Fri, Feb 21, 2020 at 7:28 AM Till Rohrmann >>> >
 > > wrote:
 > >
 > >> Congratulations Jingsong!
 > >>
 > >> Cheers,
 > >> Till
 > >>
 > >> On Fri, Feb 21, 2020 at 4:03 PM Yun Gao 
 wrote:
 > >>
 > >>>   Congratulations Jingsong!
 > >>>
 > >>>Best,
 > >>>Yun
 > >>>
 > >>> --
 > >>> From:Jingsong Li 
 > >>> Send Time:2020 Feb. 21 (Fri.) 21:42
 > >>> To:Hequn Cheng 
 > >>> Cc:Yang Wang ; Zhijiang <
 > >>> wangzhijiang...@aliyun.com>; Zhenghua Gao ;
 godfrey
 > >>> he ; dev ; user <
 > >>> u...@flink.apache.org>
 > >>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
 > >>>
 > >>> Thanks everyone~
 > >>>
 > >>> It's my pleasure to be part of the community. I hope I can make a
 > better
 > >>> contribution in future.
 > >>>
 > >>> Best,
 > >>> Jingsong Lee
 > >>>
 > >>> On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng 
 wrote:
 > >>> Congratulations Jingsong! Well deserved.
 > >>>
 > >>> Best,
 > >>> Hequn
 > >>>
 > >>> On Fri, Feb 21, 2020 at 2:42 PM Yang Wang 
 > wrote:
 > >>> Congratulations！Jingsong. Well deserved.
 > >>>
 > >>>
 > >>> Best,
 > >>> Yang
 > >>>
 > >>> Zhijiang  于2020年2月21日周五 下午1:18写道：
 > >>> Congrats Jingsong! Welcome on board!
 > >>>
 > >>> Best,
 > >>> Zhijiang
 > >>>
 > >>> --
 > >>> From:Zhenghua Gao 
 > >>> Send Time:2020 Feb. 21 (Fri.) 12:49
 > >>> To:godfrey he 
 > >>> Cc:dev ; user 
 > >>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
 > >>>
 > >>> Congrats Jingsong！
 > >>>
 > >>>
 > >>> *Best Regards,*
 > >>> *Zhenghua Gao*
 > >>>
 > >>>
 > >>> On Fri, Feb 21, 2020 at 11:59 AM godfrey he 
 > wrote:
 > >>> Congrats Jingsong! Well deserved.
 > >>>
 > >>> Best,
 > >>> godfrey
 > >>>
 > >>> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
 > >>> Congratulations！Jingsong. You deserve it
 > >>>
 > >>> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
 > >>> Congrats Jingsong!
 > >>>
 > >>> On Fri, 21 Feb 2020 at 11:41, Dian Fu 
 wrote:
 > >>>
 > >>> > Congrats Jingsong!
 > >>> >
 > >>> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
 > >>> > >
 > >>> > > Congratulations Jingsong! Well deserved.
 > >>> > >
 > >>> > > Best,
 > >>> > > Jark
 > >>> > >
 > >>> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
 > >>> > >
 > >>> > >> Congratulations! Jingsong
 > >>> > >>
 > >>> > >>
 > >>> > >> Best,
 > >>> > >> Dan Zou
 > >>> > >>
 > >>> >
 > >>> >
 > >>>
 > >>>
 > >>> --
 > >>> Best Regards
 > >>>
 > >>> Jeff Zhang
 > >>>
 > >>>
 > >>>
 > >>> --
 > >>> Best, Jingsong Lee
 > >>>
 > >>>
 > >>>
 >

>>>

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

2020-02-23 Thread Congxian Qiu 
Congratulations Jingsong!

Best,
Congxian


jincheng sun  于2020年2月24日周一 下午1:38写道：

> Congratulations Jingsong!
>
> Best,
> Jincheng
>
>
> Zhu Zhu  于2020年2月24日周一 上午11:55写道：
>
>> Congratulations Jingsong!
>>
>> Thanks,
>> Zhu Zhu
>>
>> Fabian Hueske  于2020年2月22日周六 上午1:30写道：
>>
>>> Congrats Jingsong!
>>>
>>> Cheers, Fabian
>>>
>>> Am Fr., 21. Feb. 2020 um 17:49 Uhr schrieb Rong Rong <
>>> walter...@gmail.com>:
>>>
>>> > Congratulations Jingsong!!
>>> >
>>> > Cheers,
>>> > Rong
>>> >
>>> > On Fri, Feb 21, 2020 at 8:45 AM Bowen Li  wrote:
>>> >
>>> > > Congrats, Jingsong!
>>> > >
>>> > > On Fri, Feb 21, 2020 at 7:28 AM Till Rohrmann 
>>> > > wrote:
>>> > >
>>> > >> Congratulations Jingsong!
>>> > >>
>>> > >> Cheers,
>>> > >> Till
>>> > >>
>>> > >> On Fri, Feb 21, 2020 at 4:03 PM Yun Gao 
>>> wrote:
>>> > >>
>>> > >>>   Congratulations Jingsong!
>>> > >>>
>>> > >>>Best,
>>> > >>>Yun
>>> > >>>
>>> > >>> --
>>> > >>> From:Jingsong Li 
>>> > >>> Send Time:2020 Feb. 21 (Fri.) 21:42
>>> > >>> To:Hequn Cheng 
>>> > >>> Cc:Yang Wang ; Zhijiang <
>>> > >>> wangzhijiang...@aliyun.com>; Zhenghua Gao ;
>>> godfrey
>>> > >>> he ; dev ; user <
>>> > >>> u...@flink.apache.org>
>>> > >>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>>> > >>>
>>> > >>> Thanks everyone~
>>> > >>>
>>> > >>> It's my pleasure to be part of the community. I hope I can make a
>>> > better
>>> > >>> contribution in future.
>>> > >>>
>>> > >>> Best,
>>> > >>> Jingsong Lee
>>> > >>>
>>> > >>> On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng 
>>> wrote:
>>> > >>> Congratulations Jingsong! Well deserved.
>>> > >>>
>>> > >>> Best,
>>> > >>> Hequn
>>> > >>>
>>> > >>> On Fri, Feb 21, 2020 at 2:42 PM Yang Wang 
>>> > wrote:
>>> > >>> Congratulations！Jingsong. Well deserved.
>>> > >>>
>>> > >>>
>>> > >>> Best,
>>> > >>> Yang
>>> > >>>
>>> > >>> Zhijiang  于2020年2月21日周五 下午1:18写道：
>>> > >>> Congrats Jingsong! Welcome on board!
>>> > >>>
>>> > >>> Best,
>>> > >>> Zhijiang
>>> > >>>
>>> > >>> --
>>> > >>> From:Zhenghua Gao 
>>> > >>> Send Time:2020 Feb. 21 (Fri.) 12:49
>>> > >>> To:godfrey he 
>>> > >>> Cc:dev ; user 
>>> > >>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>>> > >>>
>>> > >>> Congrats Jingsong！
>>> > >>>
>>> > >>>
>>> > >>> *Best Regards,*
>>> > >>> *Zhenghua Gao*
>>> > >>>
>>> > >>>
>>> > >>> On Fri, Feb 21, 2020 at 11:59 AM godfrey he 
>>> > wrote:
>>> > >>> Congrats Jingsong! Well deserved.
>>> > >>>
>>> > >>> Best,
>>> > >>> godfrey
>>> > >>>
>>> > >>> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
>>> > >>> Congratulations！Jingsong. You deserve it
>>> > >>>
>>> > >>> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
>>> > >>> Congrats Jingsong!
>>> > >>>
>>> > >>> On Fri, 21 Feb 2020 at 11:41, Dian Fu 
>>> wrote:
>>> > >>>
>>> > >>> > Congrats Jingsong!
>>> > >>> >
>>> > >>> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
>>> > >>> > >
>>> > >>> > > Congratulations Jingsong! Well deserved.
>>> > >>> > >
>>> > >>> > > Best,
>>> > >>> > > Jark
>>> > >>> > >
>>> > >>> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
>>> > >>> > >
>>> > >>> > >> Congratulations! Jingsong
>>> > >>> > >>
>>> > >>> > >>
>>> > >>> > >> Best,
>>> > >>> > >> Dan Zou
>>> > >>> > >>
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> > >>>
>>> > >>> --
>>> > >>> Best Regards
>>> > >>>
>>> > >>> Jeff Zhang
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>> --
>>> > >>> Best, Jingsong Lee
>>> > >>>
>>> > >>>
>>> > >>>
>>> >
>>>
>>

Re: [Discuss] Update the pull request description template.

2020-02-23 Thread tison 
Thanks for kicking off the discussion and opinions so far.

I'd like to share two of my coins.

1. Often I check the item list when review a non-trivial pull request and I
think it
works for some specific topics, such as document coverage, general
description
and test coverage description.

2. We always have a checklist which attach later by flink-bot that has some
overlaps
of the item list in pull request template. It would be a good topic how we
integrate
these two list. Maybe we don't just merge them but properly reorder items
so that
attentions can be applied gradually(for example, most of pull request is
unrelated to
per record codepath, but for those are, it is important to emphasize).

Best,
tison.


Zhijiang  于2020年2月21日周五 下午11:52写道：

> Thanks for launching this discussion and all the involved feedbacks!
>
> Since there are still many users relying on the template to raise
> attentions and guide review, it sounds reasonable to update the template
> based on demands.
>
> In my experience, I was always filling the template when submitting PRs
> before. But I found a bit trouble for filling the last two sections
>  "Does this pull request potentially affect one of the following parts:"
> and "Documentation", because I had to either remove one from "yes|no" or
> highlight one
> manually for every listed item. And I guess for most of PRs, the results
> of these items should be "no" by default.
>
> If we can refactor to another description here, E.g. "Selecting the
> following parts which this pull request potentially affects", and further
> make every item selectable instead.
> Then most of the users do not need to touch these sections by default ,
> which means without implication.  I guess it would save some efforts.
>
> My above concern is tiny and might not be the key motivation of this
> discussion. Just share my thoughts by this chance. :)
>
> Best,
> Zhijiang
>
>
> --
> From:Yangze Guo 
> Send Time:2020 Feb. 21 (Fri.) 16:16
> To:dev 
> Subject:Re: [Discuss] Update the pull request description template.
>
> In my experience, the template is helpful. Especially for the people
> just joined the community and give their first PR. I don't know how
> many people have read the contributor guide entirely before they
> commit their first PR, but I should admit that I did not read it word
> by word for the first time, since not all of the items related to my
> work. However, the template forces me to check the basic rules and
> guidelines of the community.
> Another benefit I can think of is to remind people who touch the code
> path they aren't familiar with. If that needs a special test flow, the
> template forces them to follow it.
>
> Best,
> Yangze Guo
>
> On Fri, Feb 21, 2020 at 2:39 PM Yang Wang  wrote:
> >
> > I second xintong's suggestion. When i open a PR, i also check the item
> list
> > in the template. It help to
> > know whether i should test the PR in a real cluster(Yarn/K8s/Mesos). Or i
> > should be more careful
> > when touching the per-record code paths.If we have some dependencies
> > changes, i will need to check
> > the generated jar as expected.
> >
> >
> > Best,
> > Yang
> >
> > Xintong Song  于2020年2月20日周四 上午10:33写道：
> >
> > > Thanks for the feedbacks, Chesnay and Till. And thanks for the pointer,
> > > Congxian.
> > >
> > > I don't know how often committers and reviewers checks and benefits
> from
> > > the PR description. From your feedbacks and the number of responses to
> this
> > > discussion, it's probably not often.
> > >
> > > However, as a contributor and speaking only for myself, I actually
> find the
> > > PR template very helpful. I use it as a checking list for opening a PR.
> > > Filling in the template forces me to revisit the important things,
> e.g.,
> > > have I added enough test cases to cover the all the important changes,
> does
> > > this change need to be validated with a real deployment (if it touches
> the
> > > deployment and recovery). An experienced developer might be able to
> check
> > > these things without such a checking list, but there might be more
> primary
> > > developers that can benefit from it.
> > >
> > >
> > > Therefore, if we agree that PR template is less useful for reviewers, I
> > > would like to propose to reposition it as a contributor checking list.
> The
> > > following are some examples of how the existing items might be
> > > repositioned.
> > >
> > >
> > > - The runtime per-record code paths (performance sensitive): (yes / no
> /
> > > don't know). If yes, please check the following items.
> > >
> > > - Is there a good reason to do that?
> > > - Is there an alternative non pre-record approach?
> > >
> > > - Is Java stream or Optional used in the per-recode code path? (Those
> > > should be avoid according to the code style and quality guide[1])
> > >
> > > - Do we know the exact impact on performance? (Maybe point to the
> > > performance benchmarks)
> > >
> > >
> >

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

2020-02-23 Thread jincheng sun 
Congratulations Jingsong!

Best,
Jincheng


Zhu Zhu  于2020年2月24日周一 上午11:55写道：

> Congratulations Jingsong!
>
> Thanks,
> Zhu Zhu
>
> Fabian Hueske  于2020年2月22日周六 上午1:30写道：
>
>> Congrats Jingsong!
>>
>> Cheers, Fabian
>>
>> Am Fr., 21. Feb. 2020 um 17:49 Uhr schrieb Rong Rong > >:
>>
>> > Congratulations Jingsong!!
>> >
>> > Cheers,
>> > Rong
>> >
>> > On Fri, Feb 21, 2020 at 8:45 AM Bowen Li  wrote:
>> >
>> > > Congrats, Jingsong!
>> > >
>> > > On Fri, Feb 21, 2020 at 7:28 AM Till Rohrmann 
>> > > wrote:
>> > >
>> > >> Congratulations Jingsong!
>> > >>
>> > >> Cheers,
>> > >> Till
>> > >>
>> > >> On Fri, Feb 21, 2020 at 4:03 PM Yun Gao 
>> wrote:
>> > >>
>> > >>>   Congratulations Jingsong!
>> > >>>
>> > >>>Best,
>> > >>>Yun
>> > >>>
>> > >>> --
>> > >>> From:Jingsong Li 
>> > >>> Send Time:2020 Feb. 21 (Fri.) 21:42
>> > >>> To:Hequn Cheng 
>> > >>> Cc:Yang Wang ; Zhijiang <
>> > >>> wangzhijiang...@aliyun.com>; Zhenghua Gao ;
>> godfrey
>> > >>> he ; dev ; user <
>> > >>> u...@flink.apache.org>
>> > >>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>> > >>>
>> > >>> Thanks everyone~
>> > >>>
>> > >>> It's my pleasure to be part of the community. I hope I can make a
>> > better
>> > >>> contribution in future.
>> > >>>
>> > >>> Best,
>> > >>> Jingsong Lee
>> > >>>
>> > >>> On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng 
>> wrote:
>> > >>> Congratulations Jingsong! Well deserved.
>> > >>>
>> > >>> Best,
>> > >>> Hequn
>> > >>>
>> > >>> On Fri, Feb 21, 2020 at 2:42 PM Yang Wang 
>> > wrote:
>> > >>> Congratulations！Jingsong. Well deserved.
>> > >>>
>> > >>>
>> > >>> Best,
>> > >>> Yang
>> > >>>
>> > >>> Zhijiang  于2020年2月21日周五 下午1:18写道：
>> > >>> Congrats Jingsong! Welcome on board!
>> > >>>
>> > >>> Best,
>> > >>> Zhijiang
>> > >>>
>> > >>> --
>> > >>> From:Zhenghua Gao 
>> > >>> Send Time:2020 Feb. 21 (Fri.) 12:49
>> > >>> To:godfrey he 
>> > >>> Cc:dev ; user 
>> > >>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>> > >>>
>> > >>> Congrats Jingsong！
>> > >>>
>> > >>>
>> > >>> *Best Regards,*
>> > >>> *Zhenghua Gao*
>> > >>>
>> > >>>
>> > >>> On Fri, Feb 21, 2020 at 11:59 AM godfrey he 
>> > wrote:
>> > >>> Congrats Jingsong! Well deserved.
>> > >>>
>> > >>> Best,
>> > >>> godfrey
>> > >>>
>> > >>> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
>> > >>> Congratulations！Jingsong. You deserve it
>> > >>>
>> > >>> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
>> > >>> Congrats Jingsong!
>> > >>>
>> > >>> On Fri, 21 Feb 2020 at 11:41, Dian Fu 
>> wrote:
>> > >>>
>> > >>> > Congrats Jingsong!
>> > >>> >
>> > >>> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
>> > >>> > >
>> > >>> > > Congratulations Jingsong! Well deserved.
>> > >>> > >
>> > >>> > > Best,
>> > >>> > > Jark
>> > >>> > >
>> > >>> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
>> > >>> > >
>> > >>> > >> Congratulations! Jingsong
>> > >>> > >>
>> > >>> > >>
>> > >>> > >> Best,
>> > >>> > >> Dan Zou
>> > >>> > >>
>> > >>> >
>> > >>> >
>> > >>>
>> > >>>
>> > >>> --
>> > >>> Best Regards
>> > >>>
>> > >>> Jeff Zhang
>> > >>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> Best, Jingsong Lee
>> > >>>
>> > >>>
>> > >>>
>> >
>>
>

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

2020-02-23 Thread Zhu Zhu 
Congratulations Jingsong!

Thanks,
Zhu Zhu

Fabian Hueske  于2020年2月22日周六 上午1:30写道：

> Congrats Jingsong!
>
> Cheers, Fabian
>
> Am Fr., 21. Feb. 2020 um 17:49 Uhr schrieb Rong Rong  >:
>
> > Congratulations Jingsong!!
> >
> > Cheers,
> > Rong
> >
> > On Fri, Feb 21, 2020 at 8:45 AM Bowen Li  wrote:
> >
> > > Congrats, Jingsong!
> > >
> > > On Fri, Feb 21, 2020 at 7:28 AM Till Rohrmann 
> > > wrote:
> > >
> > >> Congratulations Jingsong!
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Fri, Feb 21, 2020 at 4:03 PM Yun Gao  wrote:
> > >>
> > >>>   Congratulations Jingsong!
> > >>>
> > >>>Best,
> > >>>Yun
> > >>>
> > >>> --
> > >>> From:Jingsong Li 
> > >>> Send Time:2020 Feb. 21 (Fri.) 21:42
> > >>> To:Hequn Cheng 
> > >>> Cc:Yang Wang ; Zhijiang <
> > >>> wangzhijiang...@aliyun.com>; Zhenghua Gao ;
> godfrey
> > >>> he ; dev ; user <
> > >>> u...@flink.apache.org>
> > >>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
> > >>>
> > >>> Thanks everyone~
> > >>>
> > >>> It's my pleasure to be part of the community. I hope I can make a
> > better
> > >>> contribution in future.
> > >>>
> > >>> Best,
> > >>> Jingsong Lee
> > >>>
> > >>> On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng 
> wrote:
> > >>> Congratulations Jingsong! Well deserved.
> > >>>
> > >>> Best,
> > >>> Hequn
> > >>>
> > >>> On Fri, Feb 21, 2020 at 2:42 PM Yang Wang 
> > wrote:
> > >>> Congratulations！Jingsong. Well deserved.
> > >>>
> > >>>
> > >>> Best,
> > >>> Yang
> > >>>
> > >>> Zhijiang  于2020年2月21日周五 下午1:18写道：
> > >>> Congrats Jingsong! Welcome on board!
> > >>>
> > >>> Best,
> > >>> Zhijiang
> > >>>
> > >>> --
> > >>> From:Zhenghua Gao 
> > >>> Send Time:2020 Feb. 21 (Fri.) 12:49
> > >>> To:godfrey he 
> > >>> Cc:dev ; user 
> > >>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
> > >>>
> > >>> Congrats Jingsong！
> > >>>
> > >>>
> > >>> *Best Regards,*
> > >>> *Zhenghua Gao*
> > >>>
> > >>>
> > >>> On Fri, Feb 21, 2020 at 11:59 AM godfrey he 
> > wrote:
> > >>> Congrats Jingsong! Well deserved.
> > >>>
> > >>> Best,
> > >>> godfrey
> > >>>
> > >>> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
> > >>> Congratulations！Jingsong. You deserve it
> > >>>
> > >>> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
> > >>> Congrats Jingsong!
> > >>>
> > >>> On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:
> > >>>
> > >>> > Congrats Jingsong!
> > >>> >
> > >>> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
> > >>> > >
> > >>> > > Congratulations Jingsong! Well deserved.
> > >>> > >
> > >>> > > Best,
> > >>> > > Jark
> > >>> > >
> > >>> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
> > >>> > >
> > >>> > >> Congratulations! Jingsong
> > >>> > >>
> > >>> > >>
> > >>> > >> Best,
> > >>> > >> Dan Zou
> > >>> > >>
> > >>> >
> > >>> >
> > >>>
> > >>>
> > >>> --
> > >>> Best Regards
> > >>>
> > >>> Jeff Zhang
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Best, Jingsong Lee
> > >>>
> > >>>
> > >>>
> >
>

[jira] [Created] (FLINK-16253) Switch to Log4j 2 by default for flink-kubernetes submodule

2020-02-23 Thread Canbin Zheng (Jira) 
Canbin Zheng created FLINK-16253:


 Summary: Switch to Log4j 2 by default for flink-kubernetes 
submodule
 Key: FLINK-16253
 URL: https://issues.apache.org/jira/browse/FLINK-16253
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes
Reporter: Canbin Zheng
 Fix For: 1.11.0


Switch to Log4j 2 by default for flink-kubernetes submodule, including the 
script and the container startup command or parameters.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLINK-16194: Refactor the Kubernetes architecture design

2020-02-23 Thread Canbin Zheng 
Hi, Yang Wang,

Thanks for the feedback.

> Parameters Parser

I can think of many benefits of the dedicated parameter parsing/verifying
tools, here is some of them:
1. Reuse the parameters parsing and verifying code.
2. Ensure consistent handling logic for the same setting.
3. Simplify some of the method's signature because we can avoid defining
too many unnecessary input parameters.

> BTW, the fabric8 kubernetes-client and Kubernetes api server already has
the parameters check before starting to create the resources. I think the
exceptions
are usually meaning and enough for the users to get the root cause.

1. On the one hand, there are gaps between the declarative model used by
Kubernetes and the configuration model used by Flink. Indeed, the
Kubernetes client/server helps check the Kubernetes side parameters and
throw exceptions in case of failures; however, the exception messages are
not always easily understood from the perspective of Flink users.
Independent of the solutions, what we should make sure is that the
exception information in the Flink configuration model is meaningful enough
for the Flink users.
2. On the other hand, some prechecking is beneficial in some scenarios
since we can avoid useless effort on Kubernetes resource creation in case
of failures leading to clean up all the created resources.


Regards,
Canbin Zheng

Yang Wang  于2020年2月23日周日 下午10:37写道：

> > The new introduced decorator
> After some offline discussion with Canbin and tison, i totally understand
> the evolved decorator design. Each decorator will be self-contained and
> is responsible for just one thing. Currently, if we want to mount a new
> config
> file to jobmanager/taskmanager, then both `ConfigMapDecorator`,
> `JobManagerDecorator`, `TaskManagerDecorator` needs to be updated.
> It is not convenient for the new contributors to do this. In the new
> design,
> by leveraging the accompanying kubernetes resources in decorator, we
> could finish the creating and mounting config map in one decorator.
>
> Since now we just have a basic implementation for native Kubernetes
> integration and lots of features need to be developed. And many users
> want to participate in and contribute to the integration. So i agree to
> refactor
> the current decorator implementation and make it easier for the new
> contributor.
>
> For the detailed divergences(naming, etc.), i think we could discuss in
> the PR.
>
> > Parameters Parser
> Currently, the decorator directly use Flink configuration to get the
> parameters
> to build the Kubernetes resource. It is straightforward, however we could
> not
> have a unified parameters check. So i am not sure whether you will
> introduce
> a tool to check the parameters or just simply have our own basic check.
>
> BTW, the fabric8 kubernetes-client and Kubernetes api server already has
> the parameters check before starting to create the resources. I think the
> exceptions
> are usually meaning and enough for the users to get the root cause.
>
>
>
> Best,
> Yang
>
>
>
>
> felixzheng zheng  于2020年2月22日周六 上午10:54写道：
>
>> Great thanks for the quick feedback Till. You are right; it is not a
>> fundamentally different approach compared to
>> what we have right now, all the Kubernetes resources created are the same,
>> we aim to evolve the existing decorator approach so that,
>> 1. the decorators are monadic and smaller in size and functionality.
>> 2. the new decorator design allows reusing the decorators between the
>> client and the cluster as much as possible.
>> 3. all the decorators are independent with each other, and they could have
>> arbitrary order in the chain, they share the same APIs and follow a
>> unified
>> orchestrator architecture so that new developers could quickly understand
>> what should be done to introduce a new feature.
>>
>> Besides that, the new approach allows us adding tests for every decorator
>> alone instead of doing a final test of all the decorators in the
>> Fabric8ClientTest.java.
>>
>> Cheers,
>> Canbin Zheng
>>
>> Till Rohrmann  于2020年2月22日周六 上午12:28写道：
>>
>> > Thanks for starting this discussion Canbin. If I understand your
>> proposal
>> > correctly, then you would like to evolve the existing decorator
>> approach so
>> > that decorators are monadic and smaller in size and functionality. The
>> > latter aspect will allow to reuse them between the client and the
>> cluster.
>> > Just to make sure, it is not a fundamentally different approach
>> compared to
>> > what we have right now, is it?
>> >
>> > If this is the case, then I think it makes sense to reuse code as much
>> as
>> > possible and to create small code units which are easier to test.
>> >
>> > Cheers,
>> > Till
>> >
>> > On Fri, Feb 21, 2020 at 4:41 PM felixzheng zheng <
>> felixzhen...@gmail.com>
>> > wrote:
>> >
>> > > Thanks for the feedback @Yang Wang. I would like to discuss some of
>> the
>> > > details in depth about why I am confused about the existing design.
>> > >
>> > > Question 1: How do

Re: [VOTE] FLIP-98: Better Back Pressure Detection

2020-02-23 Thread Yun Gao 
+1 (non-binding)
  This should make tuning back pressure easier, which is one of the most 
common problems met for users. 

Best,
   Yun 


--
From:Benchao Li 
Send Time:2020 Feb. 24 (Mon.) 08:32
To:dev 
Cc:Zhijiang 
Subject:Re: [VOTE] FLIP-98: Better Back Pressure Detection

+1 (non-binding)

It's really helpful!

Yang Wang  于2020年2月21日周五 下午2:44写道：

> +1 non-binding
>
> It is really straightforward.
>
> Best,
> Yang
>
> Jark Wu  于2020年2月21日周五 下午1:52写道：
>
> > +1 (binding)
> >
> > On Fri, 21 Feb 2020 at 11:51, Kurt Young  wrote:
> >
> > > +1 (binding)
> > >
> > >
> > >
> > > On Fri, Feb 21, 2020 at 1:09 AM Zhijiang  > > .invalid>
> > > wrote:
> > >
> > > > +1 (binding).
> > > > It seems more clearly and directly to highlight the back pressured
> > vertex
> > > > in topology, which can raise the attention of users.
> > > >
> > > > Best,
> > > > Zhijiang
> > > >
> > > >
> > > > --
> > > > From:Congxian Qiu 
> > > > Send Time:2020 Feb. 20 (Thu.) 21:49
> > > > To:dev@flink.apache.org 
> > > > Subject:Re: [VOTE] FLIP-98: Better Back Pressure Detection
> > > >
> > > > +1 for this improvement, It's very helpful to know whether a task is
> > > > backpressured or not.
> > > >
> > > > Best,
> > > > Congxian
> > > >
> > > >
> > > > Yadong Xie  于2020年2月20日周四 下午5:34写道：
> > > >
> > > > > Hi all
> > > > >
> > > > > I want to start the vote for FLIP-98, which proposes to provide
> > better
> > > > > backpressure detection experience in web UI.
> > > > >
> > > > > To help everyone better understand the proposal, we spent some
> > efforts
> > > on
> > > > > making an online POC
> > > > >
> > > > > previous web:
> > > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/#/job/add9d3ea1006af15b9f872987eb3befd/overview
> > > > > POC web:
> > > > >
> > > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/add9d3ea1006af15b9f872987eb3befd/overview
> > > > >
> > > > > The vote will last for at least 72 hours, following the consensus
> > > voting
> > > > > process.
> > > > >
> > > > > FLIP wiki:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
> > > > >
> > > > > Discussion thread:
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Yadong
> > > > >
> > > >
> > > >
> > >
> >
>


-- 

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:+86-15650713730
Email: libenc...@gmail.com; libenc...@pku.edu.cn

[jira] [Created] (FLINK-16252) Optimize the progress of the process_outputs in Python UDF

2020-02-23 Thread Huang Xingbo (Jira) 
Huang Xingbo created FLINK-16252:


 Summary: Optimize the progress of the process_outputs in Python UDF
 Key: FLINK-16252
 URL: https://issues.apache.org/jira/browse/FLINK-16252
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Reporter: Huang Xingbo
 Fix For: 1.11.0


We need to optimize the function call chains in process_outputs to improve the 
performance in Python UDF



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-99: Make Max Exception Configurable

2020-02-23 Thread zoudan 
+1 (non-binding)

Best,
Dan Zou

[jira] [Created] (FLINK-16251) Optimize the cost of function call in ScalarFunctionOpertation

2020-02-23 Thread Huang Xingbo (Jira) 
Huang Xingbo created FLINK-16251:


 Summary: Optimize the cost of function call  in 
ScalarFunctionOpertation
 Key: FLINK-16251
 URL: https://issues.apache.org/jira/browse/FLINK-16251
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Reporter: Huang Xingbo
 Fix For: 1.11.0


Currently, there are too many extra function calls cost in  
ScalarFunctionOpertation.We need to optimize it to improve performance in 
Python UDF.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16250) Add interfaces for PipelineStage and Pipeline

2020-02-23 Thread Hequn Cheng (Jira) 
Hequn Cheng created FLINK-16250:
---

 Summary: Add interfaces for PipelineStage and Pipeline
 Key: FLINK-16250
 URL: https://issues.apache.org/jira/browse/FLINK-16250
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng


A pipeline is a linear workflow that chains some PipelineStages, e.g., 
Estimators and Transformers to execute an algorithm. After this issue is 
addressed, Python users can write Python Pipelines.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16249) Add interfaces for Params, ParamInfo and WithParams

2020-02-23 Thread Hequn Cheng (Jira) 
Hequn Cheng created FLINK-16249:
---

 Summary: Add interfaces for Params, ParamInfo and WithParams
 Key: FLINK-16249
 URL: https://issues.apache.org/jira/browse/FLINK-16249
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng


Parameters are widely used in machine learning realm. These classes define 
common interfaces to interact with classes with parameters.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16248) Add interfaces for MLEnvironment and MLEnvironmentFactory

2020-02-23 Thread Hequn Cheng (Jira) 
Hequn Cheng created FLINK-16248:
---

 Summary: Add interfaces for MLEnvironment and MLEnvironmentFactory
 Key: FLINK-16248
 URL: https://issues.apache.org/jira/browse/FLINK-16248
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Reporter: Hequn Cheng
Assignee: Hequn Cheng


Align interface for MLEnvironment and MLEnvironmentFactory, so Python users can 
use Python MLEnvironmentFactory to maintain execution environment and table 
environment.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-100: Add Attempt Information

2020-02-23 Thread Benchao Li 
I agree with Jark.
It's helpful to add a TM logs link for each attempt. Usually there are
multiple TMs per host, so only host name cannot determine the real TM.

Jark Wu  于2020年2月21日周五 下午2:06写道：

> Thanks Yadong,
>
> I think one of the purpose to find the failed attempts is that we want to
> look into the logs in the failed node (node may change across attempts).
> However, from the POC, I didn't find a log link on each attempts. Maybe we
> can add such feature?
>
> Best,
> Jark
>
> On Fri, 21 Feb 2020 at 11:51, Kurt Young  wrote:
>
> > Hi Yadong,
> >
> > Thanks for the proposal, it's a useful feature, especially for batch
> jobs.
> > But according
> > to the examples you gave, I can't tell whether i got required information
> > from that.
> > Can you replace the demo job to a more complex batch job and then we can
> > see some
> > differences of start/stop time of different tasks and attempts?
> >
> > Best,
> > Kurt
> >
> >
> > On Thu, Feb 20, 2020 at 5:46 PM Yadong Xie  wrote:
> >
> > > Hi all
> > >
> > > I want to start the vote for FLIP-100, which proposes to add attempt
> > > information inside subtask and timeline in web UI.
> > >
> > > To help everyone better understand the proposal, we spent some efforts
> on
> > > making an online POC
> > >
> > > Timeline Attempt (click the vertex timeline to see the differences):
> > > previous web:
> > >
> >
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/timeline
> > > POC web:
> > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/timeline
> > >
> > > Subtask Attempt (click the vertex and switch to subtask tab to see the
> > > differences):
> > > previous web:
> > >
> >
> http://101.132.122.69:8081/#/job/9d651769488466d33e7a607e85203543/overview
> > > POC web:
> > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/9d651769488466d33e7a607e85203543/overview
> > >
> > >
> > > The vote will last for at least 72 hours, following the consensus
> voting
> > > process.
> > >
> > > FLIP wiki:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-100%3A+Add+Attempt+Information
> > >
> > > Discussion thread:
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > >
> > > Thanks,
> > >
> > > Yadong
> > >
> >
>


-- 

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:+86-15650713730
Email: libenc...@gmail.com; libenc...@pku.edu.cn

Re: [VOTE] FLIP-99: Make Max Exception Configurable

2020-02-23 Thread Benchao Li 
+1 (non-binding)

Jark Wu  于2020年2月21日周五 下午1:57写道：

> +1 (binding)
>
> Best,
> Jark
>
> On Thu, 20 Feb 2020 at 23:17, Congxian Qiu  wrote:
>
> > +1 to this feature.  this is helpful when troubleshooting.
> >
> > Best,
> > Congxian
> >
> >
> > Yadong Xie  于2020年2月20日周四 下午5:39写道：
> >
> > > Hi all
> > >
> > > I want to start the vote for FLIP-99, which proposes to make the max
> > > exception configurable in web UI.
> > >
> > > To help everyone better understand the proposal, we spent some efforts
> on
> > > making an online POC
> > >
> > > previous web:
> > >
> > >
> >
> http://101.132.122.69:8081/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions
> > > POC web:
> > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions
> > >
> > > The vote will last for at least 72 hours, following the consensus
> voting
> > > process.
> > >
> > > FLIP wiki:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable
> > >
> > > Discussion thread:
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > >
> > > Thanks,
> > >
> > > Yadong
> > >
> >
>


-- 

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:+86-15650713730
Email: libenc...@gmail.com; libenc...@pku.edu.cn

Re: [VOTE] FLIP-98: Better Back Pressure Detection

2020-02-23 Thread Benchao Li 
+1 (non-binding)

It's really helpful!

Yang Wang  于2020年2月21日周五 下午2:44写道：

> +1 non-binding
>
> It is really straightforward.
>
> Best,
> Yang
>
> Jark Wu  于2020年2月21日周五 下午1:52写道：
>
> > +1 (binding)
> >
> > On Fri, 21 Feb 2020 at 11:51, Kurt Young  wrote:
> >
> > > +1 (binding)
> > >
> > >
> > >
> > > On Fri, Feb 21, 2020 at 1:09 AM Zhijiang  > > .invalid>
> > > wrote:
> > >
> > > > +1 (binding).
> > > > It seems more clearly and directly to highlight the back pressured
> > vertex
> > > > in topology, which can raise the attention of users.
> > > >
> > > > Best,
> > > > Zhijiang
> > > >
> > > >
> > > > --
> > > > From:Congxian Qiu 
> > > > Send Time:2020 Feb. 20 (Thu.) 21:49
> > > > To:dev@flink.apache.org 
> > > > Subject:Re: [VOTE] FLIP-98: Better Back Pressure Detection
> > > >
> > > > +1 for this improvement, It's very helpful to know whether a task is
> > > > backpressured or not.
> > > >
> > > > Best,
> > > > Congxian
> > > >
> > > >
> > > > Yadong Xie  于2020年2月20日周四 下午5:34写道：
> > > >
> > > > > Hi all
> > > > >
> > > > > I want to start the vote for FLIP-98, which proposes to provide
> > better
> > > > > backpressure detection experience in web UI.
> > > > >
> > > > > To help everyone better understand the proposal, we spent some
> > efforts
> > > on
> > > > > making an online POC
> > > > >
> > > > > previous web:
> > > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/#/job/add9d3ea1006af15b9f872987eb3befd/overview
> > > > > POC web:
> > > > >
> > > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/add9d3ea1006af15b9f872987eb3befd/overview
> > > > >
> > > > > The vote will last for at least 72 hours, following the consensus
> > > voting
> > > > > process.
> > > > >
> > > > > FLIP wiki:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
> > > > >
> > > > > Discussion thread:
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Yadong
> > > > >
> > > >
> > > >
> > >
> >
>


-- 

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:+86-15650713730
Email: libenc...@gmail.com; libenc...@pku.edu.cn

[jira] [Created] (FLINK-16247) Rename 'checkpoints.savepoint.' package and classes to 'checkpoint.metadata.'

2020-02-23 Thread Stephan Ewen (Jira) 
Stephan Ewen created FLINK-16247:


 Summary: Rename 'checkpoints.savepoint.' package and classes to 
'checkpoint.metadata.'
 Key: FLINK-16247
 URL: https://issues.apache.org/jira/browse/FLINK-16247
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.11.0


These classes are responsible for the persistence of checkpoint metadata.

Originally, savepoints where the only snapshots with persistent metadata, while 
checkpoints had the metadata only in memory or in ZooKeeper (Java Serialized).

Nowadays, checkpoints and savepoints both persist metadata in the same way, and 
hence the package is not actually related to savepoints any more.

Because of that, I suggest to rename the classes from "Savepoint*" to 
"Metadata*" to better describe what they are really doing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16246) Exclude "SdkMBeanRegistrySupport" from dynamically loaded AWS connectors

2020-02-23 Thread Stephan Ewen (Jira) 
Stephan Ewen created FLINK-16246:


 Summary: Exclude "SdkMBeanRegistrySupport" from dynamically loaded 
AWS connectors
 Key: FLINK-16246
 URL: https://issues.apache.org/jira/browse/FLINK-16246
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kinesis
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.10.1, 1.11.0


The AWS SDK always registers a {{MetricAdminMBean}} at the JMX MBean Server.
This allows users to turn on / off more detailed metrics via the JMX admin 
interface.

However, this registered bean keeps the user code classloader alive and thus 
causes a class leak.

Excluding the {{com.amazonaws.jmx.SdkMBeanRegistrySupport}} class from the 
connectors disables the default registration thus preventing the class leak. It 
should only disable that JMX metric admin interface and still allow users to 
manually configure/activate metrics for the AWS SDK.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16245) Use a delegating classloader as the user code classloader to prevent class leaks.

2020-02-23 Thread Stephan Ewen (Jira) 
Stephan Ewen created FLINK-16245:


 Summary: Use a delegating classloader as the user code classloader 
to prevent class leaks.
 Key: FLINK-16245
 URL: https://issues.apache.org/jira/browse/FLINK-16245
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Task
Reporter: Stephan Ewen
 Fix For: 1.11.0


As reported in FLINK-11205, a reference to the user-code ClassLoader can be 
held by some libraries, causing class leaks.

One way to circumvent this class leak is if the ClassLoader that we set as the 
user-code ClassLoader is a delegating ClassLoader to the real class loader, and 
when closing the user code ClassLoader we null out the reference.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLINK-16194: Refactor the Kubernetes architecture design

2020-02-23 Thread Yang Wang 
 > The new introduced decorator
After some offline discussion with Canbin and tison, i totally understand
the evolved decorator design. Each decorator will be self-contained and
is responsible for just one thing. Currently, if we want to mount a new
config
file to jobmanager/taskmanager, then both `ConfigMapDecorator`,
`JobManagerDecorator`, `TaskManagerDecorator` needs to be updated.
It is not convenient for the new contributors to do this. In the new design,
by leveraging the accompanying kubernetes resources in decorator, we
could finish the creating and mounting config map in one decorator.

Since now we just have a basic implementation for native Kubernetes
integration and lots of features need to be developed. And many users
want to participate in and contribute to the integration. So i agree to
refactor
the current decorator implementation and make it easier for the new
contributor.

For the detailed divergences(naming, etc.), i think we could discuss in the
PR.

> Parameters Parser
Currently, the decorator directly use Flink configuration to get the
parameters
to build the Kubernetes resource. It is straightforward, however we could
not
have a unified parameters check. So i am not sure whether you will introduce
a tool to check the parameters or just simply have our own basic check.

BTW, the fabric8 kubernetes-client and Kubernetes api server already has
the parameters check before starting to create the resources. I think the
exceptions
are usually meaning and enough for the users to get the root cause.



Best,
Yang




felixzheng zheng  于2020年2月22日周六 上午10:54写道：

> Great thanks for the quick feedback Till. You are right; it is not a
> fundamentally different approach compared to
> what we have right now, all the Kubernetes resources created are the same,
> we aim to evolve the existing decorator approach so that,
> 1. the decorators are monadic and smaller in size and functionality.
> 2. the new decorator design allows reusing the decorators between the
> client and the cluster as much as possible.
> 3. all the decorators are independent with each other, and they could have
> arbitrary order in the chain, they share the same APIs and follow a unified
> orchestrator architecture so that new developers could quickly understand
> what should be done to introduce a new feature.
>
> Besides that, the new approach allows us adding tests for every decorator
> alone instead of doing a final test of all the decorators in the
> Fabric8ClientTest.java.
>
> Cheers,
> Canbin Zheng
>
> Till Rohrmann  于2020年2月22日周六 上午12:28写道：
>
> > Thanks for starting this discussion Canbin. If I understand your proposal
> > correctly, then you would like to evolve the existing decorator approach
> so
> > that decorators are monadic and smaller in size and functionality. The
> > latter aspect will allow to reuse them between the client and the
> cluster.
> > Just to make sure, it is not a fundamentally different approach compared
> to
> > what we have right now, is it?
> >
> > If this is the case, then I think it makes sense to reuse code as much as
> > possible and to create small code units which are easier to test.
> >
> > Cheers,
> > Till
> >
> > On Fri, Feb 21, 2020 at 4:41 PM felixzheng zheng  >
> > wrote:
> >
> > > Thanks for the feedback @Yang Wang. I would like to discuss some of the
> > > details in depth about why I am confused about the existing design.
> > >
> > > Question 1: How do we mount a configuration file?
> > >
> > > For the existing design,
> > >
> > >1.
> > >
> > >We need several classes to finish it:
> > >1.
> > >
> > >   InitializerDecorator
> > >   2.
> > >
> > >   OwnerReferenceDecorator
> > >   3.
> > >
> > >   ConfigMapDecorator
> > >   4.
> > >
> > >   KubernetesUtils: providing the getConfigMapVolume method to share
> > for
> > >   the FlinkMasterDeploymentDecorator and the
> TaskManagerPodDecorator.
> > >   5.
> > >
> > >   FlinkMasterDeploymentDecorator: mounts the ConfigMap volume.
> > >   6.
> > >
> > >   TaskManagerPodDecorator: mounts the ConfigMap volume.
> > >   7.
> > >
> > >   If in the future, someone would like to introduce an init
> > Container,
> > >   the InitContainerDecorator has to mount the ConfigMap volume too.
> > >
> > >
> > > I am confused about the current solution to mounting a configuration
> > file:
> > >
> > >1.
> > >
> > >Actually, we do not need so many Decorators for mounting a file.
> > >2.
> > >
> > >If we would like to mount a new file, we have no choice but to
> repeat
> > >the same tedious and scattered routine.
> > >3.
> > >
> > >There’s no easy way to test the file mounting functionality alone;
> we
> > >have to construct the ConfigMap, the Deployment or the
> TaskManagerPod
> > > first
> > >and then do a final test.
> > >
> > >
> > > The reason why it is so complex to mount a configuration file is that
> we
> > > don’t fully consider the internal

[ANNOUNCE] Weekly Community Update 2020/07

2020-02-23 Thread Konstantin Knauf 
Dear community,

happy to share this week's community digest with updates on the next
release cycle, a set of proposal for Flink's web user interface, a couple
of discussions around our development process and a bit more.

Flink Development
==

* [releases] Stephan proposes an "anticipated feature freeze date" for
Flink 1.11 around the end of April, and hence a release in May. This would
make the next release a short one, which seems to be generally well
received. Piotr and Zhijiang will be our release managers for Flink 1.11.
[1]

* [releases] flink-shaded 10.0 has been released. [2]

* [web ui] Yadong has split up the improvement proposal to Flink's web user
interface (FLIP-75) into multiple votes. Each thread contains a live demo
of the proposed feature.
  * [FLIP-98]: Highlight backpressure task in job graph [3]
  * [FLIP-99]: Make maximally shown exceptions configurable [4]
  * [FLIP-100] Show the attempt history of tasks [5]
  * [FLIP-101] Expose Information on Pending Slots in the Web User
Interface [6]
  * [FLIP-102] More Taskmanager metrics, in particular memory resources [7]
  * [FLIP-103] List all log files and make them downloadable [8]
  * [FLIP-104] More Flink Master metrics, in particular memory resources [9]

* [development process] Hequn Cheng has started discussion on improving the
FLIP process, in particular the way we deal with discussion on the wiki,
mailing list and Google Docs. The discussion is ongoing, but there seems to
be consensus not to use Google Docs more than right now, but to maybe even
eliminate it from the process all together. In the discussion David raised
a related point, that the FLIP document is often not updated after its
implementation and changes between the original plan and the final
implementation are not documented anywhere. [10]

* [development process] Xingtong proposes to update the PR description
template as it is quite outdated. The discussion shows that the template is
generally considered useful, but it indeed needs an update. [11]

* [deployment] Aljoscha has started a discussion to either drop or extend
the support for Flink's windows scripts. [12]

* [connectors] The connector for ElasticSearch 2.x will be dropped in Flink
1.11. Elastic Search 5.x will still be supported in Flink 1.11 and
revisited for Flink 1.12. [13]

* [state] Stephan proposes to drop savepoint compatibility with Flink 1.2.
A stateful upgrade from Flink 1.2 to Flink 1.11 would still be possible by
first upgrading to an intermediate Flink version. [14]

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Kicking-off-the-1-11-release-cycle-tp37817.html
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Apache-Flink-shaded-10-0-released-tp37815.html
[3]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-98-Better-Back-Pressure-Detection-tp37893.html
[4]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-99-Make-Max-Exception-Configurable-tp37895.html
[5]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-100-Add-Attempt-Information-tp37896.html
[6]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-101-Add-Pending-Slots-Detail-tp37897.html
[7]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-102-Add-More-Metrics-to-TaskManager-tp37898.html
[8]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-103-Better-TM-JM-Log-Display-tp37899.html
[9]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-104-Add-More-Metrics-to-Jobmanager-tp37901.html
[10]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Improvements-on-FLIP-Process-tp37785.html
[11]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discuss-Update-the-pull-request-description-template-tp37755.html
[12]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Extend-or-maintain-shell-script-support-for-Windows-tp37868.html
[13]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Drop-connectors-for-Elasticsearch-2-x-and-5-x-tp37471.html
[14]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Drop-Savepoint-Compatibility-with-Flink-1-2-tp37908.html

Notable Bugs
==

* [FLINK-16111] [1.10] Flink's Kubernetes deployment does not respect the
"taskmanager.cpu.cores" configuration [15]
* [FLINK-16115] [1.10] The OSS (Alibaba Cloud's Object Storage Service)
filesystem does not work as a plugin. [16]

[15] https://issues.apache.org/jira/browse/FLINK-16111
[16] https://issues.apache.org/jira/browse/FLINK-16115

Events, Blog Posts, Misc
===

* Jingsong Lee is now an Apache Flink committer. Congratulations! [17]

* Seth has published a post on the Apache Flink blog on Flink SQL DDL: "No
Java Required: Configured Sources and Sinks in SQL" [18]

* Upcoming Meetups
* On February 26th, Prateep Kumar will host an online

[jira] [Created] (FLINK-16244) Add Asynchronous operations to state lazily.

2020-02-23 Thread Igal Shilman (Jira) 
Igal Shilman created FLINK-16244:


 Summary: Add Asynchronous operations to state lazily.
 Key: FLINK-16244
 URL: https://issues.apache.org/jira/browse/FLINK-16244
 Project: Flink
  Issue Type: Task
  Components: Stateful Functions
Reporter: Igal Shilman


Currently AsyncSink would add eagerly a registered async operation. 
An alternative approach would be to keep the async operations in an in memory 
map and only 
write them to the underlying map on snapshotState().
The rational behind this approach is the assumption that most async operations 
complete between two consecutive checkpoints, and therefore adding and removing 
them from the underlying state backend (rocksdb by default) is wasteful.

An implementation outline suggestion:

1. Add a LazyAsyncOperations class that keeps both an in memory map
and a MapStateHandle
this map would support add and remove and also flush() that would be called 
from 
snapshotState() method.

2. Use that class in AsyncSink and in AsyncMessageDecorator

3. call flush() from FunctionGroupOperator#snapshotState()

note that a special care should be taken in flash() as the current key needs to 
be set
in the keyedStateBackend.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16243) Update quickstart documentation to Log4j2

2020-02-23 Thread Chesnay Schepler (Jira) 
Chesnay Schepler created FLINK-16243:


 Summary: Update quickstart documentation to Log4j2
 Key: FLINK-16243
 URL: https://issues.apache.org/jira/browse/FLINK-16243
 Project: Flink
  Issue Type: Bug
  Components: Documentation, Quickstarts
Affects Versions: 1.11.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.11.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Apache Flink-shaded 10.0 released

2020-02-23 Thread Konstantin Knauf 
Thanks, Chesnay, for your continuous work on flink-shaded and managing this
release.

Best,

Konstantin

On Thu, Feb 20, 2020 at 2:33 PM Xingbo Huang  wrote:

> Thanks a lot Chesnay and all contributes to this release.
>
> Best,
> Xingbo
>
> Hequn Cheng  于2020年2月20日周四 下午8:17写道：
>
> > Thanks a lot for the release Chesnay!
> > Also thanks to everyone who contributes to this release!
> >
> > Best, Hequn
> >
> > On Thu, Feb 20, 2020 at 11:11 AM Yu Li  wrote:
> >
> > > Thanks Chesnay and all participants for making the release possible!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Thu, 20 Feb 2020 at 09:50, Zhu Zhu  wrote:
> > >
> > > > Thanks Chesnay for the great work and everyone who helps with the
> > > > improvements and release!
> > > >
> > > > Thanks,
> > > > Zhu Zhu
> > > >
> > > > Dian Fu  于2020年2月20日周四 上午9:44写道：
> > > >
> > > > > Thanks Chesnay for the great work and everyone involved!
> > > > >
> > > > > Regards,
> > > > > Dian
> > > > >
> > > > > > 在 2020年2月20日，上午12:21，Zhijiang  .INVALID>
> > > 写道：
> > > > > >
> > > > > > Thanks Chesnay for making the release efficiently and also thanks
> > to
> > > > all
> > > > > the other participants!
> > > > > >
> > > > > > Best,
> > > > > > Zhijiang
> > > > > >
> > > > > >
> > > > > >
> --
> > > > > > From:Till Rohrmann  > > trohrm...@apache.org
> > > > >>
> > > > > > Send Time:2020 Feb. 19 (Wed.) 22:21
> > > > > > To:dev mailto:dev@flink.apache.org>>
> > > > > > Subject:Re: [ANNOUNCE] Apache Flink-shaded 10.0 released
> > > > > >
> > > > > > Thanks for making the release possible Chesnay and everyone who
> was
> > > > > > involved!
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Wed, Feb 19, 2020 at 7:47 AM jincheng sun <
> > > sunjincheng...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Thanks a lot for the release Chesnay!
> > > > > >> And thanks to everyone who make this release possible!
> > > > > >>
> > > > > >> Best,
> > > > > >> Jincheng
> > > > > >>
> > > > > >>
> > > > > >> Chesnay Schepler  于2020年2月19日周三 上午12:45写道：
> > > > > >>
> > > > > >>> The Apache Flink community is very happy to announce the
> release
> > of
> > > > > >>> Apache Flink-shaded 10.0.
> > > > > >>>
> > > > > >>> The flink-shaded project contains a number of shaded
> dependencies
> > > for
> > > > > >>> Apache Flink.
> > > > > >>>
> > > > > >>> Apache Flink(r) is an open-source stream processing framework
> for
> > > > > >>> distributed, high-performing, always-available, and accurate
> data
> > > > > >>> streaming applications.
> > > > > >>>
> > > > > >>> The release is available for download at:
> > > > > >>> https://flink.apache.org/downloads.html
> > > > > >>>
> > > > > >>> The full release notes are available in Jira:
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346746
> > > > > >>>
> > > > > >>> We would like to thank all contributors of the Apache Flink
> > > community
> > > > > >>> who made this release possible!
> > > > > >>>
> > > > > >>> Regards,
> > > > > >>> Chesnay
> > > > >
> > > > >
> > > >
> > >
> >
>


-- 

Konstantin Knauf | Head of Product

+49 160 91394525


Follow us @VervericaData Ververica 


--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Tony) Cheng

[jira] [Created] (FLINK-16242) BinaryGeneric serialization error cause checkpoint failure

2020-02-23 Thread Jiayi Liao (Jira) 
Jiayi Liao created FLINK-16242:
--

 Summary: BinaryGeneric serialization error cause checkpoint failure
 Key: FLINK-16242
 URL: https://issues.apache.org/jira/browse/FLINK-16242
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner, Table SQL / Runtime
Affects Versions: 1.9.2
Reporter: Jiayi Liao
 Attachments: error_serialization

The serialization error occurs from time to time when we're using 
{{RoaringBitmap}} as the accumulator of a UDAF.

I've attached the screenshot of the error.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)