[jira] [Created] (FLINK-13645) Error in code-gen when using blink planner in scala shell

[Jeff Zhang (JIRA)]

Jeff Zhang created FLINK-13645:
--

 Summary: Error in code-gen when using blink planner in scala shell
 Key: FLINK-13645
 URL: https://issues.apache.org/jira/browse/FLINK-13645
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.9.0
Reporter: Jeff Zhang
 Attachments: image-2019-08-08-11-43-08-741.png

 !image-2019-08-08-11-43-08-741.png! 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13644) Translate "State Backends" page into Chinese

[fanrui (JIRA)]

fanrui created FLINK-13644:
--

 Summary: Translate "State Backends" page into Chinese
 Key: FLINK-13644
 URL: https://issues.apache.org/jira/browse/FLINK-13644
 Project: Flink
  Issue Type: Sub-task
  Components: chinese-translation, Documentation
Affects Versions: 1.10.0
Reporter: fanrui
 Fix For: 1.10.0


1、The page url is 
[https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/stream/state/state_backends.html]

The markdown file is located in "docs/dev/stream/state/state_backends.zh.md"

2、The page url is 
[https://ci.apache.org/projects/flink/flink-docs-master/zh/ops/state/state_backends.html]

The markdown file is located in "docs/ops/state/state_backends.zh.md"



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re:Re: [ANNOUNCE] Hequn becomes a Flink committer

[Haibo Sun]

Congratulations!


Best,
Haibo
At 2019-08-08 02:08:21, "Yun Tang"  wrote:
>Congratulations Hequn.
>
>Best
>Yun Tang
>
>From: Rong Rong 
>Sent: Thursday, August 8, 2019 0:41
>Cc: dev ; user 
>Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
>
>Congratulations Hequn, well deserved!
>
>--
>Rong
>
>On Wed, Aug 7, 2019 at 8:30 AM mailto:xingc...@gmail.com>> 
>wrote:
>
>Congratulations, Hequn!
>
>
>
>From: Xintong Song mailto:tonysong...@gmail.com>>
>Sent: Wednesday, August 07, 2019 10:41 AM
>To: dev@flink.apache.org
>Cc: user mailto:u...@flink.apache.org>>
>Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
>
>
>
>Congratulations~!
>
>
>Thank you~
>
>Xintong Song
>
>
>
>
>
>On Wed, Aug 7, 2019 at 4:00 PM vino yang 
>mailto:yanghua1...@gmail.com>> wrote:
>
>Congratulations!
>
>highfei2...@126.com 
>mailto:highfei2...@126.com>> 于2019年8月7日周三 下午7:09写道：
>
>> Congrats Hequn!
>>
>> Best,
>> Jeff Yang
>>
>>
>>  Original Message 
>> Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
>> From: Piotr Nowojski
>> To: JingsongLee
>> CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun ,dev 
>> ,user
>>
>>
>> Congratulations :)
>>
>> On 7 Aug 2019, at 12:09, JingsongLee 
>> mailto:lzljs3620...@aliyun.com>> wrote:
>>
>> Congrats Hequn!
>>
>> Best,
>> Jingsong Lee
>>
>> --
>> From:Biao Liu mailto:mmyy1...@gmail.com>>
>> Send Time:2019年8月7日(星期三) 12:05
>> To:Zhu Zhu mailto:reed...@gmail.com>>
>> Cc:Zili Chen mailto:wander4...@gmail.com>>; Jeff Zhang 
>> mailto:zjf...@gmail.com>>; Paul
>> Lam mailto:paullin3...@gmail.com>>; jincheng sun 
>> mailto:sunjincheng...@gmail.com>>; dev
>> mailto:dev@flink.apache.org>>; user 
>> mailto:u...@flink.apache.org>>
>> Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
>>
>> Congrats Hequn!
>>
>> Thanks,
>> Biao /'bɪ.aʊ/
>>
>>
>>
>> On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu 
>> mailto:reed...@gmail.com>> wrote:
>> Congratulations to Hequn!
>>
>> Thanks,
>> Zhu Zhu
>>
>> Zili Chen mailto:wander4...@gmail.com>> 于2019年8月7日周三 
>> 下午5:16写道：
>> Congrats Hequn!
>>
>> Best,
>> tison.
>>
>>
>> Jeff Zhang mailto:zjf...@gmail.com>> 于2019年8月7日周三 下午5:14写道：
>> Congrats Hequn!
>>
>> Paul Lam mailto:paullin3...@gmail.com>> 于2019年8月7日周三 
>> 下午5:08写道：
>> Congrats Hequn! Well deserved!
>>
>> Best,
>> Paul Lam
>>
>> 在 2019年8月7日，16:28，jincheng sun 
>> mailto:sunjincheng...@gmail.com>> 写道：
>>
>> Hi everyone,
>>
>> I'm very happy to announce that Hequn accepted the offer of the Flink PMC
>> to become a committer of the Flink project.
>>
>> Hequn has been contributing to Flink for many years, mainly working on
>> SQL/Table API features. He's also frequently helping out on the user
>> mailing lists and helping check/vote the release.
>>
>> Congratulations Hequn!
>>
>> Best, Jincheng
>> (on behalf of the Flink PMC)
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>>
>>

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Hequn Cheng]

Thanks everyone for the warm welcome!

It's a great honor for me to be a committer. Looking forward to
contributing more to the community.

Best, Hequn

On Thu, Aug 8, 2019 at 3:39 AM zhijiang  wrote:

> Congratulations Hequn!
>
> Best,
> Zhijiang
> --
> From:Xuefu Z 
> Send Time:2019年8月7日(星期三) 20:35
> To:dev 
> Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
>
> Congratulations， Hequn！
>
> On Wed, Aug 7, 2019 at 11:08 AM Yun Tang  wrote:
>
> > Congratulations Hequn.
> >
> > Best
> > Yun Tang
> > 
> > From: Rong Rong 
> > Sent: Thursday, August 8, 2019 0:41
> > Cc: dev ; user 
> > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> >
> > Congratulations Hequn, well deserved!
> >
> > --
> > Rong
> >
> > On Wed, Aug 7, 2019 at 8:30 AM  > xingc...@gmail.com>> wrote:
> >
> > Congratulations, Hequn!
> >
> >
> >
> > From: Xintong Song mailto:tonysong...@gmail.com>>
> > Sent: Wednesday, August 07, 2019 10:41 AM
> > To: dev@flink.apache.org
> > Cc: user mailto:u...@flink.apache.org>>
> > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> >
> >
> >
> > Congratulations~!
> >
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> >
> >
> > On Wed, Aug 7, 2019 at 4:00 PM vino yang  > yanghua1...@gmail.com>> wrote:
> >
> > Congratulations!
> >
> > highfei2...@126.com  > > 于2019年8月7日周三 下午7:09写道：
> >
> > > Congrats Hequn!
> > >
> > > Best,
> > > Jeff Yang
> > >
> > >
> > >  Original Message 
> > > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> > > From: Piotr Nowojski
> > > To: JingsongLee
> > > CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun
> > ,dev ,user
> > >
> > >
> > > Congratulations :)
> > >
> > > On 7 Aug 2019, at 12:09, JingsongLee  > lzljs3620...@aliyun.com>> wrote:
> > >
> > > Congrats Hequn!
> > >
> > > Best,
> > > Jingsong Lee
> > >
> > > --
> > > From:Biao Liu mailto:mmyy1...@gmail.com>>
> > > Send Time:2019年8月7日(星期三) 12:05
> > > To:Zhu Zhu mailto:reed...@gmail.com>>
> > > Cc:Zili Chen mailto:wander4...@gmail.com>>; Jeff
> > Zhang mailto:zjf...@gmail.com>>; Paul
> > > Lam mailto:paullin3...@gmail.com>>; jincheng
> sun
> > mailto:sunjincheng...@gmail.com>>; dev
> > > mailto:dev@flink.apache.org>>; user <
> > u...@flink.apache.org>
> > > Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
> > >
> > > Congrats Hequn!
> > >
> > > Thanks,
> > > Biao /'bɪ.aʊ/
> > >
> > >
> > >
> > > On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu  > reed...@gmail.com>> wrote:
> > > Congratulations to Hequn!
> > >
> > > Thanks,
> > > Zhu Zhu
> > >
> > > Zili Chen mailto:wander4...@gmail.com>>
> > 于2019年8月7日周三 下午5:16写道：
> > > Congrats Hequn!
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Jeff Zhang mailto:zjf...@gmail.com>> 于2019年8月7日周三
> > 下午5:14写道：
> > > Congrats Hequn!
> > >
> > > Paul Lam mailto:paullin3...@gmail.com>>
> > 于2019年8月7日周三 下午5:08写道：
> > > Congrats Hequn! Well deserved!
> > >
> > > Best,
> > > Paul Lam
> > >
> > > 在 2019年8月7日，16:28，jincheng sun  > sunjincheng...@gmail.com>> 写道：
> > >
> > > Hi everyone,
> > >
> > > I'm very happy to announce that Hequn accepted the offer of the Flink
> PMC
> > > to become a committer of the Flink project.
> > >
> > > Hequn has been contributing to Flink for many years, mainly working on
> > > SQL/Table API features. He's also frequently helping out on the user
> > > mailing lists and helping check/vote the release.
> > >
> > > Congratulations Hequn!
> > >
> > > Best, Jincheng
> > > (on behalf of the Flink PMC)
> > >
> > >
> > >
> > > --
> > > Best Regards
> > >
> > > Jeff Zhang
> > >
> > >
> > >
> >
>
>
> --
> Xuefu Zhang
>
> "In Honey We Trust!"
>
>

Re: [ANNOUNCE] Hequn becomes a Flink committer

[zhijiang]

Congratulations Hequn!

Best,
Zhijiang
--
From:Xuefu Z 
Send Time:2019年8月7日(星期三) 20:35
To:dev 
Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations， Hequn！

On Wed, Aug 7, 2019 at 11:08 AM Yun Tang  wrote:

> Congratulations Hequn.
>
> Best
> Yun Tang
> 
> From: Rong Rong 
> Sent: Thursday, August 8, 2019 0:41
> Cc: dev ; user 
> Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
>
> Congratulations Hequn, well deserved!
>
> --
> Rong
>
> On Wed, Aug 7, 2019 at 8:30 AM  xingc...@gmail.com>> wrote:
>
> Congratulations, Hequn!
>
>
>
> From: Xintong Song mailto:tonysong...@gmail.com>>
> Sent: Wednesday, August 07, 2019 10:41 AM
> To: dev@flink.apache.org
> Cc: user mailto:u...@flink.apache.org>>
> Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
>
>
>
> Congratulations~!
>
>
> Thank you~
>
> Xintong Song
>
>
>
>
>
> On Wed, Aug 7, 2019 at 4:00 PM vino yang  yanghua1...@gmail.com>> wrote:
>
> Congratulations!
>
> highfei2...@126.com  > 于2019年8月7日周三 下午7:09写道：
>
> > Congrats Hequn!
> >
> > Best,
> > Jeff Yang
> >
> >
> >  Original Message 
> > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> > From: Piotr Nowojski
> > To: JingsongLee
> > CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun
> ,dev ,user
> >
> >
> > Congratulations :)
> >
> > On 7 Aug 2019, at 12:09, JingsongLee  lzljs3620...@aliyun.com>> wrote:
> >
> > Congrats Hequn!
> >
> > Best,
> > Jingsong Lee
> >
> > --
> > From:Biao Liu mailto:mmyy1...@gmail.com>>
> > Send Time:2019年8月7日(星期三) 12:05
> > To:Zhu Zhu mailto:reed...@gmail.com>>
> > Cc:Zili Chen mailto:wander4...@gmail.com>>; Jeff
> Zhang mailto:zjf...@gmail.com>>; Paul
> > Lam mailto:paullin3...@gmail.com>>; jincheng sun
> mailto:sunjincheng...@gmail.com>>; dev
> > mailto:dev@flink.apache.org>>; user <
> u...@flink.apache.org>
> > Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
> >
> > Congrats Hequn!
> >
> > Thanks,
> > Biao /'bɪ.aʊ/
> >
> >
> >
> > On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu  reed...@gmail.com>> wrote:
> > Congratulations to Hequn!
> >
> > Thanks,
> > Zhu Zhu
> >
> > Zili Chen mailto:wander4...@gmail.com>>
> 于2019年8月7日周三 下午5:16写道：
> > Congrats Hequn!
> >
> > Best,
> > tison.
> >
> >
> > Jeff Zhang mailto:zjf...@gmail.com>> 于2019年8月7日周三
> 下午5:14写道：
> > Congrats Hequn!
> >
> > Paul Lam mailto:paullin3...@gmail.com>>
> 于2019年8月7日周三 下午5:08写道：
> > Congrats Hequn! Well deserved!
> >
> > Best,
> > Paul Lam
> >
> > 在 2019年8月7日，16:28，jincheng sun  sunjincheng...@gmail.com>> 写道：
> >
> > Hi everyone,
> >
> > I'm very happy to announce that Hequn accepted the offer of the Flink PMC
> > to become a committer of the Flink project.
> >
> > Hequn has been contributing to Flink for many years, mainly working on
> > SQL/Table API features. He's also frequently helping out on the user
> > mailing lists and helping check/vote the release.
> >
> > Congratulations Hequn!
> >
> > Best, Jincheng
> > (on behalf of the Flink PMC)
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
> >
> >
>


-- 
Xuefu Zhang

"In Honey We Trust!"

Re: Custom type serializer inside Flink state

[Ying Xu]

Hi Yun:

Thanks for the quick reply.   Thanks for pointing to FLINK-11333
, will take a look.

We are currently using Flink 1.8.0.  To summarize the behavior:

1) In the first version of the Flink app, there is a protobuf class getting
registered for Kyro:
env.getConfig().*registerTypeWithKryoSerializer*(MyProtoClass1.class,
ProtobufSerializer.class);

2) Then the flink app gets updated with a second protobuf class, say,
MyProtoClass2.class.
env.getConfig().*registerTypeWithKryoSerializer*(MyProtoClass1.class,
ProtobufSerializer.class);
env.getConfig().*registerTypeWithKryoSerializer*(MyProtoClass2.class,
ProtobufSerializer.class);

3) Cancel the first version of Flink app; take a savepoint;  and deploy the
second version Flink app with the savepoint. Occasionally one would
encounter the exception described above.

So far we are able to work around this issue, by simply registering the
default Kryo deserializer for Message.class

-- the super class for all the generated protobuf messages.

*NOTE*: Above is a high-level example. In practice, our app is more
complex, with a much larger number of protobuf classes. Messages may be
added/deleted across different versions of the Flink app.


On Wed, Aug 7, 2019 at 11:23 AM Yun Tang  wrote:

> Hi Ying
>
> What version of Flink are you using and please more exception stack.
> Moreover, what is the relationship between `MyProtoClass2` and
> `MyProtoClass1`? As far as I know, registering the Message class should not
> be the proper solution.
>
> For the 2nd question, you could refer to FLINK-11333 [1] for more
> information.
>
> CC @Tzu-Li (Gordon) Tai as he might provide
> more information about this.
>
> [1] https://issues.apache.org/jira/browse/FLINK-11333
>
> Best
> Yun Tang
>
> 
> From: Ying Xu 
> Sent: Thursday, August 8, 2019 1:51
> To: dev@flink.apache.org 
> Subject: Custom type serializer inside Flink state
>
> Hi Flink community:
>
> Our Flink application sends different types of protobuf messages
> on-the-wire.  Since protobuf cannot be handled by Flink type serializer, we
> had to register custom Kyro serializer:
>
> env.getConfig().*registerTypeWithKryoSerializer*(MyProtoClass1.class,
> ProtobufSerializer.class);
>
> We found registering each protobuf class is not a viable solution for
> schema evolution.  Particularly, when adding/removing new messages we would
> encounter errors when restoring state backend:
>
>
> Caused by: org.apache.flink.util.FlinkException: Could not restore operator
> state backend for StreamSource_d63019dde9166d2672f543f01f344085_(13/32)
> from any of the 1 provided restore options.
> at
>
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
> ... 5 common frames omitted
> Caused by: org.apache.flink.runtime.state.BackendBuildingException: Failed
> when trying to restore operator state backend
> at
>
> org.apache.flink.runtime.state.DefaultOperatorStateBackendBuilder.build(DefaultOperatorStateBackendBuilder.java:86)
> at
>
> org.apache.flink.runtime.state.filesystem.FsStateBackend.createOperatorStateBackend(FsStateBackend.java:504)
> at
>
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$operatorStateBackend$0(StreamTaskStateInitializerImpl.java:246)
> ... 7 common frames omitted
> Caused by: java.lang.IllegalStateException: Missing value for the key
> 'proto$*MyProtoClass2*'
> at
>
> org.apache.flink.util.LinkedOptionalMap.unwrapOptionals(LinkedOptionalMap.java:190)
>
> We then switch to registering the default protobuf class for the super
> class of all proto -- Message.class , and this issue appears to go away.
>
> see.getConfig().*addDefaultKryoSerializer*(Message.class,
> ProtobufSerializer
> .class);
>
>
> Questions:
>
> 1)  It seems custom Kyro serializers are registered with the Flink state
> backend. Can we confirm when using the default Kyro serializer, only the
> super class (e.g Message.class) is registered and no specific protobuf
> message is associated with state ?
>
> 2)  Will proto ser/de supported by Flink type serializer in the future and
> is
> there any longer term roadmap for supporting state evolution for
> protobuf-type messages?
>
> Thanks a lot.
>

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Xuefu Z]

Congratulations， Hequn！

On Wed, Aug 7, 2019 at 11:08 AM Yun Tang  wrote:

> Congratulations Hequn.
>
> Best
> Yun Tang
> 
> From: Rong Rong 
> Sent: Thursday, August 8, 2019 0:41
> Cc: dev ; user 
> Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
>
> Congratulations Hequn, well deserved!
>
> --
> Rong
>
> On Wed, Aug 7, 2019 at 8:30 AM  xingc...@gmail.com>> wrote:
>
> Congratulations, Hequn!
>
>
>
> From: Xintong Song mailto:tonysong...@gmail.com>>
> Sent: Wednesday, August 07, 2019 10:41 AM
> To: dev@flink.apache.org
> Cc: user mailto:u...@flink.apache.org>>
> Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
>
>
>
> Congratulations~!
>
>
> Thank you~
>
> Xintong Song
>
>
>
>
>
> On Wed, Aug 7, 2019 at 4:00 PM vino yang  yanghua1...@gmail.com>> wrote:
>
> Congratulations!
>
> highfei2...@126.com  > 于2019年8月7日周三 下午7:09写道：
>
> > Congrats Hequn!
> >
> > Best,
> > Jeff Yang
> >
> >
> >  Original Message 
> > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> > From: Piotr Nowojski
> > To: JingsongLee
> > CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun
> ,dev ,user
> >
> >
> > Congratulations :)
> >
> > On 7 Aug 2019, at 12:09, JingsongLee  lzljs3620...@aliyun.com>> wrote:
> >
> > Congrats Hequn!
> >
> > Best,
> > Jingsong Lee
> >
> > --
> > From:Biao Liu mailto:mmyy1...@gmail.com>>
> > Send Time:2019年8月7日(星期三) 12:05
> > To:Zhu Zhu mailto:reed...@gmail.com>>
> > Cc:Zili Chen mailto:wander4...@gmail.com>>; Jeff
> Zhang mailto:zjf...@gmail.com>>; Paul
> > Lam mailto:paullin3...@gmail.com>>; jincheng sun
> mailto:sunjincheng...@gmail.com>>; dev
> > mailto:dev@flink.apache.org>>; user <
> u...@flink.apache.org>
> > Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
> >
> > Congrats Hequn!
> >
> > Thanks,
> > Biao /'bɪ.aʊ/
> >
> >
> >
> > On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu  reed...@gmail.com>> wrote:
> > Congratulations to Hequn!
> >
> > Thanks,
> > Zhu Zhu
> >
> > Zili Chen mailto:wander4...@gmail.com>>
> 于2019年8月7日周三 下午5:16写道：
> > Congrats Hequn!
> >
> > Best,
> > tison.
> >
> >
> > Jeff Zhang mailto:zjf...@gmail.com>> 于2019年8月7日周三
> 下午5:14写道：
> > Congrats Hequn!
> >
> > Paul Lam mailto:paullin3...@gmail.com>>
> 于2019年8月7日周三 下午5:08写道：
> > Congrats Hequn! Well deserved!
> >
> > Best,
> > Paul Lam
> >
> > 在 2019年8月7日，16:28，jincheng sun  sunjincheng...@gmail.com>> 写道：
> >
> > Hi everyone,
> >
> > I'm very happy to announce that Hequn accepted the offer of the Flink PMC
> > to become a committer of the Flink project.
> >
> > Hequn has been contributing to Flink for many years, mainly working on
> > SQL/Table API features. He's also frequently helping out on the user
> > mailing lists and helping check/vote the release.
> >
> > Congratulations Hequn!
> >
> > Best, Jincheng
> > (on behalf of the Flink PMC)
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
> >
> >
>


-- 
Xuefu Zhang

"In Honey We Trust!"

Re: [ANNOUNCE] Progress updates for Apache Flink 1.9.0 release

[David Anderson]

I've spent time exploring this playground and its accompanying
documentation, and found it to be a big step forward in making it easy for
folks to experience some of Flink's key features firsthand. From a training
and educational perspective, I'd love to see this in 1.9.

*David Anderson* | Training Coordinator

Follow us @VervericaData
--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time

Re: Custom type serializer inside Flink state

[Yun Tang]

Hi Ying

What version of Flink are you using and please more exception stack. Moreover, 
what is the relationship between `MyProtoClass2` and `MyProtoClass1`? As far as 
I know, registering the Message class should not be the proper solution.

For the 2nd question, you could refer to FLINK-11333 [1] for more information.

CC @Tzu-Li (Gordon) Tai as he might provide more 
information about this.

[1] https://issues.apache.org/jira/browse/FLINK-11333

Best
Yun Tang


From: Ying Xu 
Sent: Thursday, August 8, 2019 1:51
To: dev@flink.apache.org 
Subject: Custom type serializer inside Flink state

Hi Flink community:

Our Flink application sends different types of protobuf messages
on-the-wire.  Since protobuf cannot be handled by Flink type serializer, we
had to register custom Kyro serializer:

env.getConfig().*registerTypeWithKryoSerializer*(MyProtoClass1.class,
ProtobufSerializer.class);

We found registering each protobuf class is not a viable solution for
schema evolution.  Particularly, when adding/removing new messages we would
encounter errors when restoring state backend:


Caused by: org.apache.flink.util.FlinkException: Could not restore operator
state backend for StreamSource_d63019dde9166d2672f543f01f344085_(13/32)
from any of the 1 provided restore options.
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
... 5 common frames omitted
Caused by: org.apache.flink.runtime.state.BackendBuildingException: Failed
when trying to restore operator state backend
at
org.apache.flink.runtime.state.DefaultOperatorStateBackendBuilder.build(DefaultOperatorStateBackendBuilder.java:86)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.createOperatorStateBackend(FsStateBackend.java:504)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$operatorStateBackend$0(StreamTaskStateInitializerImpl.java:246)
... 7 common frames omitted
Caused by: java.lang.IllegalStateException: Missing value for the key
'proto$*MyProtoClass2*'
at
org.apache.flink.util.LinkedOptionalMap.unwrapOptionals(LinkedOptionalMap.java:190)

We then switch to registering the default protobuf class for the super
class of all proto -- Message.class , and this issue appears to go away.

see.getConfig().*addDefaultKryoSerializer*(Message.class, ProtobufSerializer
.class);


Questions:

1)  It seems custom Kyro serializers are registered with the Flink state
backend. Can we confirm when using the default Kyro serializer, only the
super class (e.g Message.class) is registered and no specific protobuf
message is associated with state ?

2)  Will proto ser/de supported by Flink type serializer in the future and is
there any longer term roadmap for supporting state evolution for
protobuf-type messages?

Thanks a lot.

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Yun Tang]

Congratulations Hequn.

Best
Yun Tang

From: Rong Rong 
Sent: Thursday, August 8, 2019 0:41
Cc: dev ; user 
Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations Hequn, well deserved!

--
Rong

On Wed, Aug 7, 2019 at 8:30 AM mailto:xingc...@gmail.com>> 
wrote:

Congratulations, Hequn!



From: Xintong Song mailto:tonysong...@gmail.com>>
Sent: Wednesday, August 07, 2019 10:41 AM
To: dev@flink.apache.org
Cc: user mailto:u...@flink.apache.org>>
Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer



Congratulations~!


Thank you~

Xintong Song





On Wed, Aug 7, 2019 at 4:00 PM vino yang 
mailto:yanghua1...@gmail.com>> wrote:

Congratulations!

highfei2...@126.com 
mailto:highfei2...@126.com>> 于2019年8月7日周三 下午7:09写道：

> Congrats Hequn!
>
> Best,
> Jeff Yang
>
>
>  Original Message 
> Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> From: Piotr Nowojski
> To: JingsongLee
> CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun ,dev 
> ,user
>
>
> Congratulations :)
>
> On 7 Aug 2019, at 12:09, JingsongLee 
> mailto:lzljs3620...@aliyun.com>> wrote:
>
> Congrats Hequn!
>
> Best,
> Jingsong Lee
>
> --
> From:Biao Liu mailto:mmyy1...@gmail.com>>
> Send Time:2019年8月7日(星期三) 12:05
> To:Zhu Zhu mailto:reed...@gmail.com>>
> Cc:Zili Chen mailto:wander4...@gmail.com>>; Jeff Zhang 
> mailto:zjf...@gmail.com>>; Paul
> Lam mailto:paullin3...@gmail.com>>; jincheng sun 
> mailto:sunjincheng...@gmail.com>>; dev
> mailto:dev@flink.apache.org>>; user 
> mailto:u...@flink.apache.org>>
> Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
>
> Congrats Hequn!
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu 
> mailto:reed...@gmail.com>> wrote:
> Congratulations to Hequn!
>
> Thanks,
> Zhu Zhu
>
> Zili Chen mailto:wander4...@gmail.com>> 于2019年8月7日周三 
> 下午5:16写道：
> Congrats Hequn!
>
> Best,
> tison.
>
>
> Jeff Zhang mailto:zjf...@gmail.com>> 于2019年8月7日周三 下午5:14写道：
> Congrats Hequn!
>
> Paul Lam mailto:paullin3...@gmail.com>> 于2019年8月7日周三 
> 下午5:08写道：
> Congrats Hequn! Well deserved!
>
> Best,
> Paul Lam
>
> 在 2019年8月7日，16:28，jincheng sun 
> mailto:sunjincheng...@gmail.com>> 写道：
>
> Hi everyone,
>
> I'm very happy to announce that Hequn accepted the offer of the Flink PMC
> to become a committer of the Flink project.
>
> Hequn has been contributing to Flink for many years, mainly working on
> SQL/Table API features. He's also frequently helping out on the user
> mailing lists and helping check/vote the release.
>
> Congratulations Hequn!
>
> Best, Jincheng
> (on behalf of the Flink PMC)
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
>

Custom type serializer inside Flink state

[Ying Xu]

Hi Flink community:

Our Flink application sends different types of protobuf messages
on-the-wire.  Since protobuf cannot be handled by Flink type serializer, we
had to register custom Kyro serializer:

env.getConfig().*registerTypeWithKryoSerializer*(MyProtoClass1.class,
ProtobufSerializer.class);

We found registering each protobuf class is not a viable solution for
schema evolution.  Particularly, when adding/removing new messages we would
encounter errors when restoring state backend:


Caused by: org.apache.flink.util.FlinkException: Could not restore operator
state backend for StreamSource_d63019dde9166d2672f543f01f344085_(13/32)
from any of the 1 provided restore options.
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
... 5 common frames omitted
Caused by: org.apache.flink.runtime.state.BackendBuildingException: Failed
when trying to restore operator state backend
at
org.apache.flink.runtime.state.DefaultOperatorStateBackendBuilder.build(DefaultOperatorStateBackendBuilder.java:86)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.createOperatorStateBackend(FsStateBackend.java:504)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$operatorStateBackend$0(StreamTaskStateInitializerImpl.java:246)
... 7 common frames omitted
Caused by: java.lang.IllegalStateException: Missing value for the key
'proto$*MyProtoClass2*'
at
org.apache.flink.util.LinkedOptionalMap.unwrapOptionals(LinkedOptionalMap.java:190)

We then switch to registering the default protobuf class for the super
class of all proto -- Message.class , and this issue appears to go away.

see.getConfig().*addDefaultKryoSerializer*(Message.class, ProtobufSerializer
.class);


Questions:

1)  It seems custom Kyro serializers are registered with the Flink state
backend. Can we confirm when using the default Kyro serializer, only the
super class (e.g Message.class) is registered and no specific protobuf
message is associated with state ?

2)  Will proto ser/de supported by Flink type serializer in the future and is
there any longer term roadmap for supporting state evolution for
protobuf-type messages?

Thanks a lot.

Re: [ANNOUNCE] Progress updates for Apache Flink 1.9.0 release

[Konstantin Knauf]

Hi everyone,

in the context of FLIP-42 Fabian and myself were working on a docker-based
playground as part of the "Getting Started" section of our documentation.
The PR [1] was merged into master today. Besides documentation, this also
adds an additional example to `flink-streaming-examples`. For the
playground to work with the Apache Flink 1.9 this example needs to become
part of the distribution.

Would it be possible to still include this PR in the 1.9.0 release. As far
as I know, documentation can generally be added after the feature freeze,
but since this also touches the examples, this is kind of a corner case, I
suppose.

What do you think?

Best,

Konstantin


[1] https://github.com/apache/flink/pull/9192

On Wed, Aug 7, 2019 at 2:51 PM Tzu-Li (Gordon) Tai 
wrote:

> Hi all,
>
> According to the 1.9.x burndown board [1], we're approaching a releasable
> state for 1.9.0.
> Thanks to everyone who participated in the work for fixing the blockers so
> far, especially Till who has been coordinating a lot of the efforts.
>
> Below is a summary of the current state of the few remaining blockers:
>
> Pending bugs to be fixed -
>
>-
> *FLINK-13159 - Restored PojoSerializer not using correct classloader for
>deserialization [2] STATUS: *PR opened and reviewed, waiting for Travis
>run before merging
>*NOTES:* this bug is not specific to 1.9.0 only; will be backported to
>1.8.x as well. It was made a blocker for 1.9.0 as well since the fix is
>relatively low-effort.
>-
> *FLINK-13593 - Prevent failing the wrong execution attempt in
>CheckpointFailureManager [3] STATUS:* PR opened, some final passes of
>reviews pending
>
> Additional tests to be added -
>
>- *FLINK-13441 - Add batch sql E2E test which runs with fewer slots than
>parallelism to test the newly introduced batch scheduling modes [4]*
>*STATUS:* PR opened and being reviewed.
>*NOTES:* The TPC-H E2E test has also been modified to cover this
>scenario.
>
> Unstable tests:
>
>-
> *FLINK-13489 - Heavy deployment E2E test fails on Travis (agreed to make
>this a non-blocker) [5] STATUS:* The cause of this isn't a critical
>issue, and it is agreed that this would not be a blocker for the
> release.
>-
> *FLINK-13581 - BatchFineGrainedRecoveryITCase failed on Travis [6] STATUS:
> *PR
>opened and review is in progress
>-
> *FLINK-13527 - Unstable KafkaProducerExactlyOnceITCase fails [7]
> STATUS:* Blocked
>by FLINK-13593 (blocker issue mentioned above)
>*NOTES: *Yu Li already mentioned that with the fix in FLINK-13593, this
>test no longer fails
>-
> *FLINK-13607 - TCP-H E2E tests fails on Travis [8] STATUS:* Awaiting final
>confirmations on whether or not the instability still exists.
>*NOTES:* Kurt is also running a variation of this with multiple TMs and
>high parallelism (10-20 TMs, ~1000 DoP) internally in Alibaba.
>
> So, from the looks of things, it should be safe to say that we can aim for
> creating the first voting RC (RC2) by the end of this week (August 9th)!
> An official voting thread for RC2 will be established once it is ready.
>
> Cheers,
> Gordon
>
> [1] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=328
> [2] https://issues.apache.org/jira/browse/FLINK-13159
> [3] https://issues.apache.org/jira/browse/FLINK-13593
> [4] https://issues.apache.org/jira/browse/FLINK-13441
> [5] https://issues.apache.org/jira/browse/FLINK-13489
> [6] https://issues.apache.org/jira/browse/FLINK-13581
> [7] https://issues.apache.org/jira/browse/FLINK-13527
> [8] https://issues.apache.org/jira/browse/FLINK-13607
>
> On Thu, Aug 1, 2019 at 3:03 PM Kurt Young  wrote:
>
> > Update: RC1 for 1.9.0 has been created. Please see [1] for the preview
> > source / binary releases and Maven artifacts.
> >
> > Best,
> > Kurt
> >
> > [1]
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/PREVIEW-Apache-Flink-1-9-0-release-candidate-1-td31233.html
> >
> >
> > On Tue, Jul 30, 2019 at 2:36 PM Tzu-Li (Gordon) Tai  >
> > wrote:
> >
> > > Hi Biao,
> > >
> > > Thanks for working on FLINK-9900. The ticket is already assigned to you
> > > now.
> > >
> > > Cheers,
> > > Gordon
> > >
> > > On Tue, Jul 30, 2019 at 2:31 PM Biao Liu  wrote:
> > >
> > > > Hi Gordon,
> > > >
> > > > Thanks for updating progress.
> > > >
> > > > Currently I'm working on FLINK-9900. I need a committer to assign the
> > > > ticket to me.
> > > >
> > > > Tzu-Li (Gordon) Tai 于2019年7月30日 周二13:01写道：
> > > >
> > > > > Hi all,
> > > > >
> > > > > There are quite a few instabilities in our builds right now
> (master +
> > > > > release-1.9), some of which are directed or suspiciously related to
> > the
> > > > 1.9
> > > > > release.
> > > > >
> > > > > I'll categorize the instabilities into ones which we were already
> > > > tracking
> > > > > in the 1.9 Burndown Kanban board [1] prior to this email, and which
> > > ones
> > > > > seems to be new or were not monitored so 

[jira] [Created] (FLINK-13643) Document the workaround for users with a different minor Hive version

[Xuefu Zhang (JIRA)]

Xuefu Zhang created FLINK-13643:
---

 Summary: Document the workaround for users with a different minor 
Hive version
 Key: FLINK-13643
 URL: https://issues.apache.org/jira/browse/FLINK-13643
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Hive, Table SQL / API
Affects Versions: 1.9.0
Reporter: Xuefu Zhang
 Fix For: 1.9.0


We officially support two Hive versions. However, we can tell user how to work 
around the limitation if their Hive version is only minorly differently.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13642) Refactor ShuffleMaster to optionally provide preferred TM location for produced partitions

[Andrey Zagrebin (JIRA)]

Andrey Zagrebin created FLINK-13642:
---

 Summary: Refactor ShuffleMaster to optionally provide preferred TM 
location for produced partitions
 Key: FLINK-13642
 URL: https://issues.apache.org/jira/browse/FLINK-13642
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13641) Consider removing UnknownInputChannel in NettyShuffleEnvironment

[Andrey Zagrebin (JIRA)]

Andrey Zagrebin created FLINK-13641:
---

 Summary: Consider removing UnknownInputChannel in 
NettyShuffleEnvironment
 Key: FLINK-13641
 URL: https://issues.apache.org/jira/browse/FLINK-13641
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Reporter: Andrey Zagrebin


UnknownInputChannel basically is a place holder while the producer is unknown 
which contains some partition parameters to use later for creation of the known 
channel.

If NettyShuffleEnvironment#updatePartitionInfo provides enough information to 
add the known channel, one could consider removing UnknownInputChannel and 
refactoring indexed channel array into a dynamic list.

Initially suggested in 
[https://github.com/apache/flink/pull/8362#discussion_r290308989]



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13640) Consider TaskDeploymentDescriptorFactory to be Execution specific

[Andrey Zagrebin (JIRA)]

Andrey Zagrebin created FLINK-13640:
---

 Summary: Consider TaskDeploymentDescriptorFactory to be Execution 
specific
 Key: FLINK-13640
 URL: https://issues.apache.org/jira/browse/FLINK-13640
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.9.0
Reporter: Andrey Zagrebin


suggested in [https://github.com/apache/flink/pull/8362#discussion_r290297974]



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13639) Consider refactoring of IntermediateResultPartitionID to consist of IntermediateDataSetID and partitionIndex

[Andrey Zagrebin (JIRA)]

Andrey Zagrebin created FLINK-13639:
---

 Summary: Consider refactoring of IntermediateResultPartitionID to 
consist of IntermediateDataSetID and partitionIndex
 Key: FLINK-13639
 URL: https://issues.apache.org/jira/browse/FLINK-13639
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Andrey Zagrebin






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13638) Refactor RemoteChannelStateChecker#isProducerConsumerReadyOrAbortConsumption to return result action

[Andrey Zagrebin (JIRA)]

Andrey Zagrebin created FLINK-13638:
---

 Summary: Refactor 
RemoteChannelStateChecker#isProducerConsumerReadyOrAbortConsumption to return 
result action
 Key: FLINK-13638
 URL: https://issues.apache.org/jira/browse/FLINK-13638
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Affects Versions: 1.9.0, 1.10.0
Reporter: Andrey Zagrebin


RemoteChannelStateChecker#isProducerConsumerReadyOrAbortConsumption either 
triggers some action (fail or cancel) or returns a decision (trigger new 
partition check or not). It would be more symmetric if this class would not 
trigger any action but only return a decision what to do:

 
{code:java}
enum Action {
  FAIL(Throwable cause),
  CANCEL(String msg),
  TRIGGER_PARTITION_CHECK, NOOP
}
{code}
 

Then the caller would be responsible for making the action. That way this class 
would only need access to responseHandle{{.getProducerExecutionState()}} and 
not responseHandle{{ }}itself.

>From PR discussion 
>[https://github.com/apache/flink/pull/8463#discussion_r288290783]



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

RE: [ANNOUNCE] Hequn becomes a Flink committer

[xingcanc]

Congratulations, Hequn!

 

From: Xintong Song  
Sent: Wednesday, August 07, 2019 10:41 AM
To: dev@flink.apache.org
Cc: user 
Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer

 

Congratulations~!




Thank you~

Xintong Song

 

 

On Wed, Aug 7, 2019 at 4:00 PM vino yang mailto:yanghua1...@gmail.com> > wrote:

Congratulations!

highfei2...@126.com   mailto:highfei2...@126.com> > 于2019年8月7日周三 下午7:09写道：

> Congrats Hequn!
>
> Best,
> Jeff Yang
>
>
>  Original Message 
> Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> From: Piotr Nowojski
> To: JingsongLee
> CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun ,dev 
> ,user
>
>
> Congratulations :)
>
> On 7 Aug 2019, at 12:09, JingsongLee   > wrote:
>
> Congrats Hequn!
>
> Best,
> Jingsong Lee
>
> --
> From:Biao Liu mailto:mmyy1...@gmail.com> >
> Send Time:2019年8月7日(星期三) 12:05
> To:Zhu Zhu mailto:reed...@gmail.com> >
> Cc:Zili Chen mailto:wander4...@gmail.com> >; Jeff 
> Zhang mailto:zjf...@gmail.com> >; Paul
> Lam mailto:paullin3...@gmail.com> >; jincheng sun 
> mailto:sunjincheng...@gmail.com> >; dev
> mailto:dev@flink.apache.org> >; user 
> mailto:u...@flink.apache.org> >
> Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
>
> Congrats Hequn!
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu   > wrote:
> Congratulations to Hequn!
>
> Thanks,
> Zhu Zhu
>
> Zili Chen mailto:wander4...@gmail.com> > 于2019年8月7日周三 
> 下午5:16写道：
> Congrats Hequn!
>
> Best,
> tison.
>
>
> Jeff Zhang mailto:zjf...@gmail.com> > 于2019年8月7日周三 
> 下午5:14写道：
> Congrats Hequn!
>
> Paul Lam mailto:paullin3...@gmail.com> > 于2019年8月7日周三 
> 下午5:08写道：
> Congrats Hequn! Well deserved!
>
> Best,
> Paul Lam
>
> 在 2019年8月7日，16:28，jincheng sun   > 写道：
>
> Hi everyone,
>
> I'm very happy to announce that Hequn accepted the offer of the Flink PMC
> to become a committer of the Flink project.
>
> Hequn has been contributing to Flink for many years, mainly working on
> SQL/Table API features. He's also frequently helping out on the user
> mailing lists and helping check/vote the release.
>
> Congratulations Hequn!
>
> Best, Jincheng
> (on behalf of the Flink PMC)
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
>

[jira] [Created] (FLINK-13637) Anchors not working in document(building.md, common.md, queryable_state.md)

[Hequn Cheng (JIRA)]

Hequn Cheng created FLINK-13637:
---

 Summary: Anchors not working in document(building.md, common.md, 
queryable_state.md)
 Key: FLINK-13637
 URL: https://issues.apache.org/jira/browse/FLINK-13637
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.9.0
Reporter: Hequn Cheng
 Fix For: 1.9.0


Anchors not working in document(building.md, common.md, queryable_state.md).

The format should be:
{code:java}
[create an anchor](#anchors-in-markdown)
{code}

- Add - characters between each word in the heading and wrap the value in parens
- All letters should be lowercase.




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: Customize StreamingFileSink: Enable extending StreamingFileSink class

[Fabian Hueske]

Hi Kailash,

Yes, I think creating another Jira and PR would be the right thing to do.

Thank you,
Fabian

Am Fr., 2. Aug. 2019 um 10:29 Uhr schrieb Kailash Dayanand <
kailash...@gmail.com>:

> Hello,
>
> There were a few things which we missing in
> https://issues.apache.org/jira/browse/FLINK-12539, due to which it was not
> possible to extend the StreamingFileSink. To fix this I have created a
> follow-up PR but I am not sure if this is the best way to make these
> changes:
> https://github.com/apache/flink/compare/master...kailashhd:FLINK-12539.
>
> In case this is the best approach to do this, I will do ahead and create
> another JIRA and PR for this. Appreciate the help.
>
> Thanks
> Kailash
>

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Xintong Song]

Congratulations~!

Thank you~

Xintong Song



On Wed, Aug 7, 2019 at 4:00 PM vino yang  wrote:

> Congratulations!
>
> highfei2...@126.com  于2019年8月7日周三 下午7:09写道：
>
> > Congrats Hequn!
> >
> > Best,
> > Jeff Yang
> >
> >
> >  Original Message 
> > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> > From: Piotr Nowojski
> > To: JingsongLee
> > CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun
> ,dev ,user
> >
> >
> > Congratulations :)
> >
> > On 7 Aug 2019, at 12:09, JingsongLee  wrote:
> >
> > Congrats Hequn!
> >
> > Best,
> > Jingsong Lee
> >
> > --
> > From:Biao Liu 
> > Send Time:2019年8月7日(星期三) 12:05
> > To:Zhu Zhu 
> > Cc:Zili Chen ; Jeff Zhang ; Paul
> > Lam ; jincheng sun ;
> dev
> > ; user 
> > Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
> >
> > Congrats Hequn!
> >
> > Thanks,
> > Biao /'bɪ.aʊ/
> >
> >
> >
> > On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu  wrote:
> > Congratulations to Hequn!
> >
> > Thanks,
> > Zhu Zhu
> >
> > Zili Chen  于2019年8月7日周三 下午5:16写道：
> > Congrats Hequn!
> >
> > Best,
> > tison.
> >
> >
> > Jeff Zhang  于2019年8月7日周三 下午5:14写道：
> > Congrats Hequn!
> >
> > Paul Lam  于2019年8月7日周三 下午5:08写道：
> > Congrats Hequn! Well deserved!
> >
> > Best,
> > Paul Lam
> >
> > 在 2019年8月7日，16:28，jincheng sun  写道：
> >
> > Hi everyone,
> >
> > I'm very happy to announce that Hequn accepted the offer of the Flink PMC
> > to become a committer of the Flink project.
> >
> > Hequn has been contributing to Flink for many years, mainly working on
> > SQL/Table API features. He's also frequently helping out on the user
> > mailing lists and helping check/vote the release.
> >
> > Congratulations Hequn!
> >
> > Best, Jincheng
> > (on behalf of the Flink PMC)
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
> >
> >
>

Re: Suspicious watermark of operators after restore from checkpoint

[Jan Lukavský]

Hi Kostas,

yes - operators A and B are as you said. The WebUI is what would explain 
in best, if it is anyhow possible, that it would display sort of stale 
information at one operator and updated on another.


Jan

On 8/7/19 3:59 PM, Kostas Kloudas wrote:

Hi Jan,

I am not sure what is happening. Operator A does not seem to be chained to
the source (which produces the watermarks) so
the check about increasing watermarks should be also applied there. BTW, I
assume that bottom left you mean the one that
starts with "activeDevices:takePresent..." (Op. A)  and
"activeDevices:stepLength..." (Op. B).

I am wondering if it can be that the WebUi is not consistent across
different operators.
For example, the watermark of Op B was simply not updated in the WebUI.

I also cc Chesnay who may have a better insight about the WebUi.

Cheers,
Kostas


On Wed, Aug 7, 2019 at 3:25 PM Jan Lukavský  wrote:


Code would be a little complicated, because it is wrapped with several
layers of other APIs (Beam being one of them, but there is also other
layer).

I can provide the job graph [1]  a screenshot of the two watermarks [2].
The watermarks are taken from the two operators on bottom left.

Essentially, the job reads from Google cloud storage and simultaneously
from Kafka. On cloud storage are stored blob files containing historical
events and these blobs are marked with event time range (e.g. file is
named BLOB_EVENTS_TIMESTAMP1_TIMESTAMP2), and those timestamps are used
to generate watermarks from the batch storage (files are read in sorted
order).

Does that help, or would you like more details?

Jan

[1] https://transfer.sh/v473f/jobgraph.png
[2] https://transfer.sh/iDg1A/watermarks.png

On 8/7/19 3:04 PM, Kostas Kloudas wrote:

But are they chained together? Could you provide the code from your job,

at

least until operator A?

On Wed, Aug 7, 2019 at 3:03 PM Jan Lukavský  wrote:


Actually, operator A is intermediate, source is preceding it.

On 8/7/19 2:44 PM, Kostas Kloudas wrote:

Hi Jan,

After looking at the code, my point 1) is false for *intermediate*

tasks

and if you are
using a watermark assigner. This means that in these cases, Flink

checks

that the
"next" watermark is greater than the "previous" one.

But if your operator A is a source and you emit watermarks from the

source,

then
it can happen that your watermark appears to go backwards on operator

A,

but
operator B does the "correction" by discarding smaller watermarks. That

can

explain
your observation.

Cheers,
Kostas

On Wed, Aug 7, 2019 at 2:30 PM Jan Lukavský  wrote:


Hi Kostas,

thanks for reaction, comments inline.

On 8/7/19 1:59 PM, Kostas Kloudas wrote:

Hi Jan,

Two pointers that may help you explain the behavior are the

following:

1) If you have a custom watermark generator, I do not think that

Flink

checks if it emits only monotonically
increasing watermarks. This is the responsibility of the generator

itself.

This means that although you operator A
is topologically before operator B, operator A may have a smaller

watermark

if your watermark generator allows so.

I do generate watermarks by custom source, but I believe that the
generated sequence is monotonic. But still, I'm not sure, that even if
it was the case, that the generated watermark actually decreases,

would

that mean, that downstream operator after source (operator A) would
actually "go back in time"?

2) Flink currently does not checkpoint the last seen watermark (
https://issues.apache.org/jira/browse/FLINK-5601).
This means that after restoring, your (event) time is assumed to be
Long.Min until the first new watermark comes.
So if you observed late data not being late anymore or sth similar,

then

it

may not be that the two operators have
different watermarks but that after restoring event time rolls back

to

the

"beginning of time".

I actually didn't observe any wrong or unexpected behavior, exceptions
or wrong outputs. I just noticed this on Flink's WebUI and it looked
strange to me. Could it be just that the WebUI showed older watermark
for operator A? Strange was, that the watermarks were my screen long
enough to take a screenshot (so at least say 10 seconds displaying
watermark of operator A less than the one of operator B). Even if
watermarks are not checkpointed, would it still be possible for
watermark of operator B to be actually greater? I'm still confused of
how this could happen, because (in my understanding) output watermark

of

operator A should be greater or equal to input watermark of B (because
it takes minimum of inputs).

Sorry if I'm too digging into this, but I don't like things I cannot
explain, as they might point out to some bugs somewhere. :-) Or that

my

mental model it not aligned with reality.

Jan


I hope this helps,
Kostas

On Wed, Aug 7, 2019 at 12:11 PM Jan Lukavský 

wrote:

Hi all,

I have just come across a weird state of operators after restore

from

checkpoint. After the restore, two operators that are 

[DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors

[Xintong Song]

Hi everyone,

We would like to start a discussion thread on "FLIP-49: Unified Memory
Configuration for TaskExecutors"[1], where we describe how to improve
TaskExecutor memory configurations. The FLIP document is mostly based on an
early design "Memory Management and Configuration Reloaded"[2] by Stephan,
with updates from follow-up discussions both online and offline.

This FLIP addresses several shortcomings of current (Flink 1.9)
TaskExecutor memory configuration.

   - Different configuration for Streaming and Batch.
   - Complex and difficult configuration of RocksDB in Streaming.
   - Complicated, uncertain and hard to understand.


Key changes to solve the problems can be summarized as follows.

   - Extend memory manager to also account for memory usage by state
   backends.
   - Modify how TaskExecutor memory is partitioned accounted individual
   memory reservations and pools.
   - Simplify memory configuration options and calculations logics.


Please find more details in the FLIP wiki document [1].

(Please note that the early design doc [2] is out of sync, and it is
appreciated to have the discussion in this mailing list thread.)


Looking forward to your feedbacks.


Thank you~

Xintong Song


[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors

[2]
https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing

Re: [ANNOUNCE] Hequn becomes a Flink committer

[vino yang]

Congratulations!

highfei2...@126.com  于2019年8月7日周三 下午7:09写道：

> Congrats Hequn!
>
> Best,
> Jeff Yang
>
>
>  Original Message 
> Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> From: Piotr Nowojski
> To: JingsongLee
> CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun ,dev 
> ,user
>
>
> Congratulations :)
>
> On 7 Aug 2019, at 12:09, JingsongLee  wrote:
>
> Congrats Hequn!
>
> Best,
> Jingsong Lee
>
> --
> From:Biao Liu 
> Send Time:2019年8月7日(星期三) 12:05
> To:Zhu Zhu 
> Cc:Zili Chen ; Jeff Zhang ; Paul
> Lam ; jincheng sun ; dev
> ; user 
> Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
>
> Congrats Hequn!
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu  wrote:
> Congratulations to Hequn!
>
> Thanks,
> Zhu Zhu
>
> Zili Chen  于2019年8月7日周三 下午5:16写道：
> Congrats Hequn!
>
> Best,
> tison.
>
>
> Jeff Zhang  于2019年8月7日周三 下午5:14写道：
> Congrats Hequn!
>
> Paul Lam  于2019年8月7日周三 下午5:08写道：
> Congrats Hequn! Well deserved!
>
> Best,
> Paul Lam
>
> 在 2019年8月7日，16:28，jincheng sun  写道：
>
> Hi everyone,
>
> I'm very happy to announce that Hequn accepted the offer of the Flink PMC
> to become a committer of the Flink project.
>
> Hequn has been contributing to Flink for many years, mainly working on
> SQL/Table API features. He's also frequently helping out on the user
> mailing lists and helping check/vote the release.
>
> Congratulations Hequn!
>
> Best, Jincheng
> (on behalf of the Flink PMC)
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
>

Re: Suspicious watermark of operators after restore from checkpoint

[Kostas Kloudas]

Hi Jan,

I am not sure what is happening. Operator A does not seem to be chained to
the source (which produces the watermarks) so
the check about increasing watermarks should be also applied there. BTW, I
assume that bottom left you mean the one that
starts with "activeDevices:takePresent..." (Op. A)  and
"activeDevices:stepLength..." (Op. B).

I am wondering if it can be that the WebUi is not consistent across
different operators.
For example, the watermark of Op B was simply not updated in the WebUI.

I also cc Chesnay who may have a better insight about the WebUi.

Cheers,
Kostas


On Wed, Aug 7, 2019 at 3:25 PM Jan Lukavský  wrote:

> Code would be a little complicated, because it is wrapped with several
> layers of other APIs (Beam being one of them, but there is also other
> layer).
>
> I can provide the job graph [1]  a screenshot of the two watermarks [2].
> The watermarks are taken from the two operators on bottom left.
>
> Essentially, the job reads from Google cloud storage and simultaneously
> from Kafka. On cloud storage are stored blob files containing historical
> events and these blobs are marked with event time range (e.g. file is
> named BLOB_EVENTS_TIMESTAMP1_TIMESTAMP2), and those timestamps are used
> to generate watermarks from the batch storage (files are read in sorted
> order).
>
> Does that help, or would you like more details?
>
> Jan
>
> [1] https://transfer.sh/v473f/jobgraph.png
> [2] https://transfer.sh/iDg1A/watermarks.png
>
> On 8/7/19 3:04 PM, Kostas Kloudas wrote:
> > But are they chained together? Could you provide the code from your job,
> at
> > least until operator A?
> >
> > On Wed, Aug 7, 2019 at 3:03 PM Jan Lukavský  wrote:
> >
> >> Actually, operator A is intermediate, source is preceding it.
> >>
> >> On 8/7/19 2:44 PM, Kostas Kloudas wrote:
> >>> Hi Jan,
> >>>
> >>> After looking at the code, my point 1) is false for *intermediate*
> tasks
> >>> and if you are
> >>> using a watermark assigner. This means that in these cases, Flink
> checks
> >>> that the
> >>> "next" watermark is greater than the "previous" one.
> >>>
> >>> But if your operator A is a source and you emit watermarks from the
> >> source,
> >>> then
> >>> it can happen that your watermark appears to go backwards on operator
> A,
> >>> but
> >>> operator B does the "correction" by discarding smaller watermarks. That
> >> can
> >>> explain
> >>> your observation.
> >>>
> >>> Cheers,
> >>> Kostas
> >>>
> >>> On Wed, Aug 7, 2019 at 2:30 PM Jan Lukavský  wrote:
> >>>
>  Hi Kostas,
> 
>  thanks for reaction, comments inline.
> 
>  On 8/7/19 1:59 PM, Kostas Kloudas wrote:
> > Hi Jan,
> >
> > Two pointers that may help you explain the behavior are the
> following:
> >
> > 1) If you have a custom watermark generator, I do not think that
> Flink
> > checks if it emits only monotonically
> > increasing watermarks. This is the responsibility of the generator
>  itself.
> > This means that although you operator A
> > is topologically before operator B, operator A may have a smaller
>  watermark
> > if your watermark generator allows so.
>  I do generate watermarks by custom source, but I believe that the
>  generated sequence is monotonic. But still, I'm not sure, that even if
>  it was the case, that the generated watermark actually decreases,
> would
>  that mean, that downstream operator after source (operator A) would
>  actually "go back in time"?
> > 2) Flink currently does not checkpoint the last seen watermark (
> > https://issues.apache.org/jira/browse/FLINK-5601).
> > This means that after restoring, your (event) time is assumed to be
> > Long.Min until the first new watermark comes.
> > So if you observed late data not being late anymore or sth similar,
> >> then
>  it
> > may not be that the two operators have
> > different watermarks but that after restoring event time rolls back
> to
>  the
> > "beginning of time".
>  I actually didn't observe any wrong or unexpected behavior, exceptions
>  or wrong outputs. I just noticed this on Flink's WebUI and it looked
>  strange to me. Could it be just that the WebUI showed older watermark
>  for operator A? Strange was, that the watermarks were my screen long
>  enough to take a screenshot (so at least say 10 seconds displaying
>  watermark of operator A less than the one of operator B). Even if
>  watermarks are not checkpointed, would it still be possible for
>  watermark of operator B to be actually greater? I'm still confused of
>  how this could happen, because (in my understanding) output watermark
> of
>  operator A should be greater or equal to input watermark of B (because
>  it takes minimum of inputs).
> 
>  Sorry if I'm too digging into this, but I don't like things I cannot
>  explain, as they might point out to some bugs somewhere. :-) Or that
> my
>  

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Terry Wang]

Congratulations Hequn, well deserved!

Best,
Terry Wang



> 在 2019年8月7日，下午9:16，Oytun Tez  写道：
> 
> Congratulations Hequn!
> 
> ---
> Oytun Tez
> 
> M O T A W O R D
> The World's Fastest Human Translation Platform.
> oy...@motaword.com  — www.motaword.com 
> 
> 
> On Wed, Aug 7, 2019 at 9:03 AM Jark Wu  > wrote:
> Congratulations Hequn! It's great to have you in the community!
> 
> 
> 
> On Wed, 7 Aug 2019 at 21:00, Fabian Hueske  > wrote:
> Congratulations Hequn!
> 
> Am Mi., 7. Aug. 2019 um 14:50 Uhr schrieb Robert Metzger  >:
> Congratulations!
> 
> On Wed, Aug 7, 2019 at 1:09 PM highfei2...@126.com 
>   >
> wrote:
> 
> > Congrats Hequn!
> >
> > Best,
> > Jeff Yang
> >
> >
> >  Original Message 
> > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> > From: Piotr Nowojski
> > To: JingsongLee
> > CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun ,dev 
> > ,user
> >
> >
> > Congratulations :)
> >
> > On 7 Aug 2019, at 12:09, JingsongLee  > > wrote:
> >
> > Congrats Hequn!
> >
> > Best,
> > Jingsong Lee
> >
> > --
> > From:Biao Liu mailto:mmyy1...@gmail.com>>
> > Send Time:2019年8月7日(星期三) 12:05
> > To:Zhu Zhu mailto:reed...@gmail.com>>
> > Cc:Zili Chen mailto:wander4...@gmail.com>>; Jeff 
> > Zhang mailto:zjf...@gmail.com>>; Paul
> > Lam mailto:paullin3...@gmail.com>>; jincheng sun 
> > mailto:sunjincheng...@gmail.com>>; dev
> > mailto:dev@flink.apache.org>>; user 
> > mailto:u...@flink.apache.org>>
> > Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
> >
> > Congrats Hequn!
> >
> > Thanks,
> > Biao /'bɪ.aʊ/
> >
> >
> >
> > On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu  > > wrote:
> > Congratulations to Hequn!
> >
> > Thanks,
> > Zhu Zhu
> >
> > Zili Chen mailto:wander4...@gmail.com>> 于2019年8月7日周三 
> > 下午5:16写道：
> > Congrats Hequn!
> >
> > Best,
> > tison.
> >
> >
> > Jeff Zhang mailto:zjf...@gmail.com>> 于2019年8月7日周三 
> > 下午5:14写道：
> > Congrats Hequn!
> >
> > Paul Lam mailto:paullin3...@gmail.com>> 
> > 于2019年8月7日周三 下午5:08写道：
> > Congrats Hequn! Well deserved!
> >
> > Best,
> > Paul Lam
> >
> > 在 2019年8月7日，16:28，jincheng sun  > > 写道：
> >
> > Hi everyone,
> >
> > I'm very happy to announce that Hequn accepted the offer of the Flink PMC
> > to become a committer of the Flink project.
> >
> > Hequn has been contributing to Flink for many years, mainly working on
> > SQL/Table API features. He's also frequently helping out on the user
> > mailing lists and helping check/vote the release.
> >
> > Congratulations Hequn!
> >
> > Best, Jincheng
> > (on behalf of the Flink PMC)
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
> >
> >

Re: Suspicious watermark of operators after restore from checkpoint

[Jan Lukavský]

Code would be a little complicated, because it is wrapped with several 
layers of other APIs (Beam being one of them, but there is also other 
layer).


I can provide the job graph [1]  a screenshot of the two watermarks [2]. 
The watermarks are taken from the two operators on bottom left.


Essentially, the job reads from Google cloud storage and simultaneously 
from Kafka. On cloud storage are stored blob files containing historical 
events and these blobs are marked with event time range (e.g. file is 
named BLOB_EVENTS_TIMESTAMP1_TIMESTAMP2), and those timestamps are used 
to generate watermarks from the batch storage (files are read in sorted 
order).


Does that help, or would you like more details?

Jan

[1] https://transfer.sh/v473f/jobgraph.png
[2] https://transfer.sh/iDg1A/watermarks.png

On 8/7/19 3:04 PM, Kostas Kloudas wrote:

But are they chained together? Could you provide the code from your job, at
least until operator A?

On Wed, Aug 7, 2019 at 3:03 PM Jan Lukavský  wrote:


Actually, operator A is intermediate, source is preceding it.

On 8/7/19 2:44 PM, Kostas Kloudas wrote:

Hi Jan,

After looking at the code, my point 1) is false for *intermediate* tasks
and if you are
using a watermark assigner. This means that in these cases, Flink checks
that the
"next" watermark is greater than the "previous" one.

But if your operator A is a source and you emit watermarks from the

source,

then
it can happen that your watermark appears to go backwards on operator A,
but
operator B does the "correction" by discarding smaller watermarks. That

can

explain
your observation.

Cheers,
Kostas

On Wed, Aug 7, 2019 at 2:30 PM Jan Lukavský  wrote:


Hi Kostas,

thanks for reaction, comments inline.

On 8/7/19 1:59 PM, Kostas Kloudas wrote:

Hi Jan,

Two pointers that may help you explain the behavior are the following:

1) If you have a custom watermark generator, I do not think that Flink
checks if it emits only monotonically
increasing watermarks. This is the responsibility of the generator

itself.

This means that although you operator A
is topologically before operator B, operator A may have a smaller

watermark

if your watermark generator allows so.

I do generate watermarks by custom source, but I believe that the
generated sequence is monotonic. But still, I'm not sure, that even if
it was the case, that the generated watermark actually decreases, would
that mean, that downstream operator after source (operator A) would
actually "go back in time"?

2) Flink currently does not checkpoint the last seen watermark (
https://issues.apache.org/jira/browse/FLINK-5601).
This means that after restoring, your (event) time is assumed to be
Long.Min until the first new watermark comes.
So if you observed late data not being late anymore or sth similar,

then

it

may not be that the two operators have
different watermarks but that after restoring event time rolls back to

the

"beginning of time".

I actually didn't observe any wrong or unexpected behavior, exceptions
or wrong outputs. I just noticed this on Flink's WebUI and it looked
strange to me. Could it be just that the WebUI showed older watermark
for operator A? Strange was, that the watermarks were my screen long
enough to take a screenshot (so at least say 10 seconds displaying
watermark of operator A less than the one of operator B). Even if
watermarks are not checkpointed, would it still be possible for
watermark of operator B to be actually greater? I'm still confused of
how this could happen, because (in my understanding) output watermark of
operator A should be greater or equal to input watermark of B (because
it takes minimum of inputs).

Sorry if I'm too digging into this, but I don't like things I cannot
explain, as they might point out to some bugs somewhere. :-) Or that my
mental model it not aligned with reality.

Jan


I hope this helps,
Kostas

On Wed, Aug 7, 2019 at 12:11 PM Jan Lukavský  wrote:


Hi all,

I have just come across a weird state of operators after restore from
checkpoint. After the restore, two operators that are connected (i.e.
operator A is input of operator B) ended up with watermark of

operator A

being less than watermark of operator B. I don't know how to explain
this. Can it be normal or does it signal a bug somewhere? If I
understand Flink's checkpointing correctly, the checkpoint barrier

flows

from one operator to another, so the watermark should be aligned.

I'm running a Beam pipeline on Flink 1.8.1.

Am I missing something?

Many thanks for comments,

 Jan

Re: Suspicious watermark of operators after restore from checkpoint

[Kostas Kloudas]

But are they chained together? Could you provide the code from your job, at
least until operator A?

On Wed, Aug 7, 2019 at 3:03 PM Jan Lukavský  wrote:

> Actually, operator A is intermediate, source is preceding it.
>
> On 8/7/19 2:44 PM, Kostas Kloudas wrote:
> > Hi Jan,
> >
> > After looking at the code, my point 1) is false for *intermediate* tasks
> > and if you are
> > using a watermark assigner. This means that in these cases, Flink checks
> > that the
> > "next" watermark is greater than the "previous" one.
> >
> > But if your operator A is a source and you emit watermarks from the
> source,
> > then
> > it can happen that your watermark appears to go backwards on operator A,
> > but
> > operator B does the "correction" by discarding smaller watermarks. That
> can
> > explain
> > your observation.
> >
> > Cheers,
> > Kostas
> >
> > On Wed, Aug 7, 2019 at 2:30 PM Jan Lukavský  wrote:
> >
> >> Hi Kostas,
> >>
> >> thanks for reaction, comments inline.
> >>
> >> On 8/7/19 1:59 PM, Kostas Kloudas wrote:
> >>> Hi Jan,
> >>>
> >>> Two pointers that may help you explain the behavior are the following:
> >>>
> >>> 1) If you have a custom watermark generator, I do not think that Flink
> >>> checks if it emits only monotonically
> >>> increasing watermarks. This is the responsibility of the generator
> >> itself.
> >>> This means that although you operator A
> >>> is topologically before operator B, operator A may have a smaller
> >> watermark
> >>> if your watermark generator allows so.
> >> I do generate watermarks by custom source, but I believe that the
> >> generated sequence is monotonic. But still, I'm not sure, that even if
> >> it was the case, that the generated watermark actually decreases, would
> >> that mean, that downstream operator after source (operator A) would
> >> actually "go back in time"?
> >>> 2) Flink currently does not checkpoint the last seen watermark (
> >>> https://issues.apache.org/jira/browse/FLINK-5601).
> >>> This means that after restoring, your (event) time is assumed to be
> >>> Long.Min until the first new watermark comes.
> >>> So if you observed late data not being late anymore or sth similar,
> then
> >> it
> >>> may not be that the two operators have
> >>> different watermarks but that after restoring event time rolls back to
> >> the
> >>> "beginning of time".
> >> I actually didn't observe any wrong or unexpected behavior, exceptions
> >> or wrong outputs. I just noticed this on Flink's WebUI and it looked
> >> strange to me. Could it be just that the WebUI showed older watermark
> >> for operator A? Strange was, that the watermarks were my screen long
> >> enough to take a screenshot (so at least say 10 seconds displaying
> >> watermark of operator A less than the one of operator B). Even if
> >> watermarks are not checkpointed, would it still be possible for
> >> watermark of operator B to be actually greater? I'm still confused of
> >> how this could happen, because (in my understanding) output watermark of
> >> operator A should be greater or equal to input watermark of B (because
> >> it takes minimum of inputs).
> >>
> >> Sorry if I'm too digging into this, but I don't like things I cannot
> >> explain, as they might point out to some bugs somewhere. :-) Or that my
> >> mental model it not aligned with reality.
> >>
> >> Jan
> >>
> >>> I hope this helps,
> >>> Kostas
> >>>
> >>> On Wed, Aug 7, 2019 at 12:11 PM Jan Lukavský  wrote:
> >>>
>  Hi all,
> 
>  I have just come across a weird state of operators after restore from
>  checkpoint. After the restore, two operators that are connected (i.e.
>  operator A is input of operator B) ended up with watermark of
> operator A
>  being less than watermark of operator B. I don't know how to explain
>  this. Can it be normal or does it signal a bug somewhere? If I
>  understand Flink's checkpointing correctly, the checkpoint barrier
> flows
>  from one operator to another, so the watermark should be aligned.
> 
>  I'm running a Beam pipeline on Flink 1.8.1.
> 
>  Am I missing something?
> 
>  Many thanks for comments,
> 
>  Jan
> 
> 
>

Re: Suspicious watermark of operators after restore from checkpoint

[Jan Lukavský]

Actually, operator A is intermediate, source is preceding it.

On 8/7/19 2:44 PM, Kostas Kloudas wrote:

Hi Jan,

After looking at the code, my point 1) is false for *intermediate* tasks
and if you are
using a watermark assigner. This means that in these cases, Flink checks
that the
"next" watermark is greater than the "previous" one.

But if your operator A is a source and you emit watermarks from the source,
then
it can happen that your watermark appears to go backwards on operator A,
but
operator B does the "correction" by discarding smaller watermarks. That can
explain
your observation.

Cheers,
Kostas

On Wed, Aug 7, 2019 at 2:30 PM Jan Lukavský  wrote:


Hi Kostas,

thanks for reaction, comments inline.

On 8/7/19 1:59 PM, Kostas Kloudas wrote:

Hi Jan,

Two pointers that may help you explain the behavior are the following:

1) If you have a custom watermark generator, I do not think that Flink
checks if it emits only monotonically
increasing watermarks. This is the responsibility of the generator

itself.

This means that although you operator A
is topologically before operator B, operator A may have a smaller

watermark

if your watermark generator allows so.

I do generate watermarks by custom source, but I believe that the
generated sequence is monotonic. But still, I'm not sure, that even if
it was the case, that the generated watermark actually decreases, would
that mean, that downstream operator after source (operator A) would
actually "go back in time"?

2) Flink currently does not checkpoint the last seen watermark (
https://issues.apache.org/jira/browse/FLINK-5601).
This means that after restoring, your (event) time is assumed to be
Long.Min until the first new watermark comes.
So if you observed late data not being late anymore or sth similar, then

it

may not be that the two operators have
different watermarks but that after restoring event time rolls back to

the

"beginning of time".

I actually didn't observe any wrong or unexpected behavior, exceptions
or wrong outputs. I just noticed this on Flink's WebUI and it looked
strange to me. Could it be just that the WebUI showed older watermark
for operator A? Strange was, that the watermarks were my screen long
enough to take a screenshot (so at least say 10 seconds displaying
watermark of operator A less than the one of operator B). Even if
watermarks are not checkpointed, would it still be possible for
watermark of operator B to be actually greater? I'm still confused of
how this could happen, because (in my understanding) output watermark of
operator A should be greater or equal to input watermark of B (because
it takes minimum of inputs).

Sorry if I'm too digging into this, but I don't like things I cannot
explain, as they might point out to some bugs somewhere. :-) Or that my
mental model it not aligned with reality.

Jan


I hope this helps,
Kostas

On Wed, Aug 7, 2019 at 12:11 PM Jan Lukavský  wrote:


Hi all,

I have just come across a weird state of operators after restore from
checkpoint. After the restore, two operators that are connected (i.e.
operator A is input of operator B) ended up with watermark of operator A
being less than watermark of operator B. I don't know how to explain
this. Can it be normal or does it signal a bug somewhere? If I
understand Flink's checkpointing correctly, the checkpoint barrier flows
from one operator to another, so the watermark should be aligned.

I'm running a Beam pipeline on Flink 1.8.1.

Am I missing something?

Many thanks for comments,

Jan

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Fabian Hueske]

Congratulations Hequn!

Am Mi., 7. Aug. 2019 um 14:50 Uhr schrieb Robert Metzger <
rmetz...@apache.org>:

> Congratulations!
>
> On Wed, Aug 7, 2019 at 1:09 PM highfei2...@126.com 
> wrote:
>
> > Congrats Hequn!
> >
> > Best,
> > Jeff Yang
> >
> >
> >  Original Message 
> > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> > From: Piotr Nowojski
> > To: JingsongLee
> > CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun
> ,dev ,user
> >
> >
> > Congratulations :)
> >
> > On 7 Aug 2019, at 12:09, JingsongLee  wrote:
> >
> > Congrats Hequn!
> >
> > Best,
> > Jingsong Lee
> >
> > --
> > From:Biao Liu 
> > Send Time:2019年8月7日(星期三) 12:05
> > To:Zhu Zhu 
> > Cc:Zili Chen ; Jeff Zhang ; Paul
> > Lam ; jincheng sun ;
> dev
> > ; user 
> > Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
> >
> > Congrats Hequn!
> >
> > Thanks,
> > Biao /'bɪ.aʊ/
> >
> >
> >
> > On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu  wrote:
> > Congratulations to Hequn!
> >
> > Thanks,
> > Zhu Zhu
> >
> > Zili Chen  于2019年8月7日周三 下午5:16写道：
> > Congrats Hequn!
> >
> > Best,
> > tison.
> >
> >
> > Jeff Zhang  于2019年8月7日周三 下午5:14写道：
> > Congrats Hequn!
> >
> > Paul Lam  于2019年8月7日周三 下午5:08写道：
> > Congrats Hequn! Well deserved!
> >
> > Best,
> > Paul Lam
> >
> > 在 2019年8月7日，16:28，jincheng sun  写道：
> >
> > Hi everyone,
> >
> > I'm very happy to announce that Hequn accepted the offer of the Flink PMC
> > to become a committer of the Flink project.
> >
> > Hequn has been contributing to Flink for many years, mainly working on
> > SQL/Table API features. He's also frequently helping out on the user
> > mailing lists and helping check/vote the release.
> >
> > Congratulations Hequn!
> >
> > Best, Jincheng
> > (on behalf of the Flink PMC)
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
> >
> >
>

Re: [ANNOUNCE] Progress updates for Apache Flink 1.9.0 release

[Tzu-Li (Gordon) Tai]

Hi all,

According to the 1.9.x burndown board [1], we're approaching a releasable
state for 1.9.0.
Thanks to everyone who participated in the work for fixing the blockers so
far, especially Till who has been coordinating a lot of the efforts.

Below is a summary of the current state of the few remaining blockers:

Pending bugs to be fixed -

   -
*FLINK-13159 - Restored PojoSerializer not using correct classloader for
   deserialization [2] STATUS: *PR opened and reviewed, waiting for Travis
   run before merging
   *NOTES:* this bug is not specific to 1.9.0 only; will be backported to
   1.8.x as well. It was made a blocker for 1.9.0 as well since the fix is
   relatively low-effort.
   -
*FLINK-13593 - Prevent failing the wrong execution attempt in
   CheckpointFailureManager [3] STATUS:* PR opened, some final passes of
   reviews pending

Additional tests to be added -

   - *FLINK-13441 - Add batch sql E2E test which runs with fewer slots than
   parallelism to test the newly introduced batch scheduling modes [4]*
   *STATUS:* PR opened and being reviewed.
   *NOTES:* The TPC-H E2E test has also been modified to cover this
   scenario.

Unstable tests:

   -
*FLINK-13489 - Heavy deployment E2E test fails on Travis (agreed to make
   this a non-blocker) [5] STATUS:* The cause of this isn't a critical
   issue, and it is agreed that this would not be a blocker for the release.
   -
*FLINK-13581 - BatchFineGrainedRecoveryITCase failed on Travis [6] STATUS: *PR
   opened and review is in progress
   -
*FLINK-13527 - Unstable KafkaProducerExactlyOnceITCase fails [7]
STATUS:* Blocked
   by FLINK-13593 (blocker issue mentioned above)
   *NOTES: *Yu Li already mentioned that with the fix in FLINK-13593, this
   test no longer fails
   -
*FLINK-13607 - TCP-H E2E tests fails on Travis [8] STATUS:* Awaiting final
   confirmations on whether or not the instability still exists.
   *NOTES:* Kurt is also running a variation of this with multiple TMs and
   high parallelism (10-20 TMs, ~1000 DoP) internally in Alibaba.

So, from the looks of things, it should be safe to say that we can aim for
creating the first voting RC (RC2) by the end of this week (August 9th)!
An official voting thread for RC2 will be established once it is ready.

Cheers,
Gordon

[1] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=328
[2] https://issues.apache.org/jira/browse/FLINK-13159
[3] https://issues.apache.org/jira/browse/FLINK-13593
[4] https://issues.apache.org/jira/browse/FLINK-13441
[5] https://issues.apache.org/jira/browse/FLINK-13489
[6] https://issues.apache.org/jira/browse/FLINK-13581
[7] https://issues.apache.org/jira/browse/FLINK-13527
[8] https://issues.apache.org/jira/browse/FLINK-13607

On Thu, Aug 1, 2019 at 3:03 PM Kurt Young  wrote:

> Update: RC1 for 1.9.0 has been created. Please see [1] for the preview
> source / binary releases and Maven artifacts.
>
> Best,
> Kurt
>
> [1]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/PREVIEW-Apache-Flink-1-9-0-release-candidate-1-td31233.html
>
>
> On Tue, Jul 30, 2019 at 2:36 PM Tzu-Li (Gordon) Tai 
> wrote:
>
> > Hi Biao,
> >
> > Thanks for working on FLINK-9900. The ticket is already assigned to you
> > now.
> >
> > Cheers,
> > Gordon
> >
> > On Tue, Jul 30, 2019 at 2:31 PM Biao Liu  wrote:
> >
> > > Hi Gordon,
> > >
> > > Thanks for updating progress.
> > >
> > > Currently I'm working on FLINK-9900. I need a committer to assign the
> > > ticket to me.
> > >
> > > Tzu-Li (Gordon) Tai 于2019年7月30日 周二13:01写道：
> > >
> > > > Hi all,
> > > >
> > > > There are quite a few instabilities in our builds right now (master +
> > > > release-1.9), some of which are directed or suspiciously related to
> the
> > > 1.9
> > > > release.
> > > >
> > > > I'll categorize the instabilities into ones which we were already
> > > tracking
> > > > in the 1.9 Burndown Kanban board [1] prior to this email, and which
> > ones
> > > > seems to be new or were not monitored so that we draw additional
> > > attention
> > > > to them:
> > > >
> > > > *Instabilities that were already being tracked*
> > > >
> > > > - FLINK-13242: StandaloneResourceManagerTest.testStartupPeriod fails
> on
> > > > Travis [2]
> > > > A fix for this is coming with FLINK-13408 (Schedule
> > > > StandaloneResourceManager.setFailUnfulfillableRequest whenever the
> > > > leadership is acquired) [3]
> > > >
> > > > *New discovered instabilities that we should also start monitoring*
> > > >
> > > > - FLINK-13484: ConnectedComponents E2E fails with
> > > > ResourceNotAvailableException [4]
> > > > - FLINK-13487:
> > > >
> TaskExecutorPartitionLifecycleTest.testPartitionReleaseAfterReleaseCall
> > > > failed on Travis [5]. FLINK-13476 (Partitions not being properly
> > released
> > > > on cancel) could be the cause [6].
> > > > - FLINK-13488: flink-python fails to build on Travis due to Python
> 3.3
> > > > install failure [7]
> > > > - FLINK-13489: Heavy deployment E2E fails quite consistently on
> Travis

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Robert Metzger]

Congratulations!

On Wed, Aug 7, 2019 at 1:09 PM highfei2...@126.com 
wrote:

> Congrats Hequn!
>
> Best,
> Jeff Yang
>
>
>  Original Message 
> Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> From: Piotr Nowojski
> To: JingsongLee
> CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun ,dev 
> ,user
>
>
> Congratulations :)
>
> On 7 Aug 2019, at 12:09, JingsongLee  wrote:
>
> Congrats Hequn!
>
> Best,
> Jingsong Lee
>
> --
> From:Biao Liu 
> Send Time:2019年8月7日(星期三) 12:05
> To:Zhu Zhu 
> Cc:Zili Chen ; Jeff Zhang ; Paul
> Lam ; jincheng sun ; dev
> ; user 
> Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
>
> Congrats Hequn!
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu  wrote:
> Congratulations to Hequn!
>
> Thanks,
> Zhu Zhu
>
> Zili Chen  于2019年8月7日周三 下午5:16写道：
> Congrats Hequn!
>
> Best,
> tison.
>
>
> Jeff Zhang  于2019年8月7日周三 下午5:14写道：
> Congrats Hequn!
>
> Paul Lam  于2019年8月7日周三 下午5:08写道：
> Congrats Hequn! Well deserved!
>
> Best,
> Paul Lam
>
> 在 2019年8月7日，16:28，jincheng sun  写道：
>
> Hi everyone,
>
> I'm very happy to announce that Hequn accepted the offer of the Flink PMC
> to become a committer of the Flink project.
>
> Hequn has been contributing to Flink for many years, mainly working on
> SQL/Table API features. He's also frequently helping out on the user
> mailing lists and helping check/vote the release.
>
> Congratulations Hequn!
>
> Best, Jincheng
> (on behalf of the Flink PMC)
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
>

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Wei Zhong]

Congratulations, Hequn!

Best,
Wei Zhong

> 在 2019年8月7日，17:15，Zili Chen  写道：
> 
> Congrats Hequn!
> 
> Best,
> tison.
> 
> 
> Jeff Zhang mailto:zjf...@gmail.com>> 于2019年8月7日周三 下午5:14写道：
> Congrats Hequn!
> 
> Paul Lam mailto:paullin3...@gmail.com>> 于2019年8月7日周三 
> 下午5:08写道：
> Congrats Hequn! Well deserved!
> 
> Best,
> Paul Lam
> 
>> 在 2019年8月7日，16:28，jincheng sun > > 写道：
>> 
>> Hi everyone,
>> 
>> I'm very happy to announce that Hequn accepted the offer of the Flink PMC to 
>> become a committer of the Flink project.
>> 
>> Hequn has been contributing to Flink for many years, mainly working on 
>> SQL/Table API features. He's also frequently helping out on the user mailing 
>> lists and helping check/vote the release.
>> 
>> Congratulations Hequn!
>> 
>> Best, Jincheng 
>> (on behalf of the Flink PMC)
> 
> 
> 
> -- 
> Best Regards
> 
> Jeff Zhang

[jira] [Created] (FLINK-13635) Unexpectedly interrupted in AsyncFunction#timeout

[Biao Liu (JIRA)]

Biao Liu created FLINK-13635:


 Summary: Unexpectedly interrupted in AsyncFunction#timeout
 Key: FLINK-13635
 URL: https://issues.apache.org/jira/browse/FLINK-13635
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Affects Versions: 1.9.0
Reporter: Biao Liu
 Fix For: 1.10.0


Currently the way of handling {{AsyncFunction#timeout}} is a bit weird in 
{{AsyncWaitOperator#processElement}}.
 
There are two methods in {{AsyncFunction}}, {{asyncInvoke}} and {{timeout}}. 
The {{asyncInvoke}} is executed in task thread, while the {{timeout}} is 
executed in system time service. When the {{asyncInvoke}} finished, it might 
complete the {{ResultFuture}}. Then it cancels the registered timer of 
{{timeout}}. However there is no any synchronization between the 
{{asyncFunction}}, {{timeout}} and the cancelation. Moreover this cancelation 
is with interruption enabled.

The {{timeout}} must be implemented very carefully. Because when the 
{{timeout}} is executing, there might be an interruption triggered at the same 
time (due to a completion of {{ResultFuture}}). That means the {{timeout}} must 
handle {{InterruptedException}} well everywhere if there is any operation 
reacting with this exception.

My proposals are described below.
1. It should be written down in document that the {{asyncInvoke}} and 
{{timeout}} might be invoked at the same time.
2. This interruption of {{timeout}} should be avoided. There should be a 
synchronization between cancelation and {{timeout}}. If the {{timeout}} is 
executing, the cancelation should be avoided. If the cancelation has been 
invoked, this {{timeout}} should not be invoked anymore. Or we could simply 
cancel the timer without an interruption.

CC [~kkl0u], [~till.rohrmann]



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: Suspicious watermark of operators after restore from checkpoint

[Kostas Kloudas]

Hi Jan,

After looking at the code, my point 1) is false for *intermediate* tasks
and if you are
using a watermark assigner. This means that in these cases, Flink checks
that the
"next" watermark is greater than the "previous" one.

But if your operator A is a source and you emit watermarks from the source,
then
it can happen that your watermark appears to go backwards on operator A,
but
operator B does the "correction" by discarding smaller watermarks. That can
explain
your observation.

Cheers,
Kostas

On Wed, Aug 7, 2019 at 2:30 PM Jan Lukavský  wrote:

> Hi Kostas,
>
> thanks for reaction, comments inline.
>
> On 8/7/19 1:59 PM, Kostas Kloudas wrote:
> > Hi Jan,
> >
> > Two pointers that may help you explain the behavior are the following:
> >
> > 1) If you have a custom watermark generator, I do not think that Flink
> > checks if it emits only monotonically
> > increasing watermarks. This is the responsibility of the generator
> itself.
> > This means that although you operator A
> > is topologically before operator B, operator A may have a smaller
> watermark
> > if your watermark generator allows so.
> I do generate watermarks by custom source, but I believe that the
> generated sequence is monotonic. But still, I'm not sure, that even if
> it was the case, that the generated watermark actually decreases, would
> that mean, that downstream operator after source (operator A) would
> actually "go back in time"?
> >
> > 2) Flink currently does not checkpoint the last seen watermark (
> > https://issues.apache.org/jira/browse/FLINK-5601).
> > This means that after restoring, your (event) time is assumed to be
> > Long.Min until the first new watermark comes.
> > So if you observed late data not being late anymore or sth similar, then
> it
> > may not be that the two operators have
> > different watermarks but that after restoring event time rolls back to
> the
> > "beginning of time".
>
> I actually didn't observe any wrong or unexpected behavior, exceptions
> or wrong outputs. I just noticed this on Flink's WebUI and it looked
> strange to me. Could it be just that the WebUI showed older watermark
> for operator A? Strange was, that the watermarks were my screen long
> enough to take a screenshot (so at least say 10 seconds displaying
> watermark of operator A less than the one of operator B). Even if
> watermarks are not checkpointed, would it still be possible for
> watermark of operator B to be actually greater? I'm still confused of
> how this could happen, because (in my understanding) output watermark of
> operator A should be greater or equal to input watermark of B (because
> it takes minimum of inputs).
>
> Sorry if I'm too digging into this, but I don't like things I cannot
> explain, as they might point out to some bugs somewhere. :-) Or that my
> mental model it not aligned with reality.
>
> Jan
>
> >
> > I hope this helps,
> > Kostas
> >
> > On Wed, Aug 7, 2019 at 12:11 PM Jan Lukavský  wrote:
> >
> >> Hi all,
> >>
> >> I have just come across a weird state of operators after restore from
> >> checkpoint. After the restore, two operators that are connected (i.e.
> >> operator A is input of operator B) ended up with watermark of operator A
> >> being less than watermark of operator B. I don't know how to explain
> >> this. Can it be normal or does it signal a bug somewhere? If I
> >> understand Flink's checkpointing correctly, the checkpoint barrier flows
> >> from one operator to another, so the watermark should be aligned.
> >>
> >> I'm running a Beam pipeline on Flink 1.8.1.
> >>
> >> Am I missing something?
> >>
> >> Many thanks for comments,
> >>
> >>Jan
> >>
> >>
>

[jira] [Created] (FLINK-13634) StreamingFileSink - allow bulkwriter to compress data

[Joao Boto (JIRA)]

Joao Boto created FLINK-13634:
-

 Summary: StreamingFileSink - allow bulkwriter to compress data
 Key: FLINK-13634
 URL: https://issues.apache.org/jira/browse/FLINK-13634
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / FileSystem
Reporter: Joao Boto


I have developed a CompressFileWriter base on BulkWriter to compress data

but I dont know were to put this code.. inside filesystem or as flink-format 
module..

other question is that I used org.apache.commons.compress.compressors instead 
of hadoop compressor

[~kkl0u] could you guide me

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: Suspicious watermark of operators after restore from checkpoint

[Jan Lukavský]

Hi Kostas,

thanks for reaction, comments inline.

On 8/7/19 1:59 PM, Kostas Kloudas wrote:

Hi Jan,

Two pointers that may help you explain the behavior are the following:

1) If you have a custom watermark generator, I do not think that Flink
checks if it emits only monotonically
increasing watermarks. This is the responsibility of the generator itself.
This means that although you operator A
is topologically before operator B, operator A may have a smaller watermark
if your watermark generator allows so.
I do generate watermarks by custom source, but I believe that the 
generated sequence is monotonic. But still, I'm not sure, that even if 
it was the case, that the generated watermark actually decreases, would 
that mean, that downstream operator after source (operator A) would 
actually "go back in time"?


2) Flink currently does not checkpoint the last seen watermark (
https://issues.apache.org/jira/browse/FLINK-5601).
This means that after restoring, your (event) time is assumed to be
Long.Min until the first new watermark comes.
So if you observed late data not being late anymore or sth similar, then it
may not be that the two operators have
different watermarks but that after restoring event time rolls back to the
"beginning of time".


I actually didn't observe any wrong or unexpected behavior, exceptions 
or wrong outputs. I just noticed this on Flink's WebUI and it looked 
strange to me. Could it be just that the WebUI showed older watermark 
for operator A? Strange was, that the watermarks were my screen long 
enough to take a screenshot (so at least say 10 seconds displaying 
watermark of operator A less than the one of operator B). Even if 
watermarks are not checkpointed, would it still be possible for 
watermark of operator B to be actually greater? I'm still confused of 
how this could happen, because (in my understanding) output watermark of 
operator A should be greater or equal to input watermark of B (because 
it takes minimum of inputs).


Sorry if I'm too digging into this, but I don't like things I cannot 
explain, as they might point out to some bugs somewhere. :-) Or that my 
mental model it not aligned with reality.


Jan



I hope this helps,
Kostas

On Wed, Aug 7, 2019 at 12:11 PM Jan Lukavský  wrote:


Hi all,

I have just come across a weird state of operators after restore from
checkpoint. After the restore, two operators that are connected (i.e.
operator A is input of operator B) ended up with watermark of operator A
being less than watermark of operator B. I don't know how to explain
this. Can it be normal or does it signal a bug somewhere? If I
understand Flink's checkpointing correctly, the checkpoint barrier flows
from one operator to another, so the watermark should be aligned.

I'm running a Beam pipeline on Flink 1.8.1.

Am I missing something?

Many thanks for comments,

   Jan

Re: [DISCUSS] Adopting a Code Style and Quality Guide

[Robert Metzger]

Hi all,

I've now merged the guide to the Flink website:
https://flink.apache.org/contributing/code-style-and-quality-preamble.html

I'm looking forward to more pull requests enhancing the guide.


On Tue, Jul 23, 2019 at 7:22 PM Hugo Louro  wrote:

> Sounds good @stephan! Thanks.
>
> On Tue, Jul 23, 2019 at 10:15 AM Stephan Ewen  wrote:
>
> > @hugo For your suggestions, I would ask to start a separate discussion
> > thread.
> > I think this mail thread has converged towards merging the initial
> > suggestion as a starting point and refining it later based on new
> > discussions.
> >
> > Best,
> > Stephan
> >
> >
> > On Thu, Jun 27, 2019 at 10:48 PM Hugo Louro  wrote:
> >
> > > +1. Thanks for working on the guide. It's very thorough and a good
> > resource
> > > to learn good practices from.
> > >
> > > I would like use this thread as a placeholder for a couple of topics
> that
> > > may be deserving of further discussion on different threads:
> > >   - What's the best way to keep track of checkstyle version updates.
> For
> > > instance, currently there is a PR
> > >  proposing checkstyle to be
> > > updated because version 8.12 is no longer supported
> > >  - When classes import shaded dependencies, it is not trivial for
> > IntelliJ
> > > to download and associate sources and javadocs, which makes debugging
> and
> > > navigate the code base harder. I tried installing the version of the
> > > library using maven from the CLI, and associate the sources "manually"
> on
> > > IntelliJ, but it seems it does not work (perhaps IntelliJ issue). Does
> > > anyone know of a good solution? If so, we should added here
> > > <
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/flinkDev/ide_setup.html#intellij-idea
> > > >.
> > > I can volunteer for that if you tell me how to do it :)
> > > - did the community evaluate naming test methods according to these
> > > conventions  ?
> > >
> > > Thanks
> > >
> > > On Mon, Jun 24, 2019 at 11:38 AM Stephan Ewen 
> wrote:
> > >
> > > > I think it makes sense to also start individual [DISCUSS] threads
> about
> > > > extensions and follow-ups.
> > > > Various suggestions came up, partly as comments in the doc, some as
> > > > questions in other threads.
> > > >
> > > > Examples:
> > > >   - use of null in return types versus Optional versus
> > @Nullable/@Nonnull
> > > >   - initialization of collection sizes
> > > >   - logging
> > > >
> > > > I think these would be best discussed in separate threads each.
> > > > So, for contributors to whom these issues are dear, feel free to
> spawn
> > > > these additional threads.
> > > > (Bear in mind it is close to 1.9 feature freeze time, so please leave
> > > this
> > > > discussions a bit of time so that all community members have a chance
> > to
> > > > participate)
> > > >
> > > >
> > > >
> > > > On Mon, Jun 24, 2019 at 7:51 PM Stephan Ewen 
> wrote:
> > > >
> > > > > Thanks for the pointer David.
> > > > >
> > > > > I was not aware of this tool and I have no experience with its
> > > practical
> > > > > impact. For example I would be curious what the effect of this is
> for
> > > > > existing PRs, merge conflicts, etc.
> > > > >
> > > > > If you want, feel free to start another discuss thread on the
> > > > introduction
> > > > > of such a tool.
> > > > >
> > > > > On Sun, Jun 23, 2019 at 6:32 PM David Morávek 
> > wrote:
> > > > >
> > > > >> I love this kind of unification being pushed forward, the benefit
> it
> > > has
> > > > >> on
> > > > >> code reviews is enormous. Just my two cents, did you guys think
> > about
> > > > >> adopting an automatic code formatter for java, the same way as
> > Apache
> > > > Beam
> > > > >> did?
> > > > >>
> > > > >> It is super easy to use for contributors as they don't need to
> keep
> > > any
> > > > >> particular coding style in mind and they can only focus on
> > > functionality
> > > > >> they want to fix, and it's also great for reviewers, because they
> > only
> > > > see
> > > > >> the important changes. This also eliminates need for any special
> > > editor
> > > > /
> > > > >> checkstyle configs as the code formatting is part of the build
> > itself.
> > > > >>
> > > > >> The one Beam uses is https://github.com/diffplug/spotless with
> > > > >> GoogleJavaFormat, it may be worth to look into.
> > > > >>
> > > > >> Best,
> > > > >> David
> > > > >>
> > > > >> On Fri, Jun 21, 2019 at 4:40 PM Stephan Ewen 
> > > wrote:
> > > > >>
> > > > >> > Thanks, everyone, for the positive feedback :-)
> > > > >> >
> > > > >> > @Robert - It probably makes sense to break this down into
> various
> > > > pages,
> > > > >> > like PR, general code style guide, Java, component specific
> > guides,
> > > > >> > formats, etc.
> > > > >> >
> > > > >> > Best,
> > > > >> > Stephan
> > > > >> >
> > > > >> >
> > > > >> > On Fri, Jun 21, 2019 at 4:29 PM Robert Metzger <
> > rmetz...@apache.org
> > > >
> > > > >> 

Re: Suspicious watermark of operators after restore from checkpoint

[Kostas Kloudas]

Hi Jan,

Two pointers that may help you explain the behavior are the following:

1) If you have a custom watermark generator, I do not think that Flink
checks if it emits only monotonically
increasing watermarks. This is the responsibility of the generator itself.
This means that although you operator A
is topologically before operator B, operator A may have a smaller watermark
if your watermark generator allows so.

2) Flink currently does not checkpoint the last seen watermark (
https://issues.apache.org/jira/browse/FLINK-5601).
This means that after restoring, your (event) time is assumed to be
Long.Min until the first new watermark comes.
So if you observed late data not being late anymore or sth similar, then it
may not be that the two operators have
different watermarks but that after restoring event time rolls back to the
"beginning of time".

I hope this helps,
Kostas

On Wed, Aug 7, 2019 at 12:11 PM Jan Lukavský  wrote:

> Hi all,
>
> I have just come across a weird state of operators after restore from
> checkpoint. After the restore, two operators that are connected (i.e.
> operator A is input of operator B) ended up with watermark of operator A
> being less than watermark of operator B. I don't know how to explain
> this. Can it be normal or does it signal a bug somewhere? If I
> understand Flink's checkpointing correctly, the checkpoint barrier flows
> from one operator to another, so the watermark should be aligned.
>
> I'm running a Beam pipeline on Flink 1.8.1.
>
> Am I missing something?
>
> Many thanks for comments,
>
>   Jan
>
>

[DISCUSS] Repository split

[Chesnay Schepler]

Hello everyone,

The Flink project sees an ever-increasing amount of dev activity, both 
in terms of reworked and new features.


This is of course an excellent situation to be in, but we are getting to 
a point where the associate downsides are becoming increasingly troublesome.


The ever increasing build times, in addition to unstable tests, 
significantly slow down the develoment process.
Additionally, pull requests for smaller features frequently slip through 
the crasks as they are being buried under a mountain of other pull requests.


As a result I'd like to start a discussion on splitting the Flink 
repository.


In this mail I will outline the core idea, and what problems I currently 
envision.


I'd specifically like to encourage those who were part of similar 
initiatives in other projects to share the experiences and ideas.



   General Idea

For starters, the idea is to create a new repository for "flink-connectors".
For the remainder of this mail, the current Flink repository is referred 
to as "flink-main".


There are also other candidates that we could discuss in the future, 
like flink-libraries (the next top-priority repo to ease flink-ml 
development), metric reporters, filesystems and flink-formats.


Moving out flink-connectors provides the most benefits, as we straight 
away save at-least an hour of testing time, and not being included in 
the binary distribution simplifies a few things.



   Problems to solve

To make this a reality there's a number of questions we have to discuss; 
some in the short-term, others in the long-term.


1) Git history

   We have to decide whether we want to rewrite the history of sub
   repositories to only contain diffs/commits related to this part of
   Flink, or whether we just fork from some commit in flink-main and
   add a commit to the connector repo that "transforms" it from
   flink-main to flink-connectors (i.e., remove everything unrelated to
   connectors + update module structure etc.).

   The latter option would have the advantage that our commit book
   keeping in JIRA would still be correct, but it would create a
   significant divide between the current and past state of the repository.

2) Maven

   We should look into whether there's a way to share dependency/plugin
   configurations and similar, so we don't have to keep them in sync
   manually across multiple repositories.

   A new parent Flink pom that all repositories define as their parent
   could work; this would imply splicing out part of the current room
   pom.xml.

3) Documentation

   Splitting the repository realistically also implies splitting the
   documentation source files (At the beginning we can get by with
   having it still in flink-main).
   We could just move the relevant files to the respective repository
   (while maintaining the directory structure), and merge them when
   building the docs.

   We also have to look at how we can handle java-/scaladocs; e.g.
   whether it is possible to aggregate them across projects.

4) CI (end-to-end tests)

   The very basic question we have to answer is whether we want E2E
   tests in the sub repositories. If so, we need to find a way to share
   e2e-tooling.

5) Releases

   We have to discuss how our release process will look like. This may
   also have repercussions on how repositories may depend on each other
   (SNAPSHOT vs LATEST). Note that this should be discussed for each
   repo separately.

   The current options I see are the following:

   a) Single release

   Release all repositories at once as a single product.

   The source release would be a collection of repositories, like
   flink/
   |--flink-main/
   |--flink-core/
   |--flink-runtime/
   ...
   |--flink-connectors/
   ...
   |--flink-.../
   ...

   This option requires a SNAPSHOT dependency between Flink
   repositories, but it is pretty much how things work at the moment.

   b) Synced releases

   Similar to a), except that each repository gets their own source
   release that they may released independent of other repositories.
   For a given release cycle each repo would produce exactly one
   release.

   This option requires a SNAPSHOT dependency between Flink
   repositories. Once any repositories has created an RC or
   finished it's release, release-branches in other repos can
   switch to that version.

   This approach is a tad more flexible than a), but requires more
   coordination between the repos.

   c) Separate releases

   Just like we handle flink-shaded; entirely separate release
   cycles; some repositories may have more releases in a given time
   period than others.

   This option implies a LATEST dependency between Flink repositories.

   Note that hybrid approaches would also make sense, like doing b) for
   major versions and c) for bugfix releases.

   For something like 

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Piotr Nowojski]

Congratulations :)

> On 7 Aug 2019, at 12:09, JingsongLee  wrote:
> 
> Congrats Hequn!
> 
> Best,
> Jingsong Lee
> 
> --
> From:Biao Liu 
> Send Time:2019年8月7日(星期三) 12:05
> To:Zhu Zhu 
> Cc:Zili Chen ; Jeff Zhang ; Paul Lam 
> ; jincheng sun ; dev 
> ; user 
> Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
> 
> Congrats Hequn!
> 
> Thanks,
> Biao /'bɪ.aʊ/
> 
> 
> 
> On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu  > wrote:
> Congratulations to Hequn!
> 
> Thanks,
> Zhu Zhu
> 
> Zili Chen mailto:wander4...@gmail.com>> 于2019年8月7日周三 
> 下午5:16写道：
> Congrats Hequn!
> 
> Best,
> tison.
> 
> 
> Jeff Zhang mailto:zjf...@gmail.com>> 于2019年8月7日周三 下午5:14写道：
> Congrats Hequn!
> 
> Paul Lam mailto:paullin3...@gmail.com>> 于2019年8月7日周三 
> 下午5:08写道：
> Congrats Hequn! Well deserved!
> 
> Best,
> Paul Lam
> 
> 在 2019年8月7日，16:28，jincheng sun  > 写道：
> 
> Hi everyone,
> 
> I'm very happy to announce that Hequn accepted the offer of the Flink PMC to 
> become a committer of the Flink project.
> 
> Hequn has been contributing to Flink for many years, mainly working on 
> SQL/Table API features. He's also frequently helping out on the user mailing 
> lists and helping check/vote the release.
> 
> Congratulations Hequn!
> 
> Best, Jincheng 
> (on behalf of the Flink PMC)
> 
> 
> 
> -- 
> Best Regards
> 
> Jeff Zhang

[jira] [Created] (FLINK-13633) Move submittedJobGraph and completedCheckpoint to cluster-id subdirectory of high-availability storage

[Yang Wang (JIRA)]

Yang Wang created FLINK-13633:
-

 Summary: Move submittedJobGraph and completedCheckpoint to 
cluster-id subdirectory of  high-availability storage
 Key: FLINK-13633
 URL: https://issues.apache.org/jira/browse/FLINK-13633
 Project: Flink
  Issue Type: New Feature
Reporter: Yang Wang


Currently, if we enable the high-availability, the ha storage directory 
structure is stored as below. The submittedJobGraph and completedCheckpoint are 
directly stored under the ha storage path. It is reasonable when the flink 
cluster finished normally. However, when the Yarn application is failed or 
killed, the submittedJobGraph and completedCheckpoint will exist there forever. 
Even we could not know which flink cluster(Yarn application) they belongs to. 
So i suggest to move them into application subdirectory. Some external tools 
could be used to clean up these residual files.

Also, we need to do best effort clean-up before the flink cluster finishes. 

 

Current ha storage directory structure
{code:java}
└── /tmp/flink/ha
    ├── submittedJobGraph
    ├── completedCheckpoint
    ├── application__
    │   ├── blob{code}
 

The new ha storage directory structure
{code:java}
└── /tmp/flink/ha
    ├── application__
    │   ├── blob
    │   ├── submittedJobGraph
    │   ├── completedCheckpoint
{code}
 

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Suspicious watermark of operators after restore from checkpoint

[Jan Lukavský]

Hi all,

I have just come across a weird state of operators after restore from 
checkpoint. After the restore, two operators that are connected (i.e. 
operator A is input of operator B) ended up with watermark of operator A 
being less than watermark of operator B. I don't know how to explain 
this. Can it be normal or does it signal a bug somewhere? If I 
understand Flink's checkpointing correctly, the checkpoint barrier flows 
from one operator to another, so the watermark should be aligned.


I'm running a Beam pipeline on Flink 1.8.1.

Am I missing something?

Many thanks for comments,

 Jan

Re: [ANNOUNCE] Hequn becomes a Flink committer

[JingsongLee]

Congrats Hequn!

Best,
Jingsong Lee


--
From:Biao Liu 
Send Time:2019年8月7日(星期三) 12:05
To:Zhu Zhu 
Cc:Zili Chen ; Jeff Zhang ; Paul Lam 
; jincheng sun ; dev 
; user 
Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer

Congrats Hequn!

Thanks,
Biao /'bɪ.aʊ/



On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu  wrote:

Congratulations to Hequn!

Thanks,
Zhu Zhu
Zili Chen  于2019年8月7日周三 下午5:16写道：
Congrats Hequn!

Best,
tison.

Jeff Zhang  于2019年8月7日周三 下午5:14写道：
Congrats Hequn!

Paul Lam  于2019年8月7日周三 下午5:08写道：
Congrats Hequn! Well deserved!
Best,
Paul Lam 

在 2019年8月7日，16:28，jincheng sun  写道：
Hi everyone,

I'm very happy to announce that Hequn accepted the offer of the Flink PMC to 
become a committer of the Flink project.

Hequn has been contributing to Flink for many years, mainly working on 
SQL/Table API features. He's also frequently helping out on the user mailing 
lists and helping check/vote the release.

Congratulations Hequn!

Best, Jincheng 
(on behalf of the Flink PMC)



-- 
Best Regards

Jeff Zhang

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Biao Liu]

Congrats Hequn!

Thanks,
Biao /'bɪ.aʊ/



On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu  wrote:

> Congratulations to Hequn!
>
> Thanks,
> Zhu Zhu
>
> Zili Chen  于2019年8月7日周三 下午5:16写道：
>
>> Congrats Hequn!
>>
>> Best,
>> tison.
>>
>>
>> Jeff Zhang  于2019年8月7日周三 下午5:14写道：
>>
>>> Congrats Hequn!
>>>
>>> Paul Lam  于2019年8月7日周三 下午5:08写道：
>>>
 Congrats Hequn! Well deserved!

 Best,
 Paul Lam

 在 2019年8月7日，16:28，jincheng sun  写道：

 Hi everyone,

 I'm very happy to announce that Hequn accepted the offer of the Flink
 PMC to become a committer of the Flink project.

 Hequn has been contributing to Flink for many years, mainly working on
 SQL/Table API features. He's also frequently helping out on the user
 mailing lists and helping check/vote the release.

 Congratulations Hequn!

 Best, Jincheng
 (on behalf of the Flink PMC)



>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>

[jira] [Created] (FLINK-13632) Update TypeSerializerUpgradeTestBase to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13632:
-

 Summary: Update TypeSerializerUpgradeTestBase to restore from 1.9 
savepoint
 Key: FLINK-13632
 URL: https://issues.apache.org/jira/browse/FLINK-13632
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{TypeSerializerUpgradeTestBase}} to restore from 1.9 savepoint once 
FLINK-11767 has been merged.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Zhu Zhu]

Congratulations to Hequn!

Thanks,
Zhu Zhu

Zili Chen  于2019年8月7日周三 下午5:16写道：

> Congrats Hequn!
>
> Best,
> tison.
>
>
> Jeff Zhang  于2019年8月7日周三 下午5:14写道：
>
>> Congrats Hequn!
>>
>> Paul Lam  于2019年8月7日周三 下午5:08写道：
>>
>>> Congrats Hequn! Well deserved!
>>>
>>> Best,
>>> Paul Lam
>>>
>>> 在 2019年8月7日，16:28，jincheng sun  写道：
>>>
>>> Hi everyone,
>>>
>>> I'm very happy to announce that Hequn accepted the offer of the Flink
>>> PMC to become a committer of the Flink project.
>>>
>>> Hequn has been contributing to Flink for many years, mainly working on
>>> SQL/Table API features. He's also frequently helping out on the user
>>> mailing lists and helping check/vote the release.
>>>
>>> Congratulations Hequn!
>>>
>>> Best, Jincheng
>>> (on behalf of the Flink PMC)
>>>
>>>
>>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>

[jira] [Created] (FLINK-13631) Update FlinkKinesisConsumerMigrationTest to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13631:
-

 Summary: Update FlinkKinesisConsumerMigrationTest to restore from 
1.9 savepoint
 Key: FLINK-13631
 URL: https://issues.apache.org/jira/browse/FLINK-13631
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Kinesis, Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{FlinkKinesisConsumerMigrationTest}} to restore from 1.9 savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13630) StreamTableEnvironment#toAppendStream without QueryConfig overrides values set on TableConfig

[Dawid Wysakowicz (JIRA)]

Dawid Wysakowicz created FLINK-13630:


 Summary: StreamTableEnvironment#toAppendStream without QueryConfig 
overrides values set on TableConfig
 Key: FLINK-13630
 URL: https://issues.apache.org/jira/browse/FLINK-13630
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Affects Versions: 1.9.0
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.9.0


In a sequence like this:
{code}
tEnv.getConfig().setIdleStateRetentionTime(minRetention, maxRetention);
Table table = tEnv.fromDataStream(elements);
tEnv.toAppendStream(table, Row.class);
{code}
the call to {{toAppendStream}} will override the value set in 
{{tEnv.getConfig.setIdleStateRetentionTime}}. I think we should change that 
behavior, as we want to drop the version with explicit {{QueryConfig}} and 
recommend the described setup.

Thanks [~hequn8128] for pointing it out.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13629) Update AbstractNonKeyedOperatorRestoreTestBase to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13629:
-

 Summary: Update AbstractNonKeyedOperatorRestoreTestBase to restore 
from 1.9 savepoint
 Key: FLINK-13629
 URL: https://issues.apache.org/jira/browse/FLINK-13629
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{AbstractNonKeyedOperatorRestoreTestBase}} to restore from 1.9 
savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13627) Update TypeSerializerSnapshotMigrationITCase to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13627:
-

 Summary: Update TypeSerializerSnapshotMigrationITCase to restore 
from 1.9 savepoint
 Key: FLINK-13627
 URL: https://issues.apache.org/jira/browse/FLINK-13627
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{TypeSerializerSnapshotMigrationITCase}} to restore from 1.9 savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13628) Update AbstractKeyedOperatorRestoreTestBase to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13628:
-

 Summary: Update AbstractKeyedOperatorRestoreTestBase to restore 
from 1.9 savepoint
 Key: FLINK-13628
 URL: https://issues.apache.org/jira/browse/FLINK-13628
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{AbstractKeyedOperatorRestoreTestBase}} to restore from 1.9 savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13625) Update StatefulJobSavepointMigrationITCase to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13625:
-

 Summary: Update StatefulJobSavepointMigrationITCase to restore 
from 1.9 savepoint
 Key: FLINK-13625
 URL: https://issues.apache.org/jira/browse/FLINK-13625
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{StatefulJobSavepointMigrationITCase}} to restore from 1.9 savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13626) Update StatefulJobWBroadcastStateMigrationITCase to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13626:
-

 Summary: Update StatefulJobWBroadcastStateMigrationITCase to 
restore from 1.9 savepoint
 Key: FLINK-13626
 URL: https://issues.apache.org/jira/browse/FLINK-13626
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{StatefulJobWBroadcastStateMigrationITCase}} to restore from 1.9 
savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13624) Update StatefulJobWBroadcastStateMigrationITCase to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13624:
-

 Summary: Update StatefulJobWBroadcastStateMigrationITCase to 
restore from 1.9 savepoint
 Key: FLINK-13624
 URL: https://issues.apache.org/jira/browse/FLINK-13624
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{StatefulJobWBroadcastStateMigrationITCase}} to restore from 1.9 
savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13623) Update StatefulJobSavepointMigrationITCase to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13623:
-

 Summary: Update StatefulJobSavepointMigrationITCase to restore 
from 1.9 savepoint
 Key: FLINK-13623
 URL: https://issues.apache.org/jira/browse/FLINK-13623
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{StatefulJobSavepointMigrationITCase}} to restore from 1.9 savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13622) Update WindowOperatorMigrationTest to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13622:
-

 Summary: Update WindowOperatorMigrationTest to restore from 1.9 
savepoint
 Key: FLINK-13622
 URL: https://issues.apache.org/jira/browse/FLINK-13622
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{WindowOperatorMigrationTest}} to restore from 1.9 savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13621) Update ContinuousFileProcessingMigrationTest to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13621:
-

 Summary: Update ContinuousFileProcessingMigrationTest to restore 
from 1.9 savepoint
 Key: FLINK-13621
 URL: https://issues.apache.org/jira/browse/FLINK-13621
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{ContinuousFileProcessingMigrationTest}} to restore from 1.9 savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13620) Update FlinkKafkaProducerMigrationTest to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13620:
-

 Summary: Update FlinkKafkaProducerMigrationTest to restore from 
1.9 savepoint
 Key: FLINK-13620
 URL: https://issues.apache.org/jira/browse/FLINK-13620
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{FlinkKafkaProducerMigrationTest}} to restore from 1.9 savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13618) Update FlinkKafkaConsumerBaseMigrationTest to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13618:
-

 Summary: Update FlinkKafkaConsumerBaseMigrationTest to restore 
from 1.9 savepoint
 Key: FLINK-13618
 URL: https://issues.apache.org/jira/browse/FLINK-13618
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{FlinkKafkaConsumerBaseMigrationTest}} to restore from 1.9 savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13619) Update FlinkKafkaProducerMigrationOperatorTest to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13619:
-

 Summary: Update FlinkKafkaProducerMigrationOperatorTest to restore 
from 1.9 savepoint
 Key: FLINK-13619
 URL: https://issues.apache.org/jira/browse/FLINK-13619
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{FlinkKafkaProducerMigrationOperatorTest}} to restore from 1.9 
savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13617) Update FlinkKafkaProducer011MigrationTest to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13617:
-

 Summary: Update FlinkKafkaProducer011MigrationTest to restore from 
1.9 savepoint
 Key: FLINK-13617
 URL: https://issues.apache.org/jira/browse/FLINK-13617
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{FlinkKafkaProducer011MigrationTest}} to restore from 1.9 savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13616) Update BucketingSinkMigrationTest to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13616:
-

 Summary: Update BucketingSinkMigrationTest to restore from 1.9 
savepoint
 Key: FLINK-13616
 URL: https://issues.apache.org/jira/browse/FLINK-13616
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{BucketingSinkMigrationTest}} to restore from 1.9 savepoint.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Kurt Young]

Congrats Hequn!

Best,
Kurt


On Wed, Aug 7, 2019 at 5:06 PM jincheng sun 
wrote:

> Hi everyone,
>
> I'm very happy to announce that Hequn accepted the offer of the Flink PMC
> to become a committer of the Flink project.
>
> Hequn has been contributing to Flink for many years, mainly working on
> SQL/Table API features. He's also frequently helping out on the user
> mailing lists and helping check/vote the release.
>
> Congratulations Hequn!
>
> Best, Jincheng
> (on behalf of the Flink PMC)
>

[jira] [Created] (FLINK-13614) Add MigrationVersion.v1_9

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13614:
-

 Summary: Add MigrationVersion.v1_9
 Key: FLINK-13614
 URL: https://issues.apache.org/jira/browse/FLINK-13614
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Add {{MigrationVersion.v1_9}}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13615) Update CEPMigrationTest to restore from 1.9 savepoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13615:
-

 Summary: Update CEPMigrationTest to restore from 1.9 savepoint
 Key: FLINK-13615
 URL: https://issues.apache.org/jira/browse/FLINK-13615
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Update {{CEPMigrationTest}} to restore from 1.9 savepoint



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] Flink project bylaws

[Becket Qin]

Thanks Stephan.

I think we have resolved all the comments on the wiki page. There are two
minor changes made to the bylaws since last week.
1. For 2/3 majority, the required attempts to reach out to binding voters
is reduced from 3 to 2. This helps shorten the voting process a little bit.
2. Clarified what is considered as the adoption of new codebase.

I think we almost reached consensus. I'll start a voting thread in two days
if there is no new concerns.

Thanks,

Jiangjie (Becket) Qin

On Mon, Aug 5, 2019 at 1:09 PM Stephan Ewen  wrote:

> I added a clarification to the table, clarifying that the current phrasing
> means that committers do not need another +1 for their commits.
>
> On Mon, Jul 29, 2019 at 2:11 PM Fabian Hueske  wrote:
>
> > Hi Becket,
> >
> > Thanks a lot for pushing this forward and addressing the feedback.
> > I'm very happy with the current state of the bylaws.
> > +1 to wait until Friday before starting a vote.
> >
> > Best, Fabian
> >
> > Am Mo., 29. Juli 2019 um 13:47 Uhr schrieb Becket Qin <
> > becket@gmail.com
> > >:
> >
> > > Hi Everyone,
> > >
> > > Thanks for all the discussion and feedback.
> > >
> > > It seems that we have almost reached consensus. I'll leave the
> discussion
> > > thread open until this Friday. If there is no more concerns raised,
> I'll
> > > start a voting thread after that.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Fri, Jul 26, 2019 at 6:49 PM Yu Li  wrote:
> > >
> > > > Hi Becket,
> > > >
> > > > Thanks for noticing and resolving my comment around PMC removal and
> ASF
> > > > rules of PMC membership change process, which you seem to neglect in
> > the
> > > > summary of updates (smile).
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Wed, 24 Jul 2019 at 04:32, Becket Qin 
> wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Thanks for all the feedback.
> > > > >
> > > > > It seems that there are a few concerns over the emeritus status
> > after 6
> > > > > months of inactivity. Given that the main purpose is just to make
> > sure
> > > > 2/3
> > > > > majority can pass and we sort of have a solution for that, I just
> > > updated
> > > > > the draft with the following changes:
> > > > >
> > > > > 1. Removed the inactivity term for emeritus committers / PMCs. A
> > > > committer
> > > > > / PMC will only be considered emeritus by their own claim.
> > > > > 2. Removed the approval process for reinstatement of the emeritus
> > > > > committers / PMCs. An emeritus committer / PMC will be reinstated
> > when
> > > > they
> > > > > send an email to the priv...@flink.apache.org.
> > > > > 3. Adde the term to ensure 2/3 majority voting is still doable when
> > > there
> > > > > are non-emeritus committers / PMCs who do not cast the vote.
> > > > >
> > > > > Please let me know if you have any further thoughts.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On Tue, Jul 23, 2019 at 10:18 AM Becket Qin 
> > > > wrote:
> > > > >
> > > > > > Hi Fabian,
> > > > > >
> > > > > > Thanks for the feedback.
> > > > > >
> > > > > > I agree that if we don't like emeritus committers / PMCs, we
> don't
> > > have
> > > > > to
> > > > > > do it. That said, emeritus status simply means whether an
> > individual
> > > is
> > > > > > active / inactive in the community. It does not make the merits
> > > earned
> > > > to
> > > > > > go away. For our purpose, emeritus status is mostly just a way to
> > > make
> > > > > 2/3
> > > > > > majority possible. As you noticed, since reaching out to binding
> > > voters
> > > > > who
> > > > > > have not voted can achieve the same goal, the emeritus status is
> > more
> > > > of
> > > > > an
> > > > > > optimization so we don't have to ping the inactive binding voters
> > > every
> > > > > > time and wait for long. However, given that 2/3 majority votings
> > are
> > > > > rare,
> > > > > > such communication cost is probably OK. So I think we can remove
> > that
> > > > > > emeritus part from the bylaws.
> > > > > >
> > > > > > 1) We should add to the requirements of the PMC that they need to
> > > make
> > > > > >> sure the project complies with brand issues and reports misuse
> of
> > > ASF
> > > > > >> brands.
> > > > > >
> > > > > > Good point. Added.
> > > > > >
> > > > > > 2) Do we want to restrict voting days to working days, i.e., a 3
> > day
> > > > vote
> > > > > >> that starts on Friday 11:00am ends on Wednesday 11:00am?
> > > > > >
> > > > > > This might be a little tricky because people are from countries
> in
> > > > > > different time zones and with different holidays, and so on. If
> we
> > > are
> > > > > > worrying about 3 days minimum length is not enough for those who
> > want
> > > > to
> > > > > > give feedback, we can make it 5 days.
> > > > > >
> > > > > > 3) Do we need a process do decide about removal of features (like
> > the
> > > > > >> DataSet API for instance or the legacy DataSet/DataStream Python
> > > > 

[jira] [Created] (FLINK-13613) Update migration tests for Flink 1.9

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13613:
-

 Summary: Update migration tests for Flink 1.9
 Key: FLINK-13613
 URL: https://issues.apache.org/jira/browse/FLINK-13613
 Project: Flink
  Issue Type: Task
  Components: Tests
Affects Versions: 1.10.0
Reporter: Till Rohrmann
 Fix For: 1.10.0


Once the Flink {{1.9.0}} release is out, we should update existing migration 
tests to cover restoring from {{1.9.0}} savepoints.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13610) Refactor HiveTableSource Test use sql query and remove HiveInputFormatTest

[Terry Wang (JIRA)]

Terry Wang created FLINK-13610:
--

 Summary: Refactor HiveTableSource Test use sql query and remove 
HiveInputFormatTest
 Key: FLINK-13610
 URL: https://issues.apache.org/jira/browse/FLINK-13610
 Project: Flink
  Issue Type: Test
  Components: Connectors / Hive
Affects Versions: 1.10.0
Reporter: Terry Wang


Since HiveTableSource is mainly used in sql query and now blink planner support 
run sql query, it's time that we change HiveTableSourceTest using sql way 
instead of table api.

HiveTableInputFormt is tested in HiveTableSourceTest and there exists 
redundancy in code,  this ticket also aims to move some test code from 
HiveInputFormatTest and remove this file.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Till Rohrmann]

Congrats Hequn and welcome onboard as a committer :-)

Cheers,
Till

On Wed, Aug 7, 2019 at 11:30 AM Becket Qin  wrote:

> Congrats, Hequn! Well deserved!
>
> On Wed, Aug 7, 2019 at 11:16 AM Zili Chen  wrote:
>
>> Congrats Hequn!
>>
>> Best,
>> tison.
>>
>>
>> Jeff Zhang  于2019年8月7日周三 下午5:14写道：
>>
>>> Congrats Hequn!
>>>
>>> Paul Lam  于2019年8月7日周三 下午5:08写道：
>>>
 Congrats Hequn! Well deserved!

 Best,
 Paul Lam

 在 2019年8月7日，16:28，jincheng sun  写道：

 Hi everyone,

 I'm very happy to announce that Hequn accepted the offer of the Flink
 PMC to become a committer of the Flink project.

 Hequn has been contributing to Flink for many years, mainly working on
 SQL/Table API features. He's also frequently helping out on the user
 mailing lists and helping check/vote the release.

 Congratulations Hequn!

 Best, Jincheng
 (on behalf of the Flink PMC)



>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Becket Qin]

Congrats, Hequn! Well deserved!

On Wed, Aug 7, 2019 at 11:16 AM Zili Chen  wrote:

> Congrats Hequn!
>
> Best,
> tison.
>
>
> Jeff Zhang  于2019年8月7日周三 下午5:14写道：
>
>> Congrats Hequn!
>>
>> Paul Lam  于2019年8月7日周三 下午5:08写道：
>>
>>> Congrats Hequn! Well deserved!
>>>
>>> Best,
>>> Paul Lam
>>>
>>> 在 2019年8月7日，16:28，jincheng sun  写道：
>>>
>>> Hi everyone,
>>>
>>> I'm very happy to announce that Hequn accepted the offer of the Flink
>>> PMC to become a committer of the Flink project.
>>>
>>> Hequn has been contributing to Flink for many years, mainly working on
>>> SQL/Table API features. He's also frequently helping out on the user
>>> mailing lists and helping check/vote the release.
>>>
>>> Congratulations Hequn!
>>>
>>> Best, Jincheng
>>> (on behalf of the Flink PMC)
>>>
>>>
>>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Zili Chen]

Congrats Hequn!

Best,
tison.


Jeff Zhang  于2019年8月7日周三 下午5:14写道：

> Congrats Hequn!
>
> Paul Lam  于2019年8月7日周三 下午5:08写道：
>
>> Congrats Hequn! Well deserved!
>>
>> Best,
>> Paul Lam
>>
>> 在 2019年8月7日，16:28，jincheng sun  写道：
>>
>> Hi everyone,
>>
>> I'm very happy to announce that Hequn accepted the offer of the Flink PMC
>> to become a committer of the Flink project.
>>
>> Hequn has been contributing to Flink for many years, mainly working on
>> SQL/Table API features. He's also frequently helping out on the user
>> mailing lists and helping check/vote the release.
>>
>> Congratulations Hequn!
>>
>> Best, Jincheng
>> (on behalf of the Flink PMC)
>>
>>
>>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Dian Fu]

Congratulations, Hequn! Well deserved!

> 在 2019年8月7日，下午5:13，Jeff Zhang  写道：
> 
> Congrats Hequn!
> 
> Paul Lam mailto:paullin3...@gmail.com>> 于2019年8月7日周三 
> 下午5:08写道：
> Congrats Hequn! Well deserved!
> 
> Best,
> Paul Lam
> 
>> 在 2019年8月7日，16:28，jincheng sun > > 写道：
>> 
>> Hi everyone,
>> 
>> I'm very happy to announce that Hequn accepted the offer of the Flink PMC to 
>> become a committer of the Flink project.
>> 
>> Hequn has been contributing to Flink for many years, mainly working on 
>> SQL/Table API features. He's also frequently helping out on the user mailing 
>> lists and helping check/vote the release.
>> 
>> Congratulations Hequn!
>> 
>> Best, Jincheng 
>> (on behalf of the Flink PMC)
> 
> 
> 
> -- 
> Best Regards
> 
> Jeff Zhang

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Jeff Zhang]

Congrats Hequn!

Paul Lam  于2019年8月7日周三 下午5:08写道：

> Congrats Hequn! Well deserved!
>
> Best,
> Paul Lam
>
> 在 2019年8月7日，16:28，jincheng sun  写道：
>
> Hi everyone,
>
> I'm very happy to announce that Hequn accepted the offer of the Flink PMC
> to become a committer of the Flink project.
>
> Hequn has been contributing to Flink for many years, mainly working on
> SQL/Table API features. He's also frequently helping out on the user
> mailing lists and helping check/vote the release.
>
> Congratulations Hequn!
>
> Best, Jincheng
> (on behalf of the Flink PMC)
>
>
>

-- 
Best Regards

Jeff Zhang

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Paul Lam]

Congrats Hequn! Well deserved!

Best,
Paul Lam

> 在 2019年8月7日，16:28，jincheng sun  写道：
> 
> Hi everyone,
> 
> I'm very happy to announce that Hequn accepted the offer of the Flink PMC to 
> become a committer of the Flink project.
> 
> Hequn has been contributing to Flink for many years, mainly working on 
> SQL/Table API features. He's also frequently helping out on the user mailing 
> lists and helping check/vote the release.
> 
> Congratulations Hequn!
> 
> Best, Jincheng 
> (on behalf of the Flink PMC)

[jira] [Created] (FLINK-13612) 高并发初始化FlinkKafkaProducer011时StateDescriptor加载报错NPE

[weiyunqing (JIRA)]

weiyunqing created FLINK-13612:
--

 Summary: 高并发初始化FlinkKafkaProducer011时StateDescriptor加载报错NPE
 Key: FLINK-13612
 URL: https://issues.apache.org/jira/browse/FLINK-13612
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.7.2, 1.6.4, 1.6.3, shaded-7.0
Reporter: weiyunqing
 Fix For: 1.7.2, 1.6.4, 1.6.3, shaded-7.0


org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011#NEXT_TRANSACTIONAL_ID_HINT_DESCRIPTOR

FlinkKafkaProducer011中的NEXT_TRANSACTIONAL_ID_HINT_DESCRIPTOR变量state使用了static修饰

在执行initializeSerializerUnlessSet方法的时候高并发情况下会出现NPE异常

 

java.lang.NullPointerException at 
org.apache.flink.api.common.state.StateDescriptor.initializeSerializerUnlessSet(StateDescriptor.java:264)
 at 
org.apache.flink.runtime.state.DefaultOperatorStateBackend.getListState(DefaultOperatorStateBackend.java:730)
 at 
org.apache.flink.runtime.state.DefaultOperatorStateBackend.getUnionListState(DefaultOperatorStateBackend.java:271)
 at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:837)
 at 
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
 at 
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
 at 
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
 at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281)
 at 
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:730)
 at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:295) 
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:720) at 
java.lang.Thread.run(Thread.java:748)



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13609) StreamingFileSink - reset part counter on bucket change

[Joao Boto (JIRA)]

Joao Boto created FLINK-13609:
-

 Summary: StreamingFileSink - reset part counter on bucket change
 Key: FLINK-13609
 URL: https://issues.apache.org/jira/browse/FLINK-13609
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / FileSystem
Reporter: Joao Boto


When writing to files using StreamingFileSink on bucket change we expect that 
partcounter will reset its counter to 0

as a example
 * using DateTimeBucketAssigner using ({color:#6a8759}/MM/dd/HH{color}) 
 * and ten files hour (for simplicity)

this will create the:
 * bucket 2019/08/07/00 with files partfile-0-0 to partfile-0-9
 * bucket 2019/08/07/01 with files partfile-0-10 to partfile-0-19
 * bucket 2019/08/07/02 with files partfile-0-20 to partfile-0-29

and we expect this:
 * bucket 2019/08/07/00 with files partfile-0-0 to partfile-0-9
 * bucket 2019/08/07/01 with files partfile-0-0 to partfile-0-9
 * bucket 2019/08/07/02 with files partfile-0-0 to partfile-0-9

 

[~kkl0u] i don't know if it's the expected behavior  (or this can be configured)



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13611) Introduce analyze statistic utility to generate table & column statistics

[godfrey he (JIRA)]

godfrey he created FLINK-13611:
--

 Summary: Introduce analyze statistic utility to generate table & 
column statistics
 Key: FLINK-13611
 URL: https://issues.apache.org/jira/browse/FLINK-13611
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / Planner
Reporter: godfrey he
 Fix For: 1.10.0


this issue aims to introduce a utility class to generate table & column 
statistics, the main steps include: 
1. generate sql, like {{select approx_count_distinct(a) as ndv, count(1) - 
count(a) as nullCount, avg(char_length(a)) as avgLen, max(char_lenght(a)) as 
maxLen, max(a) as maxValue, min(a) as minValue, ... from MyTable }}
2. execute the query
3. convert to the result to {{TableStats}} (maybe the source table is not a 
catalog table)
4. convert to {{TableStats}} to {{CatalogTableStatistics}} if needed

This issue does not involve DDL, however the DDL could use this utility class 
once it's supported.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[ANNOUNCE] Hequn becomes a Flink committer

[jincheng sun]

Hi everyone,

I'm very happy to announce that Hequn accepted the offer of the Flink PMC
to become a committer of the Flink project.

Hequn has been contributing to Flink for many years, mainly working on
SQL/Table API features. He's also frequently helping out on the user
mailing lists and helping check/vote the release.

Congratulations Hequn!

Best, Jincheng
(on behalf of the Flink PMC)

[jira] [Created] (FLINK-13608) Update upgrade compatibility table (docs/ops/upgrading.md) for 1.9.0

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13608:
-

 Summary: Update upgrade compatibility table 
(docs/ops/upgrading.md) for 1.9.0
 Key: FLINK-13608
 URL: https://issues.apache.org/jira/browse/FLINK-13608
 Project: Flink
  Issue Type: Task
  Components: Documentation
Affects Versions: 1.9.0
Reporter: Till Rohrmann
 Fix For: 1.9.0


Update upgrade compatibility table (docs/ops/upgrading.md) for 1.9.0



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13607) TPC-H end-to-end test (Blink planner) failed on Travis

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13607:
-

 Summary: TPC-H end-to-end test (Blink planner) failed on Travis
 Key: FLINK-13607
 URL: https://issues.apache.org/jira/browse/FLINK-13607
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API, Tests
Affects Versions: 1.9.0
Reporter: Till Rohrmann
 Fix For: 1.9.0


The {{TPC-H end-to-end test (Blink planner)}} fail consistently on Travis with

{code}
Generating test data...
Error: Could not find or load main class 
org.apache.flink.table.tpch.TpchDataGenerator
{code}

https://api.travis-ci.org/v3/job/568280203/log.txt
https://api.travis-ci.org/v3/job/568280209/log.txt
https://api.travis-ci.org/v3/job/568280215/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13606) PrometheusReporterEndToEndITCase.testReporter unstable on Travis

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-13606:
-

 Summary: PrometheusReporterEndToEndITCase.testReporter unstable on 
Travis
 Key: FLINK-13606
 URL: https://issues.apache.org/jira/browse/FLINK-13606
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Metrics
Affects Versions: 1.9.0
Reporter: Till Rohrmann
 Fix For: 1.9.0


The {{PrometheusReporterEndToEndITCase.testReporter}} is unstable on Travis. It 
fails with {{java.io.IOException: Process failed due to timeout.}}

https://api.travis-ci.org/v3/job/568280216/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13605) AsyncDataStreamITCase. testUnorderedWait failed on Travis

[Kostas Kloudas (JIRA)]

Kostas Kloudas created FLINK-13605:
--

 Summary: AsyncDataStreamITCase. testUnorderedWait failed on Travis
 Key: FLINK-13605
 URL: https://issues.apache.org/jira/browse/FLINK-13605
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.9.0
Reporter: Kostas Kloudas
Assignee: Kostas Kloudas


An instance of the failure can be found here 
https://api.travis-ci.org/v3/job/568291353/log.txt



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)