Re: [DISCUSS]Support Upsert mode for Streaming Non-window FlatAggregate

[Hequn Cheng]

Hi Kurt,

Thanks for your questions. Here are my thoughts.

> if I want to write such kind function, should I make sure that this
function is used with some keys?
The key information may not be used. We can also use RetractSink to emit
the table directly.

>  If I need a use case to calculate topn without key, should I write
another function or I can reuse previous one.
For the TopN example, you can reuse the previous function if you don't care
about the key information.

So, the key information is only an indicator(or a description), not an
operator, as Jincheng mentioned above.
We do not need to change the function logic and it will not add any other
aggregations.

BTW, we have three approaches in the document. Approach 1 defines keys on
API level as we think it's more common to define keys on Table.
While approach 3 defines keys in the TableAggregateFunction which is more
precise but it is not very clear for Table users. So, we should take all
these into consideration, and make the decision in this discussion thread.

You can take a look at the document and welcome any suggestions or other
better solutions.

Best, Hequn


On Mon, Jul 1, 2019 at 12:13 PM Kurt Young  wrote:

> Hi Jincheng,
>
> Thanks for the clarification. Take 'TopN' for example, if I want to write
> such kind function,
> should I make sure that this function is used with some keys? If I need a
> use case to calculate
> topn without key, should I write another function or I can reuse previous
> one.
>
> I'm not sure about the idea of this does not involve semantic changes. To
> me, it sounds like
> we are doing another nested aggregation inside the table
> which TableAggregateFunction emits.
>
> Maybe I'm not familiar with this function enough, hope you can help me to
> understand.
>
> Best,
> Kurt
>
>
> On Mon, Jul 1, 2019 at 11:59 AM jincheng sun 
> wrote:
>
> > Hi Kurt,
> >
> > Thanks for your questions, I am glad to share my thoughts here:
> >
> > My question is, will that effect the logic ofTableAggregateFunction user
> > > wrote? Should the user know that there will a key and make some changes
> > to
> > > this function?
> >
> >
> > No, the keys information depends on the implementation of the
> > TableAggregateFunction.
> > For example, for a `topN` user defined TableAggregateFunction, we can
> only
> > use the `keys` if the `topN` contains `rankid` in the output. You can
> > treat the
> > `keys` like an indicator.
> >
> > If not, how will framework deal with the output of user's
> > > TableAggregateFunction.  if user output multiple rows with the same
> key,
> > > should be latter one replace previous ones?
> >
> >
> > If a TableAggregateFunction outputs multiple rows with the same key, the
> > latter one should replace the previous one, either with upsert mode or
> > retract mode. i.e., Whether the user defines the Key or not, the Flink
> > framework should ensure the correctness of the semantics.
> >
> > At present, the problem we are discussing does not involve semantic
> > changes. The definition of keys is to support non-window flatAggregate on
> > upsert mode. (The upsert mode is already supported in the flink
> framework.
> > The current discussion only needs to inform the framework that the keys
> > information, which is the `withKeys` API we discussing.)
> >
> > Welcome any other feedbacks :)
> >
> > Best,
> > Jincheng
> >
> > Kurt Young  于2019年7月1日周一 上午9:23写道：
> >
> > > Hi,
> > >
> > > I have a question about the key information of TableAggregateFunction.
> > > IIUC, you need to define
> > > something like primary key or unique key in the result table of
> > > TableAggregateFunction, and also
> > > need a way to let user configure this through the API. My question is,
> > will
> > > that effect the logic of
> > > TableAggregateFunction user wrote? Should the user know that there
> will a
> > > key and make some changes
> > > to this function?
> > >
> > > If so, what's the semantic the user should learned. If not, how will
> > > framework deal with the output of user's
> > > TableAggregateFunction. For example, if user output multiple rows with
> > the
> > > same key, should be latter one
> > > replace previous ones?
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Mon, Jul 1, 2019 at 7:19 AM jincheng sun 
> > > wrote:
> > >
> > > > Hi hequn, Thanks for the reply! I think `withKeys` sol

Re: [ANNOUNCE] Apache Flink 1.8.1 released

[Hequn Cheng]

Thanks for being the release manager and the great work Jincheng!
Also thanks to Gorden and the community making this release possible!

Best, Hequn

On Wed, Jul 3, 2019 at 9:40 AM jincheng sun 
wrote:

> Hi,
>
> The Apache Flink community is very happy to announce the release of Apache
> Flink 1.8.1, which is the first bugfix release for the Apache Flink 1.8
> series.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Please check out the release blog post for an overview of the
> improvements for this bugfix release:
> https://flink.apache.org/news/2019/07/02/release-1.8.1.html
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12345164
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Great thanks to @Tzu-Li (Gordon) Tai  's offline
> kind help!
>
> Regards,
> Jincheng
>

Re: [VOTE] Migrate to sponsored Travis account

[Hequn Cheng]

+1.

And thanks a lot to Chesnay for pushing this.

Best, Hequn

On Thu, Jul 4, 2019 at 8:07 PM Chesnay Schepler  wrote:

> Note that the Flinkbot approach isn't that trivial either; we can't
> _just_ trigger builds for a branch in the apache repo, but would first
> have to clone the branch/pr into a separate repository (that is owned by
> the github account that the travis account would be tied to).
>
> One roadblock after the next showing up...
>
> On 04/07/2019 11:59, Chesnay Schepler wrote:
> > Small update with mostly bad news:
> >
> > INFRA doesn't know whether it is possible, and referred my to Travis
> > support.
> > They did point out that it could be problematic in regards to
> > read/write permissions for the repository.
> >
> > From my own findings /so far/ with a test repo/organization, it does
> > not appear possible to configure the Travis account used for a
> > specific repository.
> >
> > So yeah, if we go down this route we may have to pimp the Flinkbot to
> > trigger builds through the Travis REST API.
> >
> > On 04/07/2019 10:46, Chesnay Schepler wrote:
> >> I've raised a JIRA
> >> with INFRA to
> >> inquire whether it would be possible to switch to a different Travis
> >> account, and if so what steps would need to be taken.
> >> We need a proper confirmation from INFRA since we are not in full
> >> control of the flink repository (for example, we cannot access the
> >> settings page).
> >>
> >> If this is indeed possible, Ververica is willing sponsor a Travis
> >> account for the Flink project.
> >> This would provide us with more than enough resources than we need.
> >>
> >> Since this makes the project more reliant on resources provided by
> >> external companies I would like to vote on this.
> >>
> >> Please vote on this proposal, as follows:
> >> [ ] +1, Approve the migration to a Ververica-sponsored Travis
> >> account, provided that INFRA approves
> >> [ ] -1, Do not approach the migration to a Ververica-sponsored Travis
> >> account
> >>
> >> The vote will be open for at least 24h, and until we have
> >> confirmation from INFRA. The voting period may be shorter than the
> >> usual 3 days since our current is effectively not working.
> >>
> >> On 04/07/2019 06:51, Bowen Li wrote:
> >>> Re: > Are they using their own Travis CI pool, or did the switch to
> >>> an entirely different CI service?
> >>>
> >>> I reached out to Wes and Krisztián from Apache Arrow PMC. They are
> >>> currently moving away from ASF's Travis to their own in-house metal
> >>> machines at [1] with custom CI application at [2]. They've seen
> >>> significant improvement w.r.t both much higher performance and
> >>> basically no resource waiting time, "night-and-day" difference
> >>> quoting Wes.
> >>>
> >>> Re: > If we can just switch to our own Travis pool, just for our
> >>> project, then this might be something we can do fairly quickly?
> >>>
> >>> I believe so, according to [3] and [4]
> >>>
> >>>
> >>> [1] https://ci.ursalabs.org/ 
> >>> [2] https://github.com/ursa-labs/ursabot
> >>> [3]
> >>>
> https://docs.travis-ci.com/user/migrate/open-source-repository-migration
> >>>
> >>> [4]
> >>> https://docs.travis-ci.com/user/migrate/open-source-on-travis-ci-com
> >>>
> >>>
> >>>
> >>> On Wed, Jul 3, 2019 at 12:01 AM Chesnay Schepler  >>> > wrote:
> >>>
> >>> Are they using their own Travis CI pool, or did the switch to an
> >>> entirely different CI service?
> >>>
> >>> If we can just switch to our own Travis pool, just for our
> >>> project, then
> >>> this might be something we can do fairly quickly?
> >>>
> >>> On 03/07/2019 05:55, Bowen Li wrote:
> >>> > I responded in the INFRA ticket [1] that I believe they are
> >>> using a wrong
> >>> > metric against Flink and the total build time is a completely
> >>> different
> >>> > thing than guaranteed build capacity.
> >>> >
> >>> > My response:
> >>> >
> >>> > "As mentioned above, since I started to pay attention to Flink's
> >>> build
> >>> > queue a few tens of days ago, I'm in Seattle and I saw no build
> >>> was kicking
> >>> > off in PST daytime in weekdays for Flink. Our teammates in China
> >>> and Europe
> >>> > have also reported similar observations. So we need to evaluate
> >>> how the
> >>> > large total build time came from - if 1) your number and 2) our
> >>> > observations from three locations that cover pretty much a full
> >>> day, are
> >>> > all true, I **guess** one reason can be that - highly likely the
> >>> extra
> >>> > build time came from weekends when other Apache projects may be
> >>> idle and
> >>> > Flink just drains hard its congested queue.
> >>> >
> >>> > Please be aware of that we're not complaining about the lack of
> >>> resources
> >>> > in general, I'm complaining about the lack of **s

Re: [DISCUSS]Support Upsert mode for Streaming Non-window FlatAggregate

[Hequn Cheng]

Hi Kurt and Jark,

Thanks a lot for your great inputs!

The keys of the query may not strongly be related to the UDTAGG.
It may also be related to the corresponding scenarios that a user wants to
achieve.

For example, take TopN again as an example. If the TopN table aggregate
function
outputs three columns(rankid, time, value), either rankid or rankid+time
could be
used as the key. Which one to be chosen is more likely to be decided by the
user
according to his business.

Best, Hequn

On Thu, Jul 4, 2019 at 8:11 PM Jark Wu  wrote:

> Hi jingcheng,
>
> I agree with Kurt's point. As you said "the user must know the keys of the
> output of UDTAGG clearly".
> If I understand correctly, the key information is strongly relative to the
> UDTAGG implementation.
> Users may call `flatAggregate` on a UDTAGG instance with different keys
> which may result in a wrong result.
> So I think it would be better to couple key information with UDTAGG
> interface (i.e. "Approach 3" in your design doc).
>
> Regards,
> Jark
>
> On Thu, 4 Jul 2019 at 18:06, Kurt Young  wrote:
>
>> Hi Jincheng,
>>
>> Thanks for the clarification. I think you just pointed out my concern by
>> yourself:
>>
>> > When a user uses a User-defined table aggregate function (UDTAGG), he
>> must understand the behavior of the UDTAGG, including the return type and
>> the characteristics of the returned data. such as the key fields.
>>
>> This indicates that the UDTAGG is somehow be classified to different
>> types,
>> one will no key, one with key information. So the developer of the UDTAGG
>> should choose which type of this function should be. In this case,
>> my question would be, why don't we have explicit information about keys
>> such as we split UDTAGG to keyed UDTAGG and non-keyed UDTAGG. So the user
>> and the framework will have a better understanding of
>> this UDTAGG. `withKeys` solution is letting user to choose the key and it
>> seems it will only work correctly only if the user choose the *right* key
>> this UDTAGG has.
>>
>> Let me know if this makes sense to you.
>>
>> Best,
>> Kurt
>>
>>
>> On Thu, Jul 4, 2019 at 4:32 PM jincheng sun 
>> wrote:
>>
>> > Hi All,
>> >
>> > @Kurt Young  one user-defined table aggregate
>> function
>> > can be used in both with(out) keys case, and we do not introduce any
>> other
>> > aggregations. just like the explanation from @Hequn.
>> >
>> > @Hequn Cheng  thanks for your explanation!
>> >
>> > One thing should be mentioned here:
>> >
>> > When a user uses a User-defined table aggregate function (UDTAGG), he
>> must
>> > understand the behavior of the UDTAGG, including the return type and the
>> > characteristics of the returned data. such as the key fields.  So
>> although
>> > using `withKeys` approach it is not rigorous enough(we do not need) but
>> > intuitive enough, considering that if `flatAggregate` is followed by an
>> > `upsertSink`, then the user must know the keys of the output of UDTAGG
>> > clearly, otherwise the keys of `upsertSink` cannot be defined. So I
>> still
>> > prefer the `withKeys` solution by now.
>> >
>> > Looking forward to any feedback from all of you!
>> >
>> > Best,
>> > Jincheng
>> >
>> >
>> >
>> > Hequn Cheng  于2019年7月1日周一 下午5:35写道：
>> >
>> >> Hi Kurt,
>> >>
>> >> Thanks for your questions. Here are my thoughts.
>> >>
>> >> > if I want to write such kind function, should I make sure that this
>> >> function is used with some keys?
>> >> The key information may not be used. We can also use RetractSink to
>> emit
>> >> the table directly.
>> >>
>> >> >  If I need a use case to calculate topn without key, should I write
>> >> another function or I can reuse previous one.
>> >> For the TopN example, you can reuse the previous function if you don't
>> >> care
>> >> about the key information.
>> >>
>> >> So, the key information is only an indicator(or a description), not an
>> >> operator, as Jincheng mentioned above.
>> >> We do not need to change the function logic and it will not add any
>> other
>> >> aggregations.
>> >>
>> >> BTW, we have three approaches in the document. Approach 1 defines keys
>> on
>> >> API level as we think it's more common to 

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

[Hequn Cheng]

Congratulations Rong!

Best, Hequn

On Fri, Jul 12, 2019 at 12:19 PM Jeff Zhang  wrote:

> Congrats, Rong!
>
>
> vino yang  于2019年7月12日周五 上午10:08写道：
>
>> congratulations Rong Rong!
>>
>> Fabian Hueske  于2019年7月11日周四 下午10:25写道：
>>
>>> Hi everyone,
>>>
>>> I'm very happy to announce that Rong Rong accepted the offer of the
>>> Flink PMC to become a committer of the Flink project.
>>>
>>> Rong has been contributing to Flink for many years, mainly working on
>>> SQL and Yarn security features. He's also frequently helping out on the
>>> user@f.a.o mailing lists.
>>>
>>> Congratulations Rong!
>>>
>>> Best, Fabian
>>> (on behalf of the Flink PMC)
>>>
>>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: [ANNOUNCE] Flink 1.9 release branch has been created

[Hequn Cheng]

Hi Kurt,

Great work and thanks for the update.

FYI, I find a bug[1] on Table API accidentally. The PR has already been
opened. I think it would be good if the fix can be included in the 1.9.

Best,
Hequn

[1] https://issues.apache.org/jira/browse/FLINK-13196

On Fri, Jul 12, 2019 at 2:40 PM Kurt Young  wrote:

> Hi devs,
>
> I just created the branch for the Flink 1.9 release [1] and updated the
> version on master to 1.10-SNAPSHOT. This unblocks the master from
> merging new features into it.
>
> If you are working on a 1.9 relevant bug fix, then it is important to merge
> it into the release-1.9 and master branch.
>
> I’ll create a first release candidate shortly, stay tuned!
>
> Best,
> Kurt
>
> [1] https://github.com/apache/flink/tree/release-1.9
>

Re: [ANNOUNCE] Jiangjie (Becket) Qin has been added as a committer to the Flink project

[Hequn Cheng]

Congratulations Becket!

Best, Hequn

On Thu, Jul 18, 2019 at 5:34 PM vino yang  wrote:

> Congratulations!
>
> Best,
> Vino
>
> Yun Gao  于2019年7月18日周四 下午5:31写道：
>
> > Congratulations!
> >
> > Best,
> > Yun
> >
> >
> > --
> > From:Kostas Kloudas 
> > Send Time:2019 Jul. 18 (Thu.) 17:30
> > To:dev 
> > Subject:Re: [ANNOUNCE] Jiangjie (Becket) Qin has been added as a
> committer
> > to the Flink project
> >
> > Congratulations Becket!
> >
> > Kostas
> >
> > On Thu, Jul 18, 2019 at 11:21 AM Guowei Ma  wrote:
> >
> > > Congrats Becket!
> > >
> > > Best,
> > > Guowei
> > >
> > >
> > > Terry Wang  于2019年7月18日周四 下午5:17写道：
> > >
> > > > Congratulations Becket!
> > > >
> > > > > 在 2019年7月18日，下午5:09，Dawid Wysakowicz  写道：
> > > > >
> > > > > Congratulations Becket! Good to have you onboard!
> > > > >
> > > > > On 18/07/2019 10:56, Till Rohrmann wrote:
> > > > >> Congrats Becket!
> > > > >>
> > > > >> On Thu, Jul 18, 2019 at 10:52 AM Jeff Zhang 
> > wrote:
> > > > >>
> > > > >>> Congratulations Becket!
> > > > >>>
> > > > >>> Xu Forward  于2019年7月18日周四 下午4:39写道：
> > > > >>>
> > > >  Congratulations Becket! Well deserved.
> > > > 
> > > > 
> > > >  Cheers,
> > > > 
> > > >  forward
> > > > 
> > > >  Kurt Young  于2019年7月18日周四 下午4:20写道：
> > > > 
> > > > > Congrats Becket!
> > > > >
> > > > > Best,
> > > > > Kurt
> > > > >
> > > > >
> > > > > On Thu, Jul 18, 2019 at 4:12 PM JingsongLee <
> > > lzljs3620...@aliyun.com
> > > > > .invalid>
> > > > > wrote:
> > > > >
> > > > >> Congratulations Becket!
> > > > >>
> > > > >> Best, Jingsong Lee
> > > > >>
> > > > >>
> > > > >>
> > --
> > > > >> From:Congxian Qiu 
> > > > >> Send Time:2019年7月18日(星期四) 16:09
> > > > >> To:dev@flink.apache.org 
> > > > >> Subject:Re: [ANNOUNCE] Jiangjie (Becket) Qin has been added
> as a
> > > > > committer
> > > > >> to the Flink project
> > > > >>
> > > > >> Congratulations Becket! Well deserved.
> > > > >>
> > > > >> Best,
> > > > >> Congxian
> > > > >>
> > > > >>
> > > > >> Jark Wu  于2019年7月18日周四 下午4:03写道：
> > > > >>
> > > > >>> Congratulations Becket! Well deserved.
> > > > >>>
> > > > >>> Cheers,
> > > > >>> Jark
> > > > >>>
> > > > >>> On Thu, 18 Jul 2019 at 15:56, Paul Lam <
> paullin3...@gmail.com>
> > > >  wrote:
> > > >  Congrats Becket!
> > > > 
> > > >  Best,
> > > >  Paul Lam
> > > > 
> > > > > 在 2019年7月18日，15:41，Robert Metzger 
> 写道：
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I'm excited to announce that Jiangjie (Becket) Qin just
> > became
> > > > >>> a
> > > > >> Flink
> > > > > committer!
> > > > >
> > > > > Congratulations Becket!
> > > > >
> > > > > Best,
> > > > > Robert (on behalf of the Flink PMC)
> > > > 
> > > > >>>
> > > > >>> --
> > > > >>> Best Regards
> > > > >>>
> > > > >>> Jeff Zhang
> > > > >>>
> > > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Flink project bylaws

[Hequn Cheng]

Hi Becket,

Big +1 on this.

> Regarding the vote of FLIP, preferably at least includes a PMC vote.
Perhaps what Jincheng means is to hold at least one PMC in the 3 binding
votes? i.e, the vote could have 2 binding committers and 1 PMC.
I think this makes sense considering a FLIP could somehow be a big feature
which may impacts Flink a lot. Meanwhile, there is no loss of flexibility
as committers also have a chance to vote and participate in it.

Seperator

For the nice bylaws, I agree with the general idea and most of the content.
Only share some thoughts about the "2/3 Majority". The main concern is I am
not sure if it is doable in practice. The reasons are:

1. If we follow the bylaws strictly, it means a committer or a PMC member
is active if he or she sends one email to the mailing list every 6 months.
While the minimum length of the vote is only 6 days. There are chances that
during the vote, some of the active members are still offline of the
community.
2. The code of Flink is changing fast and not everyone fully understands
every part. We don't need to force people to vote if they are not sure
about it. It may also make the final result less credible.

Given the above reasons, perhaps we can change the 2/3 Majority to lazy 2/3
Majority, just as the Hadoop bylaws[1]. It makes a higher threshold,
however, more practical.

What do you think?

[1] https://hadoop.apache.org/bylaws.html

On Sat, Jul 20, 2019 at 12:00 AM Becket Qin  wrote:

> Hi Jincheng,
>
> Thanks for the comments. I replied on the wiki page. Just a brief summary,
> the current bylaws do require some of the FLIPs to get PMC approval if
> their impact is big enough. But it leaves majority of the technical
> decisions to the committers who are supposed to be responsible for making
> such decisions.
>
> Re: Robert,
>
> I agree we can simply remove the requirement of +1 from a non-author
> committer and revisit it in a bit. After all, it does not make sense to
> have a bylaw that we cannot afford. I have just updated the bylaws wiki.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Fri, Jul 19, 2019 at 11:17 PM Robert Metzger 
> wrote:
>
> > Hi all,
> > I agree with Aljoscha that trying to reflect the current status in the
> > bylaws, and then implementing changes one by one is a very involved task.
> > Unless there's somebody who's really eager to drive this, I would stick
> to
> > Becket's initiative to come up with Bylaws for Flink, even if this means
> > some changes.
> >
> > The cross-review requirement is the last big open point in this
> discussion.
> > It seems that a there is a slight tendency in the discussion that this is
> > not feasible given the current pull request review situation.
> > For the sake of bringing this discussion to a conclusion, I'm fine with
> > leaving this requirement out. As we are currently adding more committers
> to
> > the project, we might be able to revisit this discussion in 3 - 6 months.
> >
> >
> > On Thu, Jul 18, 2019 at 4:30 AM jincheng sun 
> > wrote:
> >
> > > Hi Becket,
> > >
> > > Thanks for the proposal.
> > >
> > > Regarding the vote of FLIP, preferably at least includes a PMC vote.
> > > Because FLIP is usually a big change or affects the user's interface
> > > changes. What do you think? (I leave the comment in the wiki.)
> > >
> > > Best,
> > > Jincheng
> > >
> > > Dawid Wysakowicz  于2019年7月17日周三 下午9:12写道：
> > >
> > > > Hi all,
> > > >
> > > > Sorry for joining late. I just wanted to say that I really like the
> > > > proposed bylaws!
> > > >
> > > > One comment, I also have the same concerns as expressed by few people
> > > > before that the "committer +1" on code change might be hard to
> achieve
> > > > currently. On the other hand I would say this would be beneficial for
> > > > the quality/uniformity of our codebase and knowledge sharing.
> > > >
> > > > I was just wondering what should be the next step for this? I think
> it
> > > > would make sense to already use those bylaws and put them to PMC
> vote.
> > > >
> > > > Best,
> > > >
> > > > Dawid
> > > >
> > > > On 12/07/2019 13:35, Piotr Nowojski wrote:
> > > > > Hi Aljoscha and Becket
> > > > >
> > > > > Right, 3 days for FLIP voting is fine I think.
> > > > >
> > > > >>> I’m missing this stated somewhere clearly. If we are stating
> that a
> > > > single
> > > > >>> committers +1 is good enough for code review, with 0 hours delay
> > (de
> > > > facto
> > > > >>> the current state), we should also write down that this is
> subject
> > to
> > > > the
> > > > >>> best judgement of the committer to respect the components
> expertise
> > > and
> > > > >>> ongoing development plans (also the de facto current state).
> > > > >> Adding the statement would help clarify the intention, but it may
> > be a
> > > > >> little difficult to enforce and follow..
> > > > > I would be fine with that, it’s a soft/vague rule anyway, intended
> to
> > > be
> > > > used with your “best judgemenet". I would like to just avoid a
> > situat

Re: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

[Hequn Cheng]

Congratulations Zhijiang!

Best, Hequn

On Tue, Jul 23, 2019 at 6:17 PM JingsongLee 
wrote:

> Congratulations Zhijiang!
>
> Best, Jingsong Lee
>
>
> --
> From:wenlong.lwl 
> Send Time:2019年7月23日(星期二) 17:34
> To:dev 
> Subject:Re: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to
> the Flink project
>
> Congrats, Zhijiang!
>
> On Tue, 23 Jul 2019 at 15:59, Becket Qin  wrote:
>
> > Congrats, Zhijiang!
> >
> > On Tue, Jul 23, 2019 at 2:01 PM Jark Wu  wrote:
> >
> > > Congratulations Zhijiang!
> > >
> > >
> > > On Tue, 23 Jul 2019 at 11:30, vino yang  wrote:
> > >
> > > > Congratulations Zhijiang!
> > > >
> > > > Haibo Sun  于2019年7月23日周二 上午10:48写道：
> > > >
> > > > > Congrats, Zhejiang!
> > > > >
> > > > >
> > > > > Best,
> > > > > Haibo
> > > > > 在 2019-07-23 10:26:20，"Yun Tang"  写道：
> > > > > >Congratulations Zhijiang, well deserved.
> > > > > >
> > > > > >Best
> > > > > >
> > > > > >From: Yingjie Cao 
> > > > > >Sent: Tuesday, July 23, 2019 10:23
> > > > > >To: dev@flink.apache.org 
> > > > > >Subject: Re: [ANNOUNCE] Zhijiang Wang has been added as a
> committer
> > to
> > > > > the Flink project
> > > > > >
> > > > > >Congratulations Zhijiang!
> > > > > >
> > > > > >yangtao.yt  于2019年7月23日周二 上午10:17写道：
> > > > > >
> > > > > >> Congrats, Zhejiang!
> > > > > >>
> > > > > >> Best,
> > > > > >> Tao Yang
> > > > > >>
> > > > > >> > 在 2019年7月23日，上午9:46，boshu Zheng  写道：
> > > > > >> >
> > > > > >> > Congratulations Zhijiang
> > > > > >> >
> > > > > >> > 发自我的 iPhone
> > > > > >> >
> > > > > >> >> 在 2019年7月23日，上午12:55，Xuefu Z  写道：
> > > > > >> >>
> > > > > >> >> Congratulations, Zhijiang!
> > > > > >> >>
> > > > > >> >>> On Mon, Jul 22, 2019 at 7:42 AM Bo WANG <
> > > wbeaglewatc...@gmail.com
> > > > >
> > > > > >> wrote:
> > > > > >> >>>
> > > > > >> >>> Congratulations Zhijiang!
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >>> Best,
> > > > > >> >>>
> > > > > >> >>> Bo WANG
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >>> On Mon, Jul 22, 2019 at 10:12 PM Robert Metzger <
> > > > > rmetz...@apache.org>
> > > > > >> >>> wrote:
> > > > > >> >>>
> > > > > >>  Hey all,
> > > > > >> 
> > > > > >>  We've added another committer to the Flink project:
> Zhijiang
> > > > Wang.
> > > > > >> 
> > > > > >>  Congratulations Zhijiang!
> > > > > >> 
> > > > > >>  Best,
> > > > > >>  Robert
> > > > > >>  (on behalf of the Flink PMC)
> > > > > >> 
> > > > > >> >>>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> --
> > > > > >> >> Xuefu Zhang
> > > > > >> >>
> > > > > >> >> "In Honey We Trust!"
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

[Hequn Cheng]

Congratulations Kurt!

Best, Hequn

On Tue, Jul 23, 2019 at 7:27 PM vino yang  wrote:

> Congratulations Kurt!
>
> Bo WANG  于2019年7月23日周二 下午7:13写道：
>
> > Congratulations Kurt!
> >
> >
> > Best,
> >
> > Bo WANG
> >
> >
> > On Tue, Jul 23, 2019 at 5:24 PM Robert Metzger 
> > wrote:
> >
> > > Hi all,
> > >
> > > On behalf of the Flink PMC, I'm happy to announce that Kete Young is
> now
> > > part of the Apache Flink Project Management Committee (PMC).
> > >
> > > Kete has been a committer since February 2017, working a lot on Table
> > API /
> > > SQL. He's currently co-managing the 1.9 release! Thanks a lot for your
> > work
> > > for Flink!
> > >
> > > Congratulations & Welcome Kurt!
> > >
> > > Best,
> > > Robert
> > >
> >
>

Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds

[Hequn Cheng]

Hi Jark,

Good idea. +1!

On Tue, Jul 23, 2019 at 6:23 PM Jark Wu  wrote:

> Thank you all for your positive feedback.
>
> We have three binding +1s, so I think, we can proceed with this.
>
> Hi @Robert Metzger  , could you create a request to
> INFRA for the mailing list?
> I'm not sure if this needs a PMC permission.
>
> Thanks,
> Jark
>
> On Tue, 23 Jul 2019 at 16:42, jincheng sun 
> wrote:
>
> > +1
> >
> > Robert Metzger  于2019年7月23日周二 下午4:01写道：
> >
> > > +1
> > >
> > > On Mon, Jul 22, 2019 at 10:27 AM Biao Liu  wrote:
> > >
> > > > +1, make sense to me.
> > > > Mailing list seems to be a more "community" way.
> > > >
> > > > Timo Walther  于2019年7月22日周一 下午4:06写道：
> > > >
> > > > > +1 sounds good to inform people about instabilities or other issues
> > > > >
> > > > > Regards,
> > > > > Timo
> > > > >
> > > > >
> > > > > Am 22.07.19 um 09:58 schrieb Haibo Sun:
> > > > > > +1. Sounds good.Letting the PR creators know the build results of
> > the
> > > > > master branch can help to determine more quickly whether Travis
> > > failures
> > > > of
> > > > > their PR are an exact failure or because of the instability of test
> > > case.
> > > > > By the way, if the PR creator can abort their own Travis build, I
> > think
> > > > it
> > > > > can improve the efficient use of Travis resources (of course, this
> > > > > discussion does not deal with this issue).
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Haibo
> > > > > > At 2019-07-22 12:36:31, "Yun Tang"  wrote:
> > > > > >> +1 sounds good, more people could be encouraged to involve in
> > paying
> > > > > attention to failure commit.
> > > > > >>
> > > > > >> Best
> > > > > >> Yun Tang
> > > > > >> 
> > > > > >> From: Becket Qin 
> > > > > >> Sent: Monday, July 22, 2019 9:44
> > > > > >> To: dev 
> > > > > >> Subject: Re: [DISCUSS] Setup a bui...@flink.apache.org mailing
> > list
> > > > > for travis builds
> > > > > >>
> > > > > >> +1. Sounds a good idea to me.
> > > > > >>
> > > > > >> On Sat, Jul 20, 2019 at 7:07 PM Dian Fu 
> > > > wrote:
> > > > > >>
> > > > > >>> Thanks Jark for the proposal, sounds reasonable for me. +1.
> This
> > ML
> > > > > could
> > > > > >>> be used for all the build notifications including master and
> CRON
> > > > jobs.
> > > > > >>>
> > > > >  在 2019年7月20日，下午2:55，Xu Forward  写道：
> > > > > 
> > > > >  +1 ,Thanks jark for the proposal.
> > > > >  Best
> > > > >  Forward
> > > > > 
> > > > >  Jark Wu  于2019年7月20日周六 下午12:14写道：
> > > > > 
> > > > > > Hi all,
> > > > > >
> > > > > > As far as I know, currently, email notifications of Travis
> > builds
> > > > for
> > > > > > master branch are sent to the commit author when a build was
> > just
> > > > > >>> broken or
> > > > > > still is broken. And there is no email notifications for CRON
> > > > builds.
> > > > > >
> > > > > > Recently, we are suffering from compile errors for scala-2.12
> > and
> > > > > java-9
> > > > > > which are only ran in CRON jobs. So I'm figuring out a way to
> > get
> > > > > > notifications of CRON builds (or all builds) to quick fix
> > compile
> > > > > errors
> > > > > > and failed cron tests.
> > > > > >
> > > > > > After reaching out to @Chesnay Schepler 
> > > > (thanks
> > > > > >>> for
> > > > > > the helping), I know that we are using a Slack channel to
> > receive
> > > > all
> > > > > > failed build notifications. However, IMO, email notification
> > > might
> > > > > be a
> > > > > > better way than Slack channel to encourage more people to pay
> > > > > attention
> > > > > >>> on
> > > > > > the builds.
> > > > > >
> > > > > > So I'm here to propose to setup a bui...@flink.apache.org
> > > mailing
> > > > > list
> > > > > >>> for
> > > > > > receiving build notifications. I also find that Beam has such
> > > > mailing
> > > > > >>> list
> > > > > > too[1]. After we have such a mailing list, we can integrate
> it
> > to
> > > > > travis
> > > > > > according to the travis doc[2].
> > > > > >
> > > > > > What do you think? Do we need a formal vote for this?
> > > > > >
> > > > > > Best and thanks,
> > > > > > Jark
> > > > > >
> > > > > > [1]: https://beam.apache.org/community/contact-us/
> > > > > > [2]:
> > > > > >
> > > > > >
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > > > > <
> > > > > >
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > > > > <
> > > > > >
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://docs.travis-ci.com/user/notifications/#configuring-email-notifications
> > > > > >>>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Publish the PyFlink into PyPI

[Hequn Cheng]

+1 (non-binding)

Thanks a lot for driving this! @jincheng sun 

Best, Hequn

On Thu, Aug 1, 2019 at 11:00 PM Biao Liu  wrote:

> Thanks Jincheng for working on this.
>
> +1 (non-binding)
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Thu, Aug 1, 2019 at 8:55 PM Jark Wu  wrote:
>
> > +1 (non-binding)
> >
> > Cheers,
> > Jark
> >
> > On Thu, 1 Aug 2019 at 17:45, Yu Li  wrote:
> >
> > > +1 (non-binding)
> > >
> > > Thanks for driving this!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Thu, 1 Aug 2019 at 11:41, Till Rohrmann 
> wrote:
> > >
> > > > +1
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Thu, Aug 1, 2019 at 10:39 AM vino yang 
> > wrote:
> > > >
> > > > > +1 （non-binding）
> > > > >
> > > > > Jeff Zhang  于2019年8月1日周四 下午4:33写道：
> > > > >
> > > > > > +1 （non-binding）
> > > > > >
> > > > > > Stephan Ewen  于2019年8月1日周四 下午4:29写道：
> > > > > >
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > On Thu, Aug 1, 2019 at 9:52 AM Dian Fu 
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Jincheng,
> > > > > > > >
> > > > > > > > Thanks a lot for driving this.
> > > > > > > > +1 (non-binding).
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Dian
> > > > > > > >
> > > > > > > > > 在 2019年8月1日，下午3:24，jincheng sun 
> > 写道：
> > > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > Publish the PyFlink into PyPI is very important for our
> user,
> > > > > Please
> > > > > > > vote
> > > > > > > > > on the following proposal:
> > > > > > > > >
> > > > > > > > > 1. Create PyPI Project for Apache Flink Python API, named:
> > > > > > > "apache-flink"
> > > > > > > > > 2. Release one binary with the default Scala version same
> > with
> > > > > flink
> > > > > > > > > default config.
> > > > > > > > > 3. Create an account, named "pyflink" as owner(only PMC can
> > > > manage
> > > > > > it).
> > > > > > > > PMC
> > > > > > > > > can add account for the Release Manager, but Release
> Manager
> > > can
> > > > > not
> > > > > > > > delete
> > > > > > > > > the release.
> > > > > > > > >
> > > > > > > > > [ ] +1, Approve the proposal.
> > > > > > > > > [ ] -1, Disapprove the proposal, because ...
> > > > > > > > >
> > > > > > > > > The vote will be open for at least 72 hours. It is adopted
> > by a
> > > > > > simple
> > > > > > > > > majority with a minimum of three positive votes.
> > > > > > > > >
> > > > > > > > > See discussion threads for more details [1].
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Jincheng
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Publish-the-PyFlink-into-PyPI-td30095.html
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Regards
> > > > > >
> > > > > > Jeff Zhang
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Hequn becomes a Flink committer

[Hequn Cheng]

Thanks everyone for the warm welcome!

It's a great honor for me to be a committer. Looking forward to
contributing more to the community.

Best, Hequn

On Thu, Aug 8, 2019 at 3:39 AM zhijiang  wrote:

> Congratulations Hequn!
>
> Best,
> Zhijiang
> --
> From:Xuefu Z 
> Send Time:2019年8月7日(星期三) 20:35
> To:dev 
> Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
>
> Congratulations， Hequn！
>
> On Wed, Aug 7, 2019 at 11:08 AM Yun Tang  wrote:
>
> > Congratulations Hequn.
> >
> > Best
> > Yun Tang
> > 
> > From: Rong Rong 
> > Sent: Thursday, August 8, 2019 0:41
> > Cc: dev ; user 
> > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> >
> > Congratulations Hequn, well deserved!
> >
> > --
> > Rong
> >
> > On Wed, Aug 7, 2019 at 8:30 AM  > xingc...@gmail.com>> wrote:
> >
> > Congratulations, Hequn!
> >
> >
> >
> > From: Xintong Song mailto:tonysong...@gmail.com>>
> > Sent: Wednesday, August 07, 2019 10:41 AM
> > To: dev@flink.apache.org
> > Cc: user mailto:u...@flink.apache.org>>
> > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> >
> >
> >
> > Congratulations~!
> >
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> >
> >
> > On Wed, Aug 7, 2019 at 4:00 PM vino yang  > yanghua1...@gmail.com>> wrote:
> >
> > Congratulations!
> >
> > highfei2...@126.com  > > 于2019年8月7日周三 下午7:09写道：
> >
> > > Congrats Hequn!
> > >
> > > Best,
> > > Jeff Yang
> > >
> > >
> > >  Original Message 
> > > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer
> > > From: Piotr Nowojski
> > > To: JingsongLee
> > > CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun
> > ,dev ,user
> > >
> > >
> > > Congratulations :)
> > >
> > > On 7 Aug 2019, at 12:09, JingsongLee  > lzljs3620...@aliyun.com>> wrote:
> > >
> > > Congrats Hequn!
> > >
> > > Best,
> > > Jingsong Lee
> > >
> > > --
> > > From:Biao Liu mailto:mmyy1...@gmail.com>>
> > > Send Time:2019年8月7日(星期三) 12:05
> > > To:Zhu Zhu mailto:reed...@gmail.com>>
> > > Cc:Zili Chen mailto:wander4...@gmail.com>>; Jeff
> > Zhang mailto:zjf...@gmail.com>>; Paul
> > > Lam mailto:paullin3...@gmail.com>>; jincheng
> sun
> > mailto:sunjincheng...@gmail.com>>; dev
> > > mailto:dev@flink.apache.org>>; user <
> > u...@flink.apache.org>
> > > Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer
> > >
> > > Congrats Hequn!
> > >
> > > Thanks,
> > > Biao /'bɪ.aʊ/
> > >
> > >
> > >
> > > On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu  > reed...@gmail.com>> wrote:
> > > Congratulations to Hequn!
> > >
> > > Thanks,
> > > Zhu Zhu
> > >
> > > Zili Chen mailto:wander4...@gmail.com>>
> > 于2019年8月7日周三 下午5:16写道：
> > > Congrats Hequn!
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Jeff Zhang mailto:zjf...@gmail.com>> 于2019年8月7日周三
> > 下午5:14写道：
> > > Congrats Hequn!
> > >
> > > Paul Lam mailto:paullin3...@gmail.com>>
> > 于2019年8月7日周三 下午5:08写道：
> > > Congrats Hequn! Well deserved!
> > >
> > > Best,
> > > Paul Lam
> > >
> > > 在 2019年8月7日，16:28，jincheng sun  > sunjincheng...@gmail.com>> 写道：
> > >
> > > Hi everyone,
> > >
> > > I'm very happy to announce that Hequn accepted the offer of the Flink
> PMC
> > > to become a committer of the Flink project.
> > >
> > > Hequn has been contributing to Flink for many years, mainly working on
> > > SQL/Table API features. He's also frequently helping out on the user
> > > mailing lists and helping check/vote the release.
> > >
> > > Congratulations Hequn!
> > >
> > > Best, Jincheng
> > > (on behalf of the Flink PMC)
> > >
> > >
> > >
> > > --
> > > Best Regards
> > >
> > > Jeff Zhang
> > >
> > >
> > >
> >
>
>
> --
> Xuefu Zhang
>
> "In Honey We Trust!"
>
>

Re: [VOTE] Flink Project Bylaws

[Hequn Cheng]

+1 (non-binding)

Thanks a lot for driving this! Good job. @Becket Qin 

Best, Hequn

On Tue, Aug 13, 2019 at 6:26 PM Stephan Ewen  wrote:

> +1
>
> On Tue, Aug 13, 2019 at 12:22 PM Maximilian Michels 
> wrote:
>
> > +1 It's good that we formalize this.
> >
> > On 13.08.19 10:41, Fabian Hueske wrote:
> > > +1 for the proposed bylaws.
> > > Thanks for pushing this Becket!
> > >
> > > Cheers, Fabian
> > >
> > > Am Mo., 12. Aug. 2019 um 16:31 Uhr schrieb Robert Metzger <
> > > rmetz...@apache.org>:
> > >
> > > > I changed the permissions of the page.
> > > >
> > > > On Mon, Aug 12, 2019 at 4:21 PM Till Rohrmann 
> > > > wrote:
> > > >
> > > >> +1 for the proposal. Thanks a lot for driving this discussion
> Becket!
> > > >>
> > > >> Cheers,
> > > >> Till
> > > >>
> > > >> On Mon, Aug 12, 2019 at 3:02 PM Becket Qin 
> > wrote:
> > > >>
> > > >>> Hi Robert,
> > > >>>
> > > >>> That's a good suggestion. Will you help to change the permission on
> > > > that
> > > >>> page?
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Jiangjie (Becket) Qin
> > > >>>
> > > >>> On Mon, Aug 12, 2019 at 2:41 PM Robert Metzger <
> rmetz...@apache.org>
> > > >>> wrote:
> > > >>>
> > >  Thanks for starting the vote.
> > >  How about putting a specific version in the wiki up for voting, or
> > >  restricting edit access to the page to the PMC?
> > >  There were already two changes (very minor) to the page since the
> > > > vote
> > > >>> has
> > >  started:
> > > 
> > > 
> > > >>>
> > > >>
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=120731026
> > >  I suggest to restrict edit access to the page.
> > > 
> > > 
> > > 
> > >  On Mon, Aug 12, 2019 at 11:43 AM Timo Walther  >
> > > >>> wrote:
> > > 
> > > > +1
> > > >
> > > > Thanks for all the efforts you put into this for documenting how
> > > > the
> > > > project operates.
> > > >
> > > > Regards,
> > > > Timo
> > > >
> > > > Am 12.08.19 um 10:44 schrieb Aljoscha Krettek:
> > > >> +1
> > > >>
> > > >>> On 11. Aug 2019, at 10:07, Becket Qin 
> > > >> wrote:
> > > >>>
> > > >>> Hi all,
> > > >>>
> > > >>> I would like to start a voting thread on the project bylaws of
> > > >>> Flink.
> > >  It
> > > >>> aims to help the community coordinate more smoothly. Please see
> > > >> the
> > > > bylaws
> > > >>> wiki page below for details.
> > > >>>
> > > >>>
> > > >
> > > 
> > > >>>
> > > >>
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
> > > >>>
> > > >>> The discussion thread is following:
> > > >>>
> > > >>>
> > > >
> > > 
> > > >>>
> > > >>
> > > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-project-bylaws-td30409.html
> > > >>>
> > > >>> The vote will be open for at least 6 days. PMC members' votes
> > > > are
> > > >>> considered as binding. The vote requires 2/3 majority of the
> > > >> binding
> > > > +1s to
> > > >>> pass.
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Jiangjie (Becket) Qin
> > > >
> > > >
> > > >
> > > 
> > > >>>
> > > >>
> > > >
> > >
> >
> >
>

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

[Hequn Cheng]

Congratulations Andrey!

On Thu, Aug 15, 2019 at 3:30 PM Fabian Hueske  wrote:

> Congrats Andrey!
>
> Am Do., 15. Aug. 2019 um 07:58 Uhr schrieb Gary Yao :
>
> > Congratulations Andrey, well deserved!
> >
> > Best,
> > Gary
> >
> > On Thu, Aug 15, 2019 at 7:50 AM Bowen Li  wrote:
> >
> > > Congratulations Andrey!
> > >
> > > On Wed, Aug 14, 2019 at 10:18 PM Rong Rong 
> wrote:
> > >
> > >> Congratulations Andrey!
> > >>
> > >> On Wed, Aug 14, 2019 at 10:14 PM chaojianok 
> wrote:
> > >>
> > >> > Congratulations Andrey!
> > >> > At 2019-08-14 21:26:37, "Till Rohrmann" 
> wrote:
> > >> > >Hi everyone,
> > >> > >
> > >> > >I'm very happy to announce that Andrey Zagrebin accepted the offer
> of
> > >> the
> > >> > >Flink PMC to become a committer of the Flink project.
> > >> > >
> > >> > >Andrey has been an active community member for more than 15 months.
> > He
> > >> has
> > >> > >helped shaping numerous features such as State TTL, FRocksDB
> release,
> > >> > >Shuffle service abstraction, FLIP-1, result partition management
> and
> > >> > >various fixes/improvements. He's also frequently helping out on the
> > >> > >user@f.a.o mailing lists.
> > >> > >
> > >> > >Congratulations Andrey!
> > >> > >
> > >> > >Best, Till
> > >> > >(on behalf of the Flink PMC)
> > >> >
> > >>
> > >
> >
>

Re: CiBot Update

[Hequn Cheng]

Cool, thanks Chesnay a lot for the improvement!

Best, Hequn

On Thu, Aug 22, 2019 at 5:02 PM Zhu Zhu  wrote:

> Thanks Chesnay for the CI improvement!
> It is very helpful.
>
> Thanks,
> Zhu Zhu
>
> zhijiang  于2019年8月22日周四 下午4:18写道：
>
> > It is really very convenient now. Valuable work, Chesnay!
> >
> > Best,
> > Zhijiang
> > --
> > From:Till Rohrmann 
> > Send Time:2019年8月22日(星期四) 10:13
> > To:dev 
> > Subject:Re: CiBot Update
> >
> > Thanks for the continuous work on the CiBot Chesnay!
> >
> > Cheers,
> > Till
> >
> > On Thu, Aug 22, 2019 at 9:47 AM Jark Wu  wrote:
> >
> > > Great work! Thanks Chesnay!
> > >
> > >
> > >
> > > On Thu, 22 Aug 2019 at 15:42, Xintong Song 
> > wrote:
> > >
> > > > The re-triggering travis feature is so convenient. Thanks Chesnay~!
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Thu, Aug 22, 2019 at 9:26 AM Stephan Ewen 
> wrote:
> > > >
> > > > > Nice, thanks!
> > > > >
> > > > > On Thu, Aug 22, 2019 at 3:59 AM Zili Chen 
> > > wrote:
> > > > >
> > > > > > Thanks for your announcement. Nice work!
> > > > > >
> > > > > > Best,
> > > > > > tison.
> > > > > >
> > > > > >
> > > > > > vino yang  于2019年8月22日周四 上午8:14写道：
> > > > > >
> > > > > > > +1 for "@flinkbot run travis", it is very convenient.
> > > > > > >
> > > > > > > Chesnay Schepler  于2019年8月21日周三 下午9:12写道：
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > this is an update on recent changes to the CI bot.
> > > > > > > >
> > > > > > > >
> > > > > > > > The bot now cancels builds if a new commit was added to a PR,
> > and
> > > > > > > > cancels all builds if the PR was closed.
> > > > > > > > (This was implemented a while ago; I'm just mentioning it
> again
> > > for
> > > > > > > > discoverability)
> > > > > > > >
> > > > > > > >
> > > > > > > > Additionally, starting today you can now re-trigger a Travis
> > run
> > > by
> > > > > > > > writing a comment "@flinkbot run travis"; this means you no
> > > longer
> > > > > have
> > > > > > > > to commit an empty commit or do other shenanigans to get
> > another
> > > > > build
> > > > > > > > running.
> > > > > > > > Note that this will /not/ work if the PR was re-opened, until
> > at
> > > > > least
> > > > > > 1
> > > > > > > > new build was triggered by a push.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
>

Re: [DISCUSS] Flink Python User-Defined Function for Table API

[Hequn Cheng]

+1 for starting the vote.

Thanks Jincheng a lot for the discussion.

Best, Hequn

On Fri, Aug 23, 2019 at 10:06 AM Dian Fu  wrote:

> Hi Jincheng,
>
> +1 to start the FLIP create and VOTE on this feature. I'm willing to help
> on the FLIP create if you don't mind. As I haven't created a FLIP before,
> it will be great if you could help on this. :)
>
> Regards,
> Dian
>
> > 在 2019年8月22日，下午11:41，jincheng sun  写道：
> >
> > Hi all,
> >
> > Thanks a lot for your feedback. If there are no more suggestions and
> > comments, I think it's better to  initiate a vote to create a FLIP for
> > Apache Flink Python UDFs.
> > What do you think?
> >
> > Best, Jincheng
> >
> > jincheng sun  于2019年8月15日周四 上午12:54写道：
> >
> >> Hi Thomas,
> >>
> >> Thanks for your confirmation and the very important reminder about
> bundle
> >> processing.
> >>
> >> I have had add the description about how to perform bundle processing
> from
> >> the perspective of checkpoint and watermark. Feel free to leave
> comments if
> >> there are anything not describe clearly.
> >>
> >> Best,
> >> Jincheng
> >>
> >>
> >> Dian Fu  于2019年8月14日周三 上午10:08写道：
> >>
> >>> Hi Thomas,
> >>>
> >>> Thanks a lot the suggestions.
> >>>
> >>> Regarding to bundle processing, there is a section "Checkpoint"[1] in
> the
> >>> design doc which talks about how to handle the checkpoint.
> >>> However, I think you are right that we should talk more about it, such
> as
> >>> what's bundle processing, how it affects the checkpoint and watermark,
> how
> >>> to handle the checkpoint and watermark, etc.
> >>>
> >>> [1]
> >>>
> https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit#heading=h.urladt565yo3
> >>> <
> >>>
> https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit#heading=h.urladt565yo3
> 
> >>>
> >>> Regards,
> >>> Dian
> >>>
>  在 2019年8月14日，上午1:01，Thomas Weise  写道：
> 
>  Hi Jincheng,
> 
>  Thanks for putting this together. The proposal is very detailed,
> >>> thorough
>  and for me as a Beam Flink runner contributor easy to understand :)
> 
>  One thing that you should probably detail more is the bundle
> >>> processing. It
>  is critically important for performance that multiple elements are
>  processed in a bundle. The default bundle size in the Flink runner is
> >>> 1s or
>  1000 elements, whichever comes first. And for streaming, you can find
> >>> the
>  logic necessary to align the bundle processing with watermarks and
>  checkpointing here:
> 
> >>>
> https://github.com/apache/beam/blob/release-2.14.0/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
> 
>  Thomas
> 
> 
> 
> 
> 
> 
> 
>  On Tue, Aug 13, 2019 at 7:05 AM jincheng sun <
> sunjincheng...@gmail.com>
>  wrote:
> 
> > Hi all,
> >
> > The Python Table API(without Python UDF support) has already been
> >>> supported
> > and will be available in the coming release 1.9.
> > As Python UDF is very important for Python users, we'd like to start
> >>> the
> > discussion about the Python UDF support in the Python Table API.
> > Aljoscha Krettek, Dian Fu and I have discussed offline and have
> >>> drafted a
> > design doc[1]. It includes the following items:
> >
> > - The user-defined function interfaces.
> > - The user-defined function execution architecture.
> >
> > As mentioned by many guys in the previous discussion thread[2], a
> > portability framework was introduced in Apache Beam in latest
> >>> releases. It
> > provides well-defined, language-neutral data structures and protocols
> >>> for
> > language-neutral user-defined function execution. This design is
> based
> >>> on
> > Beam's portability framework. We will introduce how to make use of
> >>> Beam's
> > portability framework for user-defined function execution: data
> > transmission, state access, checkpoint, metrics, logging, etc.
> >
> > Considering that the design relies on Beam's portability framework
> for
> > Python user-defined function execution and not all the contributors
> in
> > Flink community are familiar with Beam's portability framework, we
> have
> > done a prototype[3] for proof of concept and also ease of
> >>> understanding of
> > the design.
> >
> > Welcome any feedback.
> >
> > Best,
> > Jincheng
> >
> > [1]
> >
> >
> >>>
> https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit?usp=sharing
> > [2]
> >
> >
> >>>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html
> > [3] https://github.com/dianfu/flink/commits/udf_poc
> >
> >>>
> >>>
>
>

Re: [VOTE] FLIP-58: Flink Python User-Defined Function for Table API

[Hequn Cheng]

Hi Dian,

+1
Thanks a lot for driving this.

Best, Hequn

On Wed, Aug 28, 2019 at 2:01 PM jincheng sun 
wrote:

> Hi Dian,
>
> +1, Thanks for your great job!
>
> Best,
> Jincheng
>
> Dian Fu  于2019年8月28日周三 上午11:04写道：
>
> > Hi all,
> >
> > I'd like to start a voting thread for FLIP-58 [1] since that we have
> > reached an agreement on the design in the discussion thread [2],
> >
> > This vote will be open for at least 72 hours. Unless there is an
> > objection, I will try to close it by Sept 2, 2019 00:00 UTC if we have
> > received sufficient votes.
> >
> > PS: This doesn't mean that we cannot further improve the design. We can
> > still discuss the implementation details case by case in the JIRA as long
> > as it doesn't affect the overall design.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Function+for+Table+API
> > <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58:+Flink+Python+User-Defined+Function+for+Table+API
> > >
> > [2]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Python-User-Defined-Function-for-Table-API-td31673.html
> > <
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Python-User-Defined-Function-for-Table-API-td31673.html
> > >
> >
> > Thanks,
> > Dian
>

Re: [DISCUSS] Releasing Flink 1.8.2

[Hequn Cheng]

Hi Jincheng,

+1 for a 1.8.2 release.
Thanks a lot for raising the discussion. It would be nice to have these
critical fixes.

Best, Hequn


On Fri, Aug 30, 2019 at 6:31 PM Maximilian Michels  wrote:

> Hi Jincheng,
>
> +1 I would be for a 1.8.2 release such that we can fix the problems with
> the nested closure cleaner which currently block 1.8.1 users with Beam:
> https://issues.apache.org/jira/browse/FLINK-13367
>
> Thanks,
> Max
>
> On 30.08.19 11:25, jincheng sun wrote:
> > Hi Flink devs,
> >
> > It has been nearly 2 months since the 1.8.1 released. So, what do you
> think
> > about releasing Flink 1.8.2 soon?
> >
> > We already have some blocker and critical fixes in the release-1.8
> branch:
> >
> > [Blocker]
> > - FLINK-13159 java.lang.ClassNotFoundException when restore job
> > - FLINK-10368 'Kerberized YARN on Docker test' unstable
> > - FLINK-12578 Use secure URLs for Maven repositories
> >
> > [Critical]
> > - FLINK-12736 ResourceManager may release TM with allocated slots
> > - FLINK-12889 Job keeps in FAILING state
> > - FLINK-13484 ConnectedComponents end-to-end test instable with
> > NoResourceAvailableException
> > - FLINK-13508 CommonTestUtils#waitUntilCondition() may attempt to sleep
> > with negative time
> > - FLINK-13806 Metric Fetcher floods the JM log with errors when TM is
> lost
> >
> > Furthermore, I think the following one blocker issue should be merged
> > before 1.8.2 release.
> >
> > - FLINK-13897: OSS FS NOTICE file is placed in wrong directory
> >
> > It would also be great if we can have the fix of Elasticsearch6.x
> connector
> > threads leaking (FLINK-13689) in 1.8.2 release which is identified as
> major.
> >
> > Please let me know what you think?
> >
> > Cheers,
> > Jincheng
> >
>

Re: [ANNOUNCE] Apache Flink-shaded 8.0 released

[Hequn Cheng]

Thanks a lot to Chesney!
Also thanks to everyone who helped to make this release possible.

Best, Hequn

On Fri, Aug 30, 2019 at 7:17 PM jincheng sun 
wrote:

> Thanks a lot Chesnay and to the community for making this release possible
> !
>
> Cheers,
> Jincheng
>
> Chesnay Schepler  于2019年8月30日周五 下午6:56写道：
>
> > The Apache Flink community is very happy to announce the release of
> > Apache Flink-shaded 8.0.
> >
> > The flink-shaded project contains a number of shaded dependencies for
> > Apache Flink.
> >
> > Apache Flink® is an open-source stream processing framework for
> > distributed, high-performing, always-available, and accurate data
> > streaming applications.
> >
> > The release is available for download at:
> > https://flink.apache.org/downloads.html
> >
> > The full release notes are available in Jira:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12345488
> >
> > We would like to thank all contributors of the Apache Flink community
> > who made this release possible!
> >
> > Regards,
> > Chesnay
> >
> >
>

Re: Build failure on flink-python

[Hequn Cheng]

Hi Biao,

Thanks a lot for reporting the problem. The fix has been merged into the
master just now. You can rebase to the master and try again.

Thanks to @Wei Zhong for the fixing.

Best, Hequn

On Mon, Sep 2, 2019 at 4:41 PM Biao Liu  wrote:

> There are already some Jira tickets opened for this failure [1] [2].
> Sorry I didn't recognize them.
>
> 1. https://issues.apache.org/jira/browse/FLINK-13906
> 2. https://issues.apache.org/jira/browse/FLINK-13932
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Mon, 2 Sep 2019 at 16:24, Biao Liu  wrote:
>
> > Hi guys,
> >
> > I just found I can't pass the Travis build due to some errors in
> > flink-python module [1].
> > I'm sure my PR has nothing related with flink-python. And there are also
> a
> > lot of builds are failing on these errors.
> >
> > I have rebased master branch and tried several times. But it doesn't
> work.
> >
> > Could somebody who is familiar with flink-python module take a look at
> > this failure?
> >
> > 1. https://travis-ci.com/flink-ci/flink/jobs/229989235
> >
> > Thanks,
> > Biao /'bɪ.aʊ/
> >
> >
>

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

[Hequn Cheng]

Congratulations Kostas! Well deserved.

Best, Hequn

On Sat, Sep 7, 2019 at 10:48 AM Jark Wu  wrote:

> Congratulations Klou!
>
>
> > 在 2019年9月7日，00:21，zhijiang  写道：
> >
> > Congratulations Klou!
> >
> > Best,
> > Zhijiang
> > --
> > From:Zhu Zhu 
> > Send Time:2019年9月6日(星期五) 17:19
> > To:dev 
> > Subject:Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC
> >
> > Congratulations Kostas!
> >
> > Thanks,
> > Zhu Zhu
> >
> > Yu Li  于2019年9月6日周五 下午10:49写道：
> >
> >> Congratulations Klou!
> >>
> >> Best Regards,
> >> Yu
> >>
> >>
> >> On Fri, 6 Sep 2019 at 22:43, Forward Xu  wrote:
> >>
> >>> Congratulations Kloudas!
> >>>
> >>>
> >>> Best,
> >>>
> >>> Forward
> >>>
> >>> Dawid Wysakowicz  于2019年9月6日周五 下午10:36写道：
> >>>
>  Congratulations Klou!
> 
>  Best,
> 
>  Dawid
> 
>  On 06/09/2019 14:55, Fabian Hueske wrote:
> > Hi everyone,
> >
> > I'm very happy to announce that Kostas Kloudas is joining the Flink
> >>> PMC.
> > Kostas is contributing to Flink for many years and puts lots of
> >> effort
> >>> in
> > helping our users and growing the Flink community.
> >
> > Please join me in congratulating Kostas!
> >
> > Cheers,
> > Fabian
> >
> 
> 
> >>>
> >>
> >
>
>

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

[Hequn Cheng]

Congratulations!

Best, Hequn

On Thu, Sep 12, 2019 at 9:24 AM Jark Wu  wrote:

> Congratulations Zili!
>
> Best,
> Jark
>
> On Wed, 11 Sep 2019 at 23:06,  wrote:
>
> > Congratulations, Zili.
> >
> >
> >
> > Best,
> >
> > Xingcan
> >
> >
> >
> > *From:* SHI Xiaogang 
> > *Sent:* Wednesday, September 11, 2019 7:43 AM
> > *To:* Guowei Ma 
> > *Cc:* Fabian Hueske ; Biao Liu ;
> > Oytun Tez ; bupt_ljy ; dev <
> > dev@flink.apache.org>; user ; Till Rohrmann <
> > trohrm...@apache.org>
> > *Subject:* Re: [ANNOUNCE] Zili Chen becomes a Flink committer
> >
> >
> >
> > Congratulations!
> >
> >
> >
> > Regards,
> >
> > Xiaogang
> >
> >
> >
> > Guowei Ma  于2019年9月11日周三 下午7:07写道：
> >
> > Congratulations Zili !
> >
> >
> > Best,
> >
> > Guowei
> >
> >
> >
> >
> >
> > Fabian Hueske  于2019年9月11日周三 下午7:02写道：
> >
> > Congrats Zili Chen :-)
> >
> >
> >
> > Cheers, Fabian
> >
> >
> >
> > Am Mi., 11. Sept. 2019 um 12:48 Uhr schrieb Biao Liu  >:
> >
> > Congrats Zili!
> >
> >
> >
> > Thanks,
> >
> > Biao /'bɪ.aʊ/
> >
> >
> >
> >
> >
> >
> >
> > On Wed, 11 Sep 2019 at 18:43, Oytun Tez  wrote:
> >
> > Congratulations!
> >
> >
> >
> > ---
> >
> > Oytun Tez
> >
> >
> >
> > *M O T A W O R D*
> >
> > *The World's Fastest Human Translation Platform.*
> >
> > oy...@motaword.com — www.motaword.com
> >
> >
> >
> >
> >
> > On Wed, Sep 11, 2019 at 6:36 AM bupt_ljy  wrote:
> >
> > Congratulations!
> >
> >
> >
> > Best,
> >
> > Jiayi Liao
> >
> >
> >
> >  Original Message
> >
> > *Sender:* Till Rohrmann
> >
> > *Recipient:* dev; user
> >
> > *Date:* Wednesday, Sep 11, 2019 17:22
> >
> > *Subject:* [ANNOUNCE] Zili Chen becomes a Flink committer
> >
> >
> >
> > Hi everyone,
> >
> >
> >
> > I'm very happy to announce that Zili Chen (some of you might also know
> > him as Tison Kun) accepted the offer of the Flink PMC to become a
> committer
> > of the Flink project.
> >
> >
> >
> > Zili Chen has been an active community member for almost 16 months now.
> > He helped pushing the Flip-6 effort over the finish line, ported a lot of
> > legacy code tests, removed a good part of the legacy code, contributed
> > numerous fixes, is involved in the Flink's client API refactoring, drives
> > the refactoring of Flink's HighAvailabilityServices and much more. Zili
> > Chen also helped the community by PR reviews, reporting Flink issues,
> > answering user mails and being very active on the dev mailing list.
> >
> >
> >
> > Congratulations Zili Chen!
> >
> >
> >
> > Best, Till
> >
> > (on behalf of the Flink PMC)
> >
> >
>

Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink

[Hequn Cheng]

+1 (binding)

Thanks,
Hequn

On Fri, Jan 12, 2024 at 2:19 PM godfrey he  wrote:

> +1 (binding)
>
> Thanks,
> Godfrey
>
> Zhu Zhu  于2024年1月12日周五 14:10写道：
> >
> > +1 (binding)
> >
> > Thanks,
> > Zhu
> >
> > Hangxiang Yu  于2024年1月11日周四 14:26写道：
> >
> > > +1 (non-binding)
> > >
> > > On Thu, Jan 11, 2024 at 11:19 AM Xuannan Su 
> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Best,
> > > > Xuannan
> > > >
> > > > On Thu, Jan 11, 2024 at 10:28 AM Xuyang  wrote:
> > > > >
> > > > > +1 (non-binding)--
> > > > >
> > > > > Best！
> > > > > Xuyang
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 在 2024-01-11 10:00:11，"Yang Wang"  写道：
> > > > > >+1 (binding)
> > > > > >
> > > > > >
> > > > > >Best,
> > > > > >Yang
> > > > > >
> > > > > >On Thu, Jan 11, 2024 at 9:53 AM liu ron 
> wrote:
> > > > > >
> > > > > >> +1 non-binding
> > > > > >>
> > > > > >> Best
> > > > > >> Ron
> > > > > >>
> > > > > >> Matthias Pohl  于2024年1月10日周三
> > > 23:05写道：
> > > > > >>
> > > > > >> > +1 (binding)
> > > > > >> >
> > > > > >> > On Wed, Jan 10, 2024 at 3:35 PM ConradJam <
> jam.gz...@gmail.com>
> > > > wrote:
> > > > > >> >
> > > > > >> > > +1 non-binding
> > > > > >> > >
> > > > > >> > > Dawid Wysakowicz  于2024年1月10日周三
> > > 21:06写道：
> > > > > >> > >
> > > > > >> > > > +1 (binding)
> > > > > >> > > > Best,
> > > > > >> > > > Dawid
> > > > > >> > > >
> > > > > >> > > > On Wed, 10 Jan 2024 at 11:54, Piotr Nowojski <
> > > > pnowoj...@apache.org>
> > > > > >> > > wrote:
> > > > > >> > > >
> > > > > >> > > > > +1 (binding)
> > > > > >> > > > >
> > > > > >> > > > > śr., 10 sty 2024 o 11:25 Martijn Visser <
> > > > martijnvis...@apache.org>
> > > > > >> > > > > napisał(a):
> > > > > >> > > > >
> > > > > >> > > > > > +1 (binding)
> > > > > >> > > > > >
> > > > > >> > > > > > On Wed, Jan 10, 2024 at 4:43 AM Xingbo Huang <
> > > > hxbks...@gmail.com
> > > > > >> >
> > > > > >> > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > +1 (binding)
> > > > > >> > > > > > >
> > > > > >> > > > > > > Best,
> > > > > >> > > > > > > Xingbo
> > > > > >> > > > > > >
> > > > > >> > > > > > > Dian Fu  于2024年1月10日周三
> 11:35写道：
> > > > > >> > > > > > >
> > > > > >> > > > > > > > +1 (binding)
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Regards,
> > > > > >> > > > > > > > Dian
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Wed, Jan 10, 2024 at 5:09 AM Sharath <
> > > > > >> dsaishar...@gmail.com
> > > > > >> > >
> > > > > >> > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Best,
> > > > > >> > > > > > > > > Sharath
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Tue, Jan 9, 2024 at 1:02 PM Venkata Sanath
> > > > Muppalla <
> > > > > >> > > > > > > > sanath...@gmail.com>
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > Sanath
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Tue, Jan 9, 2024 at 11:16 AM Peter Huang <
> > > > > >> > > > > > > > huangzhenqiu0...@gmail.com>
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Best Regards
> > > > > >> > > > > > > > > > > Peter Huang
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Tue, Jan 9, 2024 at 5:26 AM Jane Chan <
> > > > > >> > > > > qingyue@gmail.com>
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > Jane
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > On Tue, Jan 9, 2024 at 8:41 PM Lijie Wang
> <
> > > > > >> > > > > > > > wangdachui9...@gmail.com>
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > > Lijie
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Jiabao Sun  > > .invalid>
> > > > > >> > > > 于2024年1月9日周二
> > > > > >> > > > > > > > 19:28写道：
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > > > > Jiabao
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > On 2024/01/09 09:58:04 xiangyu feng
> wrote:
> > > > > >> > > > > > > > > > > > > > > +1 (non-binding)
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> 

[DISCUSS] Support Python ML Pipeline API

[Hequn Cheng]

Hi everyone,

FLIP-39[1] rebuilds the Flink ML pipeline on top of TableAPI and introduces
a new set of Java APIs. As Python is widely used in ML areas, providing
Python ML Pipeline APIs for Flink can not only make it easier to write ML
jobs for Python users but also broaden the adoption of Flink ML.

Given this, Jincheng and I discussed offline about the support of Python ML
Pipeline API and drafted a design doc[2]. We'd like to achieve three goals
for supporting Python Pipeline API:
- Add Python pipeline API according to Java pipeline API(we will adapt the
Python pipeline API if Java pipeline API changes).
- Support native Python Transformer/Estimator/Model, i.e., users can write
not only Python Transformer/Estimator/Model wrappers for calling Java ones
but also can write native Python Transformer/Estimator/Models.
- Ease of use. Support keyword arguments when defining parameters.

More details can be found in the design doc and we are looking forward to
your feedback.

Best,
Hequn

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
[2]
https://docs.google.com/document/d/1fwSO5sRNWMoYuvNgfQJUV6N2n2q5UEVA4sezCljKcVQ/edit?usp=sharing

Re: [DISCUSS] Release flink-shaded 10.0

[Hequn Cheng]

+1. It sounds great to allow us to support zk 3.4 and 3.5.
Thanks for starting the discussion.

Best,
Hequn

On Thu, Feb 6, 2020 at 12:21 AM Till Rohrmann  wrote:

> Thanks for starting this discussion Chesnay. +1 for starting a new
> flink-shaded release.
>
> Cheers,
> Till
>
> On Wed, Feb 5, 2020 at 2:10 PM Chesnay Schepler 
> wrote:
>
> > Hello,
> >
> > I would like to kick off the next release of flink-shaded. The main
> > feature are new modules that bundle zookeeper&curator, that will allow
> > us to support zk 3.4 and 3.5 .
> >
> > Additionally we fixed an issue where slightly older dependencies than
> > intended were bundled in the flink-shaded-hadoop-2-uber jar, which was
> > flagged by security checks.
> >
> > Are there any other changes that people are interested in doing?
> >
> >
> > Regards,
> >
> > Chesnay
> >
> >
>

Re: [DISCUSS] Include flink-ml-api and flink-ml-lib in opt

[Hequn Cheng]

Hi everyone,

Thank you all for the great inputs!

I think probably what we all agree on is we should try to make a leaner
flink-dist. However, we may also need to do some compromises considering
the user experience that users don't need to download the dependencies from
different places. Otherwise, we can move all the jars in the current opt
folder to the download page.

The missing of clear rules for guiding such compromises makes things more
complicated now. I would agree that the decisive factor for what goes into
Flink's binary distribution should be how core it is to Flink. Meanwhile,
it's better to treat Flink API as a (core) core to Flink. Not only it is a
very clear rule that easy to be followed but also in most cases, API is
very significant and deserved to be included in the dist.

Given this, it might make sense to put flink-ml-api and flink-ml-lib into
the opt.
What do you think?

Best,
Hequn

On Wed, Feb 5, 2020 at 12:39 AM Chesnay Schepler  wrote:

> Around a year ago I started a discussion
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Towards-a-leaner-flink-dist-tp25615.html>
> on reducing the amount of jars we ship with the distribution.
>
> While there was no definitive conclusion there was a shared sentiment that
> APIs should be shipped with the distribution.
>
> On 04/02/2020 17:25, Till Rohrmann wrote:
>
> I think there is no such rule that APIs go automatically into opt/ and
> "libraries" not. The contents of opt/ have mainly grown over time w/o
> following a strict rule.
>
> I think the decisive factor for what goes into Flink's binary distribution
> should be how core it is to Flink. Of course another important
> consideration is which use cases Flink should promote "out of the box" (not
> sure whether this is actual true for content shipped in opt/ because you
> also have to move it to lib).
>
> For example, Gelly would be an example which I would rather see as an
> optional component than shipping it with every Flink binary distribution.
>
> Cheers,
> Till
>
> On Tue, Feb 4, 2020 at 11:24 AM Becket Qin  
>  wrote:
>
>
> Thanks for the suggestion, Till.
>
> I am curious about how do we usually decide when to put the jars into the
> opt folder?
>
> Technically speaking, it seems that `flink-ml-api` should be put into the
> opt directory because they are actually API instead of libraries, just like
> CEP and Table.
>
> `flink-ml-lib` seems to be on the border. On one hand, it is a library. On
> the other hand, unlike SQL formats and Hadoop whose major code are outside
> of Flink, the algorithm codes are in Flink. So `flink-ml-lib` is more like
> those of built-in SQL UDFs. So it seems fine to either put it in the opt
> folder or in the downloads page.
>
> From the user experience perspective, it might be better to have both
> `flink-ml-lib` and `flink-ml-api` in opt folder so users needn't go to two
> places for the required dependencies.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Tue, Feb 4, 2020 at 2:32 PM Hequn Cheng  
>  wrote:
>
>
> Hi Till,
>
> Thanks a lot for your suggestion. It's a good idea to offer the flink-ml
> libraries as optional dependencies on the download page which can make
>
> the
>
> dist smaller.
>
> But I also have some concerns for it, e.g., the download page now only
> includes the latest 3 releases. We may need to find ways to support more
> versions.
> On the other hand, the size of the flink-ml libraries now is very
> small(about 246K), so it would not bring much impact on the size of dist.
>
> What do you think?
>
> Best,
> Hequn
>
> On Mon, Feb 3, 2020 at 6:24 PM Till Rohrmann  
> 
>
> wrote:
>
> An alternative solution would be to offer the flink-ml libraries as
> optional dependencies on the download page. Similar to how we offer the
> different SQL formats and Hadoop releases [1].
>
> [1] https://flink.apache.org/downloads.html
>
> Cheers,
> Till
>
> On Mon, Feb 3, 2020 at 10:19 AM Hequn Cheng  
>  wrote:
>
>
> Thank you all for your feedback and suggestions!
>
> Best, Hequn
>
> On Mon, Feb 3, 2020 at 5:07 PM Becket Qin  
> 
>
> wrote:
>
> Thanks for bringing up the discussion, Hequn.
>
> +1 on adding `flink-ml-api` and `flink-ml-lib` into opt. This would
>
> make
>
> it much easier for the users to try out some simple ml tasks.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Mon, Feb 3, 2020 at 4:34 PM jincheng sun <
>
> sunjincheng...@gmail.com
>
> wrote:
>
>
> Thank you for pushing forward @Hequn Cheng  
>  !
>
> Hi  @Becket Qin   , Do you have 
> any concerns
>
> on
>

Re: [DISCUSS] Include flink-ml-api and flink-ml-lib in opt

[Hequn Cheng]

Hi,

@Till Rohrmann  Thanks for the great inputs. I agree
with you that we should have a long term plan for this. It definitely
deserves another discussion.
@Jeff Zhang  Thanks for your reports and ideas. It's a
good idea to improve the error messages. Do we have any JIRAs for it or
maybe we can create one for it.

Thank you again for your feedback and suggestions. I will go on with the
PR. Thanks!

Best,
Hequn

On Thu, Feb 6, 2020 at 11:51 PM Jeff Zhang  wrote:

> I have another concern which may not be closely related to this thread.
> Since flink doesn't include all the necessary jars, I think it is critical
> for flink to display meaningful error message when any class is missing.
> e.g. Here's the error message when I use kafka but miss
> including flink-json.  To be honest, the kind of error message is hard to
> understand for new users.
>
>
> Reason: No factory implements
> 'org.apache.flink.table.factories.DeserializationSchemaFactory'. The
> following properties are requested:
> connector.properties.bootstrap.servers=localhost:9092
> connector.properties.group.id=testGroup
> connector.properties.zookeeper.connect=localhost:2181
> connector.startup-mode=earliest-offset connector.topic=generated.events
> connector.type=kafka connector.version=universal format.type=json
> schema.0.data-type=VARCHAR(2147483647) schema.0.name=status
> schema.1.data-type=VARCHAR(2147483647) schema.1.name=direction
> schema.2.data-type=BIGINT schema.2.name=event_ts update-mode=append The
> following factories have been considered:
> org.apache.flink.table.catalog.hive.factories.HiveCatalogFactory
> org.apache.flink.table.module.hive.HiveModuleFactory
> org.apache.flink.table.module.CoreModuleFactory
> org.apache.flink.table.catalog.GenericInMemoryCatalogFactory
> org.apache.flink.table.sources.CsvBatchTableSourceFactory
> org.apache.flink.table.sources.CsvAppendTableSourceFactory
> org.apache.flink.table.sinks.CsvBatchTableSinkFactory
> org.apache.flink.table.sinks.CsvAppendTableSinkFactory
> org.apache.flink.table.planner.delegation.BlinkPlannerFactory
> org.apache.flink.table.planner.delegation.BlinkExecutorFactory
> org.apache.flink.table.planner.StreamPlannerFactory
> org.apache.flink.table.executor.StreamExecutorFactory
> org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactory at
>
> org.apache.flink.table.factories.TableFactoryService.filterByFactoryClass(TableFactoryService.java:238)
> at
>
> org.apache.flink.table.factories.TableFactoryService.filter(TableFactoryService.java:185)
> at
>
> org.apache.flink.table.factories.TableFactoryService.findSingleInternal(TableFactoryService.java:143)
> at
>
> org.apache.flink.table.factories.TableFactoryService.find(TableFactoryService.java:113)
> at
>
> org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactoryBase.getDeserializationSchema(KafkaTableSourceSinkFactoryBase.java:277)
> at
>
> org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactoryBase.createStreamTableSource(KafkaTableSourceSinkFactoryBase.java:161)
> at
>
> org.apache.flink.table.factories.StreamTableSourceFactory.createTableSource(StreamTableSourceFactory.java:49)
> at
>
> org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:53)
> ... 36 more
>
>
>
> Till Rohrmann  于2020年2月6日周四 下午11:30写道：
>
> > I would not object given that it is rather small at the moment. However,
> I
> > also think that we should have a plan how to handle the ever growing
> Flink
> > ecosystem and how to make it easily accessible to our users. E.g. one far
> > fetched idea could be something like a configuration script which
> downloads
> > the required components for the user. But this deserves definitely a
> > separate discussion and does not really belong here.
> >
> > Cheers,
> > Till
> >
> > On Thu, Feb 6, 2020 at 3:35 PM Hequn Cheng  wrote:
> >
> > >
> > > Hi everyone,
> > >
> > > Thank you all for the great inputs!
> > >
> > > I think probably what we all agree on is we should try to make a leaner
> > > flink-dist. However, we may also need to do some compromises
> considering
> > > the user experience that users don't need to download the dependencies
> > from
> > > different places. Otherwise, we can move all the jars in the current
> opt
> > > folder to the download page.
> > >
> > > The missing of clear rules for guiding such compromises makes things
> more
> > > complicated now. I would agree that the decisive factor for what goes
> > into
> > > Flink's binary distribution should be how core it i

Re: [DISCUSS] Include flink-ml-api and flink-ml-lib in opt

[Hequn Cheng]

Hi Rong,

Thanks a lot for joining the discussion!

It would be great if we can have a long term plan. My intention is to
provide a way for users to add dependencies of Flink ML, either through the
opt or download page. This would be more and more critical along with the
improvement of the Flink ML, as you said there are multiple PRs under
review and I'm also going to support Python Pipeline API recently[1].

Meanwhile, it also makes sense to include the API into the opt, so it would
probably not break the long term plan.
However, even find something wrong in the future, we can revisit this
easily instead of blocking the improvement for users. What do you think?

Best,
Hequn

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-Python-ML-Pipeline-API-td37291.html

On Sat, Feb 8, 2020 at 1:57 AM Rong Rong  wrote:

> CC @Xu Yang 
>
> Thanks for starting the discussion @Hequn Cheng  and
> sorry for joining the discussion late.
>
> I've mainly helped merging the code in flink-ml-api and flink-ml-lib in
> the past several months.
> IMO the flink-ml-api are an extension on top of the table API and agree
> that it should be treated as a part of the "core" core.
>
> However, I think given the fact that there are multiple PRs still under
> review [1], is it a better idea to come up with a long term plan first
> before make the decision to moving it to /opt now?
>
>
> --
> Rong
>
> [1]
> https://github.com/apache/flink/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+label%3Acomponent%3DLibrary%2FMachineLearning+
>
> On Fri, Feb 7, 2020 at 5:54 AM Hequn Cheng  wrote:
>
>> Hi,
>>
>> @Till Rohrmann  Thanks for the great inputs. I
>> agree
>> with you that we should have a long term plan for this. It definitely
>> deserves another discussion.
>> @Jeff Zhang  Thanks for your reports and ideas. It's a
>> good idea to improve the error messages. Do we have any JIRAs for it or
>> maybe we can create one for it.
>>
>> Thank you again for your feedback and suggestions. I will go on with the
>> PR. Thanks!
>>
>> Best,
>> Hequn
>>
>> On Thu, Feb 6, 2020 at 11:51 PM Jeff Zhang  wrote:
>>
>> > I have another concern which may not be closely related to this thread.
>> > Since flink doesn't include all the necessary jars, I think it is
>> critical
>> > for flink to display meaningful error message when any class is missing.
>> > e.g. Here's the error message when I use kafka but miss
>> > including flink-json.  To be honest, the kind of error message is hard
>> to
>> > understand for new users.
>> >
>> >
>> > Reason: No factory implements
>> > 'org.apache.flink.table.factories.DeserializationSchemaFactory'. The
>> > following properties are requested:
>> > connector.properties.bootstrap.servers=localhost:9092
>> > connector.properties.group.id=testGroup
>> > connector.properties.zookeeper.connect=localhost:2181
>> > connector.startup-mode=earliest-offset connector.topic=generated.events
>> > connector.type=kafka connector.version=universal format.type=json
>> > schema.0.data-type=VARCHAR(2147483647) schema.0.name=status
>> > schema.1.data-type=VARCHAR(2147483647) schema.1.name=direction
>> > schema.2.data-type=BIGINT schema.2.name=event_ts update-mode=append The
>> > following factories have been considered:
>> > org.apache.flink.table.catalog.hive.factories.HiveCatalogFactory
>> > org.apache.flink.table.module.hive.HiveModuleFactory
>> > org.apache.flink.table.module.CoreModuleFactory
>> > org.apache.flink.table.catalog.GenericInMemoryCatalogFactory
>> > org.apache.flink.table.sources.CsvBatchTableSourceFactory
>> > org.apache.flink.table.sources.CsvAppendTableSourceFactory
>> > org.apache.flink.table.sinks.CsvBatchTableSinkFactory
>> > org.apache.flink.table.sinks.CsvAppendTableSinkFactory
>> > org.apache.flink.table.planner.delegation.BlinkPlannerFactory
>> > org.apache.flink.table.planner.delegation.BlinkExecutorFactory
>> > org.apache.flink.table.planner.StreamPlannerFactory
>> > org.apache.flink.table.executor.StreamExecutorFactory
>> > org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactory
>> at
>> >
>> >
>> org.apache.flink.table.factories.TableFactoryService.filterByFactoryClass(TableFactoryService.java:238)
>> > at
>> >
>> >
>> org.apache.flink.table.factories.TableFactoryService.filter(TableFactoryService.java:185)
>> > at
>> >
>> >
>> org

Re: [DISCUSS] Support scalar vectorized Python UDF in PyFlink

[Hequn Cheng]

Hi Dian,

Thanks a lot for bringing up the discussion!

It is great to see the Pandas UDFs feature is going to be introduced. I
think this would improve the performance and also the usability of
user-defined functions (UDFs) in Python.
One little suggestion: maybe it would be nice if we can add some
performance explanation in the document? (I just very curious:))

+1 to create a FLIP for this big enhancement.

Best,
Hequn

On Mon, Feb 10, 2020 at 11:15 AM jincheng sun 
wrote:

> Hi Dian,
>
> Thanks for bring up this discussion. This is very important for the
> ecological of PyFlink. Add support Pandas greatly enriches the available
> UDF library of PyFlink and greatly improves the usability of PyFlink!
>
> +1 for Support scalar vectorized Python UDF.
>
> I think we should to create a FLIP for this big enhancements. :)
>
> What do you think?
>
> Best,
> Jincheng
>
>
>
> dianfu  于2020年2月5日周三 下午6:01写道：
>
> > Hi Jingsong,
> >
> > Thanks a lot for the valuable feedback.
> >
> > 1. The configurations "python.fn-execution.bundle.size" and
> > "python.fn-execution.arrow.batch.size" are used for separate purposes
> and I
> > think they are both needed. If they are unified, the Python operator has
> to
> > wait the execution results of the previous batch of elements before
> > processing the next batch. This means that the Python UDF execution can
> not
> > be pipelined between batches. With separate configuration, there will be
> no
> > such problems.
> > 2. It means that the Java operator will convert input elements to Arrow
> > memory format and then send them to the Python worker, vice verse.
> > Regarding to the zero-copy benefits provided by Arrow, we can gain them
> > automatically using Arrow.
> > 3. Good point! As all the classes of Python module is written in Java and
> > it's not suggested to introduce new Scala classes, so I guess it's not
> easy
> > to do so right now. But I think this is definitely a good improvement we
> > can do in the future.
> > 4. You're right and we will add a series of Arrow ColumnVectors for each
> > type supported.
> >
> > Thanks,
> > Dian
> >
> > > 在 2020年2月5日，下午4:57，Jingsong Li  写道：
> > >
> > > Hi Dian,
> > >
> > > +1 for this, thanks driving.
> > > Documentation looks very good. I can imagine a huge performance
> > improvement
> > > and better integration to other Python libraries.
> > >
> > > A few thoughts:
> > > - About data split: "python.fn-execution.arrow.batch.size", can we
> unify
> > it
> > > with "python.fn-execution.bundle.size"?
> > > - Use of Apache Arrow as the exchange format: Do you mean Arrow support
> > > zero-copy between Java and Python?
> > > - ArrowFieldWriter seems we can implement it by code generation. But it
> > is
> > > OK to initial version with virtual function call.
> > > - ColumnarRow for vectorization reading seems that we need implement
> > > ArrowColumnVectors.
> > >
> > > Best,
> > > Jingsong Lee
> > >
> > > On Wed, Feb 5, 2020 at 12:45 PM dianfu  wrote:
> > >
> > >> Hi all,
> > >>
> > >> Scalar Python UDF has already been supported in the coming release
> 1.10
> > >> (FLIP-58[1]). It operates one row at a time. It works in the way that
> > the
> > >> Java operator serializes one input row to bytes and sends them to the
> > >> Python worker; the Python worker deserializes the input row and
> > evaluates
> > >> the Python UDF with it; the result row is serialized and sent back to
> > the
> > >> Java operator.
> > >>
> > >> It suffers from the following problems:
> > >> 1) High serialization/deserialization overhead
> > >> 2) It’s difficult to leverage the popular Python libraries used by
> data
> > >> scientists, such as Pandas, Numpy, etc which provide high performance
> > data
> > >> structure and functions.
> > >>
> > >> Jincheng and I have discussed offline and we want to introduce
> > vectorized
> > >> Python UDF to address the above problems. This feature has also been
> > >> mentioned in the discussion thread about the Python API plan[2]. For
> > >> vectorized Python UDF, a batch of rows are transferred between JVM and
> > >> Python VM in columnar format. The batch of rows will be converted to a
> > >> collection of Pandas.Series and given to the vectorized Python UDF
> which
> > >> could then leverage the popular Python libraries such as Pandas,
> Numpy,
> > etc
> > >> for the Python UDF implementation.
> > >>
> > >> Please refer the design doc[3] for more details and welcome any
> > feedback.
> > >>
> > >> Regards,
> > >> Dian
> > >>
> > >> [1]
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> > >> [2]
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-What-parts-of-the-Python-API-should-we-focus-on-next-tt36119.html
> > >> [3]
> > >>
> >
> https://docs.google.com/document/d/1ls0mt96YV1LWPHfLOh8v7lpgh1KsCNx8B9RrW1ilnoE/edit#heading=h.qivada1u8hwd
> > >>
> > >>
> > >
> > > --
> > > Best, Jingsong Lee
> >
> >
>

Re: [DISCUSS] Support Python ML Pipeline API

[Hequn Cheng]

Hi everyone,

Thanks a lot for your feedback. I have created the FLIP[1].

Best,
Hequn

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP+96%3A+Support+Python+ML+Pipeline+API

On Mon, Feb 10, 2020 at 12:29 PM Dian Fu  wrote:

> Hi Hequn,
>
> Thanks for bringing up the discussion. +1 to this feature. The design LGTM.
> It's great that the Python ML users could use both the Java Pipeline
> Transformer/Estimator/Model classes and the Python
> Pipeline Transformer/Estimator/Model in the same job.
>
> Regards,
> Dian
>
> On Mon, Feb 10, 2020 at 11:08 AM jincheng sun 
> wrote:
>
> > Hi Hequn,
> >
> > Thanks for bring up this discussion.
> >
> > +1 for add Python ML Pipeline API, even though the Java pipeline API may
> > change.
> >
> > I would like to suggest create a FLIP for this API changes. :)
> >
> > Best,
> > Jincheng
> >
> >
> > Hequn Cheng  于2020年2月5日周三 下午5:24写道：
> >
> > > Hi everyone,
> > >
> > > FLIP-39[1] rebuilds the Flink ML pipeline on top of TableAPI and
> > introduces
> > > a new set of Java APIs. As Python is widely used in ML areas, providing
> > > Python ML Pipeline APIs for Flink can not only make it easier to write
> ML
> > > jobs for Python users but also broaden the adoption of Flink ML.
> > >
> > > Given this, Jincheng and I discussed offline about the support of
> Python
> > ML
> > > Pipeline API and drafted a design doc[2]. We'd like to achieve three
> > goals
> > > for supporting Python Pipeline API:
> > > - Add Python pipeline API according to Java pipeline API(we will adapt
> > the
> > > Python pipeline API if Java pipeline API changes).
> > > - Support native Python Transformer/Estimator/Model, i.e., users can
> > write
> > > not only Python Transformer/Estimator/Model wrappers for calling Java
> > ones
> > > but also can write native Python Transformer/Estimator/Models.
> > > - Ease of use. Support keyword arguments when defining parameters.
> > >
> > > More details can be found in the design doc and we are looking forward
> to
> > > your feedback.
> > >
> > > Best,
> > > Hequn
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
> > > [2]
> > >
> > >
> >
> https://docs.google.com/document/d/1fwSO5sRNWMoYuvNgfQJUV6N2n2q5UEVA4sezCljKcVQ/edit?usp=sharing
> > >
> >
>

Re: [DISCUSS] Include flink-ml-api and flink-ml-lib in opt

[Hequn Cheng]

Hi Rong,

That's great! Looking forward to your feedback.

Thanks,
Hequn


On Tue, Feb 11, 2020 at 1:06 AM Rong Rong  wrote:

> Yes. I think the argument is fairly valid - we can always adjust the API
> in the future, in fact most of the APIs are labeled publicEvolving at this
> moment.
> I was only trying to provide the info, that the interfaces in flink-ml-api
> might change in the near future, for others when voting.
>
> In fact, I am actually always +1 on moving flink-ml-api to /opt :-)
> Regarding the Python ML API. sorry for not noticing it earlier as I
> haven't given it a deep look yet. will do very soon!
>
> --
> Rong
>
> On Sun, Feb 9, 2020 at 7:33 PM Hequn Cheng  wrote:
>
>> Hi Rong,
>>
>> Thanks a lot for joining the discussion!
>>
>> It would be great if we can have a long term plan. My intention is to
>> provide a way for users to add dependencies of Flink ML, either through the
>> opt or download page. This would be more and more critical along with the
>> improvement of the Flink ML, as you said there are multiple PRs under
>> review and I'm also going to support Python Pipeline API recently[1].
>>
>> Meanwhile, it also makes sense to include the API into the opt, so it
>> would probably not break the long term plan.
>> However, even find something wrong in the future, we can revisit this
>> easily instead of blocking the improvement for users. What do you think?
>>
>> Best,
>> Hequn
>>
>> [1]
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-Python-ML-Pipeline-API-td37291.html
>>
>> On Sat, Feb 8, 2020 at 1:57 AM Rong Rong  wrote:
>>
>>> CC @Xu Yang 
>>>
>>> Thanks for starting the discussion @Hequn Cheng  and
>>> sorry for joining the discussion late.
>>>
>>> I've mainly helped merging the code in flink-ml-api and flink-ml-lib in
>>> the past several months.
>>> IMO the flink-ml-api are an extension on top of the table API and agree
>>> that it should be treated as a part of the "core" core.
>>>
>>> However, I think given the fact that there are multiple PRs still under
>>> review [1], is it a better idea to come up with a long term plan first
>>> before make the decision to moving it to /opt now?
>>>
>>>
>>> --
>>> Rong
>>>
>>> [1]
>>> https://github.com/apache/flink/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+label%3Acomponent%3DLibrary%2FMachineLearning+
>>>
>>> On Fri, Feb 7, 2020 at 5:54 AM Hequn Cheng  wrote:
>>>
>>>> Hi,
>>>>
>>>> @Till Rohrmann  Thanks for the great inputs. I
>>>> agree
>>>> with you that we should have a long term plan for this. It definitely
>>>> deserves another discussion.
>>>> @Jeff Zhang  Thanks for your reports and ideas. It's
>>>> a
>>>> good idea to improve the error messages. Do we have any JIRAs for it or
>>>> maybe we can create one for it.
>>>>
>>>> Thank you again for your feedback and suggestions. I will go on with the
>>>> PR. Thanks!
>>>>
>>>> Best,
>>>> Hequn
>>>>
>>>> On Thu, Feb 6, 2020 at 11:51 PM Jeff Zhang  wrote:
>>>>
>>>> > I have another concern which may not be closely related to this
>>>> thread.
>>>> > Since flink doesn't include all the necessary jars, I think it is
>>>> critical
>>>> > for flink to display meaningful error message when any class is
>>>> missing.
>>>> > e.g. Here's the error message when I use kafka but miss
>>>> > including flink-json.  To be honest, the kind of error message is
>>>> hard to
>>>> > understand for new users.
>>>> >
>>>> >
>>>> > Reason: No factory implements
>>>> > 'org.apache.flink.table.factories.DeserializationSchemaFactory'. The
>>>> > following properties are requested:
>>>> > connector.properties.bootstrap.servers=localhost:9092
>>>> > connector.properties.group.id=testGroup
>>>> > connector.properties.zookeeper.connect=localhost:2181
>>>> > connector.startup-mode=earliest-offset
>>>> connector.topic=generated.events
>>>> > connector.type=kafka connector.version=universal format.type=json
>>>> > schema.0.data-type=VARCHAR(2147483647)

Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

[Hequn Cheng]

+1 (non-binding)

- Check signature and checksum.
- Install package successfully with Pip under Python 3.7.4.
- Run wordcount example successfully under Python 3.7.4.

Best, Hequn

On Tue, Feb 11, 2020 at 12:17 PM Dian Fu  wrote:

> +1 (non-binding)
>
> - Verified the signature and checksum
> - Pip installed the package successfully: pip install
> apache-flink-1.9.2.tar.gz
> - Run word count example successfully.
>
> Regards,
> Dian
>
> 在 2020年2月11日，上午11:44，jincheng sun  写道：
>
>
> +1 (binding)
>
> - Install the PyFlink by `pip install` [SUCCESS]
> - Run word_count in both command line and IDE [SUCCESS]
>
> Best,
> Jincheng
>
>
>
> Wei Zhong  于2020年2月11日周二 上午11:17写道：
>
>> Hi,
>>
>> Thanks for driving this, Jincheng.
>>
>> +1 (non-binding)
>>
>> - Verified signatures and checksums.
>> - Verified README.md and setup.py.
>> - Run `pip install apache-flink-1.9.2.tar.gz` in Python 2.7.15 and Python
>> 3.7.5 successfully.
>> - Start local pyflink shell in Python 2.7.15 and Python 3.7.5 via
>> `pyflink-shell.sh local` and try the examples in the help message, run well
>> and no exception.
>> - Try a word count example in IDE with Python 2.7.15 and Python 3.7.5,
>> run well and no exception.
>>
>> Best,
>> Wei
>>
>>
>> 在 2020年2月10日，19:12，jincheng sun  写道：
>>
>> Hi everyone,
>>
>> Please review and vote on the release candidate #1 for the PyFlink
>> version 1.9.2, as follows:
>>
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>> The complete staging area is available for your review, which includes:
>>
>> * the official Apache binary convenience releases to be deployed to
>> dist.apache.org [1], which are signed with the key with fingerprint
>> 8FEA1EE9D0048C0CCC70B7573211B0703B79EA0E [2] and built from source code [3].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Thanks,
>> Jincheng
>>
>> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.9.2-rc1/
>> [2] https://dist.apache.org/repos/dist/release/flink/KEYS
>> [3] https://github.com/apache/flink/tree/release-1.9.2
>>
>>
>

Re: [VOTE] Release flink-shaded 10.0, release candidate #1

[Hequn Cheng]

Hi Chesnay,

One thing needs to double-check with you.
It seems the zookeeper version is not correct in the NOTICE file
for flink-shaded-zookeeper-3.5. The version should be 3.5.6?

Best, Hequn

On Sun, Feb 9, 2020 at 6:31 PM Chesnay Schepler  wrote:

> Hi everyone,
> Please review and vote on the release candidate #1 for the version 10.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2], which are signed with the key with fingerprint 11D464BA [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-10.0-rc1 [5],
> * website pull request listing the new release [6].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Chesnay
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346746
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-10.0-rc1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1334
> [5]
>
> https://gitbox.apache.org/repos/asf?p=flink-shaded.git;a=tag;h=refs/tags/release-10.0-rc1
> [6] https://github.com/apache/flink-web/pull/304
>
>

Re: [VOTE] Support scalar vectorized Python UDF in PyFlink

[Hequn Cheng]

+1 (binding)

Best, Hequn

On Thu, Feb 13, 2020 at 11:48 AM Jingsong Li  wrote:

> +1 (non-binding)
> Thanks Dian for driving.
>
> Best,
> Jingsong Lee
>
> On Thu, Feb 13, 2020 at 11:45 AM jincheng sun 
> wrote:
>
> > +1 (binding)
> >
> > Best,
> > Jincheng
> >
> >
> > Dian Fu  于2020年2月12日周三 下午1:31写道：
> >
> > > Hi all,
> > >
> > > I'd like to start the vote of FLIP-97[1] which is discussed and reached
> > > consensus in the discussion thread[2].
> > >
> > > The vote will be open for at least 72 hours. Unless there is an
> > objection,
> > > I will try to close it by Feb 17, 2020 08:00 UTC if we have received
> > > sufficient votes.
> > >
> > > Regards,
> > > Dian
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-97%3A+Support+Scalar+Vectorized+Python+UDF+in+PyFlink
> > > [2]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-scalar-vectorized-Python-UDF-in-PyFlink-tt37264.html
> >
>
>
> --
> Best, Jingsong Lee
>

Re: [ANNOUNCE] Apache Flink Python API(PyFlink) 1.9.2 released

[Hequn Cheng]

Thanks a lot for the release, Jincheng!
Also thanks to everyone that make this release possible!

Best,
Hequn

On Thu, Feb 13, 2020 at 2:18 PM Dian Fu  wrote:

> Thanks for the great work, Jincheng.
>
> Regards,
> Dian
>
> 在 2020年2月13日，下午1:32，jincheng sun  写道：
>
> Hi everyone,
>
> The Apache Flink community is very happy to announce the release of Apache
> Flink Python API(PyFlink) 1.9.2, which is the first release to PyPI for the
> Apache Flink Python API 1.9 series.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
>
> The release is available for download at:
>
> https://pypi.org/project/apache-flink/1.9.2/#files
>
> Or installed using pip command:
>
> pip install apache-flink==1.9.2
>
> We would like to thank all contributors of the Apache Flink community who
> helped to verify this release and made this release possible!
>
> Best,
> Jincheng
>
>
>

[VOTE] Support Python ML Pipeline API

[Hequn Cheng]

Hi everyone,

I'd like to start the vote of FLIP-96[1] which is discussed and reached
consensus in the discussion thread[2].
The vote will be open for at least 72 hours. Unless there is an objection,
I will try to close it by Feb 19, 2020 02:00 UTC if we have received
sufficient votes.

Thanks,
Hequn

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-96%3A+Support+Python+ML+Pipeline+API
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-Python-ML-Pipeline-API-td37291.html

Re: [DISCUSS] Support Python ML Pipeline API

[Hequn Cheng]

Hi all,

Thanks a lot for your valuable feedback!
As it seems we have reached a consensus on the discussion now. I have
started a VOTE thread[1]. Looking forward to your vote.

Best,
Hequn

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Support-Python-ML-Pipeline-API-td37637.html

On Thu, Feb 13, 2020 at 10:40 AM Becket Qin  wrote:

> +1. I'd say this is almost a must-have for machine learning.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Thu, Feb 13, 2020 at 10:03 AM Rong Rong  wrote:
>
>> Thanks for driving this initiative @Hequn Cheng .
>>
>> Moving towards python based ML is definitely a huge win consider how large
>> the python-ML community is. a big +1 on my side!
>> Regarding the doc, I only left a few comments on the specific APIs.
>> overall
>> the architecture looks very good!
>>
>> Looking forward to it!
>> --
>> Rong
>>
>> On Sun, Feb 9, 2020 at 10:28 PM Hequn Cheng  wrote:
>>
>> > Hi everyone,
>> >
>> > Thanks a lot for your feedback. I have created the FLIP[1].
>> >
>> > Best,
>> > Hequn
>> >
>> > [1]
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP+96%3A+Support+Python+ML+Pipeline+API
>> >
>> > On Mon, Feb 10, 2020 at 12:29 PM Dian Fu  wrote:
>> >
>> > > Hi Hequn,
>> > >
>> > > Thanks for bringing up the discussion. +1 to this feature. The design
>> > LGTM.
>> > > It's great that the Python ML users could use both the Java Pipeline
>> > > Transformer/Estimator/Model classes and the Python
>> > > Pipeline Transformer/Estimator/Model in the same job.
>> > >
>> > > Regards,
>> > > Dian
>> > >
>> > > On Mon, Feb 10, 2020 at 11:08 AM jincheng sun <
>> sunjincheng...@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi Hequn,
>> > > >
>> > > > Thanks for bring up this discussion.
>> > > >
>> > > > +1 for add Python ML Pipeline API, even though the Java pipeline API
>> > may
>> > > > change.
>> > > >
>> > > > I would like to suggest create a FLIP for this API changes. :)
>> > > >
>> > > > Best,
>> > > > Jincheng
>> > > >
>> > > >
>> > > > Hequn Cheng  于2020年2月5日周三 下午5:24写道：
>> > > >
>> > > > > Hi everyone,
>> > > > >
>> > > > > FLIP-39[1] rebuilds the Flink ML pipeline on top of TableAPI and
>> > > > introduces
>> > > > > a new set of Java APIs. As Python is widely used in ML areas,
>> > providing
>> > > > > Python ML Pipeline APIs for Flink can not only make it easier to
>> > write
>> > > ML
>> > > > > jobs for Python users but also broaden the adoption of Flink ML.
>> > > > >
>> > > > > Given this, Jincheng and I discussed offline about the support of
>> > > Python
>> > > > ML
>> > > > > Pipeline API and drafted a design doc[2]. We'd like to achieve
>> three
>> > > > goals
>> > > > > for supporting Python Pipeline API:
>> > > > > - Add Python pipeline API according to Java pipeline API(we will
>> > adapt
>> > > > the
>> > > > > Python pipeline API if Java pipeline API changes).
>> > > > > - Support native Python Transformer/Estimator/Model, i.e., users
>> can
>> > > > write
>> > > > > not only Python Transformer/Estimator/Model wrappers for calling
>> Java
>> > > > ones
>> > > > > but also can write native Python Transformer/Estimator/Models.
>> > > > > - Ease of use. Support keyword arguments when defining parameters.
>> > > > >
>> > > > > More details can be found in the design doc and we are looking
>> > forward
>> > > to
>> > > > > your feedback.
>> > > > >
>> > > > > Best,
>> > > > > Hequn
>> > > > >
>> > > > > [1]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs
>> > > > > [2]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1fwSO5sRNWMoYuvNgfQJUV6N2n2q5UEVA4sezCljKcVQ/edit?usp=sharing
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] Support Python ML Pipeline API

[Hequn Cheng]

Hi Becket,

Thanks a lot for your advice. Definitely agree with you that we should
follow the FLIP process.
Will pay more attention to this next time.

Best, Hequn


On Fri, Feb 14, 2020 at 2:19 PM Becket Qin  wrote:

> I just had an offline chat with Hequn and realized that FLIP-96 has already
> been opened for this discussion. I missed that because the FLIP was not
> mentioned in the thread.
>
> I am fine with proceeding to a vote.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Fri, Feb 14, 2020 at 12:52 PM Becket Qin  wrote:
>
> > Hi Hequn,
> >
> > Given this is an addition to the public API, we probably should follow
> the
> > FLIP process. It would be a quick one though, I think.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Fri, Feb 14, 2020 at 10:03 AM Hequn Cheng  wrote:
> >
> >> Hi all,
> >>
> >> Thanks a lot for your valuable feedback!
> >> As it seems we have reached a consensus on the discussion now. I have
> >> started a VOTE thread[1]. Looking forward to your vote.
> >>
> >> Best,
> >> Hequn
> >>
> >> [1]
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Support-Python-ML-Pipeline-API-td37637.html
> >>
> >> On Thu, Feb 13, 2020 at 10:40 AM Becket Qin 
> wrote:
> >>
> >>> +1. I'd say this is almost a must-have for machine learning.
> >>>
> >>> Thanks,
> >>>
> >>> Jiangjie (Becket) Qin
> >>>
> >>> On Thu, Feb 13, 2020 at 10:03 AM Rong Rong 
> wrote:
> >>>
> >>>> Thanks for driving this initiative @Hequn Cheng .
> >>>>
> >>>> Moving towards python based ML is definitely a huge win consider how
> >>>> large
> >>>> the python-ML community is. a big +1 on my side!
> >>>> Regarding the doc, I only left a few comments on the specific APIs.
> >>>> overall
> >>>> the architecture looks very good!
> >>>>
> >>>> Looking forward to it!
> >>>> --
> >>>> Rong
> >>>>
> >>>> On Sun, Feb 9, 2020 at 10:28 PM Hequn Cheng  wrote:
> >>>>
> >>>> > Hi everyone,
> >>>> >
> >>>> > Thanks a lot for your feedback. I have created the FLIP[1].
> >>>> >
> >>>> > Best,
> >>>> > Hequn
> >>>> >
> >>>> > [1]
> >>>> >
> >>>> >
> >>>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+96%3A+Support+Python+ML+Pipeline+API
> >>>> >
> >>>> > On Mon, Feb 10, 2020 at 12:29 PM Dian Fu 
> >>>> wrote:
> >>>> >
> >>>> > > Hi Hequn,
> >>>> > >
> >>>> > > Thanks for bringing up the discussion. +1 to this feature. The
> >>>> design
> >>>> > LGTM.
> >>>> > > It's great that the Python ML users could use both the Java
> Pipeline
> >>>> > > Transformer/Estimator/Model classes and the Python
> >>>> > > Pipeline Transformer/Estimator/Model in the same job.
> >>>> > >
> >>>> > > Regards,
> >>>> > > Dian
> >>>> > >
> >>>> > > On Mon, Feb 10, 2020 at 11:08 AM jincheng sun <
> >>>> sunjincheng...@gmail.com>
> >>>> > > wrote:
> >>>> > >
> >>>> > > > Hi Hequn,
> >>>> > > >
> >>>> > > > Thanks for bring up this discussion.
> >>>> > > >
> >>>> > > > +1 for add Python ML Pipeline API, even though the Java pipeline
> >>>> API
> >>>> > may
> >>>> > > > change.
> >>>> > > >
> >>>> > > > I would like to suggest create a FLIP for this API changes. :)
> >>>> > > >
> >>>> > > > Best,
> >>>> > > > Jincheng
> >>>> > > >
> >>>> > > >
> >>>> > > > Hequn Cheng  于2020年2月5日周三 下午5:24写道：
> >>>> > > >
> >>>> > > > > Hi everyone,
> >>>> > > > >
> >>>> > > > > FLIP-39[1] rebuilds the F

Re: [VOTE] Release flink-shaded 10.0, release candidate #3

[Hequn Cheng]

Thank you Chesnay for the release!

+1 (non-binding)

- The problem that exists in RC1 has been resolved.
- Release notes looks good.
- Built from source archive successfully.
- Check commit history manually. Nothing looks weird.
- Signatures and hash are correct.
- All artifacts have been deployed to the maven central repository.
- The website pull request looks good

Best, Hequn

On Fri, Feb 14, 2020 at 1:14 AM Zhu Zhu  wrote:

> +1 (non-binding)
>
> - checked release notes, JIRA tickets and commit history
> - verified the signature and checksum
> - checked the maven central artifacts
>   * examined the zookeeper shaded jars (both 3.4.10 and 3.5.6), curator and
> zookeeper classes are there and shaded
> - built from the source archive as well as the git tag
> - checked the website pull request
>
> Thanks,
> Zhu Zhu
>
> Ufuk Celebi  于2020年2月14日周五 上午12:32写道：
>
> > PS: Also verified the NOTICE changes since the last RC.
> >
> > On Thu, Feb 13, 2020 at 5:25 PM Ufuk Celebi  wrote:
> >
> > > Hey Chensay,
> > >
> > > +1 (binding).
> > >
> > > - Verified checksum ✅
> > > - Verified signature ✅
> > > - Jira changelog looks good to me ✅
> > > - Website PR looks good to me ✅
> > > - Verified no unshaded dependencies (except the Hadoop modules which I
> > > think is expected) ✅
> > > - Verified dependency management fix FLINK-15540
> > > (commons-collections:3.2.2 as expected) ✅
> > > - Verified pom exclusion fix FLINK-15815 (no META-INF/maven except for
> > > flink-shaded-force-shading and the Hadoop modules which I think is
> > > expected) ✅
> > >
> > > – Ufuk
> > >
> > > On Thu, Feb 13, 2020 at 3:08 PM Yu Li  wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Checked issues listed in release notes: ok
> > > > Checked sums and signatures: ok
> > > > Checked the maven central artifices: ok
> > > > Built from source: ok (8u101, 11.0.4)
> > > > Built from source (with -Dshade-sources): ok (8u101, 11.0.4)
> > > > Checked contents of zookeeper shaded jars: ok
> > > > - no unshaded classes
> > > > - shading pattern is correct
> > > > Checked website pull request listing the new release: ok
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Wed, 12 Feb 2020 at 22:09, Chesnay Schepler 
> > > wrote:
> > > >
> > > > > Hi everyone,
> > > > > Please review and vote on the release candidate #3 for the version
> > > 10.0,
> > > > > as follows:
> > > > > [ ] +1, Approve the release
> > > > > [ ] -1, Do not approve the release (please provide specific
> comments)
> > > > >
> > > > >
> > > > > The complete staging area is available for your review, which
> > includes:
> > > > > * JIRA release notes [1],
> > > > > * the official Apache source release to be deployed to
> > dist.apache.org
> > > > > [2], which are signed with the key with fingerprint 11D464BA [3],
> > > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > > * source code tag "release-10.0-rc3 [5],
> > > > > * website pull request listing the new release [6].
> > > > >
> > > > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > > > approval, with at least 3 PMC affirmative votes.
> > > > >
> > > > > Thanks,
> > > > > Chesnay
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346746
> > > > > [2]
> > > https://dist.apache.org/repos/dist/dev/flink/flink-shaded-10.0-rc3/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > https://repository.apache.org/content/repositories/orgapacheflink-1337
> > > > > [5]
> > > > >
> > > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=flink-shaded.git;a=tag;h=refs/tags/release-10.0-rc3
> > > > > [6] https://github.com/apache/flink-web/pull/304
> > > > >
> > > > >
> > > > >
> > > > >
> > >
> >
>

Re: [ANNOUNCE] New Documentation Style Guide

[Hequn Cheng]

Thanks Marta and Aljoscha for the detailed document! This is very helpful.

Best,
Hequn

On Mon, Feb 17, 2020 at 9:07 PM Aljoscha Krettek 
wrote:

> Just to be clear: I didn't write this style guide. Marta (in cc) wrote
> this, I just finally merged it.
>
> Best,
> Aljoscha
>
> On 17.02.20 11:45, jincheng sun wrote:
> > Thanks for this great job Aljoscha!
> >
> > Best,
> > Jincheng
> >
> >
> >
> > Yu Li  于2020年2月17日周一 下午6:42写道：
> >
> >> I think the guide itself is a great example of the new style! Thanks for
> >> driving this, Aljoscha!
> >>
> >> Best Regards,
> >> Yu
> >>
> >>
> >> On Mon, 17 Feb 2020 at 16:44, Becket Qin  wrote:
> >>
> >>> The guideline is very practical! I like it! Thanks for putting it
> >> together,
> >>> Aljoscha.
> >>>
> >>> Jiangjie (Becket) Qin
> >>>
> >>> On Mon, Feb 17, 2020 at 10:04 AM Xintong Song 
> >>> wrote:
> >>>
>  Thanks for the summary!
> 
>  I've read through the guidelines and found it very helpful. Many of
> the
>  questions I had while working on the 1.10 docs the answer can be found
> >> in
>  the guideline. And it also inspires me with questions I have never
> >>> thought
>  of, especially the language style part.
> 
>  Thank you~
> 
>  Xintong Song
> 
> 
> 
>  On Sun, Feb 16, 2020 at 12:55 AM Zhijiang
>   wrote:
> 
> > Thanks for bringing this great and valuable document.
> >
> > I read through the document and was inspired especially by some
> >>> sections
> > in "Voice and Tone" and "General Guiding Principles".
> >   I think it is not only helpful for writing Flink documents, but
> also
> > provides guideline/benefit for other writings.
> > It also reminded me to extend the Flink glossary if necessary.
> >
> > Best,
> > Zhijiang
> >
> >
> > --
> > From:Jingsong Li 
> > Send Time:2020 Feb. 15 (Sat.) 23:21
> > To:dev 
> > Subject:Re: [ANNOUNCE] New Documentation Style Guide
> >
> > Thank for the great work,
> >
> > In 1.10, I have modified and reviewed some documents. In that
> >> process,
> > sometimes there is some confusion, how to write is the standard. How
> >> to
> > write is correct to the users.
> > Docs style now tells me. Learned a lot.
> >
> > Best,
> > Jingsong Lee
> >
> > On Sat, Feb 15, 2020 at 10:00 PM Dian Fu 
> >>> wrote:
> >
> >> Thanks for the great work! This is very helpful to keep the
>  documentation
> >> style consistent across the whole project. It's also very helpful
> >> for
> >> non-native English contributors like me.
> >>
> >>> 在 2020年2月15日，下午3:42，Jark Wu  写道：
> >>>
> >>> Great summary! Thanks for adding the translation specification in
> >>> it.
> >>> I learned a lot from the guide.
> >>>
> >>> Best,
> >>> Jark
> >>>
> >>> On Fri, 14 Feb 2020 at 23:39, Aljoscha Krettek <
> >>> aljos...@apache.org>
> >> wrote:
> >>>
>  Hi Everyone,
> 
>  we just merged a new style guide for documentation writing:
>  https://flink.apache.org/contributing/docs-style.html.
> 
>  Anyone who is writing documentation or is planning to do so
> >> should
> > check
>  this out. Please open a Jira Issue or respond here if you have
> >> any
>  comments or questions.
> 
>  Some of the most important points in the style guide are:
> 
>    - We should use direct language and address the reader as you
>  instead
>  of passive constructions. Please read the guide if you want to
>  understand what this means.
> 
>    - We should use "alert blocks" instead of simple inline alert
> >>> tags.
>  Again, please refer to the guide to see what this means exactly
> >> if
>  you're not sure.
> 
>  There's plenty more and some interesting links about
>  technical/documentation writing as well.
> 
>  Best,
>  Aljoscha
> 
> >>
> >>
> >
> > --
> > Best, Jingsong Lee
> >
> >
> 
> >>>
> >>
> >
>

[DISCUSS] Improvements on FLIP Process

[Hequn Cheng]

Hi everyone,

Currently, when we create a FLIP we follow the FLIP process in the Flink
Improvement Proposals wiki[1]. The process mainly includes the following
steps:
1. Create a FLIP wiki page.
2. Raise the discussion on the mailing list.
3. Once the proposal is finalized, call a vote to have the proposal adopted.
There is also a discussion[2] on the FLIP process which may be helpful for
you.

As it is not allowed commented on the wiki, we usually have a google doc
for the discussion at step 2 and whenever there is a change we need to pick
it to the wiki page. This makes things somehow redundant. To solve this, we
can rearrange the step a little bit and avoid the pick:
1. Raise the discussion on the mailing list. The subject of the thread is
of the format [DISCUSS][FLIP] {your FLIP heading}. Also, the design doc
should follow the FLIP-Template strictly. (The [FLIP] tag is used to inform
people that it is a FLIP discussion and more attention should be paid.)
2. Create a FLIP wiki page once we reached an agreement on the discussion.
We can simply copy the google doc into the FLIP wiki page.
3. Once the proposal is finalized, call a vote to have the proposal
adopted. It should be noted that we should always vote on a FLIP wiki page
instead of a google doc. The wiki page is the final version of the google
doc.

This can bring some benefits:
1. Make the discussion more effective as we force people to write and
discuss on a google doc that follows the FLIP template which
includes necessary information such as Motivation, Interfaces, Proposed
changes, etc.
2. Avoid redundant pick from google doc to Flink wiki page. Once we reached
an agreement on the discussion, we can simply copy the google doc into the
FLIP wiki page.
3. As adopted FLIP should mostly be "immutable", we can even make the wiki
page PMC or committer editable since it just needs a simple copy from the
google doc.

Looking forward to your feedback!

Best,
Hequn

[1]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-META-FLIP-Sticking-or-not-to-a-strict-FLIP-voting-process-td29978.html#a29988

Re: [DISCUSS] Improvements on FLIP Process

[Hequn Cheng]

; > > firstly.
> > > > > > And we can also kindly list these specific considerations/reasons
> > > > > > for google doc concerns as said below in the guideline doc.
> > > > > >
> > > > > > To do so, we still retain this option for some people who prefer
> to
> > > > > google
> > > > > > doc or willing to provide it in some corner cases.
> > > > > > Of course I am also happy to eliminate google doc completely.
> > > > > >
> > > > > > Best,
> > > > > > Zhijiang
> > > > > >
> > > > > >
> > > > > >
> --
> > > > > > From:Kostas Kloudas 
> > > > > > Send Time:2020 Feb. 18 (Tue.) 23:03
> > > > > > To:dev 
> > > > > > Subject:Re: [DISCUSS] Improvements on FLIP Process
> > > > > >
> > > > > > +1 to what Aljoscha and Timo are proposing.
> > > > > >
> > > > > > I would lean towards eliminating Google Docs altogether.
> > > > > > I think they served a purpose when discussions were among 3 to 4
> > > > > > people but with the current size of the community and the amount
> of
> > > > > > participants per discussion they become difficult to follow.
> > > > > >
> > > > > > Best,
> > > > > > Kostas
> > > > > >
> > > > > > On Tue, Feb 18, 2020 at 3:36 PM Timo Walther  >
> > > > wrote:
> > > > > > >
> > > > > > > +1 to what Aljoscha said.
> > > > > > >
> > > > > > > The past has shown that discussions in Google docs do not reach
> > all
> > > > > > > interested parties and the tracability of design decisions
> > becomes
> > > > > > > difficult. Google services are also partially inaccessible in
> > > certain
> > > > > > > parts of world.
> > > > > > >
> > > > > > > We should actually do the opposite and not allow Google docs as
> > > FLIPs
> > > > > > > anymore. Commenting should be disabled by default.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Timo
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 18.02.20 15:20, Aljoscha Krettek wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > thanks for starting this discussion!
> > > > > > > >
> > > > > > > > However, I have a somewhat opposing opinion to this: we
> should
> > > > > disallow
> > > > > > > > using Google Docs for FLIPs and FLIP discussions and follow
> the
> > > > > already
> > > > > > > > established process more strictly.
> > > > > > > >
> > > > > > > > My reasons for this are:
> > > > > > > >   - discussions on the Google Doc are not reflected in Apache
> > > > > > > > infrastructure
> > > > > > > >   - discussions on Google Docs are non-linear and hard to
> > follow
> > > > > > > >   - when discussions on Google Docs are resolved these
> > > discussions
> > > > > are
> > > > > > > > not visible/re-readable anymore (I know there's history, but
> > meh)
> > > > > > > >   - if discussion is kept purely to the ML this is easily
> > > > observable
> > > > > > for
> > > > > > > > any interested parties and it's there if somewhat want's to
> > > recheck
> > > > > the
> > > > > > > > discussion in the future
> > > > > > > >   - going from Google Doc to Wiki is an extra step that seems
> > > > > > > > unnecessary to me (but that's just personal opinion, I mean,
> I
> > > > don't
> > > > > > > > have to do the extra work here...)
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Aljoscha
> > > > > > > >
> > > > &g

Re: [ANNOUNCE] Apache Flink-shaded 10.0 released

[Hequn Cheng]

Thanks a lot for the release Chesnay!
Also thanks to everyone who contributes to this release!

Best, Hequn

On Thu, Feb 20, 2020 at 11:11 AM Yu Li  wrote:

> Thanks Chesnay and all participants for making the release possible!
>
> Best Regards,
> Yu
>
>
> On Thu, 20 Feb 2020 at 09:50, Zhu Zhu  wrote:
>
> > Thanks Chesnay for the great work and everyone who helps with the
> > improvements and release!
> >
> > Thanks,
> > Zhu Zhu
> >
> > Dian Fu  于2020年2月20日周四 上午9:44写道：
> >
> > > Thanks Chesnay for the great work and everyone involved!
> > >
> > > Regards,
> > > Dian
> > >
> > > > 在 2020年2月20日，上午12:21，Zhijiang 
> 写道：
> > > >
> > > > Thanks Chesnay for making the release efficiently and also thanks to
> > all
> > > the other participants!
> > > >
> > > > Best,
> > > > Zhijiang
> > > >
> > > >
> > > > --
> > > > From:Till Rohrmann  trohrm...@apache.org
> > >>
> > > > Send Time:2020 Feb. 19 (Wed.) 22:21
> > > > To:dev mailto:dev@flink.apache.org>>
> > > > Subject:Re: [ANNOUNCE] Apache Flink-shaded 10.0 released
> > > >
> > > > Thanks for making the release possible Chesnay and everyone who was
> > > > involved!
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Wed, Feb 19, 2020 at 7:47 AM jincheng sun <
> sunjincheng...@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Thanks a lot for the release Chesnay!
> > > >> And thanks to everyone who make this release possible!
> > > >>
> > > >> Best,
> > > >> Jincheng
> > > >>
> > > >>
> > > >> Chesnay Schepler  于2020年2月19日周三 上午12:45写道：
> > > >>
> > > >>> The Apache Flink community is very happy to announce the release of
> > > >>> Apache Flink-shaded 10.0.
> > > >>>
> > > >>> The flink-shaded project contains a number of shaded dependencies
> for
> > > >>> Apache Flink.
> > > >>>
> > > >>> Apache Flink(r) is an open-source stream processing framework for
> > > >>> distributed, high-performing, always-available, and accurate data
> > > >>> streaming applications.
> > > >>>
> > > >>> The release is available for download at:
> > > >>> https://flink.apache.org/downloads.html
> > > >>>
> > > >>> The full release notes are available in Jira:
> > > >>>
> > > >>>
> > > >>
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346746
> > > >>>
> > > >>> We would like to thank all contributors of the Apache Flink
> community
> > > >>> who made this release possible!
> > > >>>
> > > >>> Regards,
> > > >>> Chesnay
> > >
> > >
> >
>

Re: [DISCUSS] Kicking off the 1.11 release cycle

[Hequn Cheng]

Thanks a lot for kicking off the 1.11 release and thanks Piotr and Zhijiang
for volunteering as the release managers!
Also +1 for the *anticipated feature freeze date* around end of April. A
shorter release cycle would be nice.

Best,
Hequn

On Thu, Feb 20, 2020 at 6:08 PM lining jing  wrote:

> Thanks for kicking off.
>
> +1 for the *anticipated feature freeze date* end of April.
>
> Zhu Zhu  于2020年2月20日周四 下午6:04写道：
>
> > Thanks Piotr and Zhijiang for volunteering as the release managers!
> >
> > +1 for aiming with the feature freeze date for end of April.
> >
> > Thanks,
> > Zhu Zhu
> >
> > Piotr Nowojski  于2020年2月20日周四 下午4:26写道：
> >
> > > Hi,
> > >
> > > Thank you for giving me and Zhijiang opportunity and entrusting us with
> > > the release :)
> > >
> > > +1 for aiming with the feature freeze for and of April. I think all
> > things
> > > consider that’s a good timing.
> > >
> > > Piotrek
> > >
> > > > On 20 Feb 2020, at 04:22, Yu Li  wrote:
> > > >
> > > > Thanks Stephan for the kicking off! And thanks Piotr and Zhijiang for
> > > > volunteering as the release managers!
> > > >
> > > > Aiming for a shorter release cycle than 1.10 makes a lot of sense,
> and
> > +1
> > > > to target the feature freeze date around end of April.
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Thu, 20 Feb 2020 at 10:37, Xintong Song 
> > > wrote:
> > > >
> > > >> Thanks for the kicking off.
> > > >>
> > > >> +1 for Zhijiang & Piotr (if he agrees to) being the release
> managers.
> > > >> Thanks for the volunteering.
> > > >> +1 for the *anticipated feature freeze date* end of April
> > > >>
> > > >> Thank you~
> > > >>
> > > >> Xintong Song
> > > >>
> > > >>
> > > >>
> > > >> On Thu, Feb 20, 2020 at 10:30 AM Yuan Mei 
> > > wrote:
> > > >>
> > > >>> +1
> > > >>>
> > > >>> 3-month sounds a very reasonable duration for a release.
> > > >>>
> > > >>> On Thu, Feb 20, 2020 at 9:53 AM tison 
> wrote:
> > > >>>
> > >  Thanks for kicking off the discussion Stephan!
> > > 
> > >  +1 for the *anticipated feature freeze date* end of April and
> > >  Zhijiang & Piotr to be the release managers.
> > > 
> > >  Best,
> > >  tison.
> > > 
> > > 
> > >  Dian Fu  于2020年2月20日周四 上午9:41写道：
> > > 
> > > > Thanks Stephan kicking off this discussion and Zhijiang
> > volunteering
> > > >> as
> > > > one of the release managers.
> > > >
> > > > +1 for the "feature freeze date" around end of April. There are
> > still
> > >  more
> > > > than 2 months left, so I think it's a reasonable time.
> > > >
> > > > Thanks,
> > > > Dian
> > > >
> > > >> 在 2020年2月19日，下午10:52，Aljoscha Krettek  写道：
> > > >>
> > > >> +1
> > > >>
> > > >> Although I would hope that it can be more than just
> "anticipated".
> > > >>
> > > >> Best,
> > > >> Aljoscha
> > > >>
> > > >> On 19.02.20 15:40, Till Rohrmann wrote:
> > > >>> Thanks for volunteering as one of our release managers
> Zhijiang.
> > > >>> +1 for the *anticipated feature freeze date* end of April. As
> we
> > > >> go
> > > > along
> > > >>> and collect more data points we might be able to strengthen our
> > > >>> initial anticipation.
> > > >>> Cheers,
> > > >>> Till
> > > >>> On Wed, Feb 19, 2020 at 4:44 AM Zhijiang <
> > > >>> wangzhijiang...@aliyun.com
> > > > .invalid>
> > > >>> wrote:
> > >  Thanks for kicking off the next release and the introduction,
> > >  @Stephan!
> > > 
> > >  It's my pleasure to become the release manager and involve in
> > > >> other
> > >  community works. I am working together with @Piotr for a long
> > > >> time,
> > >  so
> > >  very happy to cooperate for the release manager work again.
> The
> > > > previous
> > >  release work was always well done, and I can learn a lot from
> > > >> these
> > > > rich
> > >  experiences.
> > > 
> > >  +1 for the "feature freeze date" around end of April.
> > >  Although we have the FF SF in the meantime, fortunately there
> > > >> are
> > > >>> no
> > > > long
> > >  public holidays during this period in China.
> > > 
> > >  Best,
> > >  Zhijiang
> > > 
> > > 
> > > 
> > > >> --
> > >  From:Stephan Ewen 
> > >  Send Time:2020 Feb. 19 (Wed.) 01:15
> > >  To:dev 
> > >  Cc:zhijiang ; pnowojski <
> > > >> pnowoj...@apache.org
> > > 
> > >  Subject:[DISCUSS] Kicking off the 1.11 release cycle
> > > 
> > >  Hi all!
> > > 
> > >  Now that the 1.10 release is out (congrats again, everyone!),
> I
> > >  wanted
> > > > to
> > >  bring up some questions about organizing the next release
> cycle.
> > > 
> > >  The most important questions at the beg

Re: [VOTE] Support Python ML Pipeline API

[Hequn Cheng]

Thanks all for the votes.

So far, we have
   - 4 binding +1 votes (Jincheng, Dian, Zhijiang and Rong)
   - 1 non-binding +1 votes (Yuan)
   - 0 -1 votes

I will move the FLIP to accepted!

Best, Hequn


On Sat, Feb 15, 2020 at 12:45 AM Rong Rong  wrote:

> +1 (binding)
>
> On Fri, Feb 14, 2020 at 7:15 AM Zhijiang  .invalid>
> wrote:
>
> > +1 (binding), it is valuable to enhance the python API for supporting ML
> > well.
> >
> > Best,
> > Zhijiang
> >
> >
> > --
> > From:Dian Fu 
> > Send Time:2020 Feb. 14 (Fri.) 15:07
> > To:dev 
> > Subject:Re: [VOTE] Support Python ML Pipeline API
> >
> > +1 (binding)
> >
> > Regards,
> > Dian
> >
> > > 在 2020年2月14日，下午2:49，Yuan Mei  写道：
> > >
> > > +1 vote
> > >
> > > This is one of the most important things for Flink ML framework to be
> > > widely adopted since most data scientists use python.
> > >
> > > Best
> > >
> > > Yuan
> > >
> > > On Fri, Feb 14, 2020 at 9:45 AM Hequn Cheng  wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> I'd like to start the vote of FLIP-96[1] which is discussed and
> reached
> > >> consensus in the discussion thread[2].
> > >> The vote will be open for at least 72 hours. Unless there is an
> > objection,
> > >> I will try to close it by Feb 19, 2020 02:00 UTC if we have received
> > >> sufficient votes.
> > >>
> > >> Thanks,
> > >> Hequn
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-96%3A+Support+Python+ML+Pipeline+API
> > >> [2]
> > >>
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-Python-ML-Pipeline-API-td37291.html
> > >>
> >
> >
>

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Hequn Cheng]

Congratulations Jingsong! Well deserved.

Best,
Hequn

On Fri, Feb 21, 2020 at 2:42 PM Yang Wang  wrote:

> Congratulations！Jingsong. Well deserved.
>
>
> Best,
> Yang
>
> Zhijiang  于2020年2月21日周五 下午1:18写道：
>
>> Congrats Jingsong! Welcome on board!
>>
>> Best,
>> Zhijiang
>>
>> --
>> From:Zhenghua Gao 
>> Send Time:2020 Feb. 21 (Fri.) 12:49
>> To:godfrey he 
>> Cc:dev ; user 
>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>>
>> Congrats Jingsong！
>>
>>
>> *Best Regards,*
>> *Zhenghua Gao*
>>
>>
>> On Fri, Feb 21, 2020 at 11:59 AM godfrey he  wrote:
>> Congrats Jingsong! Well deserved.
>>
>> Best,
>> godfrey
>>
>> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
>> Congratulations！Jingsong. You deserve it
>>
>> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
>> Congrats Jingsong!
>>
>> On Fri, 21 Feb 2020 at 11:41, Dian Fu  wrote:
>>
>> > Congrats Jingsong!
>> >
>> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
>> > >
>> > > Congratulations Jingsong! Well deserved.
>> > >
>> > > Best,
>> > > Jark
>> > >
>> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
>> > >
>> > >> Congratulations! Jingsong
>> > >>
>> > >>
>> > >> Best,
>> > >> Dan Zou
>> > >>
>> >
>> >
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>>
>>

Re: [PROPOSAL] Reverse the dependency from flink-streaming-java to flink-client

[Hequn Cheng]

Hi,

+1 to make flink-streaming-java an API only module and solve it sooner
rather than later.
It would be more clear to only expose an SDK module for writing jobs.

Another benefit I can see is: the flink-streaming-java would be scala-free
if we reverse the dependencies and this would be really nice for the Java
API module.

As for the issue of dependencies setup of users, I agree with Stephan that
it's ok to do so
if we add corresponding document and runtime error messages about the
changes.

Best,
Hequn


On Fri, Mar 6, 2020 at 3:03 AM Kostas Kloudas  wrote:

> Big +1 also from my side.
>
> This will eliminate some work-arounds used so far to bypass the module
> structure (like code using reflection to extract a JobGraph from a
> Pipeline).
>
> I agree with Stephan that with proper documentation, release notes and
> tooling update, it will hopefully not be a big hassle for users to
> migrate.
> Also I think it should be done as early in the release as possible, so
> that we can give it enough exposure and testing. In the past, such
> deep changes late in the release have led to longer release-testing
> periods and, eventually, longer release cycles.
>
> Cheers,
> Kostas
>
> On Thu, Mar 5, 2020 at 3:35 PM Stephan Ewen  wrote:
> >
> > +1 to this fix, in general.
> >
> > If the main issue is that users have to now add "flink-clients"
> explicitly,
> > then I think this is okay, if we spell it out prominently in the release
> > notes, and make sure quickstarts / etc are updated, and have a good error
> > message when client/runtime classes are not found.
> >
> > On Thu, Mar 5, 2020 at 2:56 PM Aljoscha Krettek 
> wrote:
> >
> > > Hi,
> > >
> > > thanks for starting the discussion, Tison!
> > >
> > > I'd like to fix this dependency mess rather sooner than later, but we
> do
> > > have to consider the fact that we are breaking the dependency setup of
> > > users. If they they only had a dependency on flink-streaming-java
> before
> > > but used classes from flink-clients they would have to explicitly add
> > > this dependency now.
> > >
> > > Let's see what others think.
> > >
> > > Best,
> > > Aljoscha
> > >
> > > On 05.03.20 02:53, tison wrote:
> > > > Hi devs,
> > > >
> > > > Here is a proposal to reverse the dependency from
> flink-streaming-java to
> > > > flink-client, for a proper
> > > > module dependency graph. Since it changes current structure, it
> should be
> > > > discussed publicly.
> > > >
> > > > The original idea comes from that flink-streaming-java acts as an API
> > > only
> > > > module just as what
> > > > we do in its batch companion flink-java. If a Flink user want to
> write a
> > > > minimum DataStream
> > > > program, the only dependency should be flink-streaming java.
> > > >
> > > > However, currently as it is implemented, flink-client and even
> > > > flink-runtime are transitively polluted
> > > > in when user depends on flink-streaming-java. These dependencies
> polluted
> > > > in as
> > > >
> > > > flink-client:
> > > >- previously, ClusterClient, which is removed by FLIP-73 Executors
> > > >- accidentally, ProgramInvocationException, we just throw in
> place as
> > > it
> > > > is accessible.
> > > >- transitively, flink-optimizer, for one utility.
> > > >- transitively, flink-java, for several utilities.
> > > > flink-runtime:
> > > >- mainly for JobGraph generating.
> > > >
> > > > With a previous discussion with @Aljoscha Krettek <
> aljos...@apache.org>
> > > our
> > > > goal is briefly making flink-streaming-java
> > > > an API only module. As a first step we can break the dependency from
> > > > flink-streaming-java to
> > > > flink-client[1][2].
> > > >
> > > > With this first step, continuously we factor out common utilities in
> > > > flink-java to
> > > > flink-core and eventually eliminate dependencies from streaming to
> batch;
> > > > while
> > > > orthogonally, we factor out job compilation logic into
> > > > flink-streaming-compiler module and
> > > > break the dependency to flink-runtime. The final dependency graph
> will
> > > be:
> > > >
> > > >
> > > > flink-client -> flink-streaming-compiler -> flink-runtime
> > > >   \->
> > > > flink-streaming-java
> > > >
> > > > Looking forward to your feedback. Basically whether or not it is in a
> > > right
> > > > direction, and if so,
> > > > how the community integrates this proposal.
> > > >
> > > > Best,
> > > > tison.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-15090
> > > > [2] https://issues.apache.org/jira/browse/FLINK-16427
> > > >
> > >
>

Re: Request for more Jira rights

[Hequn Cheng]

Hi Niels,

Thanks a lot for your contribution!

The rule has been changed now. As the contribution guide said, only
committers have the permission to assign somebody.
Feel free to ask committers to assign the jira to you with a comment under
the jira once all requirements for the ticket are met.

Thanks,
Hequn

On Sat, Mar 7, 2020 at 3:07 AM Niels Basjes  wrote:

> Hi,
>
> I've created some jira tickets and for some of them I've put up a merge
> request.
>
> I noticed that the Flinkbot warns:
>
> This pull request references an unassigned Jira ticket
> . According to the code
> contribution guide
> , tickets need
> to be assigned before starting with the implementation work.
>
> Sounds good to me.
> However at this point I cannot assign these tickets to myself.
> The strange thing is that I used to have that option in the past.
>
> I kindly request the privilege to assign tickets to myself if I choose to
> pick them up.
>
> Jira userid:  nielsbasjes
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>

Re: [DISCUSS] Disable "Squash and merge" button for Flink repository on GitHub

[Hequn Cheng]

Hi,

Thank you all for the discussion!

On one hand, due to the network problem, the "Squash and merge" button is
very helpful. I’m also getting more and more rely on it as it is also very
convenient.

On the other hand, I think the concerns raised by Stephan are valid and we
should pay attention to it, i.e., add PR id and don’t squash everything,
etc. Such changes can never be changed once been checked in. Considering
this, I have updated the committer guide wiki page[1] with some
descriptions about the GitHub web UI and some notices about merging code.
Hope it helps and feel free to add more if you find something has still
been missed.

Best,
Hequn

[1]
https://cwiki.apache.org/confluence/display/FLINK/General+Information+for+Committers


On Fri, Mar 6, 2020 at 6:55 PM Stephan Ewen  wrote:

> All right sounds fair.
> Especially that the button helps in case of unstable networks makes sense.
>
>
> On Fri, Mar 6, 2020 at 11:04 AM Aljoscha Krettek 
> wrote:
>
> > If there is a noreply email address that could be on purpose. This
> > happens when you configure github to not show your real e-mail address.
> > This also happens when contributors open a PR and don't want to show
> > their real e-mail address. I talked to at least 1 person that had it set
> > up like this on purpose.
> >
> > Best,
> > Aljoscha
> >
> > On 05.03.20 17:37, Stephan Ewen wrote:
> > > It looks like this feature still messes up email addresses, for example
> > if
> > > you do a "git log | grep noreply" in the repo.
> > >
> > > Don't most PRs consist anyways of multiple commits where we want to
> > > preserve "refactor" and "feature" differentiation in the history,
> rather
> > > than squash everything?
> > >
> > > On Thu, Mar 5, 2020 at 4:54 PM Piotr Nowojski 
> > wrote:
> > >
> > >> Hi,
> > >>
> > >> If it’s really not preserving ownership (I didn’t notice the problem
> > >> before), +1 for removing “squash and merge”.
> > >>
> > >> However -1 for removing “rebase and merge”. I didn’t see any issues
> with
> > >> it and I’m using it constantly.
> > >>
> > >> Piotrek
> > >>
> > >>> On 5 Mar 2020, at 16:40, Jark Wu  wrote:
> > >>>
> > >>> Hi all,
> > >>>
> > >>> Thanks for the feedbacks. But I want to clarify the motivation to
> > disable
> > >>> "Squash and merge" is just because of the regression/bug of the
> missing
> > >>> author information.
> > >>> If GitHub fixes this later, I think it makes sense to bring this
> button
> > >>> back.
> > >>>
> > >>> Hi Stephan & Zhijiang,
> > >>>
> > >>> To be honest, I love the "Squash and merge" button and often use it.
> It
> > >>> saves me a lot of time to merge PRs, because pulling and pushing
> > commits
> > >> in
> > >>> China is very unstable.
> > >>>
> > >>> I don't think the potential problems you mentioned is a "problem".
> > >>> For "Squash and merge",
> > >>> - "Merge commits": there is no "merge" commits, because GitHub will
> > >> squash
> > >>> commits and rebase the commit and then add to the master branch.
> > >>> - "This closes #" line to track back: when you click "Squash and
> > >>> merge", it allows you to edit the title and description, so you can
> > >>> add "This closes #" message to the description the same with in
> the
> > >>> local git. Besides, GitHub automatically append "(#)" after the
> > >> title,
> > >>> which is also helpful to track.
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>> On Thu, 5 Mar 2020 at 23:36, Robert Metzger 
> > wrote:
> > >>>
> >  +1 for disabling this feature for now.
> > 
> >  Thanks a lot for spotting this!
> > 
> >  On Thu, Mar 5, 2020 at 3:54 PM Zhijiang  >  .invalid>
> >  wrote:
> > 
> > > +1 for disabling "Squash and merge" if feasible to do that.
> > >
> > > The possible benefit to use this button is for saving some efforts
> to
> > > squash some intermediate "[fixup]" commits during PR review.
> > > But it would bring more potential problems as mentioned below,
> > missing
> > > author information and message of "This closes #", etc.
> > > Even it might cause unexpected format of long commit content
> > >> description
> > > if not handled carefully in the text box.
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > --
> > > From:tison 
> > > Send Time:2020 Mar. 5 (Thu.) 21:34
> > > To:dev 
> > > Subject:Re: [DISCUSS] Disable "Squash and merge" button for Flink
> > > repository on GitHub
> > >
> > > Hi Yadong,
> > >
> > > Maybe we firstly reach out INFRA team and see the reply from their
> > >> side.
> > >
> > > Since the actual operator is INFRA team, in the dev mailing list we
> > can
> > > focus on motivation and
> > > wait for the reply.
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Yadong Xie  于2020年3月5日周四 下午9:29写道：
> > >
> > >> Hi Jark
> > >>
> > >> I think GitHub UI can not disable 

[DISCUSS] FLIP-112: Support User-Defined Metrics for Python UDF

[Hequn Cheng]

Hi everyone,

FLIP-58 adds the support for Python UDFs, but user-defined metrics
have not been supported yet. With metrics, users can report and monitor
the UDF status to get a deeper understanding of the execution,
so in this FLIP, we want to support metrics for Python UDFs.

Previously, Jincheng and I discussed offline about the support of
metrics for Python UDFs. We'd like to achieve three goals for
supporting metrics for Python UDFs:
- Support user-defined metrics including Counters, Gauges, Meters,
  Distributions in Python UDFs.
- Support defining user scopes.
- Support defining user variables.

More details can be found in the FLIP wiki page[1] and we are looking
forward
to your feedback.

Best,
Hequn

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF

Re: [DISCUSS] Releasing Flink 1.10.1

[Hequn Cheng]

Hi Yu,

Thanks a lot for raising the discussion and volunteer as the release
manager!

I found there are some other issues[1] which are marked as a blocker:
- FLINK-16454 Update the copyright year in NOTICE files
- FLINK-16262 Class loader problem with
FlinkKafkaProducer.Semantic.EXACTLY_ONCE and usrlib directory
- FLINK-16170 SearchTemplateRequest ClassNotFoundException when use
flink-sql-connector-elasticsearch7
- FLINK-16018 Improve error reporting when submitting batch job (instead of
AskTimeoutException)

These may also need to be resolved in 1.10.1.

Best,
Hequn

[1] https://issues.apache.org/jira/projects/FLINK/versions/12346891


On Tue, Mar 10, 2020 at 6:48 PM Yu Li  wrote:

> Hi Jincheng,
>
> Yes, your help would be very helpful. Thanks a lot!
>
> Best Regards,
> Yu
>
>
> On Tue, 10 Mar 2020 at 18:24, jincheng sun 
> wrote:
>
> > Thanks for bring up the discussion Yu. I would like to give you a hand at
> > the last stage when the RC is finished.(If you need)  :)
> >
> > Best,
> > Jincheng
> >
> >
> >
> > Yu Li  于2020年3月10日周二 下午5:49写道：
> >
> > > Hi All,
> > >
> > > It has been almost one month since we released Flink 1.10.0. We already
> > > have more than 40 resolved improvements/bugs in the release-1.10
> branch,
> > > and I propose to start the 1.10.1 release cycle.
> > >
> > > Most noticeable fixes are:
> > >
> > > - FLINK-16241 [legal] Remove the license and notice file in
> flink-ml-lib
> > > module
> > > - FLINK-16313 Fix RocksDB resource leak in flink-state-processor-api
> > > - FLINK-16161 Statistics zero should be known in HiveCatalog
> > > - FLINK-2336 ArrayIndexOufOBoundsException in TypeExtractor when
> mapping
> > > - FLINK-16108 StreamSQLExample is failed if running in blink planner
> > > - FLINK-16139 Co-location constraints are not reset on task recovery in
> > > DefaultScheduler
> > > - FLINK-16414 Create udaf/udtf function using sql casuing
> > > ValidationException: SQL validation failed
> > >
> > > Furthermore, I think the following issues should be merged before
> 1.10.1
> > > release (especially the Metaspace OOM issue):
> > >
> > > - FLINK-16142 Memory Leak causes Metaspace OOM error on repeated job
> > > submission
> > > - FLINK-16406 Increase default value for JVM Metaspace to minimise its
> > > OutOfMemoryError
> > > - FLINK-16047 Blink planner produces wrong aggregate results with state
> > > clean up
> > > - FLINK-16070 Blink planner can not extract correct unique key for
> > > UpsertStreamTableSink
> > >
> > > I would volunteer as the release manager and kick off the release
> process
> > > once blocker issues are merged. What do you think?
> > >
> > > If there are any concerns or missing blocker issues need to be fixed in
> > > 1.10.1, please let me know. Thanks.
> > >
> > > Best Regards,
> > > Yu
> > >
> >
>

Re: [DISCUSS] Link Stateful Functions from the Flink Website

[Hequn Cheng]

Hi,

Thanks a lot for raising the discussion @Stephan.
+1 to increase the visibilities of the Stateful Functions.

Another option I'm think is adding a section(named Stateful Functions or
Flink Projects?)
under the "Latest Blog Posts". The advantage is we can add a picture and
some descriptions here.
A picture may attract more attention from the users when he/she visit the
website.
The picture can be the same one in [1].

In the future, if we have multiple Flink individual projects, we can also
turn the section into a Table list
to expose all of them.

What do you think?

Best,
Hequn

[1] https://ci.apache.org/projects/flink/flink-statefun-docs-master/

On Tue, Mar 10, 2020 at 11:13 PM Tzu-Li (Gordon) Tai 
wrote:

> +1 on the suggestion to add "What is Stateful Functions" to the left
> navigation bar.
> That might also mean it would be nice to have a slight rework to the main
> image on the website, illustrating the use cases of Flink (this one [1]).
> On the image it does mention "Event-Driven Applications", but there's
> somewhat missing a more direct connection from that term to the Stateful
> Functions project.
>
> As for what the "What is Stateful Functions?" button directs to, maybe that
> should point to a general concepts page. Initially, we can begin with the
> README contents on the project repo [2].
> As for the actual Statefun documentation link [3], I think we should link
> that from an item in the "Documentation" pull-down list.
>
> One last thing to increase visibility of the Statefun project just a bit
> more:
> There's a "Flink on Github" button on the very bottom of the navigation
> bar.
> What do you think about adding a "Flink Stateful Functions on Github"
> button there as well?
>
> Cheers,
> Gordon
>
> [1]
>
> https://github.com/apache/flink-web/blob/asf-site/img/flink-home-graphic.png
> [2] https://github.com/apache/flink-statefun/blob/master/README.md
> [3] https://ci.apache.org/projects/flink/flink-statefun-docs-master/
>
>
> On Tue, Mar 10, 2020 at 8:29 PM Yu Li  wrote:
>
> > +1 on adding a "What is Stateful Functions" link below the "What is
> Apache
> > Flink" entry and integrating into the Flink docs gradually (instead of
> > hiding it behind until fully integrated).
> >
> > Best Regards,
> > Yu
> >
> >
> > On Tue, 10 Mar 2020 at 19:33, Stephan Ewen  wrote:
> >
> > > Hi all!
> > >
> > > I think it would be nice to mention Stateful Function on the Flink
> > website.
> > > At the moment, Stateful Functions is very hard to discover, and with
> the
> > > first release of it under Apache Flink, it would be a good time to
> change
> > > that.
> > >
> > > My proposal would be to add a "What is Stateful Functions?" below the
> > "What
> > > is Apache Flink" entry in the sidenav, and point it to
> > > https://ci.apache.org/projects/flink/flink-statefun-docs-master/
> > > It is not ideal, yet, but it may serve as an intermediate solution
> until
> > we
> > > can make more involved attempt to rethink the website (for example to
> > also
> > > make SQL more prominent than it currently is).
> > >
> > > An alternative idea was to link it only from the docs, but this would
> be
> > a
> > > bit hidden, in my opinion.
> > >
> > > As a bit of background:
> > >   - The Stateful Functions docs a are a separate doc tree at the
> moment,
> > > because the code (with the docs) is for now in a separate repository
> and
> > > separately versioned/releases.
> > >   - The layout of the Stateful Functions docs is still in the
> > old/original
> > > format, from when it was an "outside Flink" project.
> > >   - There are plans to migrate this to the same stack as the Flink docs
> > and
> > > make it look consistent, but it would be nice to have the docs
> available
> > > in the meantime already.
> > >
> > > Best,
> > > Stephan
> > >
> >
>

Re: [DISCUSS] Features of Apache Flink 1.11

[Hequn Cheng]

Thanks Zhijiang and Piotr for kicking off the discussion and providing the
detailed list.
This would be very helpful for tracking the features.

BTW, as for PyFlink, it would be great if the feature list can also include
the following features:
- FLIP-112: Support User-Defined Metrics in Python UDF
- FLIP-114: Support Python UDF in SQL Client

Looking forward to the release!

Best,
Hequn



On Wed, Mar 11, 2020 at 1:02 PM Yu Li  wrote:

> Thanks for compiling the list of 1.11 efforts Zhijiang and Piotr! This
> helps a lot to better understand what the community is currently working
> on. Looking forward to another successful release.
>
> Best Regards,
> Yu
>
>
> On Wed, 11 Mar 2020 at 11:17, Zhijiang  .invalid>
> wrote:
>
> > Hi community,
> >
> >
> > Not more than one month ago we have released Flink 1.10. We are now
> > heading for the Flink 1.11 release and we, as release managers, would
> like
> > to share with you what are the features that the community is currently
> > working on and we are hoping that will be part of the Flink 1.11 release.
> > Currently we are aiming with the feature freeze to happen in late April.
> >
> > As for now, some of the features are in the very early stages of the
> > development or even brainstorming. Because of that, some of them do not
> > have associated JIRA tickets or FLIP documents. For the next progress
> > announcement we are hoping that this will be no longer the case.
> >
> > Please also note that because we are still relatively at the beginning of
> > the release cycle, some of the FLIPs haven’t yet been voted.
> >
> > - SQL / Table
> > - FLIP-42: Restructure documentation [1]
> > - FLIP-65: New type inference for Table API UDFs [2]
> > - FLIP-84: Improve TableEnv’s interface [3]
> > - FLIP-91 Introduce SQL client gateway and provide JDBC driver [4]
> > - FLIP-93: Introduce JDBC catalog and Postgres catalog [5]
> > - FLIP-105: Support to interpret and emit changelog in Flink SQL [6]
> > - FLIP-107: Reading table columns from different parts of source records
> > [7]
> > - [FLINK-14807] Add Table#collect API for fetching data [8]
> > - Support query and table hints
> > - ML / Connectors
> > - FLIP-27: New source API [9]
> > - [FLINK-15670] Wrap a source/sink pair to persist intermediate result
> for
> > subgraph failure recovery [10]
> > - Pulsar source / sink / catalog
> > - Update ML Pipeline API interface to better support Flink ML lib
> > algorithms
> > - PyFlink
> > - FLIP-58: Debugging and monitoring of Python UDF [11]
> > - FLIP-106: Expand the usage scope of Python UDF [12]
> > - Integration with most popular Python libraries (Pandas)
> > - Performance improvements of Python UDF
> > - Support running python UDF in docker workers
> > - Add Python ML API
> > - Fully support all kinds of Python UDF
> > - Web UI
> > - FLIP-98: Better back pressure detection [13]
> > - FLIP-99: Make max exception configurable [14]
> > - FLIP-100: Add attempt information [15]
> > - FLIP-102: Add more metrics to TaskManager [16]
> > - FLIP-103: Better TM/JM log display [17]
> > - [FLINK-14816] Add thread dump feature for TaskManager [18]
> > - Runtime
> > - FLIP-56: Support for dynamic slots on the TaskExecutor [19]
> > - FLIP-67: Support for cluster partitions [20]
> > - FLIP-76: Unaligned checkpoints [21]
> > - FLIP-83: Flink e2e performance testing framework [22]
> > - FLIP-85: Support cluster deploy mode [23]
> > - FLIP-92: Add N-Ary input stream operator in Flink [24]
> > - FLIP-108: Add GPU to the resource management (specifically for UDTF &
> > UDF) [25]
> > - FLIP-111: Consolidate docker images [26]
> > - Unified memory configuration for JobManager
> > - Specify upper bound for number of allocated TaskManagers
> > - [FLINK-9407] ORC format for StreamingFileSink [27]
> > - [FLINK-10742] Let Netty use Flink's buffers on downstream side [28]
> > - [FLINK-10934] Support per-job mode for Kubernetes integration [29]
> > - [FLINK-11395] Avro writer for StreamingFileSink [30]
> > - [FLINK-11427] Protobuf parquet writer for StreamingFileSink [31]
> > - [FLINK-11499] Extend StreamingFileSink BulkFormats to support arbitrary
> > roll policies [32]
> > - [FLINK-14106] Make SlotManager pluggable [33]
> > - [FLINK-15672] Switch to Log4j2 by default [34]
> > - [FLINK-15674] Consolidate Java and Scala type extraction stack [35]
> > - [FLINK-15679] Improve Flink’s ID system [36]
> > - [FLINK-15786] Use the separated classloader to load connectors’ jar
> [37]
> > - [FLINK-15788] Various Kubernetes improvements [38]
> > - [FLINK-15911][FLINK-15154] Support Flink work over NAT [39]
> > - [FLINK-16408] Bind user code class loader to lifetime of a slot [40]
> > - [FLINK-16428] Network memory management for backpressure [41]
> > - [FLINK-16430] Pipelined region scheduling [42]
> > - Calculate required shuffle memory before allocating slots
> > - State Backend:
> > - [FLINK-5763] Make savepoint self-contained / relocatable [43]
> > - [FLINK-8871] Complete checkpoint cancellation messages [44

Re: [DISCUSS] Drop Bucketing Sink

[Hequn Cheng]

Good idea! +1 for dropping the BucketingSink.

Best,
Hequn

> On Mar 12, 2020, at 10:40 PM, Robert Metzger  wrote:
> 
> Hi all,
> 
> I'm currently investigating a failing end to end test for the bucketing
> sink [1].
> The bucketing sink has been deprecated in the 1.9 release [2], because we
> have the new StreamingFileSink [3] for quite a while.
> Before putting any effort into fixing the end to end test for the sink, I
> wanted to propose dropping the bucketing sink from master for the upcoming
> 1.11 release.
> 
> What do you think?
> 
> 
> 
> [1] https://issues.apache.org/jira/browse/FLINK-16227
> [2] https://issues.apache.org/jira/browse/FLINK-13396
> [3] https://issues.apache.org/jira/browse/FLINK-9749

Re: [DISCUSS] FLIP-112: Support User-Defined Metrics for Python UDF

[Hequn Cheng]

@Dian Fu   Thanks a lot for the advice. The FLIP
page has been updated.

Best,
Hequn

On Thu, Mar 12, 2020 at 4:19 PM Dian Fu  wrote:

> Hi Hequn,
>
> Thanks for driving this. +1 to this feature.
>
> Just one minor comment: It seems that we will add an API get_metric_group
> for the Python class FunctionContext, could you update the FLIP reflecting
> this?
>
> Thanks,
> Dian
>
> > 在 2020年3月10日，下午3:38，Wei Zhong  写道：
> >
> > Hi Hequn,
> >
> > Thanks for driving this. +1 for the metrics support for Python UDF,
> which makes it much easier for users to monitor the execution of Python
> UDFs.
> >
> > Best,
> > Wei
> >
> >
> >> 在 2020年3月10日，15:32，Xingbo Huang  写道：
> >>
> >> Hi Hequn,
> >> thanks for drafting the FLIP and kicking off the discussion.
> >>
> >> +1 for this feature.
> >> I think this feature will be extremely convenient for PyFlink users.
> >>
> >> Best,
> >> Xingbo
> >>
> >> Hequn Cheng  于2020年3月9日周一 上午11:32写道：
> >>
> >>> Hi everyone,
> >>>
> >>> FLIP-58 adds the support for Python UDFs, but user-defined metrics
> >>> have not been supported yet. With metrics, users can report and monitor
> >>> the UDF status to get a deeper understanding of the execution,
> >>> so in this FLIP, we want to support metrics for Python UDFs.
> >>>
> >>> Previously, Jincheng and I discussed offline about the support of
> >>> metrics for Python UDFs. We'd like to achieve three goals for
> >>> supporting metrics for Python UDFs:
> >>> - Support user-defined metrics including Counters, Gauges, Meters,
> >>> Distributions in Python UDFs.
> >>> - Support defining user scopes.
> >>> - Support defining user variables.
> >>>
> >>> More details can be found in the FLIP wiki page[1] and we are looking
> >>> forward
> >>> to your feedback.
> >>>
> >>> Best,
> >>> Hequn
> >>>
> >>> [1]
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF
> >>>
> >
>
>

Re: [DISCUSS] FLIP-106: Support Python UDF in SQL Function DDL

[Hequn Cheng]

Big +1 on this feature! It would be great to extend the usage of Python UDF
in SQL scenarios.
The design doc looks good from my side now. Thank you for the update.

Best,
Hequn

On Tue, Mar 10, 2020 at 3:50 PM Wei Zhong  wrote:

> Hi Timo,
>
> Thanks for your reply.
>
> If we aim for the option 1, it makes sense for me to include the change in
> this FLIP as the option 1 does not change any public API. I'll update the
> FLIP page to illustrate this.
>
> Best,
> Wei
>
> > 在 2020年3月9日，17:58，Timo Walther  写道：
> >
> > Hi Wei,
> >
> > I agree with Dawid that we should defer the instantiation of temporary
> functions to compile time. In the long-term, we would like to integrate
> FunctionCatalog as a component of CatalogManager and unify the handling of
> catalog objects as much as possible.
> >
> > We should aim for your proposed option 1. For fluent definition of
> functions in Table API, we would still like to offer passing instances like
> `t.select(call(new ScalarFunction() { ... }))` that would be registered as
> temporary system functions.
> >
> > Regrds,
> > Timo
> >
> >
> > On 09.03.20 09:24, Wei Zhong wrote:
> >> Hi Dawid,
> >> I think defering the instantiation of temporary functions to compile
> time is quite a good idea but needs further discussion. As it is orthogonal
> with this FLIP, we could continue the discussion in a new thread later.
> What do you think?
> >> Best,
> >> Wei
> >>> 在 2020年3月5日，21:11，Wei Zhong  写道：
> >>>
> >>> Hi Dawid,
> >>>
> >>> Thanks for your suggestion.
> >>>
> >>> After some investigation, there are two designs in my mind about how
> to defer the instantiation of temporary system function and temporary
> catalog function to compile time.
> >>>
> >>> 1. FunctionCatalog accepts both FunctionDefinitions and uninstantiated
> temporary functions. The uninstantiated temporary functions will be
> instantiated when compiling. There is no public API change in this design,
> but the FunctionCatalog needs to store and process both FunctionDefinitions
> and uninstantiated temporary functions.
> >>>
> >>> 2. FunctionCatalog accepts only uninstantiated temporary functions. In
> this design we need to remove those APIs that accepts FunctionDefinitions
> from TableEnvironment, i.e. `void createTemporaryFunction(String path,
> UserDefinedFunction functionInstance)` and `void
> createTemporarySystemFunction(String name, UserDefinedFunction
> functionInstance)`. But the FunctionCatalog only needs to store and process
> uninstantiated temporary functions.
> >>>
> >>> As I don't know the details about the plan to store temporary
> functions as catalog functions instead of FunctionDefinitions, I'm not sure
> which solution fits more. It would be great if you could share more details
> or share some thoughts on these two solutions?
> >>>
> >>> Best,
> >>> Wei
> >>>
>  在 2020年3月4日，16:17，Dawid Wysakowicz  写道：
> 
>  Hi all,
>  I had a really quick look and from my perspective the proposal looks
> fine.
>  I share Jarks opinion that the instantiation could be done at a later
>  stage. I agree with Wei it requires some changes in the internal
>  implementation of the FunctionCatalog, to store temporary functions as
>  catalog functions instead of FunctionDefinitions, but we have that on
> our
>  agenda anyway. I would suggest investigating if we could do that as
> part of
>  this flip already. Nevertheless this in theory can be also done later.
> 
>  Best,
>  Dawid
> 
>  On Mon, 2 Mar 2020, 14:58 Jark Wu,  wrote:
> 
> > Thanks for the explanation, Wei!
> >
> > On Mon, 2 Mar 2020 at 20:59, Wei Zhong 
> wrote:
> >
> >> Hi Jark,
> >>
> >> Thanks for your suggestion.
> >>
> >> Actually, the timing of starting a Python process depends on the UDF
> > type,
> >> because the Python process is used to provide the necessary
> information
> > to
> >> instantiate the FunctionDefinition object of the Python UDF. For
> catalog
> >> function, the FunctionDefinition will be instantiated when
> compiling the
> >> job, which means the Python process is required during the
> compilation
> >> instead of the registeration. For temporary system function and
> temporary
> >> catalog function, the FunctionDefinition will be instantiated
> during the
> >> UDF registeration, so the Python process need to be started at that
> time.
> >>
> >> But this FLIP will only support registering the temporary system
> function
> >> and temporary catalog function in SQL DDL because registering
> Python UDF
> > to
> >> catalog is not supported yet. We plan to support the registeration
> of
> >> Python catalog function (via Table API and SQL DDL) in a separate
> FLIP.
> >> I'll add a non-goal section to the FLIP page to illustrate this.
> >>
> >> Best,
> >> Wei
> >>
> >>
> >>> 在 2020年3月2日，15:11，Jark Wu  写道：
> >>>
> >>> Hi Weizhong,
> >>>
> >>

Re: [DISCUSS] FLIP-112: Support User-Defined Metrics for Python UDF

[Hequn Cheng]

Hi everyone,

If there are no more concerns, I will raise the vote next week on Monday.
Thank you all for the feedback.

Best,
Hequn


On Fri, Mar 13, 2020 at 9:00 AM jincheng sun 
wrote:

> Hi Hequn,
>
> +1, thank you for this discussion, and metrics are very important for
> monitoring the running state of Python UDF.
>
> Best,
> Jincheng
>
>
> Hequn Cheng  于2020年3月12日周四 下午11:39写道：
>
> > @Dian Fu   Thanks a lot for the advice. The FLIP
> > page has been updated.
> >
> > Best,
> > Hequn
> >
> > On Thu, Mar 12, 2020 at 4:19 PM Dian Fu  wrote:
> >
> > > Hi Hequn,
> > >
> > > Thanks for driving this. +1 to this feature.
> > >
> > > Just one minor comment: It seems that we will add an API
> get_metric_group
> > > for the Python class FunctionContext, could you update the FLIP
> > reflecting
> > > this?
> > >
> > > Thanks,
> > > Dian
> > >
> > > > 在 2020年3月10日，下午3:38，Wei Zhong  写道：
> > > >
> > > > Hi Hequn,
> > > >
> > > > Thanks for driving this. +1 for the metrics support for Python UDF,
> > > which makes it much easier for users to monitor the execution of Python
> > > UDFs.
> > > >
> > > > Best,
> > > > Wei
> > > >
> > > >
> > > >> 在 2020年3月10日，15:32，Xingbo Huang  写道：
> > > >>
> > > >> Hi Hequn,
> > > >> thanks for drafting the FLIP and kicking off the discussion.
> > > >>
> > > >> +1 for this feature.
> > > >> I think this feature will be extremely convenient for PyFlink users.
> > > >>
> > > >> Best,
> > > >> Xingbo
> > > >>
> > > >> Hequn Cheng  于2020年3月9日周一 上午11:32写道：
> > > >>
> > > >>> Hi everyone,
> > > >>>
> > > >>> FLIP-58 adds the support for Python UDFs, but user-defined metrics
> > > >>> have not been supported yet. With metrics, users can report and
> > monitor
> > > >>> the UDF status to get a deeper understanding of the execution,
> > > >>> so in this FLIP, we want to support metrics for Python UDFs.
> > > >>>
> > > >>> Previously, Jincheng and I discussed offline about the support of
> > > >>> metrics for Python UDFs. We'd like to achieve three goals for
> > > >>> supporting metrics for Python UDFs:
> > > >>> - Support user-defined metrics including Counters, Gauges, Meters,
> > > >>> Distributions in Python UDFs.
> > > >>> - Support defining user scopes.
> > > >>> - Support defining user variables.
> > > >>>
> > > >>> More details can be found in the FLIP wiki page[1] and we are
> looking
> > > >>> forward
> > > >>> to your feedback.
> > > >>>
> > > >>> Best,
> > > >>> Hequn
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF
> > > >>>
> > > >
> > >
> > >
> >
>

[VOTE] FLIP-112: Support User-Defined Metrics for Python UDF

[Hequn Cheng]

Hi everyone,

I'd like to start the vote of FLIP-112[1] which is discussed and reached
consensus in the discussion thread[2].
The vote will be open for at least 72 hours. Unless there is an objection,
I will try to close it by March 19, 2020 03:00 UTC if we have received
sufficient votes.

Thanks,
Hequn

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-112-Support-User-Defined-Metrics-for-Python-UDF-td38609.html

Re: [VOTE] FLIP-106: Support Python UDF in SQL Function DDL

[Hequn Cheng]

+1 (binding)

Best,
Hequn

> On Mar 17, 2020, at 5:03 PM, Benchao Li  wrote:
> 
> +1 (non-binding)
> 
> BTW it's in the same thread in my gmail too.
> 
> 
> 
> Kurt Young  于2020年3月17日周二 上午11:47写道：
> 
>> Looks like I hit the gmail's bug again...
>> 
>> Best,
>> Kurt
>> 
>> 
>> On Tue, Mar 17, 2020 at 11:11 AM Wei Zhong  wrote:
>> 
>>> Hi Kurt,
>>> 
>>> This vote thread is independent from my side[1]. If this thread is
>>> combined with another thread from your side, you can try to change the
>> mail
>>> client.
>>> 
>>> Best,
>>> Wei
>>> 
>>> [1]
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-106-Support-Python-UDF-in-SQL-Function-DDL-td38895.html
>>> <
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-106-Support-Python-UDF-in-SQL-Function-DDL-td38895.html
 
>>> 
 在 2020年3月17日，10:57，Kurt Young  写道：
 
 Hi, please use a dedicated vote thread.
 
 Best,
 Kurt
 
 
 On Tue, Mar 17, 2020 at 10:36 AM jincheng sun <
>> sunjincheng...@gmail.com>
 wrote:
 
> +1
> 
> Best,
> Jincheng
> 
> 
> 
> Wei Zhong  于2020年3月13日周五 下午9:04写道：
> 
>> Hi all,
>> 
>> I would like to start the vote for FLIP-106[1] which is discussed and
>> reached consensus in the discussion thread[2].
>> 
>> The vote will be open for at least 72 hours. I'll try to close it by
>> 2020-03-18 14:00 UTC, unless there is an objection or not enough
>> votes.
>> 
>> Best,
>> Wei
>> 
>> [1]
>> 
> 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
>> [2]
>> 
> 
>>> 
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-106-Support-Python-UDF-in-SQL-Function-DDL-td38107.html
>> 
>> 
> 
>>> 
>>> 
>> 
> 
> 
> -- 
> 
> Benchao Li
> School of Electronics Engineering and Computer Science, Peking University
> Tel:+86-15650713730
> Email: libenc...@gmail.com; libenc...@pku.edu.cn

Re: [VOTE] FLIP-112: Support User-Defined Metrics for Python UDF

[Hequn Cheng]

+1 (binding)

Best, Hequn

On Thu, Mar 19, 2020 at 10:39 AM Wei Zhong  wrote:

> +1 (non-binding)
>
> Best,
> Wei
>
> > 在 2020年3月17日，19:15，Dian Fu  写道：
> >
> > +1 (binding)
> >
> > On Tue, Mar 17, 2020 at 10:35 AM jincheng sun 
> > wrote:
> >
> >> +1
> >>
> >> Best,
> >> Jincheng
> >>
> >>
> >>
> >> Hequn Cheng  于2020年3月16日周一 上午10:01写道：
> >>
> >>> Hi everyone,
> >>>
> >>> I'd like to start the vote of FLIP-112[1] which is discussed and
> reached
> >>> consensus in the discussion thread[2].
> >>> The vote will be open for at least 72 hours. Unless there is an
> >> objection,
> >>> I will try to close it by March 19, 2020 03:00 UTC if we have received
> >>> sufficient votes.
> >>>
> >>> Thanks,
> >>> Hequn
> >>>
> >>> [1]
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF
> >>> [2]
> >>>
> >>>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-112-Support-User-Defined-Metrics-for-Python-UDF-td38609.html
> >>>
> >>
>
>

Re: [VOTE] FLIP-112: Support User-Defined Metrics for Python UDF

[Hequn Cheng]

Thanks all for the votes.

So far, we have
   - 3 binding +1 votes (Jincheng, Dian, and Hequn)
   - 1 non-binding +1 votes (Wei)
   - 0 -1 votes

I will move the FLIP to accepted!

Best, Hequn

On Thu, Mar 19, 2020 at 2:07 PM Hequn Cheng  wrote:

> +1 (binding)
>
> Best, Hequn
>
> On Thu, Mar 19, 2020 at 10:39 AM Wei Zhong  wrote:
>
>> +1 (non-binding)
>>
>> Best,
>> Wei
>>
>> > 在 2020年3月17日，19:15，Dian Fu  写道：
>> >
>> > +1 (binding)
>> >
>> > On Tue, Mar 17, 2020 at 10:35 AM jincheng sun > >
>> > wrote:
>> >
>> >> +1
>> >>
>> >> Best,
>> >> Jincheng
>> >>
>> >>
>> >>
>> >> Hequn Cheng  于2020年3月16日周一 上午10:01写道：
>> >>
>> >>> Hi everyone,
>> >>>
>> >>> I'd like to start the vote of FLIP-112[1] which is discussed and
>> reached
>> >>> consensus in the discussion thread[2].
>> >>> The vote will be open for at least 72 hours. Unless there is an
>> >> objection,
>> >>> I will try to close it by March 19, 2020 03:00 UTC if we have received
>> >>> sufficient votes.
>> >>>
>> >>> Thanks,
>> >>> Hequn
>> >>>
>> >>> [1]
>> >>>
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF
>> >>> [2]
>> >>>
>> >>>
>> >>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-112-Support-User-Defined-Metrics-for-Python-UDF-td38609.html
>> >>>
>> >>
>>
>>

Re: [DISCUSS] Creating a new repo to host Stateful Functions Dockerfiles

[Hequn Cheng]

+1 for a separate repository.
The dedicated `flink-docker` repo works fine now. We can do it similarly.

Best,
Hequn

On Fri, Mar 27, 2020 at 1:16 AM Till Rohrmann  wrote:

> +1 for a separate repository.
>
> Cheers,
> Till
>
> On Thu, Mar 26, 2020 at 5:13 PM Ufuk Celebi  wrote:
>
> > +1.
> >
> > The repo creation process is a light-weight, automated process on the ASF
> > side. When Patrick Lucas contributed docker-flink back to the Flink
> > community (as flink-docker), there was virtually no overhead in creating
> > the repository. Reusing build scripts should still be possible at the
> cost
> > of some duplication which is fine imo.
> >
> > – Ufuk
> >
> > On Thu, Mar 26, 2020 at 4:18 PM Stephan Ewen  wrote:
> > >
> > > +1 to a separate repository.
> > >
> > > It seems to be best practice in the docker community.
> > > And since it does not add overhead, why not go with the best practice?
> > >
> > > Best,
> > > Stephan
> > >
> > >
> > > On Thu, Mar 26, 2020 at 4:15 PM Tzu-Li (Gordon) Tai <
> tzuli...@apache.org
> > >
> > wrote:
> > >>
> > >> Hi Flink devs,
> > >>
> > >> As part of a Stateful Functions release, we would like to publish
> > Stateful
> > >> Functions Docker images to Dockerhub as an official image.
> > >>
> > >> Some background context on Stateful Function images, for those who are
> > not
> > >> familiar with the project yet:
> > >>
> > >>- Stateful Function images are built on top of the Flink official
> > >>images, with additional StateFun dependencies being added.
> > >>You can take a look at the scripts we currently use to build the
> > images
> > >>locally for development purposes [1].
> > >>- They are quite important for user experience, since building a
> > Docker
> > >>image is the recommended go-to deployment mode for StateFun user
> > >>applications [2].
> > >>
> > >>
> > >> A prerequisite for all of this is to first decide where we host the
> > >> Stateful Functions Dockerfiles,
> > >> before we can proceed with the process of requesting a new official
> > image
> > >> repository at Dockerhub.
> > >>
> > >> We’re proposing to create a new dedicated repo for this purpose,
> > >> with the name `apache/flink-statefun-docker`.
> > >>
> > >> While we did initially consider integrating the StateFun Dockerfiles
> to
> > be
> > >> hosted together with the Flink ones in the existing
> > `apache/flink-docker`
> > >> repo, we had the following concerns:
> > >>
> > >>- In general, it is a convention that each official Dockerhub image
> > is
> > >>backed by a dedicated source repo hosting the Dockerfiles.
> > >>- The `apache/flink-docker` repo already has quite a few dedicated
> > >>tooling and CI smoke tests specific for the Flink images.
> > >>- Flink and StateFun have separate versioning schemes and
> independent
> > >>release cycles. A new Flink release does not necessarily require a
> > >>“lock-step” to release new StateFun images as well.
> > >>- Considering the above all-together, and the fact that creating a
> > new
> > >>repo is rather low-effort, having a separate repo would probably
> make
> > more
> > >>sense here.
> > >>
> > >>
> > >> What do you think?
> > >>
> > >> Cheers,
> > >> Gordon
> > >>
> > >> [1]
> > >>
> >
> >
> https://github.com/apache/flink-statefun/blob/master/tools/docker/build-stateful-functions.sh
> > >> [2]
> > >>
> >
> >
> https://ci.apache.org/projects/flink/flink-statefun-docs-master/deployment-and-operations/packaging.html
> >
>

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #2

[Hequn Cheng]

Thanks Gordon for the release and the nice release checking guide!

It seems the NOTICE file is missing in the
`statefun-ridesharing-example-simulator` module while it bundles
dependencies like
`org.springframework.boot:spring-boot-loader:2.1.6.RELEASE`.

Best,
Hequn

On Fri, Mar 27, 2020 at 3:35 PM Tzu-Li (Gordon) Tai 
wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #2 for the version 2.0.0 of
> Apache Flink Stateful Functions,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> **Testing Guideline**
>
> You can find here [1] a doc that we can use for collaborating testing
> efforts.
> The listed testing tasks in the doc also serve as a guideline in what to
> test for this release.
> If you wish to take ownership of a testing task, simply put your name down
> in the "Checked by" field of the task.
>
> **Release Overview**
>
> As an overview, the release consists of the following:
> a) Stateful Functions canonical source distribution, to be deployed to the
> release repository at dist.apache.org
> b) Stateful Functions Python SDK distributions to be deployed to PyPI
> c) Maven artifacts to be deployed to the Maven Central Repository
>
> **Staging Areas to Review**
>
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a) and b) can be found in the corresponding dev
> repository at dist.apache.org [2]
> * All artifacts for c) can be found at the Apache Nexus Repository [3]
>
> All artifacts are singed with the
> key 1C1E2394D3194E1944613488F320986D35C33D6A [4]
>
> Other links for your review:
> * JIRA release notes [5]
> * source code tag "release-2.0.0-rc2" [6] [7]
>
> **Extra Remarks**
>
> * Part of the release is also official Docker images for Stateful
> Functions. This can be a separate process, since the creation of those
> relies on the fact that we have distribution jars already deployed to
> Maven. I will follow-up with this after these artifacts are officially
> released.
> In the meantime, there is this discussion [8] ongoing about where to host
> the StateFun Dockerfiles.
> * The Flink Website and blog post is also being worked on (by Marta) as
> part of the release, to incorporate the new Stateful Functions project. We
> can follow up with a link to those changes afterwards in this vote thread,
> but that would not block you to test and cast your votes already.
> * Since the Flink website changes are still being worked on, you will not
> yet be able to find the Stateful Functions docs from there. Here are the
> links [9] [10].
>
> **Vote Duration**
>
> The vote will be open for at least 72 hours *(target end date is next
> Tuesday, April 31).*
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Gordon
>
> [1]
>
> https://docs.google.com/document/d/1P9yjwSbPQtul0z2AXMnVolWQbzhxs68suJvzR6xMjcs/edit?usp=sharing
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.0.0-rc2/
> [3]
> https://repository.apache.org/content/repositories/orgapacheflink-1340/
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346878
> [6]
>
> https://gitbox.apache.org/repos/asf?p=flink-statefun.git;a=commit;h=14ce58048a3dda792f2329cf14d30aa952f6cb24
> [7] https://github.com/apache/flink-statefun/tree/release-2.0.0-rc2
> [8]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Creating-a-new-repo-to-host-Stateful-Functions-Dockerfiles-td39342.html
> [9] https://ci.apache.org/projects/flink/flink-statefun-docs-master/
> [10] https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/
>
> TIP: You can create a `settings.xml` file with these contents:
>
> """
> 
>   
> flink-statefun-2.0.0
>   
>   
> 
>   flink-statefun-2.0.0
>   
> 
>   flink-statefun-2.0.0
>   
> https://repository.apache.org/content/repositories/orgapacheflink-1340/
> 
> 
> 
>   archetype
>   
> https://repository.apache.org/content/repositories/orgapacheflink-1340/
> 
> 
>   
> 
>   
> 
> """
>
> And reference that in you maven commands via `--settings
> path/to/settings.xml`.
> This is useful for creating a quickstart based on the staged release and
> for building against the staged jars.
>

Re: contribute to Apache Flink

[Hequn Cheng]

Hi Leping,

Welcome to the community!

You no longer need contributor permissions. You can simply create a JIRA
ticket and ask to be assigned to work on it. For some reasons[1], only
committers can assign a
Jira ticket now. You can also take a look at the Flink's contribution
guidelines [2] for more
information.

Best,
Hequn

[1]
https://flink.apache.org/contributing/contribute-code.html#create-jira-ticket-and-reach-consensus
[2] https://flink.apache.org/contributing/how-to-contribute.html

On Sat, Mar 28, 2020 at 7:09 AM Leping Huang 
wrote:

> Hi Guys,
>
> I want to contribute to Apache Flink.
> Would you please give me the permission as a contributor?
> My JIRA ID is soundhearer.
>

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #2

[Hequn Cheng]

Thanks a lot for such a quick update! @Gordon

Best, Hequn

On Sun, Mar 29, 2020 at 3:31 PM Tzu-Li (Gordon) Tai 
wrote:

> All blockers are resolved.
>
> This vote thread is cancelled.
> There is a new vote thread for RC3:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Apache-Flink-Stateful-Functions-Release-2-0-0-release-candidate-3-td39424.html
>
> On Sat, Mar 28, 2020 at 7:00 PM Tzu-Li (Gordon) Tai 
> wrote:
>
> > After a check-through on the licenses, there are in total the following 3
> > blockers due to licensing issues:
> >
> > - https://issues.apache.org/jira/browse/FLINK-16841
> > - https://issues.apache.org/jira/browse/FLINK-16842
> > - https://issues.apache.org/jira/browse/FLINK-16843
> >
> > Will open a new RC as soon as those are addressed.
> >
> > On Sat, Mar 28, 2020 at 1:34 AM Tzu-Li (Gordon) Tai  >
> > wrote:
> >
> >> Hi Hequn,
> >>
> >> That's a good catch.
> >>
> >> Unfortunately, the spring boot dependency there, while itself being
> ASLv2
> >> licensed, pulls in other dependencies that are not ASLv2.
> >> That would indeed make this problem a blocker.
> >>
> >> I'll do a thorough check again on the Maven artifacts that do bundle
> >> dependencies, before creating a new RC. AFAIK, there should be no more
> >> other than:
> >> - statefun-flink-distribution
> >> - statefun-ridesharing-example-simulator
> >>
> >> BR,
> >> Gordon
> >>
> >> On Fri, Mar 27, 2020 at 10:41 PM Hequn Cheng  wrote:
> >>
> >>> Thanks Gordon for the release and the nice release checking guide!
> >>>
> >>> It seems the NOTICE file is missing in the
> >>> `statefun-ridesharing-example-simulator` module while it bundles
> >>> dependencies like
> >>> `org.springframework.boot:spring-boot-loader:2.1.6.RELEASE`.
> >>>
> >>> Best,
> >>> Hequn
> >>>
> >>> On Fri, Mar 27, 2020 at 3:35 PM Tzu-Li (Gordon) Tai <
> tzuli...@apache.org
> >>> >
> >>> wrote:
> >>>
> >>> > Hi everyone,
> >>> >
> >>> > Please review and vote on the release candidate #2 for the version
> >>> 2.0.0 of
> >>> > Apache Flink Stateful Functions,
> >>> > as follows:
> >>> > [ ] +1, Approve the release
> >>> > [ ] -1, Do not approve the release (please provide specific comments)
> >>> >
> >>> > **Testing Guideline**
> >>> >
> >>> > You can find here [1] a doc that we can use for collaborating testing
> >>> > efforts.
> >>> > The listed testing tasks in the doc also serve as a guideline in what
> >>> to
> >>> > test for this release.
> >>> > If you wish to take ownership of a testing task, simply put your name
> >>> down
> >>> > in the "Checked by" field of the task.
> >>> >
> >>> > **Release Overview**
> >>> >
> >>> > As an overview, the release consists of the following:
> >>> > a) Stateful Functions canonical source distribution, to be deployed
> to
> >>> the
> >>> > release repository at dist.apache.org
> >>> > b) Stateful Functions Python SDK distributions to be deployed to PyPI
> >>> > c) Maven artifacts to be deployed to the Maven Central Repository
> >>> >
> >>> > **Staging Areas to Review**
> >>> >
> >>> > The staging areas containing the above mentioned artifacts are as
> >>> follows,
> >>> > for your review:
> >>> > * All artifacts for a) and b) can be found in the corresponding dev
> >>> > repository at dist.apache.org [2]
> >>> > * All artifacts for c) can be found at the Apache Nexus Repository
> [3]
> >>> >
> >>> > All artifacts are singed with the
> >>> > key 1C1E2394D3194E1944613488F320986D35C33D6A [4]
> >>> >
> >>> > Other links for your review:
> >>> > * JIRA release notes [5]
> >>> > * source code tag "release-2.0.0-rc2" [6] [7]
> >>> >
> >>> > **Extra Remarks**
> >>> >
> >>> > * Part of the release is also official Docker images for Stateful
> >>> > Functions. This can be a separate process, since

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #3

[Hequn Cheng]

Hi Gordon,

Thanks a lot for the new RC. I found some new blockers about licenses:

- Module statefun-flink-distribution
com.google.protobuf:protobuf-java:3.8.0 (The version should be 3.7.1)

- Module statefun-ridesharing-example-simulator
com.google.code.findbugs:jsr305:3.0.2:compile (Remove compile)
org.hibernate.validator:hibernate-validator:6.0.17 (Version should be
6.0.17.Final)
org.jboss.logging:jboss-logging:3.3.2  (Version should be 3.3.2.Final)

Non-blocker feedback:
- py3 is added in the name of "whl dist" but it is missing in the "source
dist"[1]. Should we make them consistent?

Best,
Hequn

[1] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.0.0-rc3/


On Sun, Mar 29, 2020 at 3:31 PM Tzu-Li (Gordon) Tai 
wrote:

> Hi everyone,
>
> Please review and vote on the *release candidate #3* for the version 2.0.0
> of Apache Flink Stateful Functions,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> **Testing Guideline**
>
> You can find here [1] a doc that we can use for collaborating testing
> efforts.
> The listed testing tasks in the doc also serve as a guideline in what to
> test for this release.
> If you wish to take ownership of a testing task, simply put your name down
> in the "Checked by" field of the task.
>
> **Release Overview**
>
> As an overview, the release consists of the following:
> a) Stateful Functions canonical source distribution, to be deployed to the
> release repository at dist.apache.org
> b) Stateful Functions Python SDK distributions to be deployed to PyPI
> c) Maven artifacts to be deployed to the Maven Central Repository
>
> **Staging Areas to Review**
>
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a) and b) can be found in the corresponding dev
> repository at dist.apache.org [2]
> * All artifacts for c) can be found at the Apache Nexus Repository [3]
>
> All artifacts are singed with the
> key 1C1E2394D3194E1944613488F320986D35C33D6A [4]
>
> Other links for your review:
> * JIRA release notes [5]
> * source code tag "release-2.0.0-rc3" [6] [7]
>
> **Extra Remarks**
>
> * Part of the release is also official Docker images for Stateful
> Functions. This can be a separate process, since the creation of those
> relies on the fact that we have distribution jars already deployed to
> Maven. I will follow-up with this after these artifacts are officially
> released.
> In the meantime, there is this discussion [8] ongoing about where to host
> the StateFun Dockerfiles.
> * The Flink Website and blog post is also being worked on (by Marta) as
> part of the release, to incorporate the new Stateful Functions project. We
> can follow up with a link to those changes afterwards in this vote thread,
> but that would not block you to test and cast your votes already.
> * Since the Flink website changes are still being worked on, you will not
> yet be able to find the Stateful Functions docs from there. Here are the
> links [9] [10].
>
> **Vote Duration**
>
> The vote will be open for at least 72 hours starting Monday
> *(target end date is Wednesday, April 1st).*
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Gordon
>
> [1]
>
> https://docs.google.com/document/d/1P9yjwSbPQtul0z2AXMnVolWQbzhxs68suJvzR6xMjcs/edit?usp=sharing
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.0.0-rc3/
> [3]
> https://repository.apache.org/content/repositories/orgapacheflink-1342/
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346878
> [6]
>
> https://gitbox.apache.org/repos/asf?p=flink-statefun.git;a=commit;h=752e07fd9987ee430eb9d1c1d3fadff632ef9213
> [7] https://github.com/apache/flink-statefun/tree/release-2.0.0-rc3
> [8]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Creating-a-new-repo-to-host-Stateful-Functions-Dockerfiles-td39342.html
> [9] https://ci.apache.org/projects/flink/flink-statefun-docs-master/
> [10] https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/
>
> TIP: You can create a `settings.xml` file with these contents:
>
> """
> 
>   
> flink-statefun-2.0.0
>   
>   
> 
>   flink-statefun-2.0.0
>   
> 
>   flink-statefun-2.0.0
>   
> https://repository.apache.org/content/repositories/orgapacheflink-1342/
> 
> 
> 
>   archetype
>   
> https://repository.apache.org/content/repositories/orgapacheflink-1342/
> 
> 
>   
> 
>   
> 
> """
>
> And reference that in you maven commands via `--settings
> path/to/settings.xml`.
> This is useful for creating a quickstart based on the staged release and
> for building against the staged jars.
>

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #3

[Hequn Cheng]

Hi,

@Gordon I have created the corresponding JIRAs and PRs for the license
problems:
- https://issues.apache.org/jira/browse/FLINK-16853
- https://issues.apache.org/jira/browse/FLINK-16854

@Igal @Gordon, Thanks a lot for the confirmation of the package name, that
addresses my concerns now.

Best,
Hequn

On Mon, Mar 30, 2020 at 4:10 AM Konstantin Knauf 
wrote:

> Hi Gordon,
>
> +1 (non-binding)
>
> * built from sources...check
> * build python SDK from sources...check
> * went through walkthrough based on local builds...check
>
> Cheers,
>
> Konstantin
>
>
> On Sun, Mar 29, 2020 at 6:30 PM Igal Shilman  wrote:
>
>> Hi @Hequn Cheng , and @Tzu-Li (Gordon) Tai
>> 
>> Indeed the names are generated automatically and are following a
>> convention.
>>
>> Cheers,
>> Igal.
>>
>> On Sun, Mar 29, 2020 at 6:28 PM Tzu-Li (Gordon) Tai 
>> wrote:
>>
>> > @Hequn Cheng 
>> > On second thought I think it would not hurt to create a RC4 to fix the
>> > version strings. Will fix those and create that now.
>> >
>> > On Mon, Mar 30, 2020 at 12:06 AM Tzu-Li (Gordon) Tai <
>> tzuli...@apache.org>
>> > wrote:
>> >
>> >>
>> >>
>> >> On Mon, Mar 30, 2020 at 12:00 AM Tzu-Li (Gordon) Tai <
>> tzuli...@apache.org>
>> >> wrote:
>> >>
>> >>> @Hequn Cheng 
>> >>> Good catches again!
>> >>>
>> >>> Regarding the incorrect versions:
>> >>> I think technically those would not be hard blockers, since what
>> matters
>> >>> is their inclusion and licenses being acknowledged.
>> >>> It would still be good to fix those though - could you open a ticket
>> for
>> >>> those?
>> >>>
>> >>
>> >> Here, I mean to fix those in future bugfix releases (if this RC does
>> >> indeed pass the vote as the official release).
>> >>
>> >>
>> >>>
>> >>> Regarding the names of the Python dists -
>> >>> the name of those distributions are generated from the setup.py file,
>> >>> and seems to be a convention used by PyPI.
>> >>> @i...@ververica.com   can you confirm here?
>> >>>
>> >>> On Sun, Mar 29, 2020 at 11:36 PM Hequn Cheng 
>> wrote:
>> >>>
>> >>>> Hi Gordon,
>> >>>>
>> >>>> Thanks a lot for the new RC. I found some new blockers about
>> licenses:
>> >>>>
>> >>>> - Module statefun-flink-distribution
>> >>>> com.google.protobuf:protobuf-java:3.8.0 (The version should be 3.7.1)
>> >>>>
>> >>>> - Module statefun-ridesharing-example-simulator
>> >>>> com.google.code.findbugs:jsr305:3.0.2:compile (Remove compile)
>> >>>> org.hibernate.validator:hibernate-validator:6.0.17 (Version should be
>> >>>> 6.0.17.Final)
>> >>>> org.jboss.logging:jboss-logging:3.3.2  (Version should be
>> 3.3.2.Final)
>> >>>>
>> >>>> Non-blocker feedback:
>> >>>> - py3 is added in the name of "whl dist" but it is missing in the
>> >>>> "source
>> >>>> dist"[1]. Should we make them consistent?
>> >>>>
>> >>>> Best,
>> >>>> Hequn
>> >>>>
>> >>>> [1]
>> >>>>
>> https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.0.0-rc3/
>> >>>>
>> >>>>
>> >>>> On Sun, Mar 29, 2020 at 3:31 PM Tzu-Li (Gordon) Tai <
>> >>>> tzuli...@apache.org>
>> >>>> wrote:
>> >>>>
>> >>>> > Hi everyone,
>> >>>> >
>> >>>> > Please review and vote on the *release candidate #3* for the
>> version
>> >>>> 2.0.0
>> >>>> > of Apache Flink Stateful Functions,
>> >>>> > as follows:
>> >>>> > [ ] +1, Approve the release
>> >>>> > [ ] -1, Do not approve the release (please provide specific
>> comments)
>> >>>> >
>> >>>> > **Testing Guideline**
>> >>>> >
>> >>>> > You can find here [1] a doc that we can use for collaborating
>> testing
>> >>>> > efforts.
>> >>>> > The listed testing tasks

Re: [DISCUSS] FLIP-114: Support Python UDF in SQL Client

[Hequn Cheng]

Hi Wei,

Thanks a lot for the proposal! +1 for the VOTE.

Best,
Hequn



On Mon, Mar 30, 2020 at 3:31 PM Dian Fu  wrote:

> Thanks Wei for this work! +1 to bring up the VOTE thread.
>
> > 在 2020年3月30日，下午2:43，jincheng sun  写道：
> >
> > Hi Wei,
> >
> > +1, Thanks for this discussion which is crucial for SQL users to use
> > PyFlink. Would be great to bring up the VOTE thread.
> >
> > Best,
> > Jincheng
> >
> >
> > Wei Zhong  于2020年3月30日周一 下午2:38写道：
> >
> >> Hi everyone,
> >>
> >> Are there more comments about this FLIP? If not, I would like to bring
> up
> >> the VOTE.
> >>
> >> Best,
> >> Wei
> >>
> >>> 在 2020年3月9日，23:18，Xingbo Huang  写道：
> >>>
> >>> Hi Godfrey,
> >>> thanks for your suggestion.
> >>> I have added two examples how to use python UDF
> >>> in SQL and how to start sql-client.sh with full python dependencies In
> >> FLIP.
> >>>
> >>> Best,
> >>> Xingo
> >>>
> >>> godfrey he  于2020年3月9日周一 下午10:24写道：
> >>>
>  Hi Wei, thanks for the proposal.
> 
>  I think it's better to give two more examples, one is how to use
> python
> >> UDF
>  in SQL, another is how to start sql-client.sh with full python
>  dependencies.
> 
>  Best,
>  Godfrey
> 
>  Wei Zhong  于2020年3月9日周一 下午10:09写道：
> 
> > Hi everyone,
> >
> > I would like to start discussion about how to support Python UDF in
> SQL
> > Client.
> >
> > Flink Python UDF(FLIP-58[1]) has already been introduced in the
> release
>  of
> > 1.10.0 and the support for SQL DDL is introduced in FLIP-106[2].
> >
> > SQL Client defines UDF via the environment file and has its own CLI
> > implementation to manage dependencies, but neither of which supports
>  Python
> > UDF. We want to introduce the support of Python UDF for SQL Client,
> > including the registration and the dependency management of Python
> UDF.
> >
> > Here is the design doc:
> >
> >
> >
> 
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-114%3A+Support+Python+UDF+in+SQL+Client
> >
> > Looking forward to your feedback!
> >
> > Best,
> > Wei
> >
> > [1]
> >
> 
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> > [2]
> >
> 
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL
> >
> >
> 
> >>
> >>
>
>

Re: [ANNOUNCE] New Committers and PMC member

[Hequn Cheng]

Congratulations to all of you!

Best, Hequn

On Wed, Apr 1, 2020 at 5:07 PM Jiayi Liao  wrote:

> Congratulations to you all!
>
> On Wed, Apr 1, 2020 at 5:05 PM Arvid Heise  wrote:
>
> > Congratulations!
> >
> > On Wed, Apr 1, 2020 at 11:03 AM Dian Fu  wrote:
> >
> > > Congratulations to you all.
> > >
> > >
> > > > 在 2020年4月1日，下午5:00，Robert Metzger  写道：
> > > >
> > > > Welcome & congratulations to all of you!
> > > >
> > > >
> > > > On Wed, Apr 1, 2020 at 10:58 AM Jingsong Li 
> > > wrote:
> > > >
> > > >> Congratulations! Konstantin, Dawid, Zhijiang. Well deserved.
> > > >>
> > > >> Best,
> > > >> Jingsong Lee
> > > >>
> > > >> On Wed, Apr 1, 2020 at 4:52 PM Stephan Ewen 
> wrote:
> > > >>
> > > >>> Hi all!
> > > >>>
> > > >>> Happy to announce that over the last few weeks, several people in
> the
> > > >>> community joined in new roles:
> > > >>>
> > > >>>  - Konstantin Knauf joined as a committer. You may know him, for
> > > >> example,
> > > >>> from the weekly community updates.
> > > >>>
> > > >>>  - Dawid Wysakowicz joined the PMC. Dawid is one of the main
> > developers
> > > >> on
> > > >>> the Table API.
> > > >>>
> > > >>>  - Zhijiang Wang joined the PMC. Zhijiang is a veteran of Flink's
> > > >> network
> > > >>> / data shuffle system.
> > > >>>
> > > >>> A warm welcome to your new roles in the Flink project!
> > > >>>
> > > >>> Best,
> > > >>> Stephan
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Best, Jingsong Lee
> > > >>
> > >
> > >
> >
>

Re: [VOTE] FLIP-114: Support Python UDF in SQL Client

[Hequn Cheng]

+1  (binding)

Best,
Hequn

On Fri, Apr 3, 2020 at 10:20 AM jincheng sun 
wrote:

> +1(binding)
>
> Best,
> Jincheng
>
>
>
> Xingbo Huang  于2020年4月1日周三 下午5:36写道：
>
> > Hi Wei,
> >
> > +1 (non-binding)
> > Thanks a lot for driving this.
> >
> > Best,
> > Xingbo
> >
> > Wei Zhong  于2020年3月31日周二 上午10:34写道：
> >
> > > Hi all,
> > >
> > > I would like to start the vote for FLIP-114[1] which is discussed and
> > > reached consensus in the discussion thread[2].
> > >
> > > The vote will be open for at least 72 hours. I'll try to close it by
> > > 2020-04-03 03:00 UTC, unless there is an objection or not enough votes.
> > >
> > > Best,
> > > Wei
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-114%3A+Support+Python+UDF+in+SQL+Client
> > > [2]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-114-Support-Python-UDF-in-SQL-Client-td38655.html
> > >
> > >
> >
>

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #6

[Hequn Cheng]

Thanks a lot for the new RC!

+1 (non-binding)

- Signatures and hash are correct.
- The source distribution contains no binaries.
- The source distribution is building properly with `-Prun-e2e-tests`
(JDK8).
- All POM files / README / Python SDK setup.py point to the same version.
- Verify license and notice.
  - Source distribution. Everything looks good and the jquery has been
added.
  - Jar artifacts. No missing dependencies, no version errors.
  - Python source distribution (source and wheel). It contains the license
and notice file.
- Flink Harness works in IDE.

Best,
Hequn

On Mon, Apr 6, 2020 at 10:05 PM Seth Wiesman  wrote:

> +1 (non-binding)
>
> legal / source
> - checked sources for binary files
> - checked license headers
>
> functional
> - built from source (mvn clean verify -Prun-e2e-tests)
> - built python sdk and ran tests
> - ran examples
> - deployed mixed python / java application on k8s with checkpointing.
> Failed TM's and watched it recover.
> - deployed application on Flink session cluster
> - created a savepoint using the bootstrap api and successfully used it to
> start an application.
>
> Seth
>
> On Mon, Apr 6, 2020 at 5:49 AM Igal Shilman  wrote:
>
> > +1 (non binding)
> >
> > legal / source:
> > - downloaded and verified the signature
> > - verified that pom and versions in the docs match
> > - no binary files in the distribution
> > - built and run e2e test with Java 8 and Java 11
> > - created a project from a maven archetype.
> >
> > functional:
> > - run all the examples
> > - deployed to Python greeter example to k8s
> > - enabled checkpointing, created an application with two Python
> functions,
> > that send both local and remote messages, restarted TMs randomly and
> > verified
> > the sequential output in the output kafka topic (exactly once test)
> > -  run the harness tests
> > -  run the ridesharing example in paraliisim 10 overnight
> > -  created a savepoint with the state bootstrapping tool and
> > successfully started a job from that.
> >
> > Kind regards,
> > Igal
> >
> > On Mon, Apr 6, 2020 at 10:23 AM Robert Metzger 
> > wrote:
> >
> > > Thanks a lot for preparing another RC!
> > >
> > > +1 (binding)
> > >
> > > - source archive looks fine (no binaries, copied sources are properly
> > > reported)
> > > - staging repository looks fine (bundled binaries seem documented,
> > versions
> > > are correct)
> > > - *mvn clean install *(mvn clean verify fails, "install" is required)
> w/
> > > e2e passes locally from source dir
> > >
> > >
> > >
> > >
> > > On Mon, Apr 6, 2020 at 9:22 AM Tzu-Li (Gordon) Tai <
> tzuli...@apache.org>
> > > wrote:
> > >
> > > > FYI -
> > > > There are these open PRs to add blog posts and update the Flink
> website
> > > for
> > > > the Stateful Functions 2.0 release:
> > > > * https://github.com/apache/flink-web/pull/322
> > > > * https://github.com/apache/flink-web/pull/321
> > > >
> > > > On Mon, Apr 6, 2020 at 2:53 PM Konstantin Knauf <
> > > konstan...@ververica.com>
> > > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > ** Functional **
> > > > > - Building from source dist with end-to-end tests enabled (mvn
> clean
> > > > verify
> > > > > -Prun-e2e-tests) passes (JDK 8)
> > > > > - Flink Harness works in IDE
> > > > > - Building Python SDK dist from source
> > > > >
> > > > > On Mon, Apr 6, 2020 at 5:12 AM Tzu-Li (Gordon) Tai <
> > > tzuli...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > ** Legal **
> > > > > > - checksums and GPG files match corresponding release files
> > > > > > - Source distribution does not contain binaries, contents are
> sane
> > > (no
> > > > > > .git* / .travis* / generated html content files)
> > > > > > - Bundled source LICENSEs and NOTICE looks good. Mentions bundled
> > > > > > font-awesome, jquery dependency in docs and copied sources from
> > > > fastutil
> > > > > (
> > > > > > http://fastutil.di.unimi.it/)
> > > > > > - Bundled LICENSEs and NOTICE files for Maven artifacts looks
> good.
> > > > > > Artifacts that do bundle dependencies are:
> > > statefun-flink-distribution,
> > > > > > statefun-ridesharing-example-simulator, statefun-flink-core
> (copied
> > > > > > sources). All non-ASLv2 deps have license files explicitly
> bundled.
> > > > > > - Python SDK distributions (source and wheel) contain ASLv2
> LICENSE
> > > and
> > > > > > NOTICE files (no bundled dependencies)
> > > > > > - All POMs / README / Python SDK setup.py / Dockerfiles / doc
> > configs
> > > > > point
> > > > > > to same version “2.0.0”
> > > > > > - README looks good
> > > > > >
> > > > > > ** Functional **
> > > > > > - Building from source dist with end-to-end tests enabled (mvn
> > clean
> > > > > verify
> > > > > > -Prun-e2e-tests) passes (JDK 8)
> > > > > > - Generated quickstart from archetype looks good (correct POM /
> > > > > Dockerfile
> > > > > > / service file)
> > > > > > - Examples run: Java Greeter / Java Ridesharing / Python Greeter
> /
> > > > Pyt

Re: [ANNOUNCE] New Flink committer: Seth Wiesman

[Hequn Cheng]

Congratulations Seth!

Best, Hequn

On Tue, Apr 7, 2020 at 4:11 PM Fabian Hueske  wrote:

> Congrats Seth! Well deserved :-)
>
> Cheers, Fabian
>
> Am Di., 7. Apr. 2020 um 10:09 Uhr schrieb Yangze Guo :
>
> > Congratulations Seth!
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, Apr 7, 2020 at 4:07 PM Jiayi Liao 
> wrote:
> > >
> > > >
> > > > Congratulations Seth :)
> > > >
> > > >
> >
>

Re: [DISCUSS] FLIP-121: Support Cython Optimizing Python User Defined Function

[Hequn Cheng]

Hi,

+1 on integrating with Azure, it is consistent with the long term goal and
we are also going to switch from Travis to Azure.
The performance improvement is very impressive. Looking forward to the vote.

Best, Hequn

On Tue, Apr 7, 2020 at 9:10 PM Xingbo Huang  wrote:

> Hi everyone,
>
> Thanks all of you for the discussion.
> If there are no objections, I would like to start a vote thread tomorrow.
>
> Best,
> Xingbo
>
> jincheng sun  于2020年4月7日周二 下午6:22写道：
>
> > Hi Xingbo,
> >
> > Thanks for bring up this discussion!
> >
> > I agree with Robert, +1 for integration with Azure.
> >
> > Best,
> > Jincheng
> >
> > Dian Fu  于2020年4月7日周二 下午2:21写道：
> >
> > > Hi Xingbo,
> > >
> > > Thanks a lot for the great work. Big +1 to this feature. The
> performance
> > > improvement is impressive.
> > >
> > > Regards,
> > > Dian
> > >
> > > > 在 2020年4月7日，下午12:38，Robert Metzger  写道：
> > > >
> > > > Thank you for posting the FLIP.
> > > >
> > > > The proposed integration with Azure Pipelines looks good to me.
> > > >
> > > > On Tue, Mar 31, 2020 at 1:23 PM Xingbo Huang 
> > wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> I would like to start a discussion thread on "Support Cython
> > Optimizing
> > > >> Python User Defined Function"
> > > >>
> > > >> Scalar Python UDF FLIP-58[1] has already been supported in release
> > 1.10
> > > and
> > > >> Python UDTF will be supported in the coming release of 1.11. In
> > release
> > > >> 1.10, we focused on supporting UDF features and did not make many
> > > >> optimizations in terms of performance. Although we have made a lot
> of
> > > >> optimizations in master[2], Cython can further greatly improve the
> > > >> performance of Python UDF.
> > > >>
> > > >> Robert Metzger, Jincheng Sun and I have discussed offline and have
> > > drafted
> > > >> the FLIP-121[3]. It includes the following items:
> > > >>
> > > >> - Introduces Cython implementation of coder and operations
> > > >>
> > > >> - Doc changes for building sdist and wheel packages from source code
> > > >>
> > > >> - Solutions for packages building
> > > >>
> > > >>
> > > >> Looking forward to your feedback!
> > > >>
> > > >> Best,
> > > >>
> > > >> Xingbo
> > > >>
> > > >> [1]
> > > >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table
> > > >>
> > > >> [2] https://issues.apache.org/jira/browse/FLINK-16747
> > > >>
> > > >> [3]
> > > >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-121%3A+Support+Cython+Optimizing+Python+User+Defined+Function
> > > >>
> > >
> > >
> >
>

Re: [ANNOUNCE] Apache Flink Stateful Functions 2.0.0 released

[Hequn Cheng]

Thanks a lot for the release and your great job, Gordon!
Also thanks to everyone who made this release possible!

Best,
Hequn

On Tue, Apr 7, 2020 at 8:58 PM Tzu-Li (Gordon) Tai 
wrote:

> The Apache Flink community is very happy to announce the release of Apache
> Flink Stateful Functions 2.0.0.
>
> Stateful Functions is an API that simplifies building distributed stateful
> applications.
> It's based on functions with persistent state that can interact
> dynamically with strong consistency guarantees.
>
> Please check out the release blog post for an overview of the release:
> https://flink.apache.org/news/2020/04/07/release-statefun-2.0.0.html
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Maven artifacts for Stateful Functions can be found at:
> https://search.maven.org/search?q=g:org.apache.flink%20statefun
>
> Python SDK for Stateful Functions published to the PyPI index can be found
> at:
> https://pypi.org/project/apache-flink-statefun/
>
> Official Docker image for building Stateful Functions applications is
> currently being published to Docker Hub.
> Dockerfiles for this release can be found at:
> https://github.com/apache/flink-statefun-docker/tree/master/2.0.0
> Progress for creating the Docker Hub repository can be tracked at:
> https://github.com/docker-library/official-images/pull/7749
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346878
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Cheers,
> Gordon
>

Re: Configuring autolinks to Flink JIRA ticket in github repos

[Hequn Cheng]

It’s much more convenient now. Thanks you!

> On Apr 9, 2020, at 8:01 PM, Aljoscha Krettek  wrote:
> 
> That is very nice! Thanks for taking care of this ~3q
> 
> On 09.04.20 11:08, Dian Fu wrote:
>> Cool! Thanks Yun for this effort. Very useful feature.
>> Regards,
>> Dian
>>> 在 2020年4月9日，下午4:32，Yu Li  写道：
>>> 
>>> Great! Thanks for the efforts Yun.
>>> 
>>> Best Regards,
>>> Yu
>>> 
>>> 
>>> On Thu, 9 Apr 2020 at 16:15, Jark Wu  wrote:
>>> 
 Thanks Yun,
 
 This's a great feature! I was surprised by the autolink feature yesterday
 (didn't know your work at that time).
 
 Best,
 Jark
 
 On Thu, 9 Apr 2020 at 16:12, Yun Tang  wrote:
 
> Hi community
> 
> The autolink to Flink JIRA ticket has taken effect. You could refer to
 the
> commit details page[1] to see all Flink JIRA titles within commits has
 the
> hyper link underline. Moreover, you don't need to use markdown language
 to
> create hyper link to Flink JIRA ticket when discussing in the pull
> requests. e.g FLINK-16850 could point to the link instead of
 [FLINK-16850](
> https://issues.apache.org/jira/browse/FLINK-16850)
> 
> 
> [1] https://github.com/apache/flink/commits/master
> 
> Best
> Yun Tang
> 
> 
> From: Till Rohrmann 
> Sent: Thursday, April 2, 2020 23:11
> To: dev 
> Subject: Re: Configuring autolinks to Flink JIRA ticket in github repos
> 
> Nice, this is a cool feature. Thanks for asking INFRA for it.
> 
> Cheers,
> Till
> 
> On Wed, Apr 1, 2020 at 6:52 PM Yun Tang  wrote:
> 
>> Hi community.
>> 
>> I noticed that Github supports autolink reference recently [1]. This is
>> helpful to allow developers could open Jira ticket link from pull
> requests
>> title directly when accessing github repo.
>> 
>> I have already created INFRA-20055 [2] to ask for configuration for
 seven
>> Flink related github repos. Hope it could be resolved soon 🙂
>> 
>> 
>> [1]
>> 
> 
 https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources
>> [2] https://issues.apache.org/jira/browse/INFRA-20055
>> 
>> Best
>> Yun Tang
>> 
> 
 
> 

Re: [VOTE] FLIP-120: Support conversion between PyFlink Table and Pandas DataFrame

[Hequn Cheng]

+1 (binding)

Best,
Hequn

On Mon, Apr 13, 2020 at 1:28 PM Jeff Zhang  wrote:

> +1
>
>
> jincheng sun  于2020年4月13日周一 下午1:24写道：
>
> > +1(binding)
> >
> > Best,
> > Jincheng
> >
> >
> >
> > Xingbo Huang  于2020年4月9日周四 下午8:27写道：
> >
> > > Hi Dian,
> > >
> > > +1 (non-binding)
> > > Thanks a lot for driving this.
> > >
> > > Best,
> > > Xingbo
> > >
> > > Dian Fu  于2020年4月8日周三 上午10:03写道：
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start the vote for FLIP-120[1] which is discussed and
> > reached
> > > > consensus in the discussion thread[2].
> > > >
> > > > The vote will be open for at least 72 hours unless there is an
> > objection
> > > > or we have not received sufficient votes.
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120%3A+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-120:+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame
> > > > >
> > > > [2]
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
> > > > <
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-120-Support-conversion-between-PyFlink-Table-and-Pandas-DataFrame-tt39611.html
> > > > >
> > >
> >
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: [VOTE] FLIP-121: Support Cython Optimizing Python User Defined Function

[Hequn Cheng]

+1 (binding)

Best,
Hequn

On Mon, Apr 13, 2020 at 1:50 PM jincheng sun 
wrote:

> +1(binding)
>
> Best,
> Jincheng
>
>
>
> Dian Fu  于2020年4月8日周三 上午9:59写道：
>
> > +1 (binding)
> >
> > > 在 2020年4月8日，上午9:53，Xingbo Huang  写道：
> > >
> > > Hi all,
> > > I would like to start the vote for FLIP-121[1], which is discussed and
> > > reached a consensus in the discussion thread[2].
> > >
> > > The vote will be open for at least 72h, unless there is an objection or
> > not
> > > enough votes.
> > >
> > > Best,
> > > Xingbo
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-121%3A+Support+Cython+Optimizing+Python+User+Defined+Function
> > > [2]
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-121-Support-Cython-Optimizing-Python-User-Defined-Function-tt39577.html
> >
> >
>

Re: [DISCUSS] Releasing Flink 1.9.3

[Hequn Cheng]

+1 for the release and for Dian being the RM.
Thanks Jincheng for your continuous efforts on helping the releasing.

Best,
Hequn

On Wed, Apr 15, 2020 at 3:45 PM Till Rohrmann  wrote:

> Hi Dian,
>
> creating a new 1.9 bug fix release is a very good idea. +1 for creating it
> soon. Also thanks for volunteering as our release manager.
>
> Cheers,
> Till
>
> On Fri, Apr 10, 2020 at 7:27 AM Dian Fu  wrote:
>
> > Hi Jincheng,
> >
> > Thanks a lot for offering help. It would be very helpful. Thanks again!
> >
> > Regards,
> > Dian
> >
> > > 在 2020年4月10日，下午12:46，jincheng sun  写道：
> > >
> > > Hi Dian,
> > >
> > > Thanks for bring up the discussion. I would like to give you a hand at
> > the
> > > last stage when the RC is finished.  :)
> > >
> > > Best,
> > > Jincheng
> > >
> > >
> > >
> > > Dian Fu  于2020年4月10日周五 上午11:08写道：
> > >
> > >> Hi all,
> > >>
> > >> It has been more than two months since we released Flink 1.9.2. There
> > are
> > >> already 36 improvements/bugfixes in the release-1.9 branch.
> Therefore, I
> > >> propose to create the next bugfix release 1.9.3 for Flink 1.9.
> > >>
> > >> Most notable fixes are:
> > >>
> > >> - [FLINK-15085] HistoryServer dashboard config json out of sync
> > >> - [FLINK-15575] Azure Filesystem Shades Wrong Package "httpcomponents"
> > >> - [FLINK-15638] releasing/create_release_branch.sh does not set
> version
> > in
> > >> flink-python/pyflink/version.py
> > >> - [FLINK-16242] BinaryGeneric serialization error cause checkpoint
> > failure
> > >> - [FLINK-16573] Kinesis consumer does not properly shutdown
> > RecordFetcher
> > >> threads
> > >> - [FLINK-16047] Blink planner produces wrong aggregate results with
> > state
> > >> clean up
> > >> - [FLINK-16860] TableException: Failed to push filter into table
> source!
> > >> when upgrading flink to 1.9.2
> > >> - [FLINK-16916] The logic of NullableSerializer#copy is wrong
> > >> - [FLINK-16389] Bump Kafka 0.10 to 0.10.2.2
> > >> - [FLINK-15812] HistoryServer archiving is done in Dispatcher main
> > thread
> > >> - [FLINK-17062] Fix the conversion from Java row type to Python row
> type
> > >>
> > >> Furthermore, there is one blocker issue which should be merged before
> > >> 1.9.3 release:
> > >>
> > >> - [FLINK-16576] State inconsistency on restore with memory state
> > backends
> > >> (reviewing)
> > >>
> > >> I would volunteer as the release manager and kick off the release
> > process.
> > >> What do you think?
> > >>
> > >> Please let me know if there are any concerns or any other blocker
> issues
> > >> need to be fixed in 1.9.3. Thanks.
> > >>
> > >> Appreciated if there is any PMC could help with the final steps of the
> > >> release process.
> > >>
> > >> Regards,
> > >> Dian
> >
> >
>

Re: [DISCUSS] Releasing Flink 1.10.1

[Hequn Cheng]

.apa...@gmail.com>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>  - For 1.10.1 I am not completely sure,
> >> because
> >>>>>> users
> >>>>>>>>>> expect
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> upgrade
> >>>>>>>>>>>>>>>>>> that without config adjustments. That might not
> >>> be
> >>>>>>>>>> possible
> >>>>>>>>>>>> with
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> change.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Ok, makes sense, I will revert it for 1.10 and
> >> only
> >>>>>> try to
> >>>>>>>>>>>> improve
> >>>>>>>>>>>>>>> error
> >>>>>>>>>>>>>>>>> message and docs.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 12 Mar 2020, at 13:15, Stephan Ewen <
> >>>>>>>>>> se...@apache.org>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> @Andrey about the increase in metaspace size
> >>>>>>>>>>>>>>>>>>  - I have no concerns for 1.11.0.
> >>>>>>>>>>>>>>>>>>  - For 1.10.1 I am not completely sure,
> >> because
> >>>>>> users
> >>>>>>>>>> expect
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> upgrade
> >>>>>>>>>>>>>>>>>> that without config adjustments. That might not
> >>> be
> >>>>>>>>>> possible
> >>>>>>>>>>>> with
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> change.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, Mar 12, 2020 at 12:55 PM Andrey
> >> Zagrebin
> >>> <
> >>>>>>>>>>>>>>>>> azagrebin.apa...@gmail.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> About "FLINK-16142 Memory Leak causes
> >> Metaspace
> >>>> OOM
> >>>>>>>>>> error
> >>>>>>>>>>> on
> >>>>>>>>>>>>>>> repeated
> >>>>>>>>>>>>>>>>>>> job”
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> My understanding that the issue is basically
> >>>> covered
> >>>>>>>>>> by:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> - [FLINK-16225] Metaspace Out Of Memory should
> >>> be
> >>>>>>>>>> handled as
> >>>>>>>>>>>>> Fatal
> >>>>>>>>>>>>>>>> Error
> >>>>>>>>>>>>>>>>>>> in TaskManager
> >>>>>>>>>>>>>>>>>>>  no full consensus there but improving error
> >>>>>> message
> >>>>>>>>>> for
> >>>>>>>>>>>>> existing
> >>&

Re: [ANNOUNCE] New Apache Flink PMC Member - Hequn Chen

[Hequn Cheng]

Many thanks for your support. Thank you!

Best,
Hequn

On Sat, Apr 18, 2020 at 1:27 AM Jacky Bai  wrote:

> Congratulations！Hequn Chen.I hope to make so many contributions to Flink
> like you.
>
> Best
> Bai Xu
>
> Congxian Qiu  于2020年4月17日周五 下午10:47写道：
>
> > Congratulations, Hequn!
> >
> > Best,
> > Congxian
> >
> >
> > Yu Li  于2020年4月17日周五 下午9:36写道：
> >
> > > Congratulations, Hequn!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Fri, 17 Apr 2020 at 21:22, Kurt Young  wrote:
> > >
> > > > Congratulations Hequn!
> > > >
> > > > Best,
> > > > Kurt
> > > >
> > > >
> > > > On Fri, Apr 17, 2020 at 8:57 PM Till Rohrmann 
> > > > wrote:
> > > >
> > > > > Congratulations Hequn!
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Fri, Apr 17, 2020 at 2:49 PM Shuo Cheng 
> > wrote:
> > > > >
> > > > > > Congratulations, Hequn
> > > > > >
> > > > > > Best,
> > > > > > Shuo
> > > > > >
> > > > > > On 4/17/20, hufeih...@mails.ucas.ac.cn <
> hufeih...@mails.ucas.ac.cn
> > >
> > > > > wrote:
> > > > > > > Congratulations , Hequn
> > > > > > >
> > > > > > > Best wish
> > > > > > >
> > > > > > >
> > > > > > > hufeih...@mails.ucas.ac.cn
> > > > > > > Congratulations, Hequn!
> > > > > > >
> > > > > > > Paul Lam  于2020年4月17日周五 下午3:02写道：
> > > > > > >
> > > > > > >> Congrats Hequn! Thanks a lot for your contribution to the
> > > community!
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Paul Lam
> > > > > > >>
> > > > > > >> Dian Fu  于2020年4月17日周五 下午2:58写道：
> > > > > > >>
> > > > > > >> > Congratulations, Hequn!
> > > > > > >> >
> > > > > > >> > > 在 2020年4月17日，下午2:36，Becket Qin  写道：
> > > > > > >> > >
> > > > > > >> > > Hi all,
> > > > > > >> > >
> > > > > > >> > > I am glad to announce that Hequn Chen has joined the Flink
> > > PMC.
> > > > > > >> > >
> > > > > > >> > > Hequn has contributed to Flink for years. He has worked on
> > > > several
> > > > > > >> > > components including Table / SQL，PyFlink and Flink ML
> > > Pipeline.
> > > > > > >> Besides,
> > > > > > >> > > Hequn is also very active in the community since the
> > > beginning.
> > > > > > >> > >
> > > > > > >> > > Congratulations, Hequn! Looking forward to your future
> > > > > > contributions.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > >
> > > > > > >> > > Jiangjie (Becket) Qin
> > > > > > >> > > (On behalf of the Apache Flink PMC)
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Regards
> > > > > > >
> > > > > > > Jeff Zhang
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: New contributor

[Hequn Cheng]

Welcome, Etienne :)

Best,
Hequn

On Fri, Apr 24, 2020 at 10:04 PM Etienne Chauchot 
wrote:

> Hi Till,
>
> Looking forward too ...
>
> Thanks
>
> Etienne
>
> On 24/04/2020 15:09, Till Rohrmann wrote:
> > Hi Etienne,
> >
> > welcome to the Flink community. Looking forward to working with you on
> > Flink :-)
> >
> > Cheers,
> > Till
> >
> > On Fri, Apr 24, 2020 at 11:20 AM Etienne Chauchot 
> > wrote:
> >
> >> Hi everyone,
> >>
> >> Let me introduce myself, I'm Etienne Chauchot, I'm an Apache Beam
> >> committer and a PMC member and I would like to start working on Flink as
> >> well.
> >>
> >> For now I only did 3 simple Flink PRs but that will grow :)
> >>
> >> https://github.com/apache/flink/pull/11886
> >>
> >> https://github.com/apache/flink/pull/11740
> >>
> >> https://github.com/apache/flink/pull/11703
> >>
> >> Here is a link to my blog: https://echauchot.blogspot.com/
> >>
> >> Best.
> >>
> >> Etienne
> >>
> >>
>

Re: [ANNOUNCE] Apache Flink 1.9.3 released

[Hequn Cheng]

@Dian, thanks a lot for the release and for being the release manager.
Also thanks to everyone who made this release possible!

Best,
Hequn

On Sat, Apr 25, 2020 at 7:57 PM Dian Fu  wrote:

> Hi everyone,
>
> The Apache Flink community is very happy to announce the release of Apache
> Flink 1.9.3, which is the third bugfix release for the Apache Flink 1.9
> series.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Please check out the release blog post for an overview of the improvements
> for this bugfix release:
> https://flink.apache.org/news/2020/04/24/release-1.9.3.html
>
> The full release notes are available in Jira:
> https://issues.apache.org/jira/projects/FLINK/versions/12346867
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
> Also great thanks to @Jincheng for helping finalize this release.
>
> Regards,
> Dian
>

Re: [COMMITTER] repo locked due to synchronization issues

[Hequn Cheng]

I met the same problem. Pushing to the GitHub repo directly works fine and
it seems will resync the two repos.

Best, Hequn

On Tue, Sep 24, 2019 at 4:59 PM Fabian Hueske  wrote:

> Maybe it's a mix of pushing to the ASF repository and Github mirrors?
> I'm only pushing to the ASF repositories (although not that frequently
> anymore...).
>
> Cheers, Fabian
>
> Am Di., 24. Sept. 2019 um 10:50 Uhr schrieb Till Rohrmann <
> trohrm...@apache.org>:
>
> > Pushing directly to Github also works for me without a problem.
> >
> > Cheers,
> > Till
> >
> > On Tue, Sep 24, 2019 at 10:28 AM Jark Wu  wrote:
> >
> > > Hi Bowen,
> > >
> > > I have also encountered this problem. I don't know how to fix this.
> > > But pushing to GitHub repo always works for me.
> > >
> > > Best,
> > > Jark
> > >
> > > On Tue, 24 Sep 2019 at 06:05, Bowen Li  wrote:
> > >
> > > > Hi committers,
> > > >
> > > > Recently I've run a repo issue multiple times in different days.
> When I
> > > > tried to push a commit to master, git reports the following error:
> > > >
> > > > ```
> > > > remote: This repository has been locked due to synchronization
> issues:
> > > > remote:  - /x1/gitbox/broken/flink.txt exists due to a previous
> error,
> > > and
> > > > prevents pushes.
> > > > remote: This could either be a benign issue, or the repositories
> could
> > be
> > > > out of sync.
> > > > remote: Please contact us...@infra.apache.org to have infrastructure
> > > > resolve the issue.
> > > > remote:
> > > > To https://gitbox.apache.org/repos/asf/flink.git
> > > >  ! [remote rejected]   master -> master (pre-receive hook
> declined)
> > > > error: failed to push some refs to '
> > > > https://gitbox.apache.org/repos/asf/flink.git'
> > > > ```
> > > >
> > > > This is quite a new issue that didn't come till two or three weeks
> > ago. I
> > > > researched online with no luck. I also reported it to ASF INFRA [1]
> but
> > > > their suggested solution doesn't work.
> > > >
> > > > The issue however usually goes away the next morning in PST, so I
> > assume
> > > > someone from a different timezone in Asia or Europe fixes it somehow?
> > Has
> > > > anyone run into it before? How did you fix it?
> > > >
> > > > Thanks,
> > > > Bowen
> > > >
> > > > [1] https://issues.apache.org/jira/projects/INFRA/issues/INFRA-18992
> > > >
> > >
> >
>

Re: [DISCUSS] Releasing Flink 1.9.1

[Hequn Cheng]

Hi,

@jincheng sun  @Dian Fu
 Thanks
a lot for reporting and fixing the problem!
@Jark Wu  The PR of FLINK-14288 has been merged into both
master and release-1.9. Looking forward to the first RC of 1.9.1.

Best, Hequn

On Mon, Sep 30, 2019 at 10:54 AM Jark Wu  wrote:

> Thanks Jincheng and Dian for reporting and fixing this.
> Please let me know when FLINK-14288 is merged.
>
> Best,
> Jark
>
>
> On Mon, 30 Sep 2019 at 09:29, jincheng sun 
> wrote:
>
> > Yes, Thanks for the quick response @Dian Fu .
> >
> > Dian Fu  于2019年9月30日周一 上午9:22写道：
> >
> > > Hi Jincheng,
> > >
> > > Thanks a lot for reporting this issue. Good catch! I'd like to take
> this
> > > issue and fix it ASAP. Would you please assign it to me?
> > >
> > > Regards,
> > > Dian
> > >
> > > On Mon, Sep 30, 2019 at 9:18 AM jincheng sun  >
> > > wrote:
> > >
> > > > Hi Jark,
> > > >
> > > > I just found that there is NOTICE problem for source release, we
> should
> > > fix
> > > > it in 1.9.1. The detail can be found in FLINK-14288.
> > > >
> > > > Best,
> > > > Jincheng
> > > >
> > > >
> > > > Debasish Ghosh  于2019年9月29日周日 下午4:14写道：
> > > >
> > > > > Thanks a lot!
> > > > >
> > > > > On Sun, Sep 29, 2019 at 1:42 PM Jark Wu  wrote:
> > > > >
> > > > > > Hi Debasish,
> > > > > >
> > > > > > Yes. The maven artifacts will also be updated. I can upgrade the
> > > > version
> > > > > of
> > > > > > flink-avro to 1.9.1 once it is released.
> > > > > >
> > > > > > Cheers,
> > > > > > Jark
> > > > > >
> > > > > > On Sun, 29 Sep 2019 at 16:08, Debasish Ghosh <
> > > ghosh.debas...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > 1 more question .. since it has been cherry picked to 1.9
> branch
> > > will
> > > > > the
> > > > > > > maven artifacts of 1.9 also be updated accordingly ?
> > > > > > >
> > > > > > > regards.
> > > > > > >
> > > > > > > On Sun, Sep 29, 2019 at 1:37 PM Debasish Ghosh <
> > > > > ghosh.debas...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Thanks a lot ..
> > > > > > >>
> > > > > > >> On Sun, Sep 29, 2019 at 1:36 PM Jark Wu 
> > wrote:
> > > > > > >>
> > > > > > >>> Hi Debasish,
> > > > > > >>>
> > > > > > >>> +1 have FLINK-12501 in 1.9.1. As there is nobody against it
> > under
> > > > the
> > > > > > >>> JIRA issue. I just cherry-picked it to 1.9 branch.
> > > > > > >>>
> > > > > > >>> Best,
> > > > > > >>> Jark
> > > > > > >>>
> > > > > > >>> On Sun, 29 Sep 2019 at 13:09, Debasish Ghosh <
> > > > > ghosh.debas...@gmail.com
> > > > > > >
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > >  Hi -
> > > > > > 
> > > > > >  Will FLINK-12501 be included in 1.9.1 ?
> > > > > > 
> > > > > >  regards
> > > > > > 
> > > > > >  On Sun, 29 Sep 2019 at 9:04 AM, Jark Wu 
> > > wrote:
> > > > > > 
> > > > > >  > Hi all,
> > > > > >  >
> > > > > >  > I am here to update the progress of the issues that need
> to
> > be
> > > > > >  tracked:
> > > > > >  >
> > > > > >  > - FLINK-14010 (merged)
> > > > > >  > - FLINK-13386 (merged)
> > > > > >  > - FLINK-14145 (merged)
> > > > > >  > - FLINK-14118 (not back port to 1.9)
> > > > > >  > - FLINK-13708 (as discussed with Jeff offline, postpone it
> > to
> > > > > 1.9.2)
> > > > > >  >
> > > > > >  > Great thanks to all of you for helping fix and reviewing!
> > All
> > > > the
> > > > > >  blocker
> > > > > >  > issues for 1.9.1 has been resolved.
> > > > > >  > Thanks @Jincheng for creating 1.9.2 version in JIRA.
> > > > > >  > I'll proceed to create the first release candidate for
> 1.9.1
> > > now
> > > > > > then.
> > > > > >  >
> > > > > >  > Cheers,
> > > > > >  > Jark
> > > > > >  >
> > > > > >  > On Thu, 26 Sep 2019 at 18:25, Jark Wu 
> > > wrote:
> > > > > >  >
> > > > > >  > > Thanks a lot for the kindly help @jincheng. That will be
> > > > really
> > > > > >  helpful.
> > > > > >  > > Could you help to create 1.9.2 release version in JIRA
> > when
> > > > you
> > > > > > are
> > > > > >  free?
> > > > > >  > >
> > > > > >  > > Best,
> > > > > >  > > Jark
> > > > > >  > >
> > > > > >  > > On Thu, 26 Sep 2019 at 17:59, jincheng sun <
> > > > > >  sunjincheng...@gmail.com>
> > > > > >  > > wrote:
> > > > > >  > >
> > > > > >  > >> Good Job Jark, Looking forward the first RC of 1.9.1.
> > And I
> > > > > would
> > > > > >  like
> > > > > >  > to
> > > > > >  > >> give you a hand at the last stage when the RC is
> > > finished.(If
> > > > > you
> > > > > >  need)
> > > > > >  > >> Best,
> > > > > >  > >> Jincheng
> > > > > >  > >>
> > > > > >  > >> Jark Wu  于2019年9月25日周三 下午8:04写道：
> > > > > >  > >>
> > > > > >  > >> > Hi all,
> > > > > >  > >> >
> > > > > >  > >> > I am here to update the progress of the issue that
> > needs
> > > to
> > > > > be
> > > > > >  > tracked:
> > > > > >  > >> >
> > > > > >  > >> > - FLINK-14010 (

Re: [DISCUSS] Drop Python 2 support for 1.10

[Hequn Cheng]

Hi Dian,

+1 to drop Python 2 directly.

Just as @jincheng said, things would be more complicated if we are going to
support python UDFs.
The python UDFs will introduce a lot of python dependencies which will also
drop the support of Python 2, such as beam, pandas, pyarrow, etc.
Given this and Python 2 will reach EOL on Jan 1 2020. I think we can drop
Python 2 in Flink as well.

As for the two options, I think we can drop it directly in 1.10. The
flink-python is introduced just from 1.9, I think it's safe to drop it for
now.
And we can also benefit from it when we add support for python UDFs.

Best, Hequn


On Wed, Oct 9, 2019 at 8:40 AM jincheng sun 
wrote:

> Hi Dian，
>
> Thanks for bringing this discussion!
>
> In Flink 1.9 we only add Python Table API mapping to Java Table API(without
> Python UDFs), there no special requirements for Python version, so we add
> python 2,7 support. But for Flink 1.10, we add the Python UDFs support,
> i.e., user will add more python code in Flink job and more requirements for
> the features of the Python language.So I think It's better to follow the
> rhythm of Python official.
>
> Option 2 is the most conservative and correct approach, but for the current
> situation, we cooperate with the Beam community and use Beam's portability
> framework for UDFs support, so we prefer to adopt the Option 1.
>
> Best,
> Jincheng
>
>
>
> Dian Fu  于2019年10月8日周二 下午10:34写道：
>
> > Hi everyone,
> >
> > I would like to propose to drop Python 2 support(Currently Python 2.7,
> > 3.5, 3.6, 3.7 are all supported in Flink) as it's coming to an end at Jan
> > 1, 2020 [1]. A lot of projects [2][3][4] has already stated or are
> planning
> > to drop Python 2 support.
> >
> > The benefits of dropping Python 2 support are:
> > 1. Maintaining Python 2/3 compatibility is a burden and it makes the code
> > complicate as Python 2 and Python 3 is not compatible.
> > 2. There are many features which are only available in Python 3.x such as
> > Type Hints[5]. We can only make use of this kind of features after
> dropping
> > the Python 2 support.
> > 3. Flink-python depends on third-part projects, such as Apache Beam (may
> > add more dependencies such as pandas, etc in the near future), it's not
> > possible to upgrade them to the latest version once they drop the Python
> 2
> > support.
> >
> > Here are the options we have:
> > 1. Drop Python 2 support in 1.10:
> > As flink-python module is a new module added since 1.9.0 and so dropping
> > Python 2 support at the early stage seems a good choice for us.
> > 2. Deprecate Python 2 in 1.10 and drop its support in 1.11:
> > As 1.10 is planned to be released around the beginning of 2020. This is
> > also aligned with the official Python 2 support.
> >
> > Personally I prefer option 1 as flink-python is new module and there is
> no
> > much history reasons to consider.
> >
> > Looking forward to your feedback!
> >
> > Regards,
> > Dian
> >
> > [1] https://pythonclock.org/ 
> > [2] https://python3statement.org/ 
> > [3]
> https://spark.apache.org/news/plan-for-dropping-python-2-support.html
> > 
> > [4]
> >
> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
> > <
> >
> https://lists.apache.org/thread.html/eba6caa58ea79a7ecbc8560d1c680a366b44c531d96ce5c699d41535@%3Cdev.beam.apache.org%3E
> > >
> > [5]
> >
> https://stackoverflow.com/questions/32557920/what-are-type-hints-in-python-3-5
> > <
> >
> https://stackoverflow.com/questions/32557920/what-are-type-hints-in-python-3-5
> > >
>

Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

[Hequn Cheng]

Hi Stephan,

Big +1 for adding this to Apache Flink!

As for the problem of whether this should be added to the Flink main
repository, from my side, I prefer to put it in the main repository. Not
only Stateful Functions shares very close relations with the current Flink,
but also other libs or modules in Flink can make use of it the other way
round in the future. At that time the Flink API stack would also be changed
a bit and this would be cool.

Best, Hequn

On Sat, Oct 12, 2019 at 9:16 PM Biao Liu  wrote:

> Hi Stehpan,
>
> +1 for having Stateful Functions in Flink.
>
> Before discussing which repository it should belong, I was wondering if we
> have reached an agreement of "splitting flink repository" as Piotr
> mentioned or not. It seems that it's just no more further discussion.
> It's OK for me to add it to core repository. After all almost everything
> is in core repository now. But if we decide to split the core repository
> someday, I tend to create a separate repository for Stateful Functions. It
> might be good time to take the first step of splitting.
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Sat, 12 Oct 2019 at 19:31, Yu Li  wrote:
>
>> Hi Stephan,
>>
>> Big +1 for adding stateful functions to Flink. I believe a lot of user
>> would be interested to try this out and I could imagine how this could
>> contribute to reduce the TCO for business requiring both streaming
>> processing and stateful functions.
>>
>> And my 2 cents is to put it into flink core repository since I could see
>> a tight connection between this library and flink state.
>>
>> Best Regards,
>> Yu
>>
>>
>> On Sat, 12 Oct 2019 at 17:31, jincheng sun 
>> wrote:
>>
>>> Hi Stephan,
>>>
>>> bit +1 for adding this great features to Apache Flink.
>>>
>>> Regarding where we should place it, put it into Flink core repository or
>>> create a separate repository? I prefer put it into main repository and
>>> looking forward the more detail discussion for this decision.
>>>
>>> Best,
>>> Jincheng
>>>
>>>
>>> Jingsong Li  于2019年10月12日周六 上午11:32写道：
>>>
 Hi Stephan,

 big +1 for this contribution. It provides another user interface that
 is easy to use and popular at this time. these functions, It's hard for
 users to write in SQL/TableApi, while using DataStream is too complex.
 (We've done some stateFun kind jobs using DataStream before). With
 statefun, it is very easy.

 I think it's also a good opportunity to exercise Flink's core
 capabilities. I looked at stateful-functions-flink briefly, it is very
 interesting. I think there are many other things Flink can improve. So I
 think it's a better thing to put it into Flink, and the improvement for it
 will be more natural in the future.

 Best,
 Jingsong Lee

 On Fri, Oct 11, 2019 at 7:33 PM Dawid Wysakowicz <
 dwysakow...@apache.org> wrote:

> Hi Stephan,
>
> I think this is a nice library, but what I like more about it is that
> it suggests exploring different use-cases. I think it definitely makes
> sense for the Flink community to explore more lightweight applications 
> that
> reuses resources. Therefore I definitely think it is a good idea for Flink
> community to accept this contribution and help maintaining it.
>
> Personally I'd prefer to have it in a separate repository. There were
> a few discussions before where different people were suggesting to extract
> connectors and other libraries to separate repositories. Moreover I think
> it could serve as an example for the Flink ecosystem website[1]. This 
> could
> be the first project in there and give a good impression that the 
> community
> sees potential in the ecosystem website.
>
> Lastly, I'm wondering if this should go through PMC vote according to
> our bylaws[2]. In the end the suggestion is to adopt an existing code base
> as is. It also proposes a new programs concept that could result in a 
> shift
> of priorities for the community in a long run.
>
> Best,
>
> Dawid
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html
>
> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
> On 11/10/2019 13:12, Till Rohrmann wrote:
>
> Hi Stephan,
>
> +1 for adding stateful functions to Flink. I believe the new set of
> applications this feature will unlock will be super interesting for new 
> and
> existing Flink users alike.
>
> One reason for not including it in the main repository would to not
> being bound to Flink's release cadence. This would allow to release faster
> and more often. However, I believe that having it eventually in Flink's
> main repository would be beneficial in the long run.
>
> Cheers,
> Till
>
> On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <

Re: [VOTE] Release 1.9.1, release candidate #1

[Hequn Cheng]

+1 (non-binding)

Do the following checks and all are success.
- Verified signatures and hashes.
- Built from the source archive.
- Check repository contains all artifacts.
- Test WordCount on a local standalone cluster.
a. Both streaming and batch
b. Web UI works fine
- Test WordCount on yarn cluster, with 4 nodes.
a. Both streaming and batch
b. Web UI works fine
c. Read write hdfs files.
d. Test session mode and non-session mode.

Minor comments(not blocker)
- One comment about the website pr.
- Found an issue that there are
ClassNotFoundException(java.lang.ClassNotFoundException:
org.apache.hadoop.yarn.exceptions.YarnException) when run the setup
example(under Local Flink Cluster). It may be confusing for new users and
would be nice if can be improved. (
https://issues.apache.org/jira/browse/FLINK-14385)

Best, Hequn

On Sat, Oct 12, 2019 at 6:01 PM Jark Wu  wrote:

> Hi Jingsong,
>
> Thanks for verifying. I updated the fixVersion to 1.9.2 for these issues.
>
> Best,
> Jark
>
> > 在 2019年10月12日，16:45，Jingsong Li  写道：
> >
> > +1 (non-binding)
> >
> > - Check if checksums files match the corresponding release files
> > - Check if GPG files match the corresponding release files
> > - Verify that the source archives do not contains any binaries
> > - Build the source with Maven to ensure all source files have Apache
> headers
> > - Check that all POM files point to the same version (1.9.1)
> > - Start a local cluster both Scala 2.11 and 2.12, and shut down. verified
> > out and log, verified we ui. run examples.
> > All succeeded.
> >
> > Hi Jark, there are some JIRA issue still use fix version 1.9.0, do you
> need
> > modify fix version?
> > https://issues.apache.org/jira/browse/FLINK-14328
> > https://issues.apache.org/jira/browse/FLINK-14327
> > https://issues.apache.org/jira/browse/FLINK-14215
> > https://issues.apache.org/jira/browse/FLINK-14072
> > https://issues.apache.org/jira/browse/FLINK-12576
> >
> > Best,
> > Jingsong Lee
> >
> >
> > On Wed, Oct 9, 2019 at 3:32 PM Jark Wu  wrote:
> >
> >> +1 from my side.
> >>
> >> - checked signatures and hashes
> >> - checked that all POM files point to the same version
> >> - verified that the source archives do not contains any binaries
> >> - build the source release with Scala 2.12 and Scala 2.11 successfully
> >> - manually verified the diff pom files between 1.9.0 and 1.9.1 to check
> >> dependencies, looks good
> >> - started cluster for both Scala 2.11 and 2.12, ran examples, verified
> web
> >> ui and log output, nothing unexpected
> >>
> >> Best,
> >> Jark
> >>
> >> On Wed, 9 Oct 2019 at 11:18, Jark Wu  wrote:
> >>
> >>> Thanks Jincheng and Till, then let's keep on verifying the RC1.
> >>>
> >>> Best,
> >>> Jark
> >>>
> >>> On Wed, 9 Oct 2019 at 11:00, jincheng sun 
> >>> wrote:
> >>>
>  I think we should create the new RC when we find the blocker issues.
>  We can looking forward the other check result, we can add the fix of
>  FLINK-14315 in to 1.9.1 only we find the blockers.
> 
>  Best,
>  Jincheng
> 
>  Till Rohrmann  于2019年10月8日周二 下午8:20写道：
> 
> > FLINK-14315 has been merged into the release-1.9 branch. I've marked
> >> the
> > fix version of this ticket as 1.9.2. If we should create a new RC,
> then
> > we
> > could include this fix. If this happens, then we need to update the
> fix
> > version to 1.9.1.
> >
> > Cheers,
> > Till
> >
> > On Tue, Oct 8, 2019 at 1:51 PM Till Rohrmann 
> > wrote:
> >
> >> If people already spent time on verifying the current RC I would
> also
> > be
> >> fine to release the fix for FLINK-14315 with Flink 1.9.2.
> >>
> >> I will try to merge the PR as soon as possible. When I close the
> > ticket, I
> >> will update the fix version field to 1.9.2.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Tue, Oct 8, 2019 at 4:43 AM Jark Wu  wrote:
> >>
> >>> Hi Zili,
> >>>
> >>> Thanks for reminding me this, because of the Chinese National Day
> >> and
> >>> Flink Forward Europe,
> >>> we didn't receive any verification on the 1.9.1 RC1. And I guess we
> > have
> >>> to extend the voting time after Flink Forward.
> >>> So I'm fine to have FLINK-14315 and rebuild another RC. What do you
> > think
> >>> @Till @Jincheng?
> >>>
> >>> I guess FLINK-14315 will be merged soon as it is approved 4 days
> >> ago?
> >>> Could you help to merge it once it is passed ? @Zili Chen
> >>> 
> >>>
> >>> Best,
> >>> Jark
> >>>
> >>> On Tue, 8 Oct 2019 at 09:14, Zili Chen 
> >> wrote:
> >>>
>  Hi Jark,
> 
>  I notice a critical bug[1] is marked resolved in 1.9.1 but given
> > 1.9.1
>  has been cut I'd like to throw the issue here so that we're sure
>  whether or not it is included in 1.9.1.
> 
>  Best,
>  tison.
> 
>  [1] https://issue

Re: [VOTE] Drop Python 2 support for 1.10

[Hequn Cheng]

+1

Thanks a lot for driving this, Dian!

On Mon, Oct 14, 2019 at 1:46 PM jincheng sun 
wrote:

> +1
>
> Dian Fu  于2019年10月14日周一 下午1:21写道：
>
> > Hi all,
> >
> > I would like to start the vote for "Drop Python 2 support for 1.10",
> which
> > is discussed and reached a consensus in the discussion thread[1].
> >
> > The vote will be open for at least 72 hours. Unless there is an
> objection,
> > I will try to close it by Oct 17, 2019 18:00 UTC if we have received
> > sufficient votes.
> >
> > Regards,
> > Dian
> >
> > [1]
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Drop-Python-2-support-for-1-10-td33824.html
> > <
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Drop-Python-2-support-for-1-10-td33824.html
> > >
>

Re: [VOTE] FLIP-78: Flink Python UDF Environment and Dependency Management

[Hequn Cheng]

+1

Good job, Wei!

Best, Hequn

On Mon, Oct 14, 2019 at 2:54 PM Dian Fu  wrote:

> Hi Wei,
>
> +1 (non-binding). Thanks for driving this.
>
> Thanks,
> Dian
>
> > 在 2019年10月14日，下午1:40，jincheng sun  写道：
> >
> > +1
> >
> > Wei Zhong  于2019年10月12日周六 下午8:41写道：
> >
> >> Hi all,
> >>
> >> I would like to start the vote for FLIP-78[1] which is discussed and
> >> reached consensus in the discussion thread[2].
> >>
> >> The vote will be open for at least 72 hours. I'll try to close it by
> >> 2019-10-16 18:00 UTC, unless there is an objection or not enough votes.
> >>
> >> Thanks,
> >> Wei
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-78%3A+Flink+Python+UDF+Environment+and+Dependency+Management
> >> <
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-78:+Flink+Python+UDF+Environment+and+Dependency+Management
> >>>
> >> [2]
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Python-UDF-Environment-and-Dependency-Management-td33514.html
> >> <
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Python-UDF-Environment-and-Dependency-Management-td33514.html
> >>>
> >>
> >>
> >>
>
>

Re: [DISCUSS] FLIP policy for introducing config option keys

[Hequn Cheng]

Hi all,

+1 to have a looser FLIP policy for these API changes.

I think the concerns raised above are all valid. Besides the feedbacks from
@Jark, if we want to keep track of these changes, maybe we can create a new
kind of FLIP that is dedicated to these minor API changes? For example, we
can add a single wiki page and list all related JIRAs in it. The design
details can be described in the JIRA.
Another option is to simply add a new JIRA label to track these changes.

What do you think?

Best, Hequn

On Tue, Oct 15, 2019 at 8:43 PM Zili Chen  wrote:

> The naming concern above can be a separated issue since it looks also
> affect FLIP-54 and isn't limited for config option changes FLIP.
>
> Best,
> tison.
>
>
> Aljoscha Krettek  于2019年10月15日周二 下午8:37写道：
>
> > Another PR that introduces new config options:
> > https://github.com/apache/flink/pull/9759
> >
> > > On 15. Oct 2019, at 14:31, Zili Chen  wrote:
> > >
> > > Hi Aljoscha & Dawid & Kostas,
> > >
> > > I agree that changes on config option keys deserve a FLIP and it is
> > > reasonable
> > > we commit the changes with a standard FLIP process so that ensure the
> > change
> > > given proper visibility.
> > >
> > > My concern is about naming. Given FLIP-73 as an example, if FLIPs
> > > associated to
> > > FLIP-73(actually can be regarded as sub-FLIP of it) grows FLIP numbers
> > and
> > > appears
> > > like FLIP-80 FLIP-85 FLIP-91 and so on, then we possibly run into a
> state
> > > flooded by
> > > quite a few config option only FLIP. Maybe it makes sense to number
> these
> > > FLIP as
> > > FLIP-73.1 FLIP-73.2, which shows the association and doesn't pollute
> > other
> > > FLIPs.
> > >
> > > Remind the general thoughts, IMO changes on config option keys deserve
> a
> > > standard
> > > FLIP process, e.g. FLIP-61.
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Kostas Kloudas  于2019年10月15日周二 下午8:20写道：
> > >
> > >> Hi Aljoscha,
> > >>
> > >> Given that config option keys are user-facing and any change there is
> > >> breaking, I think there should be a discussion about them and a FLIP,
> > >> where people have to actually vote for it seems to be the right place.
> > >> I understand that this is tedious (and actually I will also have to
> > >> open some FLIPs as part of FLIP-73), but this contributes to the
> > >> uniformity of our parameters and also giving them some more
> > >> visibility.
> > >>
> > >> Cheers,
> > >> Kostas
> > >>
> > >> On Tue, Oct 15, 2019 at 2:05 PM Aljoscha Krettek  >
> > >> wrote:
> > >>>
> > >>> Hi Everyone,
> > >>>
> > >>> The title says it all, do you think we need to cover all config
> options
> > >> that we introduce/change by FLIPs? I was thinking about this because
> of
> > the
> > >> FLIP-73 work, which will introduce some new config options and also
> > because
> > >> I just spotted a PR [1] that introduces some config options.
> > >>>
> > >>> Best,
> > >>> Aljoscha
> > >>>
> > >>> [1] https://github.com/apache/flink/pull/9836
> > >>
> >
> >
>

Re: [NOTICE] Binary licensing is now auto-generated

[Hequn Cheng]

Cool! Thanks a lot for the improvement, Chesnay!

On Fri, Oct 18, 2019 at 9:51 AM Jark Wu  wrote:

> Thanks Chesnay! This is really a great job!
>
> Best,
> Jark
>
> > 在 2019年10月17日，22:03，未来阳光 <2217232...@qq.com> 写道：
> >
> > Thanks for this improvement Chesnay !
> >
> >
> >
> >
> > ---Original---
> > From: "Chesnay Schepler" > Date: Thu, Oct 17, 2019 21:37 PM
> > To: "dev@flink.apache.org" > Subject: [NOTICE] Binary licensing is now auto-generated
> >
> >
> > Hello,
> >
> > I just merged FLINK-14008 to 1.8, 1.9 and 1.10, which means that from
> > now on the tricky part of the binary licensing (NOTICE-binary,
> > licenses-binary) is automatically generated during the release process.
> >
> > As such these files have been removed from the root directory of the
> > project (thus, you don't have to update these things anymore ;)).
> >
> > This also means that only builds of flink-dist that were built as part
> > of the release process will have these files attached.
> >
> > I have updated the Licensing guide
> > ;
> accordingly.
>
>

Re: [DISCUSS] FLIP-65: New type inference for Table API UDFs

[Hequn Cheng]

+1 to start a vote.

Good to see that it is backward compatibility, as python UDFs will also be
affected.

BTW, do we target this Flip to release-1.10?

Best, Hequn

On Wed, Oct 16, 2019 at 6:19 PM Timo Walther  wrote:

> Hi everyone,
>
> thanks for the comments in the doc. The feedback so far was very
> positive. I will convert the doc into a FLIP wiki page now.
>
> If there are no further concerns, I would like to start a voting process
> soon.
>
> Thanks,
> Timo
>
> On 09.10.19 04:58, Jingsong Li wrote:
> > Thanks Timo for your pretty nice proposal, big +1 to the FLIP. Left some
> > minor comments.
> >
> > A minor concern about flink-planner, precision things maybe cannot be
> > supported.
> >
> > Best,
> > Jingsong Lee
> >
> > On Tue, Oct 8, 2019 at 5:58 PM zha...@lenovocloud.com <
> > zha...@lenovocloud.com> wrote:
> >
> >> unsubscribe
> >>
> >>
> >>
> >> zha...@lenovocloud.com
> >>
> >> From: Jark Wu
> >> Date: 2019-10-08 17:29
> >> To: dev
> >> Subject: Re: [DISCUSS] FLIP-65: New type inference for Table API UDFs
> >> Hi Timo,
> >>
> >> Thanks for the proposal, a big +1 to the FLIP, especially this enables
> the
> >> unified `TableEnvironment.registerFunction()`.
> >>
> >> I think the design documentation is pretty good enough, I only left some
> >> minor comments there.
> >>
> >> Best,
> >> Jark
> >>
> >> On Fri, 4 Oct 2019 at 23:54, Timo Walther  wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> I would like to propose FLIP-65 that describes how we want to deal with
> >>> data types and their inference/extraction in the Table API in the
> >>> future. I have collected many comments, shortcomings, issues from users
> >>> and trainings in last years that went into the design. It completes the
> >>> work started in FLIP-37 by upgrading the type system to DataTypes and
> >>> allowing new use cases. Some key features of this FLIP are:
> >>>
> >>> - Type extraction closely coupled to the SQL standard (with
> >>> precision/scale specification and complex data types)
> >>>
> >>> - Simple stuff is simple, missing information can be provided with
> >>> annotations without the need to specify everything from scratch.
> >>>
> >>> - Full access to the planner's type inference which allows to create
> >>> UDFs as powerful as built-in system functions
> >>>
> >>> - Unification of Scala and Java API to enable the unified
> >>> TableEnvironment.registerFunction()
> >>>
> >>>
> >>> The design document can be found here:
> >>>
> >>>
> >>>
> >>
> https://docs.google.com/document/d/1Zf8-okGvCiTTRaTN0IqtTGrrjXYNGFJnhUQyinF4xcU/edit#
> >>> I will convert it to a wiki page after the first review comments.
> >>>
> >>> Happy to hear your thoughts.
> >>>
> >>> Thanks,
> >>>
> >>> Timo
> >>>
> >>> [1]
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> >>>
> >
>
>

Re: [ANNOUNCE] Apache Flink 1.9.1 released

[Hequn Cheng]

Thanks a lot to Jark, Jincheng, and everyone that make this release
possible.

Best, Hequn

On Sat, Oct 19, 2019 at 10:29 PM Zili Chen  wrote:

> Thanks a lot for being release manager Jark. Great work!
>
> Best,
> tison.
>
>
> Till Rohrmann  于2019年10月19日周六 下午10:15写道：
>
>> Thanks a lot for being our release manager Jark and thanks to everyone
>> who has helped to make this release possible.
>>
>> Cheers,
>> Till
>>
>> On Sat, Oct 19, 2019 at 3:26 PM Jark Wu  wrote:
>>
>>>  Hi,
>>>
>>> The Apache Flink community is very happy to announce the release of
>>> Apache Flink 1.9.1, which is the first bugfix release for the Apache Flink
>>> 1.9 series.
>>>
>>> Apache Flink® is an open-source stream processing framework for
>>> distributed, high-performing, always-available, and accurate data streaming
>>> applications.
>>>
>>> The release is available for download at:
>>> https://flink.apache.org/downloads.html
>>>
>>> Please check out the release blog post for an overview of the
>>> improvements for this bugfix release:
>>> https://flink.apache.org/news/2019/10/18/release-1.9.1.html
>>>
>>> The full release notes are available in Jira:
>>> https://issues.apache.org/jira/projects/FLINK/versions/12346003
>>>
>>> We would like to thank all contributors of the Apache Flink community
>>> who helped to verify this release and made this release possible!
>>> Great thanks to @Jincheng for helping finalize this release.
>>>
>>> Regards,
>>> Jark Wu
>>>
>>

Re: [VOTE] FLIP-65: New type inference for Table API UDFs

[Hequn Cheng]

+1 (binding)

Best, Hequn

On Mon, Oct 21, 2019 at 4:55 PM Jark Wu  wrote:

> +1 (binding)
>
> Best,
> Jark
>
> > 在 2019年10月21日，16:52，Aljoscha Krettek  写道：
> >
> > +1 (binding)
> >
> > Aljoscha
> >
> >> On 21. Oct 2019, at 10:50, Timo Walther  wrote:
> >>
> >> Hi everyone,
> >>
> >> I would like to start a vote on FLIP-65. The discussion seems to have
> >> reached an agreement.
> >>
> >> Please vote for the following design document:
> >>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-65%3A+New+type+inference+for+Table+API+UDFs
> >>
> >>
> >> The discussion can be found at:
> >>
> >>
> https://lists.apache.org/thread.html/2b8cfd811a927d64a79f47387b7412c3b98a11fcb9358d3c23ef666c@%3Cdev.flink.apache.org%3E
> >>
> >> This voting will be open for at least 72 hours. I'll try to close it on
> >> 2019-10-24 9:00 UTC, unless there is an objection or not enough votes.
> >>
> >> Best,
> >>
> >> Timo
> >>
> >
>
>

Re: [VOTE] Accept Stateful Functions into Apache Flink

[Hequn Cheng]

+1 (non-binding)

Best, Hequn

On Tue, Oct 22, 2019 at 9:21 AM Dian Fu  wrote:

> +1 (non-binding)
>
> Regards,
> Dian
>
> > 在 2019年10月22日，上午9:10，Kurt Young  写道：
> >
> > +1 (binding)
> >
> > Best,
> > Kurt
> >
> >
> > On Tue, Oct 22, 2019 at 12:56 AM Fabian Hueske 
> wrote:
> >
> >> +1 (binding)
> >>
> >> Am Mo., 21. Okt. 2019 um 16:18 Uhr schrieb Thomas Weise  >:
> >>
> >>> +1 (binding)
> >>>
> >>>
> >>> On Mon, Oct 21, 2019 at 7:10 AM Timo Walther 
> wrote:
> >>>
>  +1 (binding)
> 
>  Thanks,
>  Timo
> 
> 
>  On 21.10.19 15:59, Till Rohrmann wrote:
> > +1 (binding)
> >
> > Cheers,
> > Till
> >
> > On Mon, Oct 21, 2019 at 12:13 PM Robert Metzger  >>>
>  wrote:
> >
> >> +1 (binding)
> >>
> >> On Mon, Oct 21, 2019 at 12:06 PM Stephan Ewen 
> >>> wrote:
> >>
> >>> This is the official vote whether to accept the Stateful Functions
> >>> code
> >>> contribution to Apache Flink.
> >>>
> >>> The current Stateful Functions code, documentation, and website can
> >>> be
> >>> found here:
> >>> https://statefun.io/
> >>> https://github.com/ververica/stateful-functions
> >>>
> >>> This vote should capture whether the Apache Flink community is
>  interested
> >>> in accepting, maintaining, and evolving Stateful Functions.
> >>>
> >>> Reiterating my original motivation, I believe that this project is
> >> a
> >> great
> >>> match for Apache Flink, because it helps Flink to grow the
> >> community
> >> into a
> >>> new set of use cases. We see current users interested in such use
>  cases,
> >>> but they are not well supported by Flink as it currently is.
> >>>
> >>> I also personally commit to put time into making sure this
> >> integrates
> >> well
> >>> with Flink and that we grow contributors and committers to maintain
>  this
> >>> new component well.
> >>>
> >>> This is a "Adoption of a new Codebase" vote as per the Flink bylaws
>  [1].
> >>> Only PMC votes are binding. The vote will be open at least 6 days
> >>> (excluding weekends), meaning until Tuesday Oct.29th 12:00 UTC, or
>  until
> >> we
> >>> achieve the 2/3rd majority.
> >>>
> >>> Happy voting!
> >>>
> >>> Best,
> >>> Stephan
> >>>
> >>> [1]
> >>>
> >>
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
> 
> 
> 
> >>>
> >>
>
>

Re: [ANNOUNCE] Becket Qin joins the Flink PMC

[Hequn Cheng]

Congratulations Becket! Well deserved.

Best, Hequn

On Tue, Oct 29, 2019 at 9:01 AM Thomas Weise  wrote:

> Congrats!
>
> --
> sent from mobile
>
> On Mon, Oct 28, 2019, 12:56 PM Shuyi Chen  wrote:
>
> > Congratulations!
> >
> > On Mon, Oct 28, 2019 at 11:18 AM Xingcan Cui  wrote:
> >
> > > Congratulations, Becket!
> > >
> > > Best,
> > > Xingcan
> > >
> > > > On Oct 28, 2019, at 1:23 PM, Xuefu Z  wrote:
> > > >
> > > > Congratulations, Becket!
> > > >
> > > > On Mon, Oct 28, 2019 at 10:08 AM Zhu Zhu  wrote:
> > > >
> > > >> Congratulations Becket!
> > > >>
> > > >> Thanks,
> > > >> Zhu Zhu
> > > >>
> > > >> Peter Huang  于2019年10月29日周二 上午1:01写道：
> > > >>
> > > >>> Congratulations Becket Qin!
> > > >>>
> > > >>>
> > > >>> Best Regards
> > > >>> Peter Huang
> > > >>>
> > > >>> On Mon, Oct 28, 2019 at 9:19 AM Rong Rong 
> > wrote:
> > > >>>
> > >  Congratulations Becket!!
> > > 
> > >  --
> > >  Rong
> > > 
> > >  On Mon, Oct 28, 2019, 7:53 AM Jark Wu  wrote:
> > > 
> > > > Congratulations Becket!
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Mon, 28 Oct 2019 at 20:26, Benchao Li 
> > > >> wrote:
> > > >
> > > >> Congratulations Becket.
> > > >>
> > > >> Dian Fu  于2019年10月28日周一 下午7:22写道：
> > > >>
> > > >>> Congrats, Becket.
> > > >>>
> > >  在 2019年10月28日，下午6:07，Fabian Hueske  写道：
> > > 
> > >  Hi everyone,
> > > 
> > >  I'm happy to announce that Becket Qin has joined the Flink
> PMC.
> > >  Let's congratulate and welcome Becket as a new member of the
> > > >>> Flink
> > > > PMC!
> > > 
> > >  Cheers,
> > >  Fabian
> > > >>>
> > > >>>
> > > >>
> > > >> --
> > > >>
> > > >> Benchao Li
> > > >> School of Electronics Engineering and Computer Science, Peking
> > >  University
> > > >> Tel:+86-15650713730
> > > >> Email: libenc...@gmail.com; libenc...@pku.edu.cn
> > > >>
> > > >
> > > 
> > > >>>
> > > >>
> > > >
> > > >
> > > > --
> > > > Xuefu Zhang
> > > >
> > > > "In Honey We Trust!"
> > >
> > >
> >
>

Re: [DISCUSS] How to prepare the Python environment of PyFlink release in the current Flink release process

[Hequn Cheng]

Hi Dian,

Thanks a lot for bringing the discussion.
It would be a headache to address these environmental problems, so +1 for
option #1 to use the virtual environment.

Best, Hequn

On Tue, Oct 29, 2019 at 9:31 PM Till Rohrmann  wrote:

> Thanks for bringing this topic up Dian. I'd be in favour of option #1
> because this would also allow to create reproducible builds.
>
> Cheers,
> Till
>
> On Tue, Oct 29, 2019 at 5:28 AM jincheng sun 
> wrote:
>
> > Hi,
> > Thanks for bringing up the discussion Dian.
> > +1 for the #1.
> >
> > Hi Jeff, this changes is for the PyFlink release, i.e.,The release
> manager
> > should build the release package for Pyflink, and prepare the python
> > environment during the building. Since 1.10 we only support python 3.5+,
> so
> > it will throw an exception if you use python 3.4.
> >
> > Best,
> > Jincheng
> >
> >
> > Jeff Zhang  于2019年10月29日周二 上午11:55写道：
> >
> > > I am a little confused, why we need to prepare python environment in
> > > release. Shouldn't that be done when user start to use pyflink ?
> > > Or do you mean to set up python environment for pyflink's CI build ?
> > >
> > > Regarding this problem  "It needs a proper Python environment(i.e.
> Python
> > > 3.5+, setuptools, etc) to build the PyFlink package"
> > > Would the build fail if I use python 3.4 ?
> > >
> > >
> > > Dian Fu  于2019年10月29日周二 上午11:01写道：
> > >
> > > > Hi all,
> > > >
> > > > We have reached a consensus that the PyFlink package should be
> > published
> > > > to PyPI in [1]. Thanks to Jincheng's effort, the PyPI account has
> > already
> > > > been created and available to use now [2]. It means that we could
> > publish
> > > > PyFlink to PyPI in the coming releases and it also means that
> > additional
> > > > steps will be added to the normal process of the Flink release to
> > prepare
> > > > the PyFlink release package.
> > > >
> > > > It needs a proper Python environment(i.e. Python 3.5+, setuptools,
> etc)
> > > to
> > > > build the PyFlink package. There are two options in my mind to
> prepare
> > > the
> > > > Python environment:
> > > > 1) Reuse the script lint-python.sh defined in flink-python module to
> > > > create the required virtual environment and build the PyFlink package
> > > using
> > > > the created virtual environment.
> > > > 2) It's assumed that the local Python environment is properly
> installed
> > > > and ready to use. The Python environment requirement will be
> documented
> > > at
> > > > the page "Create a Flink Release" and validation check could also be
> > > added
> > > > in create_binary_release.sh to throw an meaningful error with hints
> how
> > > to
> > > > fix it if it's not correct.
> > > >
> > > > Option 1:
> > > > Pros:
> > > > - It's transparent for release managers.
> > > > Cons:
> > > > - It needs to prepare the virtual environment during preparing the
> > > PyFlink
> > > > release package and it will take some several minutes as it need to
> > > > download a few binaries.
> > > >
> > > > Option 2:
> > > > Pros:
> > > > - There is no need to prepare the virtual environment if the local
> > > > environment is already properly configured.
> > > > Cons:
> > > > - It requires the release managers to prepare the local Python
> > > environment
> > > > and not all the people are familiar with Python and it's a burden for
> > > > release managers.
> > > >
> > > > Personally I prefer to option 1).
> > > >
> > > > Looking forward to your feedback!
> > > >
> > > > PS: I think this issue could also be discussed in the JIRA. But I
> tend
> > to
> > > > bring up the discussion to ML as it introduces an additional step to
> > the
> > > > release process and I think this should be visible to the community
> and
> > > it
> > > > should be well discussed. Besides, we could also get more feedback.
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > > [1]
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Publish-the-PyFlink-into-PyPI-tt31201.html
> > > > [2]
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-13011?focusedCommentId=16947307&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16947307
> > >
> > >
> > >
> > > --
> > > Best Regards
> > >
> > > Jeff Zhang
> > >
> >
>

Re: [VOTE] Move flink-orc to flink-formats from flink-connectors

[Hequn Cheng]

+1. Good catch Jingsong!

Best, Hequn

On Wed, Oct 30, 2019 at 7:26 PM Zhenghua Gao  wrote:

> +1 (non-binding)
>
> *Best Regards,*
> *Zhenghua Gao*
>
>
> On Wed, Oct 30, 2019 at 12:05 PM Jingsong Li 
> wrote:
>
> > Hi all:
> >
> > We already have the parent model of formats. we have put other
> > formats(flink-avro, flink-json, flink-parquet, flink-json, flink-csv,
> > flink-sequence-file) to flink-formats. flink-orc is a format too. So we
> can
> > move it to flink-formats.
> >
> > In theory, there should be no compatibility problem, only the parent
> model
> > needs to be changed, and no other changes are needed.
> >
> > I would like to start the vote for it. The vote will be open for at least
> > 72 hours.
> >
> > Discuss thread: [1]
> >
> > [1]
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Move-flink-orc-to-flink-formats-from-flink-connectors-td34438.html
> >
> > --
> > Best, Jingsong Lee
> >
>

Re: [ANNOUNCE] Jark Wu is now part of the Flink PMC

[Hequn Cheng]

Congratulations Jark, well deserved!

On Fri, Nov 8, 2019 at 5:52 PM jincheng sun 
wrote:

> Hi all,
>
> On behalf of the Flink PMC, I'm happy to announce that Jark Wu is now
> part of the Apache Flink Project Management Committee (PMC).
>
> Jark has been a committer since February 2017. He has been very active on
> Flink's Table API / SQL component, as well as frequently helping
> manage/verify/vote releases. He has been writing many blogs about Flink,
> also driving the translation work of Flink website and documentation. He is
> very active in China community as he gives talks about Flink at many events
> in China.
>
> Congratulations & Welcome Jark!
>
> Best,
> Jincheng (on behalf of the Flink PMC)
>