[jira] [Created] (FLINK-6978) Add documentation for Register User-Defined Functions.

[sunjincheng (JIRA)]

sunjincheng created FLINK-6978:
--

 Summary: Add documentation for Register User-Defined Functions.
 Key: FLINK-6978
 URL: https://issues.apache.org/jira/browse/FLINK-6978
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng


The User-defined Functions documentation is currently lacking a description of 
Register User-Defined Functions.
The page has a placeholder section with a TODO: ./docs/dev/table/udfs.md.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[hzyuemeng1]

Congratulations Shaoxuan!

2017-06-22 

hzyuemeng1 



发件人：Jark Wu 
发送时间：2017-06-22 11:21
主题：Re: [ANNOUNCE] New Flink committer Shaoxuan Wang
收件人："dev"
抄送：

Congratulations Shaoxuan! 

Regards, 
Jark 

2017-06-22 11:09 GMT+08:00 JingsongLee : 

> Congrats! 
> Best, Jingsong Lee 
> --From:Andrew 
> Psaltis Time:2017 Jun 22 (Thu) 11:06To:dev < 
> dev@flink.apache.org>Subject:Re: [ANNOUNCE] New Flink committer Shaoxuan 
> Wang 
> Congrats Shaoxuan! 
> 
> On Wed, Jun 21, 2017 at 10:28 PM, Zhuoluo Yang  alibaba-inc.com> 
> wrote: 
> 
> > Congrats! 
> > 
> > Thanks, 
> > 
> > Zhuoluo  
> > 
> > 
> > 
> > 
> > 
> > 在 2017年6月22日，上午10:14，Haohui Mai  写道： 
> > 
> > Congrats! 
> > On Thu, Jun 22, 2017 at 9:58 AM SHI Xiaogang  
> > wrote: 
> > 
> > Congrats, Shaoxuan 
> > 
> > Regards, 
> > Xiaogang 
> > 
> > 2017-06-22 9:08 GMT+08:00 jincheng sun : 
> > 
> > Congratulations Shaoxuan. 
> > 
> > 
> > 2017-06-22 8:56 GMT+08:00 Zhangrucong : 
> > 
> > Congrats Shaoxuan! 
> > 
> > -邮件原件- 
> > 发件人: Fabian Hueske [mailto:fhue...@gmail.com ] 
> > 发送时间: 2017年6月22日 4:19 
> > 收件人: dev@flink.apache.org 
> > 主题: [ANNOUNCE] New Flink committer Shaoxuan Wang 
> > 
> > Hi everybody, 
> > 
> > On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has 
> > accepted the invitation of the PMC to become a Flink committer. 
> > 
> > Shaoxuan has contributed several major features to the Table API / SQL 
> > 
> > and 
> > 
> > is very engaged in discussions about the design of new features and the 
> > future direction of Flink's relational APIs. 
> > 
> > Please join in me congratulating Shaoxuan for becoming a Flink 
> > 
> > committer. 
> > 
> > 
> > Thanks, Fabian 
> > 
> > 
> > 
> > 
> > 
> 
> 
> -- 
> Thanks, 
> Andrew 
> 
> Subscribe to my book: Streaming Data  
>  
> twiiter: @itmdata  
> 
> 

Re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[Jark Wu]

Congratulations Shaoxuan!

Regards,
Jark

2017-06-22 11:09 GMT+08:00 JingsongLee :

> Congrats!
> Best, Jingsong Lee
> --From:Andrew
> Psaltis Time:2017 Jun 22 (Thu) 11:06To:dev <
> dev@flink.apache.org>Subject:Re: [ANNOUNCE] New Flink committer Shaoxuan
> Wang
> Congrats Shaoxuan!
>
> On Wed, Jun 21, 2017 at 10:28 PM, Zhuoluo Yang  alibaba-inc.com>
> wrote:
>
> > Congrats!
> >
> > Thanks,
> >
> > Zhuoluo 
> >
> >
> >
> >
> >
> > 在 2017年6月22日，上午10:14，Haohui Mai  写道：
> >
> > Congrats!
> > On Thu, Jun 22, 2017 at 9:58 AM SHI Xiaogang 
> > wrote:
> >
> > Congrats, Shaoxuan
> >
> > Regards,
> > Xiaogang
> >
> > 2017-06-22 9:08 GMT+08:00 jincheng sun :
> >
> > Congratulations Shaoxuan.
> >
> >
> > 2017-06-22 8:56 GMT+08:00 Zhangrucong :
> >
> > Congrats Shaoxuan!
> >
> > -邮件原件-
> > 发件人: Fabian Hueske [mailto:fhue...@gmail.com ]
> > 发送时间: 2017年6月22日 4:19
> > 收件人: dev@flink.apache.org
> > 主题: [ANNOUNCE] New Flink committer Shaoxuan Wang
> >
> > Hi everybody,
> >
> > On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has
> > accepted the invitation of the PMC to become a Flink committer.
> >
> > Shaoxuan has contributed several major features to the Table API / SQL
> >
> > and
> >
> > is very engaged in discussions about the design of new features and the
> > future direction of Flink's relational APIs.
> >
> > Please join in me congratulating Shaoxuan for becoming a Flink
> >
> > committer.
> >
> >
> > Thanks, Fabian
> >
> >
> >
> >
> >
>
>
> --
> Thanks,
> Andrew
>
> Subscribe to my book: Streaming Data 
> 
> twiiter: @itmdata 
>
>

Re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[JingsongLee]

Congrats!
Best, Jingsong Lee
--From:Andrew 
Psaltis Time:2017 Jun 22 (Thu) 11:06To:dev 
Subject:Re: [ANNOUNCE] New Flink committer Shaoxuan Wang
Congrats Shaoxuan!

On Wed, Jun 21, 2017 at 10:28 PM, Zhuoluo Yang 
wrote:

> Congrats!
>
> Thanks,
>
> Zhuoluo 
>
>
>
>
>
> 在 2017年6月22日，上午10:14，Haohui Mai  写道：
>
> Congrats!
> On Thu, Jun 22, 2017 at 9:58 AM SHI Xiaogang 
> wrote:
>
> Congrats, Shaoxuan
>
> Regards,
> Xiaogang
>
> 2017-06-22 9:08 GMT+08:00 jincheng sun :
>
> Congratulations Shaoxuan.
>
>
> 2017-06-22 8:56 GMT+08:00 Zhangrucong :
>
> Congrats Shaoxuan!
>
> -邮件原件-
> 发件人: Fabian Hueske [mailto:fhue...@gmail.com ]
> 发送时间: 2017年6月22日 4:19
> 收件人: dev@flink.apache.org
> 主题: [ANNOUNCE] New Flink committer Shaoxuan Wang
>
> Hi everybody,
>
> On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has
> accepted the invitation of the PMC to become a Flink committer.
>
> Shaoxuan has contributed several major features to the Table API / SQL
>
> and
>
> is very engaged in discussions about the design of new features and the
> future direction of Flink's relational APIs.
>
> Please join in me congratulating Shaoxuan for becoming a Flink
>
> committer.
>
>
> Thanks, Fabian
>
>
>
>
>


-- 
Thanks,
Andrew

Subscribe to my book: Streaming Data 

twiiter: @itmdata 

Re: [ANNOUNCE] New committer: Dawid Wysakowicz

[Andrew Psaltis]

 Congratulations Dawid!

On Wed, Jun 21, 2017 at 10:15 PM, Haohui Mai  wrote:

> Congrats!
> On Thu, Jun 22, 2017 at 5:51 AM Stavros Kontopoulos <
> st.kontopou...@gmail.com> wrote:
>
> > Congratulations Dawid!
> >
> > On Tue, Jun 20, 2017 at 12:06 PM, Vasudevan, Ramkrishna S <
> > ramkrishna.s.vasude...@intel.com> wrote:
> >
> > > Congratulations !!
> > >
> > > -Original Message-
> > > From: Henry Saputra [mailto:henry.sapu...@gmail.com]
> > > Sent: Tuesday, June 20, 2017 2:19 PM
> > > To: dev@flink.apache.org
> > > Subject: Re: [ANNOUNCE] New committer: Dawid Wysakowicz
> > >
> > > Congrats and welcome! =)
> > >
> > > - Henry
> > >
> > > On Mon, Jun 19, 2017 at 6:55 PM, SHI Xiaogang 
> > > wrote:
> > >
> > > > Congrats  Dawid.
> > > > Great thanks for your contribution!
> > > >
> > > > Xiaogang
> > > >
> > > > 2017-06-19 18:52 GMT+08:00 Dawid Wysakowicz <
> > wysakowicz.da...@gmail.com
> > > >:
> > > >
> > > > > Thank you all for the warm welcome. I will do my best to be as
> > > > > helpful as possible.
> > > > >
> > > >
> > >
> >
>



-- 
Thanks,
Andrew

Subscribe to my book: Streaming Data 

twiiter: @itmdata 

[jira] [Created] (FLINK-6977) Add MD5/SHA1/SHA2 supported in TableAPI

[sunjincheng (JIRA)]

sunjincheng created FLINK-6977:
--

 Summary: Add MD5/SHA1/SHA2 supported in TableAPI
 Key: FLINK-6977
 URL: https://issues.apache.org/jira/browse/FLINK-6977
 Project: Flink
  Issue Type: Sub-task
  Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng


See FLINK-6895 for detail.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-6975) Add CONCAT/CONCAT_WS supported in TableAPI

[sunjincheng (JIRA)]

sunjincheng created FLINK-6975:
--

 Summary: Add CONCAT/CONCAT_WS supported in TableAPI
 Key: FLINK-6975
 URL: https://issues.apache.org/jira/browse/FLINK-6975
 Project: Flink
  Issue Type: Sub-task
  Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng


See FLINK-6925 for detail.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[Zhuoluo Yang]

Congrats!

Thanks,

Zhuoluo 





> 在 2017年6月22日，上午10:14，Haohui Mai  写道：
> 
> Congrats!
> On Thu, Jun 22, 2017 at 9:58 AM SHI Xiaogang  wrote:
> 
>> Congrats, Shaoxuan
>> 
>> Regards,
>> Xiaogang
>> 
>> 2017-06-22 9:08 GMT+08:00 jincheng sun :
>> 
>>> Congratulations Shaoxuan.
>>> 
>>> 
>>> 2017-06-22 8:56 GMT+08:00 Zhangrucong :
>>> 
 Congrats Shaoxuan!
 
 -邮件原件-
 发件人: Fabian Hueske [mailto:fhue...@gmail.com]
 发送时间: 2017年6月22日 4:19
 收件人: dev@flink.apache.org
 主题: [ANNOUNCE] New Flink committer Shaoxuan Wang
 
 Hi everybody,
 
 On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has
 accepted the invitation of the PMC to become a Flink committer.
 
 Shaoxuan has contributed several major features to the Table API / SQL
>>> and
 is very engaged in discussions about the design of new features and the
 future direction of Flink's relational APIs.
 
 Please join in me congratulating Shaoxuan for becoming a Flink
>> committer.
 
 Thanks, Fabian
 
>>> 
>> 



smime.p7s
Description: S/MIME cryptographic signature

[jira] [Created] (FLINK-6974) Add BIN supported in TableAPI

[sunjincheng (JIRA)]

sunjincheng created FLINK-6974:
--

 Summary: Add BIN supported in TableAPI
 Key: FLINK-6974
 URL: https://issues.apache.org/jira/browse/FLINK-6974
 Project: Flink
  Issue Type: Sub-task
  Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng


See FLINK-6893 for detail.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-6973) Add L/RPAD supported in TableAPI

[sunjincheng (JIRA)]

sunjincheng created FLINK-6973:
--

 Summary: Add L/RPAD supported in TableAPI
 Key: FLINK-6973
 URL: https://issues.apache.org/jira/browse/FLINK-6973
 Project: Flink
  Issue Type: Sub-task
  Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng


See FLINK-6892 for detail.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-6972) Flink REPL api

[Praveen Kanamarlapudi (JIRA)]

Praveen Kanamarlapudi created FLINK-6972:


 Summary: Flink REPL api
 Key: FLINK-6972
 URL: https://issues.apache.org/jira/browse/FLINK-6972
 Project: Flink
  Issue Type: Improvement
  Components: Client
Reporter: Praveen Kanamarlapudi


Can you please develop FlinkIMap (Similar to 
[SparkIMain|https://github.com/apache/spark/blob/master/repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkIMain.scala])
 developer api for creating interactive sessions.

I am thinking to add flink support to [livy|https://github.com/cloudera/livy/]. 
For enabling flink interactive sessions it would be really helpful.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[Haohui Mai]

Congrats!
On Thu, Jun 22, 2017 at 9:58 AM SHI Xiaogang  wrote:

> Congrats, Shaoxuan
>
> Regards,
> Xiaogang
>
> 2017-06-22 9:08 GMT+08:00 jincheng sun :
>
> > Congratulations Shaoxuan.
> >
> >
> > 2017-06-22 8:56 GMT+08:00 Zhangrucong :
> >
> > > Congrats Shaoxuan!
> > >
> > > -邮件原件-
> > > 发件人: Fabian Hueske [mailto:fhue...@gmail.com]
> > > 发送时间: 2017年6月22日 4:19
> > > 收件人: dev@flink.apache.org
> > > 主题: [ANNOUNCE] New Flink committer Shaoxuan Wang
> > >
> > > Hi everybody,
> > >
> > > On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has
> > > accepted the invitation of the PMC to become a Flink committer.
> > >
> > > Shaoxuan has contributed several major features to the Table API / SQL
> > and
> > > is very engaged in discussions about the design of new features and the
> > > future direction of Flink's relational APIs.
> > >
> > > Please join in me congratulating Shaoxuan for becoming a Flink
> committer.
> > >
> > > Thanks, Fabian
> > >
> >
>

Re: [ANNOUNCE] New committer: Dawid Wysakowicz

[Haohui Mai]

Congrats!
On Thu, Jun 22, 2017 at 5:51 AM Stavros Kontopoulos <
st.kontopou...@gmail.com> wrote:

> Congratulations Dawid!
>
> On Tue, Jun 20, 2017 at 12:06 PM, Vasudevan, Ramkrishna S <
> ramkrishna.s.vasude...@intel.com> wrote:
>
> > Congratulations !!
> >
> > -Original Message-
> > From: Henry Saputra [mailto:henry.sapu...@gmail.com]
> > Sent: Tuesday, June 20, 2017 2:19 PM
> > To: dev@flink.apache.org
> > Subject: Re: [ANNOUNCE] New committer: Dawid Wysakowicz
> >
> > Congrats and welcome! =)
> >
> > - Henry
> >
> > On Mon, Jun 19, 2017 at 6:55 PM, SHI Xiaogang 
> > wrote:
> >
> > > Congrats  Dawid.
> > > Great thanks for your contribution!
> > >
> > > Xiaogang
> > >
> > > 2017-06-19 18:52 GMT+08:00 Dawid Wysakowicz <
> wysakowicz.da...@gmail.com
> > >:
> > >
> > > > Thank you all for the warm welcome. I will do my best to be as
> > > > helpful as possible.
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[SHI Xiaogang]

Congrats, Shaoxuan

Regards,
Xiaogang

2017-06-22 9:08 GMT+08:00 jincheng sun :

> Congratulations Shaoxuan.
>
>
> 2017-06-22 8:56 GMT+08:00 Zhangrucong :
>
> > Congrats Shaoxuan!
> >
> > -邮件原件-
> > 发件人: Fabian Hueske [mailto:fhue...@gmail.com]
> > 发送时间: 2017年6月22日 4:19
> > 收件人: dev@flink.apache.org
> > 主题: [ANNOUNCE] New Flink committer Shaoxuan Wang
> >
> > Hi everybody,
> >
> > On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has
> > accepted the invitation of the PMC to become a Flink committer.
> >
> > Shaoxuan has contributed several major features to the Table API / SQL
> and
> > is very engaged in discussions about the design of new features and the
> > future direction of Flink's relational APIs.
> >
> > Please join in me congratulating Shaoxuan for becoming a Flink committer.
> >
> > Thanks, Fabian
> >
>

[jira] [Created] (FLINK-6971) Add Alluxio Filesystem in Flink Ecosystem page

[Bin Fan (JIRA)]

Bin Fan created FLINK-6971:
--

 Summary: Add Alluxio Filesystem in Flink Ecosystem page
 Key: FLINK-6971
 URL: https://issues.apache.org/jira/browse/FLINK-6971
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Bin Fan
Priority: Minor


Flink Ecosystem page (http://flink.apache.org/ecosystem.html) lists a set of 
third-party projects
that supports working with Flink.

Alluxio (www.alluxio.org) can work with Flink as a Hadoop-compatible file 
system, see more description in 
http://www.alluxio.org/docs/master/en/Running-Flink-on-Alluxio.html. I am 
wondering if I could submit patch to add a paragraph of Alluxio under 
http://flink.apache.org/ecosystem.html#third-party-projects and points users 
the Alluxio-flink integration page?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[jincheng sun]

Congratulations Shaoxuan.


2017-06-22 8:56 GMT+08:00 Zhangrucong :

> Congrats Shaoxuan!
>
> -邮件原件-
> 发件人: Fabian Hueske [mailto:fhue...@gmail.com]
> 发送时间: 2017年6月22日 4:19
> 收件人: dev@flink.apache.org
> 主题: [ANNOUNCE] New Flink committer Shaoxuan Wang
>
> Hi everybody,
>
> On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has
> accepted the invitation of the PMC to become a Flink committer.
>
> Shaoxuan has contributed several major features to the Table API / SQL and
> is very engaged in discussions about the design of new features and the
> future direction of Flink's relational APIs.
>
> Please join in me congratulating Shaoxuan for becoming a Flink committer.
>
> Thanks, Fabian
>

re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[Zhangrucong]

Congrats Shaoxuan!

-邮件原件-
发件人: Fabian Hueske [mailto:fhue...@gmail.com] 
发送时间: 2017年6月22日 4:19
收件人: dev@flink.apache.org
主题: [ANNOUNCE] New Flink committer Shaoxuan Wang

Hi everybody,

On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has 
accepted the invitation of the PMC to become a Flink committer.

Shaoxuan has contributed several major features to the Table API / SQL and is 
very engaged in discussions about the design of new features and the future 
direction of Flink's relational APIs.

Please join in me congratulating Shaoxuan for becoming a Flink committer.

Thanks, Fabian

Re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[Neelesh Salian]

Congratulations Shaoxuan

On Jun 21, 2017 1:19 PM, "Fabian Hueske"  wrote:

> Hi everybody,
>
> On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has
> accepted the invitation of the PMC to become a Flink committer.
>
> Shaoxuan has contributed several major features to the Table API / SQL and
> is very engaged in discussions about the design of new features and the
> future direction of Flink's relational APIs.
>
> Please join in me congratulating Shaoxuan for becoming a Flink committer.
>
> Thanks, Fabian
>

Re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[Kostas Kloudas]

Congratulations Shaoxuan!
More hands are always welcomed!

Kostas

On Jun 21, 2017 22:45, "Greg Hogan"  wrote:

> Congrats and welcome, Shaoxuan!
>
>
> > On Jun 21, 2017, at 4:19 PM, Fabian Hueske  wrote:
> >
> > Hi everybody,
> >
> > On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has
> > accepted the invitation of the PMC to become a Flink committer.
> >
> > Shaoxuan has contributed several major features to the Table API / SQL
> and
> > is very engaged in discussions about the design of new features and the
> > future direction of Flink's relational APIs.
> >
> > Please join in me congratulating Shaoxuan for becoming a Flink committer.
> >
> > Thanks, Fabian
>
>

Re: [DISCUSS] Table API / SQL features for Flink 1.4.0

[Fabian Hueske]

Hi Haohui,

thanks for your input!

Can you describe the semantics of the join you'd like to see in Flink 1.4?
I can think of three types of joins that match your description:
1) `table` is an external table stored in an external database (redis,
cassandra, MySQL, etc) and we join with the current value that is in that
table. This could be implemented with an async TableFunction (based on
Flink's AsyncFunction).
2) `table` is static: In this case we need support for side-inputs to read
the whole table before starting to process the other (streaming) side.
There is a FLIP [1] for side inputs. I don't know what's the status of this
feature though.
3) `table` changing and each record of the stream should be joined with the
most recent update (but no future updates). In this case, the query is more
complex to express and requires some time-bound logic which is quite
cumbersome to express in SQL. I think this is a very important type of
join, but IMO it is more challenging to implement than the other joins. We
had also a discussion about this type of join on the dev ML a few months
back [2].

Which type of join are you looking for (external table, static table,
dynamic table)?

Regarding the bottleneck of committers, the situation should become a bit
better in the near future as we have two committer more working on the
relational APIs (Jark is spending more time here and Shaoxuan recently
became a committer). However, we will of course continue to encourage and
help contributors to earn the merits to become committers and grow the
number of committers.

Thank you very much,
Fabian

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/STREAM-SQL-inner-queries-tp15585.html

[jira] [Created] (FLINK-6969) Add support for deferred computation for group window aggregates

[Fabian Hueske (JIRA)]

Fabian Hueske created FLINK-6969:


 Summary: Add support for deferred computation for group window 
aggregates
 Key: FLINK-6969
 URL: https://issues.apache.org/jira/browse/FLINK-6969
 Project: Flink
  Issue Type: New Feature
  Components: Table API & SQL
Reporter: Fabian Hueske


Deferred computation is a strategy to deal with late arriving data and avoid 
updates of previous results. Instead of computing a result as soon as it is 
possible (i.e., when a corresponding watermark was received), deferred 
computation adds a configurable amount of slack time in which late data is 
accepted before the result is compute. For example, instead of computing a 
tumbling window of 1 hour at each full hour, we can add a deferred computation 
interval of 15 minute to compute the result quarter past each full hour.

This approach adds latency but can reduce the number of update esp. in use 
cases where the user cannot influence the generation of watermarks. It is also 
useful if the data is emitted to a system that cannot update result (files or 
Kafka). The deferred computation interval should be configured via the 
{{QueryConfig}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-6968) Store streaming, updating tables with unique key in queryable state

[Fabian Hueske (JIRA)]

Fabian Hueske created FLINK-6968:


 Summary: Store streaming, updating tables with unique key in 
queryable state
 Key: FLINK-6968
 URL: https://issues.apache.org/jira/browse/FLINK-6968
 Project: Flink
  Issue Type: New Feature
  Components: Table API & SQL
Reporter: Fabian Hueske


Streaming tables with unique key are continuously updated. For example queries 
with a non-windowed aggregation generate such tables. Commonly, such updating 
tables are emitted via an upsert table sink to an external datastore (k-v 
store, database) to make it accessible to applications.

This issue is about adding a feature to store and maintain such a table as 
queryable state in Flink. By storing the table in Flnk's queryable state, we do 
not need an external data store to access the results of the query but can 
query the results directly from Flink.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[Greg Hogan]

Congrats and welcome, Shaoxuan!


> On Jun 21, 2017, at 4:19 PM, Fabian Hueske  wrote:
> 
> Hi everybody,
> 
> On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has
> accepted the invitation of the PMC to become a Flink committer.
> 
> Shaoxuan has contributed several major features to the Table API / SQL and
> is very engaged in discussions about the design of new features and the
> future direction of Flink's relational APIs.
> 
> Please join in me congratulating Shaoxuan for becoming a Flink committer.
> 
> Thanks, Fabian

Re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[Dawid Wysakowicz]

Congrats Shaoxuan!

Z pozdrowieniami! / Cheers!

Dawid Wysakowicz

*Data/Software Engineer*

Skype: dawid_wys | Twitter: @OneMoreCoder



2017-06-21 22:22 GMT+02:00 Ted Yu :

> Congratulations, Shaoxuan
>
> On Wed, Jun 21, 2017 at 1:19 PM, Fabian Hueske  wrote:
>
> > Hi everybody,
> >
> > On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has
> > accepted the invitation of the PMC to become a Flink committer.
> >
> > Shaoxuan has contributed several major features to the Table API / SQL
> and
> > is very engaged in discussions about the design of new features and the
> > future direction of Flink's relational APIs.
> >
> > Please join in me congratulating Shaoxuan for becoming a Flink committer.
> >
> > Thanks, Fabian
> >
>

Re: [ANNOUNCE] New Flink committer Shaoxuan Wang

[Ted Yu]

Congratulations, Shaoxuan

On Wed, Jun 21, 2017 at 1:19 PM, Fabian Hueske  wrote:

> Hi everybody,
>
> On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has
> accepted the invitation of the PMC to become a Flink committer.
>
> Shaoxuan has contributed several major features to the Table API / SQL and
> is very engaged in discussions about the design of new features and the
> future direction of Flink's relational APIs.
>
> Please join in me congratulating Shaoxuan for becoming a Flink committer.
>
> Thanks, Fabian
>

[ANNOUNCE] New Flink committer Shaoxuan Wang

[Fabian Hueske]

Hi everybody,

On behalf of the PMC, I'm very happy to announce that Shaoxuan Wang has
accepted the invitation of the PMC to become a Flink committer.

Shaoxuan has contributed several major features to the Table API / SQL and
is very engaged in discussions about the design of new features and the
future direction of Flink's relational APIs.

Please join in me congratulating Shaoxuan for becoming a Flink committer.

Thanks, Fabian

Re: [DISCUSS] GitBox

[Robert Metzger]

+1 for trying out Gitbox!

On Sun, Jun 18, 2017 at 6:50 PM, Greg Hogan  wrote:

> My understanding is that with GitBox project committers who have linked
> Apache and GitHub accounts are given organization write permissions. Other
> contributors will continue to have read permissions.
>   https://help.github.com/articles/repository-permission-levels-for-an-
> organization/
>
> The last comment noting the “split-brain” shouldn’t preclude the use of
> GitBox but we should come to a general consensus before switching to commit
> into the GitHub repo.
>
> If we want to try GitHub for flink-web, a second step could to switch and
> use with the nascent flink-libraries.
>
>
> > On Jun 18, 2017, at 6:50 AM, Chesnay Schepler 
> wrote:
> >
> > Found some info in this JIRA: https://issues.apache.org/
> jira/browse/INFRA-14191
> >
> > Apparently, Gitbox is still in the beta phase. There are no public docs
> for it yet.
> >
> > Committers are required to link their apache & GitHub accounts, which
> requires 2FA on GitHub.
> >
> > As it stands I would be in favor of Gregs original suggestion of
> activating it for flink-web as a test bed.
> > I would wait with the main repo until we actually have more info and it
> is a bit more proven.
> >
> > On 11.06.2017 19:37, Ufuk Celebi wrote:
> >> I would also like to see this happening for both flink-web and flink
> >> if it allows committers to have control over the respective repos.
> >>
> >> On Sat, Jun 10, 2017 at 4:05 PM, Chesnay Schepler 
> wrote:
> >>> What are the downsides of this? Actually, is there any ASF resource
> that
> >>> outlines what this would enable?
> >>>
> >>> In one of the threads i saw said that this would also allow committers
> to
> >>> close PR's, assign labels and such.
> >>> This sounds very interesting to me for the main repo actually.
> >>>
> >>>
> >>> On 09.06.2017 17:41, Greg Hogan wrote:
>  Robert has an open PR from March. I’ve found, for example, PRs adding
>  links to talks or slides left open for months.
> 
>  I’d suggest Fluo is to Accumulo as flink-web is to the flink repo, and
>  that migration looks to be satisfactory.
> 
> 
> > On Jun 9, 2017, at 11:15 AM, Ted Yu  wrote:
> >
> > bq. better track the oft-neglected contributions
> >
> > Do you have estimate on how many contributions were not paid
> attention in
> > the current infrastructure.
> >
> > Looking at #2, it seems Accumulo community hasn't reached consensus
> yet.
> >
> > Cheers
> >
> > On Fri, Jun 9, 2017 at 7:54 AM, Greg Hogan 
> wrote:
> >
> >> All,
> >>
> >> ASF now has available (and maybe mandatory for new projects or
> repos)
> >> GitBox [0] which enables bi-directional sync to GitHub and links
> >> committers' accounts, allowing for greater use of GitHub
> functionality
> >> by
> >> contributors and for committers to perform many tasks otherwise
> >> requiring
> >> INFRA tickets.
> >>
> >> I'd like to propose moving flink-web [1] to GitBox, using GitHub
> issues,
> >> and enabling notifications to the mailing lists. Apache Accumulo has
> >> recently discussed [2] this topic with a list of benefits after
> >> migrating
> >> Fluo. By migrating flink-web we can better track the oft-neglected
> >> contributions and also test the waters for future migrations
> (perhaps
> >> for
> >> the future sub-projects).
> >>
> >> [0] https://gitbox.apache.org/
> >> [1] https://github.com/apache/flink-web/pulls
> >> [2]
> >> http://apache-accumulo.1065345.n5.nabble.com/DISCUSS-
> >> GitBox-tp21160p21497.html
> >>
> >> Greg
> >>>
> >>>
> >
>
>

Re: [VOTE] Release Apache Flink 1.3.1 (RC2)

[Ted Yu]

+1

Ran unit test suite which passed.

On Wed, Jun 21, 2017 at 8:27 AM, Robert Metzger  wrote:

> +1 (binding)
>
> - Checked staging repository: ES5 connector is present
> - quickstart-java points to 1.3.1
> - yarn session starts
> - per-job YARN submission works / log files look good.
> - Executed all tests listed here [1] without any issues / log files looked
> good afterwards
> - The source doesn't contain any binaries
> - checked (some) md5sums for correctness
>
>
> [1] https://github.com/aljoscha/FliRTT/blob/master/test-flink.sh#L36
>
>
>
>
> On Wed, Jun 21, 2017 at 5:06 PM, Robert Metzger 
> wrote:
>
> > I see that this is an issue we should address, but on the other hand,
> > there's a large list of issues we need to address with 1.3.1 already.
> > If we keep delaying minor releases for every issue we find, we'll never
> > release it :)
> >
> >
> >
> > On Wed, Jun 21, 2017 at 3:51 PM, Stefan Richter <
> > s.rich...@data-artisans.com> wrote:
> >
> >> Hi,
> >>
> >> I suggest to include https://issues.apache.org/jira/browse/FLINK-6964 <
> >> https://issues.apache.org/jira/browse/FLINK-6964> to the release.
> >>
> >> Best,
> >> Stefan
> >>
> >> > Am 21.06.2017 um 09:46 schrieb Robert Metzger :
> >> >
> >> > Dear Flink community,
> >> >
> >> > Please vote on releasing the following candidate as Apache Flink
> version
> >> > 1.3.1.
> >> >
> >> > The commit to be voted on:
> >> > *http://git-wip-us.apache.org/repos/asf/flink/commit/1ca6e5b6
> >> > *
> >> >
> >> > Branch:
> >> > release-1.3.1-rc2
> >> >
> >> > The release artifacts to be voted on can be found at:
> >> > *http://people.apache.org/~rmetzger/flink-1.3.1-rc2/
> >> > *
> >> >
> >> > The release artifacts are signed with the key with fingerprint
> D9839159:
> >> > http://www.apache.org/dist/flink/KEYS
> >> >
> >> > The staging repository for this release can be found at:
> >> > *https://repository.apache.org/content/repositories/
> orgapacheflink-1125
> >> >  orgapacheflink-1125
> >> >*
> >> >
> >> >
> >> > -
> >> >
> >> >
> >> > The vote ends on Thursday (5pm CEST), June 22, 2016.
> >> > IMPORTANT: I've reduced the voting time to only one day because the
> >> number
> >> > of changes between RC1 and RC2 are mostly in the table API (mostly
> >> > documentation) and the serializer changes Till and Gordon were working
> >> on.
> >> > The list of changes is the following
> >> > - Reworked Table API documentation (this is a set of commits)
> >> > - [FLINK-6817] [table] Add OverWindowWithPreceding class to guide
> users
> >> > - [FLINK-6859] [table] Do not delete timers in
> StateCleaningCountTrigger
> >> > - [FLINK-6930] [table] Forbid selecting window start/end on row-based
> T…
> >> > - [FLINK-6886] [table] Fix conversion of Row Table to POJO
> >> > - [FLINK-6602] [table] Prevent TableSources with empty time attribute
> n…
> >> > - [FLINK-6941] [table] Validate that start and end window properties
> ar…
> >> > - [FLINK-6881] [FLINK-6896] [table] Creating a table from a POJO and
> de…
> >> > - [FLINK-6921] [serializer] Allow EnumValueSerializer to deal with
> appe…
> >> > - [FLINK-6948] [serializer] Harden EnumValueSerializer to detect
> change…
> >> > - [FLINK-6922] [serializer] Remove Java serialization from
> Enum(Value)S…
> >> > - [FLINK-6652] [core] Fix handling of delimiters split by buffers in
> De…
> >> >
> >> >
> >> >
> >> > [ ] +1 Release this package as Apache Flink 1.3.1
> >> > [ ] -1 Do not release this package, because ...
> >>
> >>
> >
>

Re: [VOTE] Release Apache Flink 1.3.1 (RC2)

[Robert Metzger]

+1 (binding)

- Checked staging repository: ES5 connector is present
- quickstart-java points to 1.3.1
- yarn session starts
- per-job YARN submission works / log files look good.
- Executed all tests listed here [1] without any issues / log files looked
good afterwards
- The source doesn't contain any binaries
- checked (some) md5sums for correctness


[1] https://github.com/aljoscha/FliRTT/blob/master/test-flink.sh#L36




On Wed, Jun 21, 2017 at 5:06 PM, Robert Metzger  wrote:

> I see that this is an issue we should address, but on the other hand,
> there's a large list of issues we need to address with 1.3.1 already.
> If we keep delaying minor releases for every issue we find, we'll never
> release it :)
>
>
>
> On Wed, Jun 21, 2017 at 3:51 PM, Stefan Richter <
> s.rich...@data-artisans.com> wrote:
>
>> Hi,
>>
>> I suggest to include https://issues.apache.org/jira/browse/FLINK-6964 <
>> https://issues.apache.org/jira/browse/FLINK-6964> to the release.
>>
>> Best,
>> Stefan
>>
>> > Am 21.06.2017 um 09:46 schrieb Robert Metzger :
>> >
>> > Dear Flink community,
>> >
>> > Please vote on releasing the following candidate as Apache Flink version
>> > 1.3.1.
>> >
>> > The commit to be voted on:
>> > *http://git-wip-us.apache.org/repos/asf/flink/commit/1ca6e5b6
>> > *
>> >
>> > Branch:
>> > release-1.3.1-rc2
>> >
>> > The release artifacts to be voted on can be found at:
>> > *http://people.apache.org/~rmetzger/flink-1.3.1-rc2/
>> > *
>> >
>> > The release artifacts are signed with the key with fingerprint D9839159:
>> > http://www.apache.org/dist/flink/KEYS
>> >
>> > The staging repository for this release can be found at:
>> > *https://repository.apache.org/content/repositories/orgapacheflink-1125
>> > > >*
>> >
>> >
>> > -
>> >
>> >
>> > The vote ends on Thursday (5pm CEST), June 22, 2016.
>> > IMPORTANT: I've reduced the voting time to only one day because the
>> number
>> > of changes between RC1 and RC2 are mostly in the table API (mostly
>> > documentation) and the serializer changes Till and Gordon were working
>> on.
>> > The list of changes is the following
>> > - Reworked Table API documentation (this is a set of commits)
>> > - [FLINK-6817] [table] Add OverWindowWithPreceding class to guide users
>> > - [FLINK-6859] [table] Do not delete timers in StateCleaningCountTrigger
>> > - [FLINK-6930] [table] Forbid selecting window start/end on row-based T…
>> > - [FLINK-6886] [table] Fix conversion of Row Table to POJO
>> > - [FLINK-6602] [table] Prevent TableSources with empty time attribute n…
>> > - [FLINK-6941] [table] Validate that start and end window properties ar…
>> > - [FLINK-6881] [FLINK-6896] [table] Creating a table from a POJO and de…
>> > - [FLINK-6921] [serializer] Allow EnumValueSerializer to deal with appe…
>> > - [FLINK-6948] [serializer] Harden EnumValueSerializer to detect change…
>> > - [FLINK-6922] [serializer] Remove Java serialization from Enum(Value)S…
>> > - [FLINK-6652] [core] Fix handling of delimiters split by buffers in De…
>> >
>> >
>> >
>> > [ ] +1 Release this package as Apache Flink 1.3.1
>> > [ ] -1 Do not release this package, because ...
>>
>>
>

Re: [VOTE] Release Apache Flink 1.3.1 (RC2)

[Chesnay Schepler]

+1 to release 1.3.1 as is.

On 21.06.2017 17:06, Robert Metzger wrote:

I see that this is an issue we should address, but on the other hand,
there's a large list of issues we need to address with 1.3.1 already.
If we keep delaying minor releases for every issue we find, we'll never
release it :)



On Wed, Jun 21, 2017 at 3:51 PM, Stefan Richter  to the release.

Best,
Stefan


Am 21.06.2017 um 09:46 schrieb Robert Metzger :

Dear Flink community,

Please vote on releasing the following candidate as Apache Flink version
1.3.1.

The commit to be voted on:
*http://git-wip-us.apache.org/repos/asf/flink/commit/1ca6e5b6
*

Branch:
release-1.3.1-rc2

The release artifacts to be voted on can be found at:
*http://people.apache.org/~rmetzger/flink-1.3.1-rc2/
*

The release artifacts are signed with the key with fingerprint D9839159:
http://www.apache.org/dist/flink/KEYS

The staging repository for this release can be found at:
*https://repository.apache.org/content/repositories/orgapacheflink-1125

Re: [VOTE] Release Apache Flink 1.3.1 (RC2)

[Robert Metzger]

I see that this is an issue we should address, but on the other hand,
there's a large list of issues we need to address with 1.3.1 already.
If we keep delaying minor releases for every issue we find, we'll never
release it :)



On Wed, Jun 21, 2017 at 3:51 PM, Stefan Richter  wrote:

> Hi,
>
> I suggest to include https://issues.apache.org/jira/browse/FLINK-6964 <
> https://issues.apache.org/jira/browse/FLINK-6964> to the release.
>
> Best,
> Stefan
>
> > Am 21.06.2017 um 09:46 schrieb Robert Metzger :
> >
> > Dear Flink community,
> >
> > Please vote on releasing the following candidate as Apache Flink version
> > 1.3.1.
> >
> > The commit to be voted on:
> > *http://git-wip-us.apache.org/repos/asf/flink/commit/1ca6e5b6
> > *
> >
> > Branch:
> > release-1.3.1-rc2
> >
> > The release artifacts to be voted on can be found at:
> > *http://people.apache.org/~rmetzger/flink-1.3.1-rc2/
> > *
> >
> > The release artifacts are signed with the key with fingerprint D9839159:
> > http://www.apache.org/dist/flink/KEYS
> >
> > The staging repository for this release can be found at:
> > *https://repository.apache.org/content/repositories/orgapacheflink-1125
> >  >*
> >
> >
> > -
> >
> >
> > The vote ends on Thursday (5pm CEST), June 22, 2016.
> > IMPORTANT: I've reduced the voting time to only one day because the
> number
> > of changes between RC1 and RC2 are mostly in the table API (mostly
> > documentation) and the serializer changes Till and Gordon were working
> on.
> > The list of changes is the following
> > - Reworked Table API documentation (this is a set of commits)
> > - [FLINK-6817] [table] Add OverWindowWithPreceding class to guide users
> > - [FLINK-6859] [table] Do not delete timers in StateCleaningCountTrigger
> > - [FLINK-6930] [table] Forbid selecting window start/end on row-based T…
> > - [FLINK-6886] [table] Fix conversion of Row Table to POJO
> > - [FLINK-6602] [table] Prevent TableSources with empty time attribute n…
> > - [FLINK-6941] [table] Validate that start and end window properties ar…
> > - [FLINK-6881] [FLINK-6896] [table] Creating a table from a POJO and de…
> > - [FLINK-6921] [serializer] Allow EnumValueSerializer to deal with appe…
> > - [FLINK-6948] [serializer] Harden EnumValueSerializer to detect change…
> > - [FLINK-6922] [serializer] Remove Java serialization from Enum(Value)S…
> > - [FLINK-6652] [core] Fix handling of delimiters split by buffers in De…
> >
> >
> >
> > [ ] +1 Release this package as Apache Flink 1.3.1
> > [ ] -1 Do not release this package, because ...
>
>

[jira] [Created] (FLINK-6967) Fully separate batch and storm examples

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-6967:
---

 Summary: Fully separate batch and storm examples
 Key: FLINK-6967
 URL: https://issues.apache.org/jira/browse/FLINK-6967
 Project: Flink
  Issue Type: Improvement
  Components: Examples, Storm Compatibility
Affects Versions: 1.4.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Priority: Minor
 Fix For: 1.4.0


Like the streaming examples (see FLINK-6863) the storm examples have a 
dependency on the batch examples, exclusively for the WordCount example data.

I propose to duplicate the test data again for the storm examples.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Changing Flink's shading model

[Robert Metzger]

Okay, I'll request a repo for the shading.

On Wed, Jun 21, 2017 at 1:38 PM, Chesnay Schepler 
wrote:

> I like your suggestion Robert. A lot actually.
>
> Having the actual dependency version (i.e 18 for guava) in the version
> should improve clarity a lot.
>
> Originally i intended to release 1 artifact per Flink version, with the
> normal versioning scheme
> that we use. But given that the shaded dependencies aren't changed often
> (even rarely might be a stretch),
> and aren't actually coupled to the Flink release cycle this doesn't make a
> lot of sense.
>
> Having separate repos looks like a reasonable separation of concerns. The
> release for Flink itself
> would work just like it does now; we don't have to modify any scripts or
> do extra steps.
>
> Since the build, release and development process are separate (since
> flink-shaded isn't part of Flink build
> process, has a separate release process and changes to it will /never
> /require immediate changes to Flink)
> it seems like a very good candidate to move it into a separate repo.
>
>
> On 21.06.2017 11:26, Robert Metzger wrote:
>
>> Its not completely clear to me how we want to version the shaded
>> dependencies, and where we are putting them.
>>
>> One concern are the official apache release rules. If we want to release
>> something to maven central, we need to do a proper vote over a source
>> archive.
>> I would propose to create a new repository "flink-shaded.git" that
>> contains
>> the following maven module structure:
>> - flink-shaded: 1
>> - flink-shaded-asm: 1-5.2
>> - flink-shaded-guava: 1-18.0
>> - ...
>>
>> The number indicates the version (for ASM, I've just guessed).
>> The version for the parent "flink-shaded" needs to be updated on each
>> parent pom change (new module added, new / changed plugins, ...)
>>
>> We could create a separate release script in this repository that creates
>> the flink-shaded-src.zip from the code and deploys the artifacts to the
>> maven staging area.
>>
>> The advantage of a separate repo would be that we don't need to maintain
>> separate maven projects in the same git repo.
>> Also, the src archives for the release vote can be created from the repo
>> content (without much filtering).
>>
>>
>> On Tue, Jun 20, 2017 at 9:44 PM, Stephan Ewen  wrote:
>>
>> I like this approach.
>>>
>>> Two additional things can be mention here:
>>>
>>>- We need to deploy these artifacts independently and not as part of
>>> the
>>> build. That is a manual step once per "bump" in the dependency of that
>>> library.
>>>
>>>- We reduce the shading complexity of the original build and should
>>> thus
>>> also speed up build times :-)
>>>
>>> Stephan
>>>
>>>
>>>
>>>
>>> On Tue, Jun 20, 2017 at 1:15 PM, Chesnay Schepler 
>>> wrote:
>>>
>>> I would like to start working on this.

 I've looked into adding a flink-shaded-guava module. Working against the
 shaded namespaces seems
 to work without problems from the IDE, and we could forbid un-shaded
 usages with checkstyle.

 So for the list of dependencies that we want to shade we currently got:

   * asm
   * guava
   * netty
   * hadoop
   * curator

 I've had a chat with Stephan Ewan and he brought up kryo + chill as
 well.

 The nice thing is that we can do this incrementally, one dependency at a
 time. As such i would propose
 to go through the whole process for guava and see what problems arise.

 This would include adding a flink-shaded module and a child
 flink-shaded-guava module to the flink repository
 that are not part of the build process, replacing all usages of guava in
 Flink, adding the
 checkstyle rule (optional) and deploying the artifact to maven central.


 On 11.05.2017 10:54, Stephan Ewen wrote:

 @Ufuk  - I have never set up artifact deployment in Maven, could need
>
 some
>>>
 help there.
>
> Regarding shading Netty, I agree, would be good to do that as well...
>
> On Thu, May 11, 2017 at 10:52 AM, Ufuk Celebi  wrote:
>
> The advantages you've listed sound really compelling to me.
>
>> - Do you have time to implement these changes or do we need a
>>
> volunteer?
>>>
 ;)
>>
>> - I assume that republishing the artifacts as you propose doesn't have
>> any new legal implications since we already publish them with our
>> JARs, right?
>>
>> - We might think about adding Netty to the list of shaded artifacts
>> since some dependency conflicts were reported recently. Would have to
>> double check the reported issues before doing that though. ;-)
>>
>> – Ufuk
>>
>>
>> On Wed, May 10, 2017 at 8:45 PM, Stephan Ewen 
>>
> wrote:
>>>
 @chesnay: I used ASM as an example in the proposal. Maybe I did not
>>>

Re: [DISCUSS] Table API / SQL features for Flink 1.4.0

[Haohui Mai]

Hi,

We are interested in building the simplest case of stream-table joins --
essentially calling stream.map(x => (x, table.get(x)). It solves the use
cases of augmenting the streams with the information of the database. The
operation itself can be batched for better performance.

We are happy to contribute to the the scalar functions as well as we
internally also share similar requirements.

Fabian mentioned that the development of Table / SQL API was bottlenecked
by committers, which shows that there are thriving developments happening
in the space. I think it is a good problem to have. :-)

I wonder, is it a good time to nominate new batches of committers and to
keep the momentum of developments?

Regards,
Haohui



On Fri, Jun 16, 2017 at 7:28 AM jincheng sun 
wrote:

> Hi Fabian,
> Thanks for bring up this discuss.
> In order to enrich Flink's built-in scalar function, friendly user
> experience, I recommend adding as much scalar functions as possible in
> version 1.4 release. I have filed the JIRAs(
> https://issues.apache.org/jira/browse/FLINK-6810), and try my best to work
> on them.
>
> Of course, welcome anybody to add sub-tasks or take the JIRAs.
>
> Cheers,
> SunJincheng
>
> 2017-06-16 16:07 GMT+08:00 Fabian Hueske :
>
> > Thanks for your response Shaoxuan,
> >
> > My "Table-table join with retraction" is probably the same as your
> > "unbounded stream-stream join with retraction".
> > Basically, a join between two dynamic tables with unique keys (either
> > because of an upsert stream->table conversion or an unbounded
> aggregation).
> >
> > Best, Fabian
> >
> > 2017-06-16 0:56 GMT+02:00 Shaoxuan Wang :
> >
> > > Nice timing, Fabian!
> > >
> > > Your checklist aligns our plans very well. Here are the things we are
> > > working on & planning to contribute to release 1.4:
> > > 1. DDL (with property waterMark config for source-table, and emit
> config
> > on
> > > result-table)
> > > 2. unbounded stream-stream joins (with retraction supported)
> > > 3. backend state user interface for UDAGG
> > > 4. UDOP (as oppose to UDF(scalars to scalar)/UDTF(scalar to
> > > table)/UDAGG(table to scalar), this allows user to define a table to
> > table
> > > conversion business logic)
> > >
> > > Some of them already have PR/jira, while some are not. We will send out
> > the
> > > design doc for the missing ones very soon. Looking forward to the 1.4
> > > release.
> > >
> > > Btw, what is "Table-Table (with retraction)" you have mentioned in your
> > > plan?
> > >
> > > Regards,
> > > Shaoxuan
> > >
> > >
> > >
> > > On Thu, Jun 15, 2017 at 10:29 PM, Fabian Hueske 
> > wrote:
> > >
> > > > Hi everybody,
> > > >
> > > > I would like to start a discussion about the targeted feature set of
> > the
> > > > Table API / SQL for Flink 1.4.0.
> > > > Flink 1.3.0 was released about 2 weeks ago and we have 2.5 months
> (~11
> > > > weeks, until begin of September) left until the feature freeze for
> > Flink
> > > > 1.4.0.
> > > >
> > > > I think it makes sense to start with a collection of desired
> features.
> > > Once
> > > > we have a list of requested features, we might want to prioritize and
> > > maybe
> > > > also assign responsibilities.
> > > >
> > > > When we prioritize, we should keep in mind that:
> > > > - we want to have a consistent API. Larger features should be
> developed
> > > in
> > > > a feature branch first.
> > > > - the next months are typical time for vacations
> > > > - we have been bottlenecked by committer resources in the last
> release.
> > > >
> > > > I think the following features would be a nice addition to the
> current
> > > > state:
> > > >
> > > > - Conversion of a stream into an upsert table (with retraction,
> > updating
> > > to
> > > > the last row per key)
> > > > - Joins for streaming tables
> > > >   - Stream-Stream (time-range predicate) there is already a PR for
> > > > processing time joins
> > > >   - Table-Table (with retraction)
> > > > - Support for late arriving records in group window aggregations
> > > > - Exposing a keyed result table as queryable state
> > > >
> > > > Which features are others looking for?
> > > >
> > > > Cheers,
> > > > Fabian
> > > >
> > >
> >
>

Looking for Contributors: Apache Flink meets Apache Mesos and DC/OS

[Till Rohrmann]

Hi,

we are actively looking for contributors (and anyone interested) for the
Flink DC/OS package, as well as the Mesos Flink framework.

We will have a first meeting to explain the current package and
outline/develop a potential roadmap in the next virtual DC/OS office hours:
https://www.meetup.com/DC-OS-Online-Meetup/events/240919616/

If you cannot make it to this virtual event, feel free to reach out via the
#flink channel in the  DC/OS community slack (chat.dcos.io).

Looking forwards to many cool features around Flink and Mesos/ DC/OS,
Jörg and Till

[jira] [Created] (FLINK-6965) Avro is missing snappy dependency

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-6965:
---

 Summary: Avro is missing snappy dependency
 Key: FLINK-6965
 URL: https://issues.apache.org/jira/browse/FLINK-6965
 Project: Flink
  Issue Type: Bug
  Components: Type Serialization System
Affects Versions: 1.3.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.3.2


The shading rework made before 1.3 removed a snappy dependency that was 
accidentally pulled in through hadoop. This is technically alright, until 
class-loaders rear their ugly heads.

Our kafka connector can read avro records, which may or may not require snappy. 
Usually this _should _be solvable by including the snappy dependency in the 
user-jar if necessary, however since the kafka connector loads classes that it 
requires using the system class loader this doesn't work.

As such we have to add a separate snappy dependency to flink-core.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] Release Apache Flink 1.3.1 (RC2)

[Stefan Richter]

Hi,

I suggest to include https://issues.apache.org/jira/browse/FLINK-6964 
 to the release.

Best,
Stefan

> Am 21.06.2017 um 09:46 schrieb Robert Metzger :
> 
> Dear Flink community,
> 
> Please vote on releasing the following candidate as Apache Flink version
> 1.3.1.
> 
> The commit to be voted on:
> *http://git-wip-us.apache.org/repos/asf/flink/commit/1ca6e5b6
> *
> 
> Branch:
> release-1.3.1-rc2
> 
> The release artifacts to be voted on can be found at:
> *http://people.apache.org/~rmetzger/flink-1.3.1-rc2/
> *
> 
> The release artifacts are signed with the key with fingerprint D9839159:
> http://www.apache.org/dist/flink/KEYS
> 
> The staging repository for this release can be found at:
> *https://repository.apache.org/content/repositories/orgapacheflink-1125
> *
> 
> 
> -
> 
> 
> The vote ends on Thursday (5pm CEST), June 22, 2016.
> IMPORTANT: I've reduced the voting time to only one day because the number
> of changes between RC1 and RC2 are mostly in the table API (mostly
> documentation) and the serializer changes Till and Gordon were working on.
> The list of changes is the following
> - Reworked Table API documentation (this is a set of commits)
> - [FLINK-6817] [table] Add OverWindowWithPreceding class to guide users
> - [FLINK-6859] [table] Do not delete timers in StateCleaningCountTrigger
> - [FLINK-6930] [table] Forbid selecting window start/end on row-based T…
> - [FLINK-6886] [table] Fix conversion of Row Table to POJO
> - [FLINK-6602] [table] Prevent TableSources with empty time attribute n…
> - [FLINK-6941] [table] Validate that start and end window properties ar…
> - [FLINK-6881] [FLINK-6896] [table] Creating a table from a POJO and de…
> - [FLINK-6921] [serializer] Allow EnumValueSerializer to deal with appe…
> - [FLINK-6948] [serializer] Harden EnumValueSerializer to detect change…
> - [FLINK-6922] [serializer] Remove Java serialization from Enum(Value)S…
> - [FLINK-6652] [core] Fix handling of delimiters split by buffers in De…
> 
> 
> 
> [ ] +1 Release this package as Apache Flink 1.3.1
> [ ] -1 Do not release this package, because ...

Re: [DISCUSS] Changing Flink's shading model

[Chesnay Schepler]

I like your suggestion Robert. A lot actually.

Having the actual dependency version (i.e 18 for guava) in the version 
should improve clarity a lot.


Originally i intended to release 1 artifact per Flink version, with the 
normal versioning scheme
that we use. But given that the shaded dependencies aren't changed often 
(even rarely might be a stretch),
and aren't actually coupled to the Flink release cycle this doesn't make 
a lot of sense.


Having separate repos looks like a reasonable separation of concerns. 
The release for Flink itself
would work just like it does now; we don't have to modify any scripts or 
do extra steps.


Since the build, release and development process are separate (since 
flink-shaded isn't part of Flink build
process, has a separate release process and changes to it will /never 
/require immediate changes to Flink)

it seems like a very good candidate to move it into a separate repo.

On 21.06.2017 11:26, Robert Metzger wrote:

Its not completely clear to me how we want to version the shaded
dependencies, and where we are putting them.

One concern are the official apache release rules. If we want to release
something to maven central, we need to do a proper vote over a source
archive.
I would propose to create a new repository "flink-shaded.git" that contains
the following maven module structure:
- flink-shaded: 1
- flink-shaded-asm: 1-5.2
- flink-shaded-guava: 1-18.0
- ...

The number indicates the version (for ASM, I've just guessed).
The version for the parent "flink-shaded" needs to be updated on each
parent pom change (new module added, new / changed plugins, ...)

We could create a separate release script in this repository that creates
the flink-shaded-src.zip from the code and deploys the artifacts to the
maven staging area.

The advantage of a separate repo would be that we don't need to maintain
separate maven projects in the same git repo.
Also, the src archives for the release vote can be created from the repo
content (without much filtering).


On Tue, Jun 20, 2017 at 9:44 PM, Stephan Ewen  wrote:


I like this approach.

Two additional things can be mention here:

   - We need to deploy these artifacts independently and not as part of the
build. That is a manual step once per "bump" in the dependency of that
library.

   - We reduce the shading complexity of the original build and should thus
also speed up build times :-)

Stephan




On Tue, Jun 20, 2017 at 1:15 PM, Chesnay Schepler 
wrote:


I would like to start working on this.

I've looked into adding a flink-shaded-guava module. Working against the
shaded namespaces seems
to work without problems from the IDE, and we could forbid un-shaded
usages with checkstyle.

So for the list of dependencies that we want to shade we currently got:

  * asm
  * guava
  * netty
  * hadoop
  * curator

I've had a chat with Stephan Ewan and he brought up kryo + chill as well.

The nice thing is that we can do this incrementally, one dependency at a
time. As such i would propose
to go through the whole process for guava and see what problems arise.

This would include adding a flink-shaded module and a child
flink-shaded-guava module to the flink repository
that are not part of the build process, replacing all usages of guava in
Flink, adding the
checkstyle rule (optional) and deploying the artifact to maven central.


On 11.05.2017 10:54, Stephan Ewen wrote:


@Ufuk  - I have never set up artifact deployment in Maven, could need

some

help there.

Regarding shading Netty, I agree, would be good to do that as well...

On Thu, May 11, 2017 at 10:52 AM, Ufuk Celebi  wrote:

The advantages you've listed sound really compelling to me.

- Do you have time to implement these changes or do we need a

volunteer?

;)

- I assume that republishing the artifacts as you propose doesn't have
any new legal implications since we already publish them with our
JARs, right?

- We might think about adding Netty to the list of shaded artifacts
since some dependency conflicts were reported recently. Would have to
double check the reported issues before doing that though. ;-)

– Ufuk


On Wed, May 10, 2017 at 8:45 PM, Stephan Ewen 

wrote:

@chesnay: I used ASM as an example in the proposal. Maybe I did not

say

that clearly.

If we like that approach, we should deal with the other libraries (at


least


the frequently used ones) in the same way.


I would imagine to have a project layout like that:

flink-shaded-deps
- flink-shaded-asm
- flink-shaded-guava
- flink-shaded-curator
- flink-shaded-hadoop


"flink-shaded-deps" would not be built every time (and not be released
every time), but only when needed.






On Wed, May 10, 2017 at 7:28 PM, Chesnay Schepler 

[jira] [Created] (FLINK-6964) Fix recovery for incremental checkpoints in StandaloneCompletedCheckpointStore

[Stefan Richter (JIRA)]

Stefan Richter created FLINK-6964:
-

 Summary: Fix recovery for incremental checkpoints in 
StandaloneCompletedCheckpointStore
 Key: FLINK-6964
 URL: https://issues.apache.org/jira/browse/FLINK-6964
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing
Reporter: Stefan Richter
Assignee: Stefan Richter


{{StandaloneCompletedCheckpointStore}} does not register shared states in the 
{{recover}} method. However, for externalized checkpoints, it register the 
checkpoint from which it resumed. This checkpoint gets added to the completed 
checkpoint store as part of resume.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-6963) User Defined Operator

[Shaoxuan Wang (JIRA)]

Shaoxuan Wang created FLINK-6963:


 Summary: User Defined Operator
 Key: FLINK-6963
 URL: https://issues.apache.org/jira/browse/FLINK-6963
 Project: Flink
  Issue Type: New Feature
  Components: Table API & SQL
Reporter: Shaoxuan Wang
Assignee: Jark Wu
 Fix For: 1.4.0


We are proposing to add an User Defined Operator (UDOP) interface. As oppose to 
UDF(scalars to scalar)/UDTF(scalar to table)/UDAGG(table to scalar), this UDOP 
allows user to describe a business logic for a table to table conversion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-6962) SQL DDL for input and output tables

[Shaoxuan Wang (JIRA)]

Shaoxuan Wang created FLINK-6962:


 Summary: SQL DDL for input and output tables
 Key: FLINK-6962
 URL: https://issues.apache.org/jira/browse/FLINK-6962
 Project: Flink
  Issue Type: New Feature
  Components: Table API & SQL
Reporter: Shaoxuan Wang
Assignee: lincoln.lee
 Fix For: 1.4.0


This Jira adds support to allow user define the DDL for source and sink tables, 
including the waterMark(on source table) and emit SLA (on result table). The 
detailed design doc will be attached soon.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Changing Flink's shading model

[Robert Metzger]

Its not completely clear to me how we want to version the shaded
dependencies, and where we are putting them.

One concern are the official apache release rules. If we want to release
something to maven central, we need to do a proper vote over a source
archive.
I would propose to create a new repository "flink-shaded.git" that contains
the following maven module structure:
- flink-shaded: 1
   - flink-shaded-asm: 1-5.2
   - flink-shaded-guava: 1-18.0
   - ...

The number indicates the version (for ASM, I've just guessed).
The version for the parent "flink-shaded" needs to be updated on each
parent pom change (new module added, new / changed plugins, ...)

We could create a separate release script in this repository that creates
the flink-shaded-src.zip from the code and deploys the artifacts to the
maven staging area.

The advantage of a separate repo would be that we don't need to maintain
separate maven projects in the same git repo.
Also, the src archives for the release vote can be created from the repo
content (without much filtering).


On Tue, Jun 20, 2017 at 9:44 PM, Stephan Ewen  wrote:

> I like this approach.
>
> Two additional things can be mention here:
>
>   - We need to deploy these artifacts independently and not as part of the
> build. That is a manual step once per "bump" in the dependency of that
> library.
>
>   - We reduce the shading complexity of the original build and should thus
> also speed up build times :-)
>
> Stephan
>
>
>
>
> On Tue, Jun 20, 2017 at 1:15 PM, Chesnay Schepler 
> wrote:
>
> > I would like to start working on this.
> >
> > I've looked into adding a flink-shaded-guava module. Working against the
> > shaded namespaces seems
> > to work without problems from the IDE, and we could forbid un-shaded
> > usages with checkstyle.
> >
> > So for the list of dependencies that we want to shade we currently got:
> >
> >  * asm
> >  * guava
> >  * netty
> >  * hadoop
> >  * curator
> >
> > I've had a chat with Stephan Ewan and he brought up kryo + chill as well.
> >
> > The nice thing is that we can do this incrementally, one dependency at a
> > time. As such i would propose
> > to go through the whole process for guava and see what problems arise.
> >
> > This would include adding a flink-shaded module and a child
> > flink-shaded-guava module to the flink repository
> > that are not part of the build process, replacing all usages of guava in
> > Flink, adding the
> > checkstyle rule (optional) and deploying the artifact to maven central.
> >
> >
> > On 11.05.2017 10:54, Stephan Ewen wrote:
> >
> >> @Ufuk  - I have never set up artifact deployment in Maven, could need
> some
> >> help there.
> >>
> >> Regarding shading Netty, I agree, would be good to do that as well...
> >>
> >> On Thu, May 11, 2017 at 10:52 AM, Ufuk Celebi  wrote:
> >>
> >> The advantages you've listed sound really compelling to me.
> >>>
> >>> - Do you have time to implement these changes or do we need a
> volunteer?
> >>> ;)
> >>>
> >>> - I assume that republishing the artifacts as you propose doesn't have
> >>> any new legal implications since we already publish them with our
> >>> JARs, right?
> >>>
> >>> - We might think about adding Netty to the list of shaded artifacts
> >>> since some dependency conflicts were reported recently. Would have to
> >>> double check the reported issues before doing that though. ;-)
> >>>
> >>> – Ufuk
> >>>
> >>>
> >>> On Wed, May 10, 2017 at 8:45 PM, Stephan Ewen 
> wrote:
> >>>
>  @chesnay: I used ASM as an example in the proposal. Maybe I did not
> say
>  that clearly.
> 
>  If we like that approach, we should deal with the other libraries (at
> 
> >>> least
> >>>
>  the frequently used ones) in the same way.
> 
> 
>  I would imagine to have a project layout like that:
> 
>  flink-shaded-deps
> - flink-shaded-asm
> - flink-shaded-guava
> - flink-shaded-curator
> - flink-shaded-hadoop
> 
> 
>  "flink-shaded-deps" would not be built every time (and not be released
>  every time), but only when needed.
> 
> 
> 
> 
> 
> 
>  On Wed, May 10, 2017 at 7:28 PM, Chesnay Schepler  >
>  wrote:
> 
>  I like the idea, thank you for bringing it up.
> >
> > Given that the raised problems aren't really ASM specific would it
> make
> > sense to create one flink-shaded module that contains all frequently
> >
>  shaded
> >>>
>  libraries? (or maybe even all shaded dependencies by core modules) The
> > proposal limits the scope of this to ASM and i was wondering why.
> >
> > I also remember that there was a discussion recently about why we
> shade
> > things at all, and the idea of working against the shaded namespaces
> > was
> > brought up. Back then i was expressing doubts as to whether IDE's
> would
> > 

[jira] [Created] (FLINK-6961) Enable configurable early-firing rate

[Shaoxuan Wang (JIRA)]

Shaoxuan Wang created FLINK-6961:


 Summary: Enable configurable early-firing rate 
 Key: FLINK-6961
 URL: https://issues.apache.org/jira/browse/FLINK-6961
 Project: Flink
  Issue Type: New Feature
  Components: Table API & SQL
Reporter: Shaoxuan Wang


There are cases that we need to emit the result earlier to allow user get an 
observation/sample of the result. Right now, only the unbounded aggregate works 
in early-firing mode (in the future we will support early firing for all 
different scenarios, like windowed aggregate, unbounded/windowed join, etc.). 
But in unbounded aggregate, the result is prepared and emitted for each input. 
This may not be necessary, as user may not need to get the result so frequent 
in most cases.
We create this Jira to track all the efforts (sub-jira) to enable configurable 
early-firing rate. It should be noted that the early-firing rate will not be 
exposed to the user, it will be smartly decided by the query optimizer 
depending on the SLA(allowed latency) of the final result.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[VOTE] Release Apache Flink 1.3.1 (RC2)

[Robert Metzger]

Dear Flink community,

Please vote on releasing the following candidate as Apache Flink version
1.3.1.

The commit to be voted on:
*http://git-wip-us.apache.org/repos/asf/flink/commit/1ca6e5b6
*

Branch:
release-1.3.1-rc2

The release artifacts to be voted on can be found at:
*http://people.apache.org/~rmetzger/flink-1.3.1-rc2/
*

The release artifacts are signed with the key with fingerprint D9839159:
http://www.apache.org/dist/flink/KEYS

The staging repository for this release can be found at:
*https://repository.apache.org/content/repositories/orgapacheflink-1125
*


-


The vote ends on Thursday (5pm CEST), June 22, 2016.
IMPORTANT: I've reduced the voting time to only one day because the number
of changes between RC1 and RC2 are mostly in the table API (mostly
documentation) and the serializer changes Till and Gordon were working on.
The list of changes is the following
- Reworked Table API documentation (this is a set of commits)
- [FLINK-6817] [table] Add OverWindowWithPreceding class to guide users
- [FLINK-6859] [table] Do not delete timers in StateCleaningCountTrigger
- [FLINK-6930] [table] Forbid selecting window start/end on row-based T…
- [FLINK-6886] [table] Fix conversion of Row Table to POJO
- [FLINK-6602] [table] Prevent TableSources with empty time attribute n…
- [FLINK-6941] [table] Validate that start and end window properties ar…
- [FLINK-6881] [FLINK-6896] [table] Creating a table from a POJO and de…
- [FLINK-6921] [serializer] Allow EnumValueSerializer to deal with appe…
- [FLINK-6948] [serializer] Harden EnumValueSerializer to detect change…
- [FLINK-6922] [serializer] Remove Java serialization from Enum(Value)S…
- [FLINK-6652] [core] Fix handling of delimiters split by buffers in De…



[ ] +1 Release this package as Apache Flink 1.3.1
[ ] -1 Do not release this package, because ...

[jira] [Created] (FLINK-6960) Add E(2.7182818284590452354),PI(3.14159265358979323846) supported in SQL

[sunjincheng (JIRA)]

sunjincheng created FLINK-6960:
--

 Summary: Add E(2.7182818284590452354),PI(3.14159265358979323846) 
supported in SQL
 Key: FLINK-6960
 URL: https://issues.apache.org/jira/browse/FLINK-6960
 Project: Flink
  Issue Type: Sub-task
  Components: Table API & SQL
Reporter: sunjincheng
Assignee: sunjincheng


E=Math.E 
PI=Math.PI



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-6959) Add documentation for SQL OVER windows

[sunjincheng (JIRA)]

sunjincheng created FLINK-6959:
--

 Summary: Add documentation for SQL OVER windows
 Key: FLINK-6959
 URL: https://issues.apache.org/jira/browse/FLINK-6959
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng
Assignee: sunjincheng


Add documentation for SQL OVER windows.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)