Re: [DISCUSS] FLIP-32: Restructure flink-table for future contributions

2019-01-29 Thread Timo Walther

Hi everyone,

thanks for the positive feedback we received so far. I converted the 
design document into a actual FLIP in the wiki. You can find it here:


https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions

I'm in the process of creating the first JIRA issues to kick off the 
preparation efforts. Most of the issues will be first or second level 
children of FLINK-11439:


https://issues.apache.org/jira/browse/FLINK-11439

Feel free to further join the discussions and provide feedback. I'm sure 
there will be a lot of stuff to discuss :-)


Thanks,
Timo


Am 25.01.19 um 16:43 schrieb fudian.fd:

Hi Timo,

Thanks a lot for bringing up the discussion and the detailed implementation 
plan. The implementation plan makes much sense to me. +1 to the FLIP.

Regards,
Dian



在 2019年1月25日,下午11:00,Jark Wu  写道:

Hi Timo,

Thanks for the detailed design. +1 for the FLIP.

It's very nice to have Blink planner and runtime as a plugin in the early
stages. This will keep flink-table stable as much as possible.

Best,
Jark


On Fri, 25 Jan 2019 at 22:26, Hequn Cheng  wrote:


Hi Timo,

+1 for the FLIP!

Great work and thanks a lot for the detailed document! The task dependency
graph is very helpful for teasing out relationships between tasks.
Looking forward to the JIRAs and hoping to contribute to it!

Best,
Hequn


On Fri, Jan 25, 2019 at 7:54 PM Piotr Nowojski 
wrote:


Hi,

+1 from my side for this plan. The proposed decoupling TypeSystem and

most

interface (TableSource/Sinks, UDFs, Catalogs, …) discussions from the of
merging MVP Blink's runtime gives us best shot of handling Blink merging

as

fluently and as painless as possible. I’m also looking forward to the
follow up discussions that will need to take place in order to achieve

the

final goal :)

Piotrek


On 25 Jan 2019, at 07:53, jincheng sun 

wrote:

Hi Timo,

Thanks a lot for bring up the FLIP-32 discussion and the very detailed
implementation plan document !

Restructure `flink-table` is an important part of flink merge blink,
looking forward to the JIRAs which will be opened !

Cheers,
Jincheng


Timo Walther  于2019年1月24日周四 下午9:06写道:


Hi everyone,

as Stephan already announced on the mailing list [1], the Flink
community will receive a big code contribution from Alibaba. The
flink-table module is one of the biggest parts that will receive many
new features and major architectural improvements. Instead of waiting
until the next major version of Flink or introducing big API-breaking
changes, we would like to gradually build up the Blink-based planner

and

runtime while keeping the Table & SQL API mostly stable. Users will be
able to play around with the current merge status of the new planner

or

fall back to the old planner until the new one is stable.

We have prepared a design document that discusses a restructuring of

the

flink-table module and suggests a rough implementation plan:




https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing

I will briefly summarize the steps we would like to do:

- Split the flink-table module similar to the proposal of FLIP-28 [3]
which is outdated. This is a preparation to separate API from core
(targeted for Flink 1.8).
- Perform minor API changes to separate API from actual implementation
(targeted for Flink 1.8).
- Merge a MVP Blink SQL planner given that necessary Flink

core/runtime

changes have been completed.
The merging will happen in stages (e.g. basic planner framework,

then

operator by operator). The exact merging plan still needs to be

determined.

- Rework the type system in order to unblock work on unified table
environments, UDFs, sources/sinks, and catalog.
- Enable full end-to-end batch and stream execution features.

Our mid-term goal:

Run full TPC-DS on a unified batch/streaming runtime. Initially, we

will

only support ingesting data coming from the DataStream API. Once we
reworked the sources/sink interfaces, we will target full end-to-end
TPC-DS query execution with table connectors.

A rough task dependency graph is illustrated in the design document. A
more detailed task dependency structure will be added to JIRA after we
agreed on this FLIP.

Looking forward to any feedback.

Thanks,
Timo

[1]



https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E

[2]



https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E

[3]



https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free








Re: [DISCUSS] FLIP-32: Restructure flink-table for future contributions

2019-01-25 Thread fudian . fd
Hi Timo,

Thanks a lot for bringing up the discussion and the detailed implementation 
plan. The implementation plan makes much sense to me. +1 to the FLIP.

Regards,
Dian


> 在 2019年1月25日,下午11:00,Jark Wu  写道:
> 
> Hi Timo,
> 
> Thanks for the detailed design. +1 for the FLIP.
> 
> It's very nice to have Blink planner and runtime as a plugin in the early
> stages. This will keep flink-table stable as much as possible.
> 
> Best,
> Jark
> 
> 
> On Fri, 25 Jan 2019 at 22:26, Hequn Cheng  wrote:
> 
>> Hi Timo,
>> 
>> +1 for the FLIP!
>> 
>> Great work and thanks a lot for the detailed document! The task dependency
>> graph is very helpful for teasing out relationships between tasks.
>> Looking forward to the JIRAs and hoping to contribute to it!
>> 
>> Best,
>> Hequn
>> 
>> 
>> On Fri, Jan 25, 2019 at 7:54 PM Piotr Nowojski 
>> wrote:
>> 
>>> Hi,
>>> 
>>> +1 from my side for this plan. The proposed decoupling TypeSystem and
>> most
>>> interface (TableSource/Sinks, UDFs, Catalogs, …) discussions from the of
>>> merging MVP Blink's runtime gives us best shot of handling Blink merging
>> as
>>> fluently and as painless as possible. I’m also looking forward to the
>>> follow up discussions that will need to take place in order to achieve
>> the
>>> final goal :)
>>> 
>>> Piotrek
>>> 
 On 25 Jan 2019, at 07:53, jincheng sun 
>> wrote:
 
 Hi Timo,
 
 Thanks a lot for bring up the FLIP-32 discussion and the very detailed
 implementation plan document !
 
 Restructure `flink-table` is an important part of flink merge blink,
 looking forward to the JIRAs which will be opened !
 
 Cheers,
 Jincheng
 
 
 Timo Walther  于2019年1月24日周四 下午9:06写道:
 
> Hi everyone,
> 
> as Stephan already announced on the mailing list [1], the Flink
> community will receive a big code contribution from Alibaba. The
> flink-table module is one of the biggest parts that will receive many
> new features and major architectural improvements. Instead of waiting
> until the next major version of Flink or introducing big API-breaking
> changes, we would like to gradually build up the Blink-based planner
>> and
> runtime while keeping the Table & SQL API mostly stable. Users will be
> able to play around with the current merge status of the new planner
>> or
> fall back to the old planner until the new one is stable.
> 
> We have prepared a design document that discusses a restructuring of
>> the
> flink-table module and suggests a rough implementation plan:
> 
> 
> 
>>> 
>> https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing
> 
> I will briefly summarize the steps we would like to do:
> 
> - Split the flink-table module similar to the proposal of FLIP-28 [3]
> which is outdated. This is a preparation to separate API from core
> (targeted for Flink 1.8).
> - Perform minor API changes to separate API from actual implementation
> (targeted for Flink 1.8).
> - Merge a MVP Blink SQL planner given that necessary Flink
>> core/runtime
> changes have been completed.
> The merging will happen in stages (e.g. basic planner framework,
>> then
> operator by operator). The exact merging plan still needs to be
>>> determined.
> - Rework the type system in order to unblock work on unified table
> environments, UDFs, sources/sinks, and catalog.
> - Enable full end-to-end batch and stream execution features.
> 
> Our mid-term goal:
> 
> Run full TPC-DS on a unified batch/streaming runtime. Initially, we
>> will
> only support ingesting data coming from the DataStream API. Once we
> reworked the sources/sink interfaces, we will target full end-to-end
> TPC-DS query execution with table connectors.
> 
> A rough task dependency graph is illustrated in the design document. A
> more detailed task dependency structure will be added to JIRA after we
> agreed on this FLIP.
> 
> Looking forward to any feedback.
> 
> Thanks,
> Timo
> 
> [1]
> 
> 
>>> 
>> https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
> [2]
> 
> 
>>> 
>> https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
> [3]
> 
> 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free
> 
> 
>>> 
>>> 
>> 



smime.p7s
Description: S/MIME cryptographic signature


Re: [DISCUSS] FLIP-32: Restructure flink-table for future contributions

2019-01-25 Thread Jark Wu
Hi Timo,

Thanks for the detailed design. +1 for the FLIP.

It's very nice to have Blink planner and runtime as a plugin in the early
stages. This will keep flink-table stable as much as possible.

Best,
Jark


On Fri, 25 Jan 2019 at 22:26, Hequn Cheng  wrote:

> Hi Timo,
>
> +1 for the FLIP!
>
> Great work and thanks a lot for the detailed document! The task dependency
> graph is very helpful for teasing out relationships between tasks.
> Looking forward to the JIRAs and hoping to contribute to it!
>
> Best,
> Hequn
>
>
> On Fri, Jan 25, 2019 at 7:54 PM Piotr Nowojski 
> wrote:
>
> > Hi,
> >
> > +1 from my side for this plan. The proposed decoupling TypeSystem and
> most
> > interface (TableSource/Sinks, UDFs, Catalogs, …) discussions from the of
> > merging MVP Blink's runtime gives us best shot of handling Blink merging
> as
> > fluently and as painless as possible. I’m also looking forward to the
> > follow up discussions that will need to take place in order to achieve
> the
> > final goal :)
> >
> > Piotrek
> >
> > > On 25 Jan 2019, at 07:53, jincheng sun 
> wrote:
> > >
> > > Hi Timo,
> > >
> > > Thanks a lot for bring up the FLIP-32 discussion and the very detailed
> > > implementation plan document !
> > >
> > > Restructure `flink-table` is an important part of flink merge blink,
> > > looking forward to the JIRAs which will be opened !
> > >
> > > Cheers,
> > > Jincheng
> > >
> > >
> > > Timo Walther  于2019年1月24日周四 下午9:06写道:
> > >
> > >> Hi everyone,
> > >>
> > >> as Stephan already announced on the mailing list [1], the Flink
> > >> community will receive a big code contribution from Alibaba. The
> > >> flink-table module is one of the biggest parts that will receive many
> > >> new features and major architectural improvements. Instead of waiting
> > >> until the next major version of Flink or introducing big API-breaking
> > >> changes, we would like to gradually build up the Blink-based planner
> and
> > >> runtime while keeping the Table & SQL API mostly stable. Users will be
> > >> able to play around with the current merge status of the new planner
> or
> > >> fall back to the old planner until the new one is stable.
> > >>
> > >> We have prepared a design document that discusses a restructuring of
> the
> > >> flink-table module and suggests a rough implementation plan:
> > >>
> > >>
> > >>
> >
> https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing
> > >>
> > >> I will briefly summarize the steps we would like to do:
> > >>
> > >> - Split the flink-table module similar to the proposal of FLIP-28 [3]
> > >> which is outdated. This is a preparation to separate API from core
> > >> (targeted for Flink 1.8).
> > >> - Perform minor API changes to separate API from actual implementation
> > >> (targeted for Flink 1.8).
> > >> - Merge a MVP Blink SQL planner given that necessary Flink
> core/runtime
> > >> changes have been completed.
> > >>   The merging will happen in stages (e.g. basic planner framework,
> then
> > >> operator by operator). The exact merging plan still needs to be
> > determined.
> > >> - Rework the type system in order to unblock work on unified table
> > >> environments, UDFs, sources/sinks, and catalog.
> > >> - Enable full end-to-end batch and stream execution features.
> > >>
> > >> Our mid-term goal:
> > >>
> > >> Run full TPC-DS on a unified batch/streaming runtime. Initially, we
> will
> > >> only support ingesting data coming from the DataStream API. Once we
> > >> reworked the sources/sink interfaces, we will target full end-to-end
> > >> TPC-DS query execution with table connectors.
> > >>
> > >> A rough task dependency graph is illustrated in the design document. A
> > >> more detailed task dependency structure will be added to JIRA after we
> > >> agreed on this FLIP.
> > >>
> > >> Looking forward to any feedback.
> > >>
> > >> Thanks,
> > >> Timo
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
> > >> [2]
> > >>
> > >>
> >
> https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
> > >> [3]
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free
> > >>
> > >>
> >
> >
>


Re: [DISCUSS] FLIP-32: Restructure flink-table for future contributions

2019-01-25 Thread Hequn Cheng
Hi Timo,

+1 for the FLIP!

Great work and thanks a lot for the detailed document! The task dependency
graph is very helpful for teasing out relationships between tasks.
Looking forward to the JIRAs and hoping to contribute to it!

Best,
Hequn


On Fri, Jan 25, 2019 at 7:54 PM Piotr Nowojski 
wrote:

> Hi,
>
> +1 from my side for this plan. The proposed decoupling TypeSystem and most
> interface (TableSource/Sinks, UDFs, Catalogs, …) discussions from the of
> merging MVP Blink's runtime gives us best shot of handling Blink merging as
> fluently and as painless as possible. I’m also looking forward to the
> follow up discussions that will need to take place in order to achieve the
> final goal :)
>
> Piotrek
>
> > On 25 Jan 2019, at 07:53, jincheng sun  wrote:
> >
> > Hi Timo,
> >
> > Thanks a lot for bring up the FLIP-32 discussion and the very detailed
> > implementation plan document !
> >
> > Restructure `flink-table` is an important part of flink merge blink,
> > looking forward to the JIRAs which will be opened !
> >
> > Cheers,
> > Jincheng
> >
> >
> > Timo Walther  于2019年1月24日周四 下午9:06写道:
> >
> >> Hi everyone,
> >>
> >> as Stephan already announced on the mailing list [1], the Flink
> >> community will receive a big code contribution from Alibaba. The
> >> flink-table module is one of the biggest parts that will receive many
> >> new features and major architectural improvements. Instead of waiting
> >> until the next major version of Flink or introducing big API-breaking
> >> changes, we would like to gradually build up the Blink-based planner and
> >> runtime while keeping the Table & SQL API mostly stable. Users will be
> >> able to play around with the current merge status of the new planner or
> >> fall back to the old planner until the new one is stable.
> >>
> >> We have prepared a design document that discusses a restructuring of the
> >> flink-table module and suggests a rough implementation plan:
> >>
> >>
> >>
> https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing
> >>
> >> I will briefly summarize the steps we would like to do:
> >>
> >> - Split the flink-table module similar to the proposal of FLIP-28 [3]
> >> which is outdated. This is a preparation to separate API from core
> >> (targeted for Flink 1.8).
> >> - Perform minor API changes to separate API from actual implementation
> >> (targeted for Flink 1.8).
> >> - Merge a MVP Blink SQL planner given that necessary Flink core/runtime
> >> changes have been completed.
> >>   The merging will happen in stages (e.g. basic planner framework, then
> >> operator by operator). The exact merging plan still needs to be
> determined.
> >> - Rework the type system in order to unblock work on unified table
> >> environments, UDFs, sources/sinks, and catalog.
> >> - Enable full end-to-end batch and stream execution features.
> >>
> >> Our mid-term goal:
> >>
> >> Run full TPC-DS on a unified batch/streaming runtime. Initially, we will
> >> only support ingesting data coming from the DataStream API. Once we
> >> reworked the sources/sink interfaces, we will target full end-to-end
> >> TPC-DS query execution with table connectors.
> >>
> >> A rough task dependency graph is illustrated in the design document. A
> >> more detailed task dependency structure will be added to JIRA after we
> >> agreed on this FLIP.
> >>
> >> Looking forward to any feedback.
> >>
> >> Thanks,
> >> Timo
> >>
> >> [1]
> >>
> >>
> https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
> >> [2]
> >>
> >>
> https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
> >> [3]
> >>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free
> >>
> >>
>
>


Re: [DISCUSS] FLIP-32: Restructure flink-table for future contributions

2019-01-25 Thread Piotr Nowojski
Hi,

+1 from my side for this plan. The proposed decoupling TypeSystem and most 
interface (TableSource/Sinks, UDFs, Catalogs, …) discussions from the of 
merging MVP Blink's runtime gives us best shot of handling Blink merging as 
fluently and as painless as possible. I’m also looking forward to the follow up 
discussions that will need to take place in order to achieve the final goal :)

Piotrek 

> On 25 Jan 2019, at 07:53, jincheng sun  wrote:
> 
> Hi Timo,
> 
> Thanks a lot for bring up the FLIP-32 discussion and the very detailed
> implementation plan document !
> 
> Restructure `flink-table` is an important part of flink merge blink,
> looking forward to the JIRAs which will be opened !
> 
> Cheers,
> Jincheng
> 
> 
> Timo Walther  于2019年1月24日周四 下午9:06写道:
> 
>> Hi everyone,
>> 
>> as Stephan already announced on the mailing list [1], the Flink
>> community will receive a big code contribution from Alibaba. The
>> flink-table module is one of the biggest parts that will receive many
>> new features and major architectural improvements. Instead of waiting
>> until the next major version of Flink or introducing big API-breaking
>> changes, we would like to gradually build up the Blink-based planner and
>> runtime while keeping the Table & SQL API mostly stable. Users will be
>> able to play around with the current merge status of the new planner or
>> fall back to the old planner until the new one is stable.
>> 
>> We have prepared a design document that discusses a restructuring of the
>> flink-table module and suggests a rough implementation plan:
>> 
>> 
>> https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing
>> 
>> I will briefly summarize the steps we would like to do:
>> 
>> - Split the flink-table module similar to the proposal of FLIP-28 [3]
>> which is outdated. This is a preparation to separate API from core
>> (targeted for Flink 1.8).
>> - Perform minor API changes to separate API from actual implementation
>> (targeted for Flink 1.8).
>> - Merge a MVP Blink SQL planner given that necessary Flink core/runtime
>> changes have been completed.
>>   The merging will happen in stages (e.g. basic planner framework, then
>> operator by operator). The exact merging plan still needs to be determined.
>> - Rework the type system in order to unblock work on unified table
>> environments, UDFs, sources/sinks, and catalog.
>> - Enable full end-to-end batch and stream execution features.
>> 
>> Our mid-term goal:
>> 
>> Run full TPC-DS on a unified batch/streaming runtime. Initially, we will
>> only support ingesting data coming from the DataStream API. Once we
>> reworked the sources/sink interfaces, we will target full end-to-end
>> TPC-DS query execution with table connectors.
>> 
>> A rough task dependency graph is illustrated in the design document. A
>> more detailed task dependency structure will be added to JIRA after we
>> agreed on this FLIP.
>> 
>> Looking forward to any feedback.
>> 
>> Thanks,
>> Timo
>> 
>> [1]
>> 
>> https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
>> [2]
>> 
>> https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
>> [3]
>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free
>> 
>> 



Re: [DISCUSS] FLIP-32: Restructure flink-table for future contributions

2019-01-24 Thread jincheng sun
Hi Timo,

Thanks a lot for bring up the FLIP-32 discussion and the very detailed
implementation plan document !

Restructure `flink-table` is an important part of flink merge blink,
looking forward to the JIRAs which will be opened !

Cheers,
Jincheng


Timo Walther  于2019年1月24日周四 下午9:06写道:

> Hi everyone,
>
> as Stephan already announced on the mailing list [1], the Flink
> community will receive a big code contribution from Alibaba. The
> flink-table module is one of the biggest parts that will receive many
> new features and major architectural improvements. Instead of waiting
> until the next major version of Flink or introducing big API-breaking
> changes, we would like to gradually build up the Blink-based planner and
> runtime while keeping the Table & SQL API mostly stable. Users will be
> able to play around with the current merge status of the new planner or
> fall back to the old planner until the new one is stable.
>
> We have prepared a design document that discusses a restructuring of the
> flink-table module and suggests a rough implementation plan:
>
>
> https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing
>
> I will briefly summarize the steps we would like to do:
>
> - Split the flink-table module similar to the proposal of FLIP-28 [3]
> which is outdated. This is a preparation to separate API from core
> (targeted for Flink 1.8).
> - Perform minor API changes to separate API from actual implementation
> (targeted for Flink 1.8).
> - Merge a MVP Blink SQL planner given that necessary Flink core/runtime
> changes have been completed.
>The merging will happen in stages (e.g. basic planner framework, then
> operator by operator). The exact merging plan still needs to be determined.
> - Rework the type system in order to unblock work on unified table
> environments, UDFs, sources/sinks, and catalog.
> - Enable full end-to-end batch and stream execution features.
>
> Our mid-term goal:
>
> Run full TPC-DS on a unified batch/streaming runtime. Initially, we will
> only support ingesting data coming from the DataStream API. Once we
> reworked the sources/sink interfaces, we will target full end-to-end
> TPC-DS query execution with table connectors.
>
> A rough task dependency graph is illustrated in the design document. A
> more detailed task dependency structure will be added to JIRA after we
> agreed on this FLIP.
>
> Looking forward to any feedback.
>
> Thanks,
> Timo
>
> [1]
>
> https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
> [2]
>
> https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
> [3]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free
>
>


[DISCUSS] FLIP-32: Restructure flink-table for future contributions

2019-01-24 Thread Timo Walther

Hi everyone,

as Stephan already announced on the mailing list [1], the Flink 
community will receive a big code contribution from Alibaba. The 
flink-table module is one of the biggest parts that will receive many 
new features and major architectural improvements. Instead of waiting 
until the next major version of Flink or introducing big API-breaking 
changes, we would like to gradually build up the Blink-based planner and 
runtime while keeping the Table & SQL API mostly stable. Users will be 
able to play around with the current merge status of the new planner or 
fall back to the old planner until the new one is stable.


We have prepared a design document that discusses a restructuring of the 
flink-table module and suggests a rough implementation plan:


https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing

I will briefly summarize the steps we would like to do:

- Split the flink-table module similar to the proposal of FLIP-28 [3] 
which is outdated. This is a preparation to separate API from core 
(targeted for Flink 1.8).
- Perform minor API changes to separate API from actual implementation 
(targeted for Flink 1.8).
- Merge a MVP Blink SQL planner given that necessary Flink core/runtime 
changes have been completed.
  The merging will happen in stages (e.g. basic planner framework, then 
operator by operator). The exact merging plan still needs to be determined.
- Rework the type system in order to unblock work on unified table 
environments, UDFs, sources/sinks, and catalog.

- Enable full end-to-end batch and stream execution features.

Our mid-term goal:

Run full TPC-DS on a unified batch/streaming runtime. Initially, we will 
only support ingesting data coming from the DataStream API. Once we 
reworked the sources/sink interfaces, we will target full end-to-end 
TPC-DS query execution with table connectors.


A rough task dependency graph is illustrated in the design document. A 
more detailed task dependency structure will be added to JIRA after we 
agreed on this FLIP.


Looking forward to any feedback.

Thanks,
Timo

[1] 
https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
[2] 
https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
[3] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free