Re: [DISCUSS] Linkis Proposal (a Computation Middleware project)

2021-07-16 Thread Lidong Dai 
+1

Interesting!
Linkis builds a computation middleware layer to reduce the difficulty for
users to use multiple compute engines, and it seems that they have built a
community around Links
I'd like to see Links enter the Apache incubator for better development

Best Regards



---
Apache DolphinScheduler PMC Chair
David
lidong...@apache.org
Linkedin: https://www.linkedin.com/in/dailidong
Twitter: @WorkflowEasy 
---


On Fri, Jul 16, 2021 at 10:07 AM Xun Liu  wrote:

> +1
>
> Linkis builds a layer of computing middleware between the upper application
> and the underlying engine.
> Solve the docking problems between various big data systems.
> Linkis helps users to use Apache's Hadoop, hive, spark, zeppelin and other
> systems more conveniently.
> I think that allowing Linkis to enter Apache for better development can
> also help other Apache Better use of the project.
> I think Linkis is a very valuable project.
> Good luck!
>
> Best regards,
> Xun Liu
>
> On Thu, Jul 15, 2021 at 3:32 PM ShaoFeng Shi 
> wrote:
>
> > I'd like to second this proposal. Linkis builds an active open source
> > community and has been adopted by many companies. Bringing it to
> > Apache software foundation can accelerate its development, and add value
> to
> > Apache. Good luck!
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
> >
> >
> > 俊平堵  于2021年7月13日周二 上午9:32写道：
> >
> > > +1.
> > > Linkis is an interesting project that builds a unified layer to
> decouple
> > > the upper layer computation/application and under layer data engines,
> > such
> > > as Spark, Presto, Flink, etc. Also it looks already build a community
> > > around the project.
> > > Good luck!
> > >
> > >
> > > Shuai Di 于2021年7月12日 周一上午11:13写道：
> > >
> > > > Greetings!
> > > >
> > > >
> > > > We would like to start an open discussion on bringing Linkis (
> > > > https://github.com/WeBankFinTech/Linkis), a computation middleware
> > > > project, to the Apache Incubator.
> > > > The proposal can be found below and is also listed in the Incubator
> > wiki:
> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/LinkisProposal
> ,
> > > > thanks @Junping Du for creating the page!
> > > > We appreciate anyone who would give guidance or be willing to support
> > us
> > > a
> > > > an additional mentor.
> > > > ==
> > > > Linkis Proposal
> > > >
> > > >
> > > > =Abstract=
> > > > Linkis builds a computation middleware layer to decouple the upper
> > > > applications and the underlying data engines, provides standardized
> > > > interfaces (REST, JDBC, WebSocket etc.) to easily connect to various
> > > > underlying engines (Spark, Presto, Flink, etc.), while enables cross
> > > engine
> > > > context sharing, unified job& engine governance and orchestration.
> > > > Linkis codebase: https://github.com/WeBankFinTech/Linkis
> > > >
> > > >
> > > > =Proposal=
> > > > Linkis is designed to solve computation governance problems in
> complex
> > > > distributed environments (typically in a big data platform), where
> you
> > > have
> > > > to deal with different types, versions, or clusters of underlying
> data
> > > > engines and hundreds of diversified engine clients at the upper
> > > application
> > > > layer as well.
> > > > Linkis acts as a proxy between the upper applications layer and
> > > underlying
> > > > engines layer. By abstracting and implementing the 3 common phases
> of a
> > > > job/request for submit, prepare and execute, Linkis is able to
> > facilitate
> > > > the connectivity, governance and orchestration capabilities of
> > different
> > > > kind of engines like OLAP, OLTP (developing), Streaming, and handle
> all
> > > > these "computation governance" affairs in a standardized reusable
> way.
> > > > We are actively operating the Linkis community and we are looking
> > forward
> > > > to increase community activity continuously.We propose to contribute
> > the
> > > > Linkis codebase to the Apache Software Foundation. We believe that
> > > bringing
> > > > Linkis into Apache Software Foundation and following the
> COMMUNITY-LED
> > > > DEVELOPMENT "APACHE WAY" could continuously improve project quality
> and
> > > > community vitality.
> > > >
> > > >
> > > > =Background=
> > > > In today's complex and distributed environment, the communication,
> > > > coordination and governance of application services have developed
> > mature
> > > > solutions from SOA to micro-services, and many practices from ESB to
> > > > Service Mesh to decouple different services.
> > > > However, things go different while an application service needs to
> > > > communicate with the underlying engines. Engines are isolated from
> each
> > > > other,

Re: [DISCUSS] Linkis Proposal (a Computation Middleware project)

2021-07-15 Thread Xun Liu 
+1

Linkis builds a layer of computing middleware between the upper application
and the underlying engine.
Solve the docking problems between various big data systems.
Linkis helps users to use Apache's Hadoop, hive, spark, zeppelin and other
systems more conveniently.
I think that allowing Linkis to enter Apache for better development can
also help other Apache Better use of the project.
I think Linkis is a very valuable project.
Good luck!

Best regards,
Xun Liu

On Thu, Jul 15, 2021 at 3:32 PM ShaoFeng Shi  wrote:

> I'd like to second this proposal. Linkis builds an active open source
> community and has been adopted by many companies. Bringing it to
> Apache software foundation can accelerate its development, and add value to
> Apache. Good luck!
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>
>
> 俊平堵  于2021年7月13日周二 上午9:32写道：
>
> > +1.
> > Linkis is an interesting project that builds a unified layer to decouple
> > the upper layer computation/application and under layer data engines,
> such
> > as Spark, Presto, Flink, etc. Also it looks already build a community
> > around the project.
> > Good luck!
> >
> >
> > Shuai Di 于2021年7月12日 周一上午11:13写道：
> >
> > > Greetings!
> > >
> > >
> > > We would like to start an open discussion on bringing Linkis (
> > > https://github.com/WeBankFinTech/Linkis), a computation middleware
> > > project, to the Apache Incubator.
> > > The proposal can be found below and is also listed in the Incubator
> wiki:
> > > https://cwiki.apache.org/confluence/display/INCUBATOR/LinkisProposal,
> > > thanks @Junping Du for creating the page!
> > > We appreciate anyone who would give guidance or be willing to support
> us
> > a
> > > an additional mentor.
> > > ==
> > > Linkis Proposal
> > >
> > >
> > > =Abstract=
> > > Linkis builds a computation middleware layer to decouple the upper
> > > applications and the underlying data engines, provides standardized
> > > interfaces (REST, JDBC, WebSocket etc.) to easily connect to various
> > > underlying engines (Spark, Presto, Flink, etc.), while enables cross
> > engine
> > > context sharing, unified job& engine governance and orchestration.
> > > Linkis codebase: https://github.com/WeBankFinTech/Linkis
> > >
> > >
> > > =Proposal=
> > > Linkis is designed to solve computation governance problems in complex
> > > distributed environments (typically in a big data platform), where you
> > have
> > > to deal with different types, versions, or clusters of underlying data
> > > engines and hundreds of diversified engine clients at the upper
> > application
> > > layer as well.
> > > Linkis acts as a proxy between the upper applications layer and
> > underlying
> > > engines layer. By abstracting and implementing the 3 common phases of a
> > > job/request for submit, prepare and execute, Linkis is able to
> facilitate
> > > the connectivity, governance and orchestration capabilities of
> different
> > > kind of engines like OLAP, OLTP (developing), Streaming, and handle all
> > > these "computation governance" affairs in a standardized reusable way.
> > > We are actively operating the Linkis community and we are looking
> forward
> > > to increase community activity continuously.We propose to contribute
> the
> > > Linkis codebase to the Apache Software Foundation. We believe that
> > bringing
> > > Linkis into Apache Software Foundation and following the COMMUNITY-LED
> > > DEVELOPMENT "APACHE WAY" could continuously improve project quality and
> > > community vitality.
> > >
> > >
> > > =Background=
> > > In today's complex and distributed environment, the communication,
> > > coordination and governance of application services have developed
> mature
> > > solutions from SOA to micro-services, and many practices from ESB to
> > > Service Mesh to decouple different services.
> > > However, things go different while an application service needs to
> > > communicate with the underlying engines. Engines are isolated from each
> > > other, and the client-server tight coupling pattern goes everywhere.
> Each
> > > and every upper application has to directly connect to and access
> various
> > > underlying engines in a tightly coupled way, and solves the
> "computation
> > > governance" problems on its own, including maintaining different client
> > > environments, submiting the job, monitoring job status, fetching the
> > > output, handling large number of concurrent client instances, watching
> > the
> > > bad jobs, adapt to engine version changes, etc.
> > > It lacks a common layer of "computation middleware" between the
> numerous
> > > upper-layer applications and the countless underlying engines to handle
> > all
> > > these "computation governance" affairs in a standardized reusable way,
> > >

[DISCUSS] Linkis Proposal (a Computation Middleware project)

2021-07-15 Thread 俊平堵 
Greetings!


We would like to start an open discussion on bringing Linkis (
https://github.com/WeBankFinTech/Linkis), a computation middleware project,
to the Apache Incubator.


The proposal can be found below and is also listed in the Incubator wiki:
https://cwiki.apache.org/confluence/display/INCUBATOR/LinkisProposal. We
appreciate anyone who would give guidance or be willing to support us as an
additional mentor.


==
Linkis Proposal




=Abstract=

Linkis builds a computation middleware layer to decouple the upper
applications and the underlying data engines, provides standardized
interfaces (REST, JDBC, WebSocket etc.) to easily connect to various
underlying engines (Spark, Presto, Flink, etc.), while enables cross engine
context sharing, unified job& engine governance and orchestration.

Linkis codebase: https://github.com/WeBankFinTech/Linkis




=Proposal=

Linkis is designed to solve computation governance problems in complex
distributed environments (typically in a big data platform), where you have
to deal with different types, versions, or clusters of underlying data
engines and hundreds of diversified engine clients at the upper application
layer as well.

Linkis acts as a proxy between the upper applications layer and underlying
engines layer. By abstracting and implementing the 3 common phases of a
job/request for submit, prepare and execute, Linkis is able to facilitate
the connectivity, governance and orchestration capabilities of different
kind of engines like OLAP, OLTP (developing), Streaming, and handle all
these "computation governance" affairs in a standardized reusable way.

We are actively operating the Linkis community and we are looking forward
to increase community activity continuously.

We propose to contribute the Linkis codebase to the Apache Software
Foundation. We believe that bringing Linkis into Apache Software Foundation
and following the COMMUNITY-LED DEVELOPMENT "APACHE WAY" could continuously
improve project quality and community vitality.




=Background=

In today's complex and distributed environment, the communication,
coordination and governance of application services have developed mature
solutions from SOA to micro-services, and many practices from ESB to
Service Mesh to decouple different services.

However, things go different while an application service needs to
communicate with the underlying engines. Engines are isolated from each
other, and the client-server tight coupling pattern goes everywhere. Each
and every upper application has to directly connect to and access various
underlying engines in a tightly coupled way, and solves the "computation
governance" problems on its own, including maintaining different client
environments, submiting the job, monitoring job status, fetching the
output, handling large number of concurrent client instances, watching the
bad jobs, adapt to engine version changes, etc.

It lacks a common layer of "computation middleware" between the numerous
upper-layer applications and the countless underlying engines to handle all
these "computation governance" affairs in a standardized reusable way,
that's why we started the Linkis project.

Firstly, Linkis could reduce the complexity of connectivity. Instead of
maintaining a variety of engine client environments, users now only need to
install the Linkis client, or even just HTTP client while using the REST
interface. Routing query to desired clusters could be done by simply
providing a tag.

Secondly, Links provides governance capabilities such as multi-tenancy,
concurrency control, resource management, query validation, privilege
enhancement and auditing.

Meanwhile, Linkis enables orchestration strategies such as routing,
load-balance, active-active and hybrid computation across engines (some
still under development).




=Rationale=

Linkis is built on distributed microservice architecture with great
scalability and extendibility. The enhancements of high concurrency and
fault tolerance make it more stable and reliable. It has already supported
many production environments with large number of daily jobs over a long
term.

Linkis's microservices are divided into 3 groups: Computation Governance
Services, Public Enhancement Services, and Microservice Governance Services.

Computation Governance Services(CGS) group is responsible for the core
process of job/request submission, preparation and execution, lifecycle
management, resource management, validation and orchestration.

Public enhancement Services(PES) group provides basic public functions
including job context sharing, material management and data source
management, to serve other Linkis services and upper application systems.

Microservice Governance Services(MGS) group includes customized Spring
Cloud Gateway, Eureka and Open Feign, to provide basic functions like
routing, service registration and discovery, and RPC framework.

By providing capabilities of multi-tenant, high concurrency, job

Re: [DISCUSS] Linkis Proposal (a Computation Middleware project)

2021-07-15 Thread ShaoFeng Shi 
I'd like to second this proposal. Linkis builds an active open source
community and has been adopted by many companies. Bringing it to
Apache software foundation can accelerate its development, and add value to
Apache. Good luck!

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofeng...@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org



俊平堵  于2021年7月13日周二 上午9:32写道：

> +1.
> Linkis is an interesting project that builds a unified layer to decouple
> the upper layer computation/application and under layer data engines, such
> as Spark, Presto, Flink, etc. Also it looks already build a community
> around the project.
> Good luck!
>
>
> Shuai Di 于2021年7月12日 周一上午11:13写道：
>
> > Greetings!
> >
> >
> > We would like to start an open discussion on bringing Linkis (
> > https://github.com/WeBankFinTech/Linkis), a computation middleware
> > project, to the Apache Incubator.
> > The proposal can be found below and is also listed in the Incubator wiki:
> > https://cwiki.apache.org/confluence/display/INCUBATOR/LinkisProposal,
> > thanks @Junping Du for creating the page!
> > We appreciate anyone who would give guidance or be willing to support us
> a
> > an additional mentor.
> > ==
> > Linkis Proposal
> >
> >
> > =Abstract=
> > Linkis builds a computation middleware layer to decouple the upper
> > applications and the underlying data engines, provides standardized
> > interfaces (REST, JDBC, WebSocket etc.) to easily connect to various
> > underlying engines (Spark, Presto, Flink, etc.), while enables cross
> engine
> > context sharing, unified job& engine governance and orchestration.
> > Linkis codebase: https://github.com/WeBankFinTech/Linkis
> >
> >
> > =Proposal=
> > Linkis is designed to solve computation governance problems in complex
> > distributed environments (typically in a big data platform), where you
> have
> > to deal with different types, versions, or clusters of underlying data
> > engines and hundreds of diversified engine clients at the upper
> application
> > layer as well.
> > Linkis acts as a proxy between the upper applications layer and
> underlying
> > engines layer. By abstracting and implementing the 3 common phases of a
> > job/request for submit, prepare and execute, Linkis is able to facilitate
> > the connectivity, governance and orchestration capabilities of different
> > kind of engines like OLAP, OLTP (developing), Streaming, and handle all
> > these "computation governance" affairs in a standardized reusable way.
> > We are actively operating the Linkis community and we are looking forward
> > to increase community activity continuously.We propose to contribute the
> > Linkis codebase to the Apache Software Foundation. We believe that
> bringing
> > Linkis into Apache Software Foundation and following the COMMUNITY-LED
> > DEVELOPMENT "APACHE WAY" could continuously improve project quality and
> > community vitality.
> >
> >
> > =Background=
> > In today's complex and distributed environment, the communication,
> > coordination and governance of application services have developed mature
> > solutions from SOA to micro-services, and many practices from ESB to
> > Service Mesh to decouple different services.
> > However, things go different while an application service needs to
> > communicate with the underlying engines. Engines are isolated from each
> > other, and the client-server tight coupling pattern goes everywhere. Each
> > and every upper application has to directly connect to and access various
> > underlying engines in a tightly coupled way, and solves the "computation
> > governance" problems on its own, including maintaining different client
> > environments, submiting the job, monitoring job status, fetching the
> > output, handling large number of concurrent client instances, watching
> the
> > bad jobs, adapt to engine version changes, etc.
> > It lacks a common layer of "computation middleware" between the numerous
> > upper-layer applications and the countless underlying engines to handle
> all
> > these "computation governance" affairs in a standardized reusable way,
> > that's why we started the Linkis project.
> > Firstly, Linkis could reduce the complexity of connectivity. Instead of
> > maintaining a variety of engine client environments, users now only need
> to
> > install the Linkis client, or even just HTTP client while using the REST
> > interface. Routing query to desired clusters could be done by simply
> > providing a tag.
> > Secondly, Links provides governance capabilities such as multi-tenancy,
> > concurrency control, resource management, query validation, privilege
> > enhancement and auditing.
> > Meanwhile, Linkis enables orchestration strategies such as routing,
> > load-balance, active-active and hybrid computation across engines (some
> > still under development).
> >
> >
> >

Re: [DISCUSS] Linkis Proposal (a Computation Middleware project)

2021-07-12 Thread 俊平堵 
+1.
Linkis is an interesting project that builds a unified layer to decouple
the upper layer computation/application and under layer data engines, such
as Spark, Presto, Flink, etc. Also it looks already build a community
around the project.
Good luck!


Shuai Di 于2021年7月12日 周一上午11:13写道：

> Greetings!
>
>
> We would like to start an open discussion on bringing Linkis (
> https://github.com/WeBankFinTech/Linkis), a computation middleware
> project, to the Apache Incubator.
> The proposal can be found below and is also listed in the Incubator wiki:
> https://cwiki.apache.org/confluence/display/INCUBATOR/LinkisProposal,
> thanks @Junping Du for creating the page!
> We appreciate anyone who would give guidance or be willing to support us a
> an additional mentor.
> ==
> Linkis Proposal
>
>
> =Abstract=
> Linkis builds a computation middleware layer to decouple the upper
> applications and the underlying data engines, provides standardized
> interfaces (REST, JDBC, WebSocket etc.) to easily connect to various
> underlying engines (Spark, Presto, Flink, etc.), while enables cross engine
> context sharing, unified job& engine governance and orchestration.
> Linkis codebase: https://github.com/WeBankFinTech/Linkis
>
>
> =Proposal=
> Linkis is designed to solve computation governance problems in complex
> distributed environments (typically in a big data platform), where you have
> to deal with different types, versions, or clusters of underlying data
> engines and hundreds of diversified engine clients at the upper application
> layer as well.
> Linkis acts as a proxy between the upper applications layer and underlying
> engines layer. By abstracting and implementing the 3 common phases of a
> job/request for submit, prepare and execute, Linkis is able to facilitate
> the connectivity, governance and orchestration capabilities of different
> kind of engines like OLAP, OLTP (developing), Streaming, and handle all
> these "computation governance" affairs in a standardized reusable way.
> We are actively operating the Linkis community and we are looking forward
> to increase community activity continuously.We propose to contribute the
> Linkis codebase to the Apache Software Foundation. We believe that bringing
> Linkis into Apache Software Foundation and following the COMMUNITY-LED
> DEVELOPMENT "APACHE WAY" could continuously improve project quality and
> community vitality.
>
>
> =Background=
> In today's complex and distributed environment, the communication,
> coordination and governance of application services have developed mature
> solutions from SOA to micro-services, and many practices from ESB to
> Service Mesh to decouple different services.
> However, things go different while an application service needs to
> communicate with the underlying engines. Engines are isolated from each
> other, and the client-server tight coupling pattern goes everywhere. Each
> and every upper application has to directly connect to and access various
> underlying engines in a tightly coupled way, and solves the "computation
> governance" problems on its own, including maintaining different client
> environments, submiting the job, monitoring job status, fetching the
> output, handling large number of concurrent client instances, watching the
> bad jobs, adapt to engine version changes, etc.
> It lacks a common layer of "computation middleware" between the numerous
> upper-layer applications and the countless underlying engines to handle all
> these "computation governance" affairs in a standardized reusable way,
> that's why we started the Linkis project.
> Firstly, Linkis could reduce the complexity of connectivity. Instead of
> maintaining a variety of engine client environments, users now only need to
> install the Linkis client, or even just HTTP client while using the REST
> interface. Routing query to desired clusters could be done by simply
> providing a tag.
> Secondly, Links provides governance capabilities such as multi-tenancy,
> concurrency control, resource management, query validation, privilege
> enhancement and auditing.
> Meanwhile, Linkis enables orchestration strategies such as routing,
> load-balance, active-active and hybrid computation across engines (some
> still under development).
>
>
> =Rationale=
> Linkis is built on distributed microservice architecture with great
> scalability and extendibility. The enhancements of high concurrency and
> fault tolerance make it more stable and reliable. It has already supported
> many production environments with large number of daily jobs over a long
> term.
> Linkis's microservices are divided into 3 groups: Computation Governance
> Services, Public Enhancement Services, and Microservice Governance Services.
> Computation Governance Services(CGS) group is responsible for the core
> process of job/request submission, preparation and execution, lifecycle
> management, resource management, validation and orchestration.
> Public enhancement Services(PES) group

[DISCUSS] Linkis Proposal (a Computation Middleware project)

2021-07-11 Thread Shuai Di 
Greetings!


We would like to start an open discussion on bringing Linkis 
(https://github.com/WeBankFinTech/Linkis), a computation middleware project, to 
the Apache Incubator.
The proposal can be found below and is also listed in the Incubator wiki: 
https://cwiki.apache.org/confluence/display/INCUBATOR/LinkisProposal, thanks 
@Junping Du for creating the page!
We appreciate anyone who would give guidance or be willing to support us a an 
additional mentor.
==
Linkis Proposal


=Abstract=
Linkis builds a computation middleware layer to decouple the upper applications 
and the underlying data engines, provides standardized interfaces (REST, JDBC, 
WebSocket etc.) to easily connect to various underlying engines (Spark, Presto, 
Flink, etc.), while enables cross engine context sharing, unified job& engine 
governance and orchestration.
Linkis codebase: https://github.com/WeBankFinTech/Linkis


=Proposal=
Linkis is designed to solve computation governance problems in complex 
distributed environments (typically in a big data platform), where you have to 
deal with different types, versions, or clusters of underlying data engines and 
hundreds of diversified engine clients at the upper application layer as well.
Linkis acts as a proxy between the upper applications layer and underlying 
engines layer. By abstracting and implementing the 3 common phases of a 
job/request for submit, prepare and execute, Linkis is able to facilitate the 
connectivity, governance and orchestration capabilities of different kind of 
engines like OLAP, OLTP (developing), Streaming, and handle all these 
"computation governance" affairs in a standardized reusable way.
We are actively operating the Linkis community and we are looking forward to 
increase community activity continuously.We propose to contribute the Linkis 
codebase to the Apache Software Foundation. We believe that bringing Linkis 
into Apache Software Foundation and following the COMMUNITY-LED DEVELOPMENT 
"APACHE WAY" could continuously improve project quality and community vitality.


=Background=
In today's complex and distributed environment, the communication, coordination 
and governance of application services have developed mature solutions from SOA 
to micro-services, and many practices from ESB to Service Mesh to decouple 
different services.
However, things go different while an application service needs to communicate 
with the underlying engines. Engines are isolated from each other, and the 
client-server tight coupling pattern goes everywhere. Each and every upper 
application has to directly connect to and access various underlying engines in 
a tightly coupled way, and solves the "computation governance" problems on its 
own, including maintaining different client environments, submiting the job, 
monitoring job status, fetching the output, handling large number of concurrent 
client instances, watching the bad jobs, adapt to engine version changes, etc.
It lacks a common layer of "computation middleware" between the numerous 
upper-layer applications and the countless underlying engines to handle all 
these "computation governance" affairs in a standardized reusable way, that's 
why we started the Linkis project.
Firstly, Linkis could reduce the complexity of connectivity. Instead of 
maintaining a variety of engine client environments, users now only need to 
install the Linkis client, or even just HTTP client while using the REST 
interface. Routing query to desired clusters could be done by simply providing 
a tag.
Secondly, Links provides governance capabilities such as multi-tenancy, 
concurrency control, resource management, query validation, privilege 
enhancement and auditing.
Meanwhile, Linkis enables orchestration strategies such as routing, 
load-balance, active-active and hybrid computation across engines (some still 
under development).


=Rationale=
Linkis is built on distributed microservice architecture with great scalability 
and extendibility. The enhancements of high concurrency and fault tolerance 
make it more stable and reliable. It has already supported many production 
environments with large number of daily jobs over a long term.
Linkis's microservices are divided into 3 groups: Computation Governance 
Services, Public Enhancement Services, and Microservice Governance Services.
Computation Governance Services(CGS) group is responsible for the core process 
of job/request submission, preparation and execution, lifecycle management, 
resource management, validation and orchestration.
Public enhancement Services(PES) group provides basic public functions 
including job context sharing, material management and data source management, 
to serve other Linkis services and upper application systems.
Microservice Governance Services(MGS) group includes customized Spring Cloud 
Gateway, Eureka and Open Feign, to provide basic functions like routing, 
service registration and discovery, and RPC framework.
By providing

Re:[DISCUSS] Linkis Proposal (a Computation Middleware project)

2021-07-11 Thread Shuai Di 
Sorry about the format problem, will resend later.



在 2021-07-12 10:21:47，"Shuai Di"  写道：

Greetings!


We would like to start an open discussion on bringing Linkis 
(https://github.com/WeBankFinTech/Linkis), a computation middleware project, to 
the Apache Incubator.


The proposal can be found below and is also listed in the Incubator wiki: 
https://cwiki.apache.org/confluence/display/INCUBATOR/LinkisProposal, thanks 
@Junping Du for creating the page! We appreciate anyone who would give guidance 
or be willing to support us a an additional mentor.


==
Linkis Proposal




=Abstract=

Linkis builds a computation middleware layer to decouple the upper applications 
and the underlying data engines, provides standardized interfaces (REST, JDBC, 
WebSocket etc.) to easily connect to various underlying engines (Spark, Presto, 
Flink, etc.), while enables cross engine context sharing, unified job& engine 
governance and orchestration.

Linkis codebase: https://github.com/WeBankFinTech/Linkis




=Proposal=

Linkis is designed to solve computation governance problems in complex 
distributed environments (typically in a big data platform), where you have to 
deal with different types, versions, or clusters of underlying data engines and 
hundreds of diversified engine clients at the upper application layer as well.

Linkis acts as a proxy between the upper applications layer and underlying 
engines layer. By abstracting and implementing the 3 common phases of a 
job/request for submit, prepare and execute, Linkis is able to facilitate the 
connectivity, governance and orchestration capabilities of different kind of 
engines like OLAP, OLTP (developing), Streaming, and handle all these 
"computation governance" affairs in a standardized reusable way.

We are actively operating the Linkis community and we are looking forward to 
increase community activity continuously.

We propose to contribute the Linkis codebase to the Apache Software Foundation. 
We believe that bringing Linkis into Apache Software Foundation and following 
the COMMUNITY-LED DEVELOPMENT "APACHE WAY" could continuously improve project 
quality and community vitality.




=Background=

In today's complex and distributed environment, the communication, coordination 
and governance of application services have developed mature solutions from SOA 
to micro-services, and many practices from ESB to Service Mesh to decouple 
different services.

However, things go different while an application service needs to communicate 
with the underlying engines. Engines are isolated from each other, and the 
client-server tight coupling pattern goes everywhere. Each and every upper 
application has to directly connect to and access various underlying engines in 
a tightly coupled way, and solves the "computation governance" problems on its 
own, including maintaining different client environments, submiting the job, 
monitoring job status, fetching the output, handling large number of concurrent 
client instances, watching the bad jobs, adapt to engine version changes, etc.

It lacks a common layer of "computation middleware" between the numerous 
upper-layer applications and the countless underlying engines to handle all 
these "computation governance" affairs in a standardized reusable way, that's 
why we started the Linkis project.

Firstly, Linkis could reduce the complexity of connectivity. Instead of 
maintaining a variety of engine client environments, users now only need to 
install the Linkis client, or even just HTTP client while using the REST 
interface. Routing query to desired clusters could be done by simply providing 
a tag.

Secondly, Links provides governance capabilities such as multi-tenancy, 
concurrency control, resource management, query validation, privilege 
enhancement and auditing.

Meanwhile, Linkis enables orchestration strategies such as routing, 
load-balance, active-active and hybrid computation across engines (some still 
under development).




=Rationale=

Linkis is built on distributed microservice architecture with great scalability 
and extendibility. The enhancements of high concurrency and fault tolerance 
make it more stable and reliable. It has already supported many production 
environments with large number of daily jobs over a long term.

Linkis's microservices are divided into 3 groups: Computation Governance 
Services, Public Enhancement Services, and Microservice Governance Services.

Computation Governance Services(CGS) group is responsible for the core process 
of job/request submission, preparation and execution, lifecycle management, 
resource management, validation and orchestration.

Public enhancement Services(PES) group provides basic public functions 
including job context sharing, material management and data source management, 
to serve other Linkis services and upper application systems.

Microservice Governance Services(MGS) group includes customized Spring Cloud 
Gateway, Eureka and Open

[DISCUSS] Linkis Proposal (a Computation Middleware project)

2021-07-11 Thread Shuai Di 
Greetings!


We would like to start an open discussion on bringing Linkis 
(https://github.com/WeBankFinTech/Linkis), a computation middleware project, to 
the Apache Incubator.


The proposal can be found below and is also listed in the Incubator wiki: 
https://cwiki.apache.org/confluence/display/INCUBATOR/LinkisProposal, thanks 
@Junping Du for creating the page! We appreciate anyone who would give guidance 
or be willing to support us a an additional mentor.


==
Linkis Proposal




=Abstract=

Linkis builds a computation middleware layer to decouple the upper applications 
and the underlying data engines, provides standardized interfaces (REST, JDBC, 
WebSocket etc.) to easily connect to various underlying engines (Spark, Presto, 
Flink, etc.), while enables cross engine context sharing, unified job& engine 
governance and orchestration.

Linkis codebase: https://github.com/WeBankFinTech/Linkis




=Proposal=

Linkis is designed to solve computation governance problems in complex 
distributed environments (typically in a big data platform), where you have to 
deal with different types, versions, or clusters of underlying data engines and 
hundreds of diversified engine clients at the upper application layer as well.

Linkis acts as a proxy between the upper applications layer and underlying 
engines layer. By abstracting and implementing the 3 common phases of a 
job/request for submit, prepare and execute, Linkis is able to facilitate the 
connectivity, governance and orchestration capabilities of different kind of 
engines like OLAP, OLTP (developing), Streaming, and handle all these 
"computation governance" affairs in a standardized reusable way.

We are actively operating the Linkis community and we are looking forward to 
increase community activity continuously.

We propose to contribute the Linkis codebase to the Apache Software Foundation. 
We believe that bringing Linkis into Apache Software Foundation and following 
the COMMUNITY-LED DEVELOPMENT "APACHE WAY" could continuously improve project 
quality and community vitality.




=Background=

In today's complex and distributed environment, the communication, coordination 
and governance of application services have developed mature solutions from SOA 
to micro-services, and many practices from ESB to Service Mesh to decouple 
different services.

However, things go different while an application service needs to communicate 
with the underlying engines. Engines are isolated from each other, and the 
client-server tight coupling pattern goes everywhere. Each and every upper 
application has to directly connect to and access various underlying engines in 
a tightly coupled way, and solves the "computation governance" problems on its 
own, including maintaining different client environments, submiting the job, 
monitoring job status, fetching the output, handling large number of concurrent 
client instances, watching the bad jobs, adapt to engine version changes, etc.

It lacks a common layer of "computation middleware" between the numerous 
upper-layer applications and the countless underlying engines to handle all 
these "computation governance" affairs in a standardized reusable way, that's 
why we started the Linkis project.

Firstly, Linkis could reduce the complexity of connectivity. Instead of 
maintaining a variety of engine client environments, users now only need to 
install the Linkis client, or even just HTTP client while using the REST 
interface. Routing query to desired clusters could be done by simply providing 
a tag.

Secondly, Links provides governance capabilities such as multi-tenancy, 
concurrency control, resource management, query validation, privilege 
enhancement and auditing.

Meanwhile, Linkis enables orchestration strategies such as routing, 
load-balance, active-active and hybrid computation across engines (some still 
under development).




=Rationale=

Linkis is built on distributed microservice architecture with great scalability 
and extendibility. The enhancements of high concurrency and fault tolerance 
make it more stable and reliable. It has already supported many production 
environments with large number of daily jobs over a long term.

Linkis's microservices are divided into 3 groups: Computation Governance 
Services, Public Enhancement Services, and Microservice Governance Services.

Computation Governance Services(CGS) group is responsible for the core process 
of job/request submission, preparation and execution, lifecycle management, 
resource management, validation and orchestration.

Public enhancement Services(PES) group provides basic public functions 
including job context sharing, material management and data source management, 
to serve other Linkis services and upper application systems.

Microservice Governance Services(MGS) group includes customized Spring Cloud 
Gateway, Eureka and Open Feign, to provide basic functions like routing, 
service registration and discovery, and RPC