Re: [DISCUSS] Gluten proposal

[tan zhongyi]

Cool.

Can't wait to see this project to be an ASF project.

在 2024/1/4 10:12，“ShaoFeng Shi”mailto:shaofeng...@apache.org>> 写入:


After discuss with the project team, we invited Felix Cheung into the
mentor list (he is the first one expressed the willing for mentoring), and
the proposal has been updated [1]; so far we have 5 mentors, we think that
is a reasonable number for entering incubator. Thanks also to all others
that have chimed in expressing their support.


As this discussion has lasted 13 days which is a little bit long, I will
start the vote shortly.


[1] https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal 



Best regards,


Shaofeng Shi 史少锋
Apache Kylin PMC member,
Apache Incubator PMC member,
Email: shaofeng...@apache.org 


Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html 

Join Kylin user mail group: user-subscr...@kylin.apache.org 

Join Kylin dev mail group: dev-subscr...@kylin.apache.org 











ShaoFeng Shi mailto:shaofeng...@apache.org>> 
于2023年12月22日周五 20:58写道：


> Hi IPMC members,
>
> I would like to propose a new project to the ASF incubator - Gluten.
>
> Gluten[1] is a middle layer responsible for offloading Apache Spark SQL
> queries to native engines. This project aims to address the CPU
> computational bottleneck to offload SparkSQL operators to native engines in
> data loading scenarios based on Apache Spark.
>
> Here is the proposal -
> https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal 
> 
>
> I would be the Champion of the project. I will mentor and help the project
> through the incubator with Yu Li [l...@apache.org ], 
> Kent Yao [
> y...@apache.org ] and Wenli Zhang [ovi...@apache.org 
> ] .
>
> We are open to hearing the feedback from the incubator.
>
> Best,
> Shaofeng Shi.
>
> [1] https://oap-project.github.io/gluten/ 
> 
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofeng...@apache.org 
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html 
> 
> Join Kylin user mail group: user-subscr...@kylin.apache.org 
> 
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org 
> 
>
>
>

Re: [DISCUSS] Gluten proposal

[ShaoFeng Shi]

After discuss with the project team, we invited Felix Cheung into the
mentor list (he is the first one expressed the willing for mentoring), and
the proposal has been updated [1]; so far we have 5 mentors, we think that
is a reasonable number for entering incubator. Thanks also to all others
that have chimed in expressing their support.

As this discussion has lasted 13 days which is a little bit long, I will
start the vote shortly.

[1] https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC member,
Apache Incubator PMC member,
Email: shaofeng...@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org





ShaoFeng Shi  于2023年12月22日周五 20:58写道：

> Hi IPMC members,
>
> I would like to propose a new project to the ASF incubator - Gluten.
>
> Gluten[1] is a middle layer responsible for offloading Apache Spark SQL
> queries to native engines. This project aims to address the CPU
> computational bottleneck to offload SparkSQL operators to native engines in
> data loading scenarios based on Apache Spark.
>
> Here is the proposal -
> https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
>
> I would be the Champion of the project. I will mentor and help the project
> through the incubator with Yu Li [l...@apache.org], Kent Yao [
> y...@apache.org] and Wenli Zhang [ovi...@apache.org] .
>
> We are open to hearing the feedback from the incubator.
>
> Best,
> Shaofeng Shi.
>
> [1] https://oap-project.github.io/gluten/
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>
>

Re: [DISCUSS] Gluten proposal

[ShaoFeng Shi]

Hi Kent,

Thank you for the information from NetEase. As I know, Kyuubi + Spark +
Gluten can be a typical high performance Lakehouse. Glad to see more
innovations appears among them.

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofeng...@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org




Kent Yao  于2023年12月28日周四 12:34写道：

> +1
>
> We have been using Glutin with Apache Kyuubi and Apache Spark
> at NetEase for about a year now. It's a promising project.
>
> And I'm glad to be one of the nominated mentors.
>
> Thanks,
> Kent Yao
>
> On 2023/12/28 03:25:56 ShaoFeng Shi wrote:
> > Hi Justin,
> >
> > Yeah, 12 months is a little tough; I have discussed this with the team,
> and
> > updated the proposal to 18 months.
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC,
> > Apache Incubator PMC,
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
> >
> >
> >
> > Justin Mclean  于2023年12月27日周三 13:53写道：
> >
> > > Hi,
> > >
> > > Looks like a good proposal. Only one thing stood out to me, while not
> > > impossible, planning to graduate in a year might not be easily
> achievable.
> > >
> > > Kind Regards,
> > > Justn
> > > -
> > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > For additional commands, e-mail: general-h...@incubator.apache.org
> > >
> > >
> >
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSS] Gluten proposal

[ShaoFeng Shi]

Hi Rakesh,

Thank you for the comment! The Gluten community welcome everyone. I will
discuss the mentorship with the project members, and let you know if need
more mentors.

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofeng...@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org




Rakesh Radhakrishnan  于2023年12月29日周五 23:05写道：

> +1
>
> Thank you Shaofeng for the details.
>
> I've connected with my colleagues at Microsoft and the project roadmap is
> super exciting to me. As my colleagues are part of various Apache projects
> Spark, Hadoop, Flink, Trino projects, I believe I can provide insights and
> add diversity to this project.
>
> Happy to be a mentor on this proposal during the incubation process, if you
> are interested.
>
> Thanks,
> Rakesh
>
> On Thu, Dec 28, 2023 at 6:47 AM ShaoFeng Shi 
> wrote:
>
> > Hi Rakesh,
> >
> > Yeah, the team aware that, they plan to merge the Trino plugin into the
> > main repository at some point of time, depends on its maturity. The Hive
> > plugin is not on current roadmap I think, but this is open for
> discussion.
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC,
> > Apache Incubator PMC,
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
> >
> >
> >
> > Rakesh Radhakrishnan  于2023年12月27日周三 14:06写道：
> >
> > > OK, that's really interesting. I hope you will add  "Gluten-Trino
> > > " and "Gluten-Flink" as
> > > sub-projects(plugin module) under the main Gluten project and follow a
> > > single release cycle?
> > >
> > > Is it technically/architecturally feasible to develop a "Gluten-Hive"
> > > plugin (as we know Hive uses Apache Tez DAG) as a long term plan ?
> > >
> > > Thanks,
> > > Rakesh
> > >
> > > On Tue, Dec 26, 2023 at 4:30 PM ShaoFeng Shi 
> > > wrote:
> > >
> > > > Hi Rakesh,
> > > >
> > > > Thanks for your comment. The Trino plugin is under PoC, not ready for
> > > > widely use at this moment, so it is staged in another git repository:
> > > > https://github.com/oap-project/Gluten-Trino
> > > > Flink support is still in early discussion stage, need more
> > collaboration
> > > > in the future.
> > > >
> > > > Best regards,
> > > >
> > > > Shaofeng Shi 史少锋
> > > > Apache Kylin PMC,
> > > > Apache Incubator PMC,
> > > > Email: shaofeng...@apache.org
> > > >
> > > > Apache Kylin FAQ:
> > https://kylin.apache.org/docs/gettingstarted/faq.html
> > > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > > >
> > > >
> > > >
> > > >
> > > > Rakesh Radhakrishnan  于2023年12月26日周二 11:24写道：
> > > >
> > > > > Thanks Shaofeng and team for the efforts!
> > > > >
> > > > > Its really interesting to see that Flink and Trino projects can get
> > the
> > > > > benefits of Gluten and boost their performance. Much appreciated,
> if
> > > you
> > > > > could share more on this integration, perf_numbers,
> > > > pullrequest_reference,
> > > > > jira_reference etc. Will Flink and Trino already have a plugin
> > > mechanism
> > > > in
> > > > > place to seamlessly integrate with Gluten ?
> > > > >
> > > > > Thanks,
> > > > > Rakesh
> > > > >
> > > > > On Sat, Dec 23, 2023 at 6:13 PM ShaoFeng Shi <
> shaofeng...@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Enrico,
> > > > > >
> > > > > > This is a good question. Actually I asked the same to the team
> when
> > > > they
> > > > > > find me for coaching.
> > > > > >
> > > > > > As the proposal mentioned, Gluten is a middle layer between the
> > > > JVM-based
> > > > > > SQL engine and native libraries. It connects the two sides, and
> > build
> > > > > > common services like unified plan transformation, seamless native
> > > > > > integration, clear JNI interfaces, etc. Spark is the first engine
> > it
> > > > > > supports now, but the framework and mechanism can also work for
> > other
> > > > > > engines like Trino, Flink SQL in the future (the proposal also
> > > > mentioned
> > > > > > this). Underlying Gluten, it can integrate with different
> > libararies,
> > > > > like
> > > > > > Velox, Clickhouse and Arrow (not ready yet). So, its mission and
> > > scope
> > > > is
> > > > > > different, that's why we put it as a new project.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Shaofeng Shi 史少锋
> > > > > > Apache Kylin PMC,
> > > > > > Apache Incubator PMC,
> > > > > > Email: shaofeng...@apache.org
> > > > > >
> > > > > > Apache Kylin FAQ:
> > > > https://kylin.apache.org/docs/gettingstarted/faq.html
> > > > > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > > > > Join 

Re: [DISCUSS] Gluten proposal

[ShaoFeng Shi]

Yu, thank you for the information. Glad to hear the feedback, and expect to
see the collaboration with Flink in the new year!

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofeng...@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org




Yu Li  于2023年12月29日周五 19:37写道：

> +1, and I'm glad to be one of the nominated mentors.
>
> My team members at Alibaba deeply joined in the development of Gluten and
> Gluten-Trino, so I'm well aware that it's a diverse and open community, and
> really glad that the Gluten community decides to join the incubator.
>
> AFAIK, with Gluten-Trino plugin we could have some 50%~100% performance
> gain, and I believe it could further improve after joining the Gluten main
> branch later (not yet since that requires quite some code refactor, if I
> understand it correctly, but that's in the plan).
>
> About Flink plugin, I'm thrilled to see how things will develop, with my
> Flink PMC Member hat on (smile).
>
> Good luck team, and Happy New Year all!
>
> Best Regards,
> Yu
>
>
> On Thu, 28 Dec 2023 at 12:34, Kent Yao  wrote:
>
> > +1
> >
> > We have been using Glutin with Apache Kyuubi and Apache Spark
> > at NetEase for about a year now. It's a promising project.
> >
> > And I'm glad to be one of the nominated mentors.
> >
> > Thanks,
> > Kent Yao
> >
> > On 2023/12/28 03:25:56 ShaoFeng Shi wrote:
> > > Hi Justin,
> > >
> > > Yeah, 12 months is a little tough; I have discussed this with the team,
> > and
> > > updated the proposal to 18 months.
> > >
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > > Apache Kylin PMC,
> > > Apache Incubator PMC,
> > > Email: shaofeng...@apache.org
> > >
> > > Apache Kylin FAQ:
> https://kylin.apache.org/docs/gettingstarted/faq.html
> > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > >
> > >
> > >
> > >
> > > Justin Mclean  于2023年12月27日周三 13:53写道：
> > >
> > > > Hi,
> > > >
> > > > Looks like a good proposal. Only one thing stood out to me, while not
> > > > impossible, planning to graduate in a year might not be easily
> > achievable.
> > > >
> > > > Kind Regards,
> > > > Justn
> > > > -
> > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > > For additional commands, e-mail: general-h...@incubator.apache.org
> > > >
> > > >
> > >
> >
> > -
> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > For additional commands, e-mail: general-h...@incubator.apache.org
> >
> >
>

Re: [DISCUSS] Gluten proposal

[Rakesh Radhakrishnan]

+1

Thank you Shaofeng for the details.

I've connected with my colleagues at Microsoft and the project roadmap is
super exciting to me. As my colleagues are part of various Apache projects
Spark, Hadoop, Flink, Trino projects, I believe I can provide insights and
add diversity to this project.

Happy to be a mentor on this proposal during the incubation process, if you
are interested.

Thanks,
Rakesh

On Thu, Dec 28, 2023 at 6:47 AM ShaoFeng Shi  wrote:

> Hi Rakesh,
>
> Yeah, the team aware that, they plan to merge the Trino plugin into the
> main repository at some point of time, depends on its maturity. The Hive
> plugin is not on current roadmap I think, but this is open for discussion.
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>
>
>
> Rakesh Radhakrishnan  于2023年12月27日周三 14:06写道：
>
> > OK, that's really interesting. I hope you will add  "Gluten-Trino
> > " and "Gluten-Flink" as
> > sub-projects(plugin module) under the main Gluten project and follow a
> > single release cycle?
> >
> > Is it technically/architecturally feasible to develop a "Gluten-Hive"
> > plugin (as we know Hive uses Apache Tez DAG) as a long term plan ?
> >
> > Thanks,
> > Rakesh
> >
> > On Tue, Dec 26, 2023 at 4:30 PM ShaoFeng Shi 
> > wrote:
> >
> > > Hi Rakesh,
> > >
> > > Thanks for your comment. The Trino plugin is under PoC, not ready for
> > > widely use at this moment, so it is staged in another git repository:
> > > https://github.com/oap-project/Gluten-Trino
> > > Flink support is still in early discussion stage, need more
> collaboration
> > > in the future.
> > >
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > > Apache Kylin PMC,
> > > Apache Incubator PMC,
> > > Email: shaofeng...@apache.org
> > >
> > > Apache Kylin FAQ:
> https://kylin.apache.org/docs/gettingstarted/faq.html
> > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > >
> > >
> > >
> > >
> > > Rakesh Radhakrishnan  于2023年12月26日周二 11:24写道：
> > >
> > > > Thanks Shaofeng and team for the efforts!
> > > >
> > > > Its really interesting to see that Flink and Trino projects can get
> the
> > > > benefits of Gluten and boost their performance. Much appreciated, if
> > you
> > > > could share more on this integration, perf_numbers,
> > > pullrequest_reference,
> > > > jira_reference etc. Will Flink and Trino already have a plugin
> > mechanism
> > > in
> > > > place to seamlessly integrate with Gluten ?
> > > >
> > > > Thanks,
> > > > Rakesh
> > > >
> > > > On Sat, Dec 23, 2023 at 6:13 PM ShaoFeng Shi  >
> > > > wrote:
> > > >
> > > > > Hi Enrico,
> > > > >
> > > > > This is a good question. Actually I asked the same to the team when
> > > they
> > > > > find me for coaching.
> > > > >
> > > > > As the proposal mentioned, Gluten is a middle layer between the
> > > JVM-based
> > > > > SQL engine and native libraries. It connects the two sides, and
> build
> > > > > common services like unified plan transformation, seamless native
> > > > > integration, clear JNI interfaces, etc. Spark is the first engine
> it
> > > > > supports now, but the framework and mechanism can also work for
> other
> > > > > engines like Trino, Flink SQL in the future (the proposal also
> > > mentioned
> > > > > this). Underlying Gluten, it can integrate with different
> libararies,
> > > > like
> > > > > Velox, Clickhouse and Arrow (not ready yet). So, its mission and
> > scope
> > > is
> > > > > different, that's why we put it as a new project.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Shaofeng Shi 史少锋
> > > > > Apache Kylin PMC,
> > > > > Apache Incubator PMC,
> > > > > Email: shaofeng...@apache.org
> > > > >
> > > > > Apache Kylin FAQ:
> > > https://kylin.apache.org/docs/gettingstarted/faq.html
> > > > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Enrico Olivelli  于2023年12月22日周五 22:43写道：
> > > > >
> > > > > > Shaofeng,
> > > > > >
> > > > > > Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi
> > > > > >  ha scritto:
> > > > > > >
> > > > > > > Hi IPMC members,
> > > > > > >
> > > > > > > I would like to propose a new project to the ASF incubator -
> > > Gluten.
> > > > > > >
> > > > > > > Gluten[1] is a middle layer responsible for offloading Apache
> > Spark
> > > > SQL
> > > > > > > queries to native engines. This project aims to address the CPU
> > > > > > > computational bottleneck to offload SparkSQL operators to
> native
> > > > > engines
> > > > > > in
> > > > > > > data loading scenarios based on Apache Spark.
> > > > > >
> > > > > > 

Re: [DISCUSS] Gluten proposal

[Yu Li]

+1, and I'm glad to be one of the nominated mentors.

My team members at Alibaba deeply joined in the development of Gluten and
Gluten-Trino, so I'm well aware that it's a diverse and open community, and
really glad that the Gluten community decides to join the incubator.

AFAIK, with Gluten-Trino plugin we could have some 50%~100% performance
gain, and I believe it could further improve after joining the Gluten main
branch later (not yet since that requires quite some code refactor, if I
understand it correctly, but that's in the plan).

About Flink plugin, I'm thrilled to see how things will develop, with my
Flink PMC Member hat on (smile).

Good luck team, and Happy New Year all!

Best Regards,
Yu


On Thu, 28 Dec 2023 at 12:34, Kent Yao  wrote:

> +1
>
> We have been using Glutin with Apache Kyuubi and Apache Spark
> at NetEase for about a year now. It's a promising project.
>
> And I'm glad to be one of the nominated mentors.
>
> Thanks,
> Kent Yao
>
> On 2023/12/28 03:25:56 ShaoFeng Shi wrote:
> > Hi Justin,
> >
> > Yeah, 12 months is a little tough; I have discussed this with the team,
> and
> > updated the proposal to 18 months.
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC,
> > Apache Incubator PMC,
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
> >
> >
> >
> > Justin Mclean  于2023年12月27日周三 13:53写道：
> >
> > > Hi,
> > >
> > > Looks like a good proposal. Only one thing stood out to me, while not
> > > impossible, planning to graduate in a year might not be easily
> achievable.
> > >
> > > Kind Regards,
> > > Justn
> > > -
> > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > For additional commands, e-mail: general-h...@incubator.apache.org
> > >
> > >
> >
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSS] Gluten proposal

[Kent Yao]

+1

We have been using Glutin with Apache Kyuubi and Apache Spark 
at NetEase for about a year now. It's a promising project.

And I'm glad to be one of the nominated mentors.

Thanks,
Kent Yao

On 2023/12/28 03:25:56 ShaoFeng Shi wrote:
> Hi Justin,
> 
> Yeah, 12 months is a little tough; I have discussed this with the team, and
> updated the proposal to 18 months.
> 
> Best regards,
> 
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofeng...@apache.org
> 
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> 
> 
> 
> 
> Justin Mclean  于2023年12月27日周三 13:53写道：
> 
> > Hi,
> >
> > Looks like a good proposal. Only one thing stood out to me, while not
> > impossible, planning to graduate in a year might not be easily achievable.
> >
> > Kind Regards,
> > Justn
> > -
> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > For additional commands, e-mail: general-h...@incubator.apache.org
> >
> >
> 

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [DISCUSS] Gluten proposal

[ShaoFeng Shi]

Hi Justin,

Yeah, 12 months is a little tough; I have discussed this with the team, and
updated the proposal to 18 months.

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofeng...@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org




Justin Mclean  于2023年12月27日周三 13:53写道：

> Hi,
>
> Looks like a good proposal. Only one thing stood out to me, while not
> impossible, planning to graduate in a year might not be easily achievable.
>
> Kind Regards,
> Justn
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSS] Gluten proposal

[ShaoFeng Shi]

Hi Junping,

Thank you for the suggestion. I have discussed this with the team, they
agree with this. I have updated abstract part of the proposal. The slogan
on the document website will be updated later.

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofeng...@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org




俊平堵  于2023年12月27日周三 13:46写道：

> This is definitely an interesting project and I noticed this project from
> a very early stage.
> I saw many discussions above on if Gluten should be a subproject of Spark
> or not, and I can see the value to keep it independent just like Shaofeng's
> previous points. However, the project name or slogan - "Gluten: Plugin to
> Double SparkSQL's Performance" could be updated a bit later to get rid of
> the confusion to serve SparkSQL only.
> +1 on the project going to apache. I am interested in helping as mentor if
> needed.
>
>
> Thanks,
>
> JP
>
> ShaoFeng Shi  于2023年12月22日周五 20:59写道：
>
> > Hi IPMC members,
> >
> > I would like to propose a new project to the ASF incubator - Gluten.
> >
> > Gluten[1] is a middle layer responsible for offloading Apache Spark SQL
> > queries to native engines. This project aims to address the CPU
> > computational bottleneck to offload SparkSQL operators to native engines
> in
> > data loading scenarios based on Apache Spark.
> >
> > Here is the proposal -
> > https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
> >
> > I would be the Champion of the project. I will mentor and help the
> project
> > through the incubator with Yu Li [l...@apache.org], Kent Yao [
> > y...@apache.org]
> > and Wenli Zhang [ovi...@apache.org] .
> >
> > We are open to hearing the feedback from the incubator.
> >
> > Best,
> > Shaofeng Shi.
> >
> > [1] https://oap-project.github.io/gluten/
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC,
> > Apache Incubator PMC,
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
>

Re: [DISCUSS] Gluten proposal

[ShaoFeng Shi]

Hi Rakesh,

Yeah, the team aware that, they plan to merge the Trino plugin into the
main repository at some point of time, depends on its maturity. The Hive
plugin is not on current roadmap I think, but this is open for discussion.

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofeng...@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org




Rakesh Radhakrishnan  于2023年12月27日周三 14:06写道：

> OK, that's really interesting. I hope you will add  "Gluten-Trino
> " and "Gluten-Flink" as
> sub-projects(plugin module) under the main Gluten project and follow a
> single release cycle?
>
> Is it technically/architecturally feasible to develop a "Gluten-Hive"
> plugin (as we know Hive uses Apache Tez DAG) as a long term plan ?
>
> Thanks,
> Rakesh
>
> On Tue, Dec 26, 2023 at 4:30 PM ShaoFeng Shi 
> wrote:
>
> > Hi Rakesh,
> >
> > Thanks for your comment. The Trino plugin is under PoC, not ready for
> > widely use at this moment, so it is staged in another git repository:
> > https://github.com/oap-project/Gluten-Trino
> > Flink support is still in early discussion stage, need more collaboration
> > in the future.
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC,
> > Apache Incubator PMC,
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
> >
> >
> >
> > Rakesh Radhakrishnan  于2023年12月26日周二 11:24写道：
> >
> > > Thanks Shaofeng and team for the efforts!
> > >
> > > Its really interesting to see that Flink and Trino projects can get the
> > > benefits of Gluten and boost their performance. Much appreciated, if
> you
> > > could share more on this integration, perf_numbers,
> > pullrequest_reference,
> > > jira_reference etc. Will Flink and Trino already have a plugin
> mechanism
> > in
> > > place to seamlessly integrate with Gluten ?
> > >
> > > Thanks,
> > > Rakesh
> > >
> > > On Sat, Dec 23, 2023 at 6:13 PM ShaoFeng Shi 
> > > wrote:
> > >
> > > > Hi Enrico,
> > > >
> > > > This is a good question. Actually I asked the same to the team when
> > they
> > > > find me for coaching.
> > > >
> > > > As the proposal mentioned, Gluten is a middle layer between the
> > JVM-based
> > > > SQL engine and native libraries. It connects the two sides, and build
> > > > common services like unified plan transformation, seamless native
> > > > integration, clear JNI interfaces, etc. Spark is the first engine it
> > > > supports now, but the framework and mechanism can also work for other
> > > > engines like Trino, Flink SQL in the future (the proposal also
> > mentioned
> > > > this). Underlying Gluten, it can integrate with different libararies,
> > > like
> > > > Velox, Clickhouse and Arrow (not ready yet). So, its mission and
> scope
> > is
> > > > different, that's why we put it as a new project.
> > > >
> > > > Best regards,
> > > >
> > > > Shaofeng Shi 史少锋
> > > > Apache Kylin PMC,
> > > > Apache Incubator PMC,
> > > > Email: shaofeng...@apache.org
> > > >
> > > > Apache Kylin FAQ:
> > https://kylin.apache.org/docs/gettingstarted/faq.html
> > > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > > >
> > > >
> > > >
> > > >
> > > > Enrico Olivelli  于2023年12月22日周五 22:43写道：
> > > >
> > > > > Shaofeng,
> > > > >
> > > > > Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi
> > > > >  ha scritto:
> > > > > >
> > > > > > Hi IPMC members,
> > > > > >
> > > > > > I would like to propose a new project to the ASF incubator -
> > Gluten.
> > > > > >
> > > > > > Gluten[1] is a middle layer responsible for offloading Apache
> Spark
> > > SQL
> > > > > > queries to native engines. This project aims to address the CPU
> > > > > > computational bottleneck to offload SparkSQL operators to native
> > > > engines
> > > > > in
> > > > > > data loading scenarios based on Apache Spark.
> > > > >
> > > > > Have you considered making this project a subproject of Apache
> Spark
> > ?
> > > > >
> > > > > Enrico
> > > > >
> > > > > >
> > > > > > Here is the proposal -
> > > > > >
> > https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
> > > > > >
> > > > > > I would be the Champion of the project. I will mentor and help
> the
> > > > > project
> > > > > > through the incubator with Yu Li [l...@apache.org], Kent Yao [
> > > > > y...@apache.org]
> > > > > > and Wenli Zhang [ovi...@apache.org] .
> > > > > >
> > > > > > We are open to hearing the feedback from the incubator.
> > > > > >
> > > > > > Best,
> > > > > > Shaofeng Shi.
> > > > > >
> > > > > > [1] https://oap-project.github.io/gluten/
> > > > > >
> > > > > > Best 

Re: [DISCUSS] Gluten proposal

[Rakesh Radhakrishnan]

OK, that's really interesting. I hope you will add  "Gluten-Trino
" and "Gluten-Flink" as
sub-projects(plugin module) under the main Gluten project and follow a
single release cycle?

Is it technically/architecturally feasible to develop a "Gluten-Hive"
plugin (as we know Hive uses Apache Tez DAG) as a long term plan ?

Thanks,
Rakesh

On Tue, Dec 26, 2023 at 4:30 PM ShaoFeng Shi  wrote:

> Hi Rakesh,
>
> Thanks for your comment. The Trino plugin is under PoC, not ready for
> widely use at this moment, so it is staged in another git repository:
> https://github.com/oap-project/Gluten-Trino
> Flink support is still in early discussion stage, need more collaboration
> in the future.
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>
>
>
> Rakesh Radhakrishnan  于2023年12月26日周二 11:24写道：
>
> > Thanks Shaofeng and team for the efforts!
> >
> > Its really interesting to see that Flink and Trino projects can get the
> > benefits of Gluten and boost their performance. Much appreciated, if you
> > could share more on this integration, perf_numbers,
> pullrequest_reference,
> > jira_reference etc. Will Flink and Trino already have a plugin mechanism
> in
> > place to seamlessly integrate with Gluten ?
> >
> > Thanks,
> > Rakesh
> >
> > On Sat, Dec 23, 2023 at 6:13 PM ShaoFeng Shi 
> > wrote:
> >
> > > Hi Enrico,
> > >
> > > This is a good question. Actually I asked the same to the team when
> they
> > > find me for coaching.
> > >
> > > As the proposal mentioned, Gluten is a middle layer between the
> JVM-based
> > > SQL engine and native libraries. It connects the two sides, and build
> > > common services like unified plan transformation, seamless native
> > > integration, clear JNI interfaces, etc. Spark is the first engine it
> > > supports now, but the framework and mechanism can also work for other
> > > engines like Trino, Flink SQL in the future (the proposal also
> mentioned
> > > this). Underlying Gluten, it can integrate with different libararies,
> > like
> > > Velox, Clickhouse and Arrow (not ready yet). So, its mission and scope
> is
> > > different, that's why we put it as a new project.
> > >
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > > Apache Kylin PMC,
> > > Apache Incubator PMC,
> > > Email: shaofeng...@apache.org
> > >
> > > Apache Kylin FAQ:
> https://kylin.apache.org/docs/gettingstarted/faq.html
> > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > >
> > >
> > >
> > >
> > > Enrico Olivelli  于2023年12月22日周五 22:43写道：
> > >
> > > > Shaofeng,
> > > >
> > > > Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi
> > > >  ha scritto:
> > > > >
> > > > > Hi IPMC members,
> > > > >
> > > > > I would like to propose a new project to the ASF incubator -
> Gluten.
> > > > >
> > > > > Gluten[1] is a middle layer responsible for offloading Apache Spark
> > SQL
> > > > > queries to native engines. This project aims to address the CPU
> > > > > computational bottleneck to offload SparkSQL operators to native
> > > engines
> > > > in
> > > > > data loading scenarios based on Apache Spark.
> > > >
> > > > Have you considered making this project a subproject of Apache Spark
> ?
> > > >
> > > > Enrico
> > > >
> > > > >
> > > > > Here is the proposal -
> > > > >
> https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
> > > > >
> > > > > I would be the Champion of the project. I will mentor and help the
> > > > project
> > > > > through the incubator with Yu Li [l...@apache.org], Kent Yao [
> > > > y...@apache.org]
> > > > > and Wenli Zhang [ovi...@apache.org] .
> > > > >
> > > > > We are open to hearing the feedback from the incubator.
> > > > >
> > > > > Best,
> > > > > Shaofeng Shi.
> > > > >
> > > > > [1] https://oap-project.github.io/gluten/
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Shaofeng Shi 史少锋
> > > > > Apache Kylin PMC,
> > > > > Apache Incubator PMC,
> > > > > Email: shaofeng...@apache.org
> > > > >
> > > > > Apache Kylin FAQ:
> > > https://kylin.apache.org/docs/gettingstarted/faq.html
> > > > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > > >
> > > > -
> > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > > For additional commands, e-mail: general-h...@incubator.apache.org
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Gluten proposal

[Venkat Ranganathan]

I am currently looking into gluten as well and have been looking ap spark
acceleration in general.  Available for contribution as well as mentoring.

Thanks

Venkat

On Tue, Dec 26, 2023 at 9:53 PM Justin Mclean 
wrote:

> Hi,
>
> Looks like a good proposal. Only one thing stood out to me, while not
> impossible, planning to graduate in a year might not be easily achievable.
>
> Kind Regards,
> Justn
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

-- 
Regards

Venkat

Re: [DISCUSS] Gluten proposal

[Justin Mclean]

Hi,

Looks like a good proposal. Only one thing stood out to me, while not 
impossible, planning to graduate in a year might not be easily achievable.

Kind Regards,
Justn
-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [DISCUSS] Gluten proposal

[俊平堵]

This is definitely an interesting project and I noticed this project from
a very early stage.
I saw many discussions above on if Gluten should be a subproject of Spark
or not, and I can see the value to keep it independent just like Shaofeng's
previous points. However, the project name or slogan - "Gluten: Plugin to
Double SparkSQL's Performance" could be updated a bit later to get rid of
the confusion to serve SparkSQL only.
+1 on the project going to apache. I am interested in helping as mentor if
needed.


Thanks,

JP

ShaoFeng Shi  于2023年12月22日周五 20:59写道：

> Hi IPMC members,
>
> I would like to propose a new project to the ASF incubator - Gluten.
>
> Gluten[1] is a middle layer responsible for offloading Apache Spark SQL
> queries to native engines. This project aims to address the CPU
> computational bottleneck to offload SparkSQL operators to native engines in
> data loading scenarios based on Apache Spark.
>
> Here is the proposal -
> https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
>
> I would be the Champion of the project. I will mentor and help the project
> through the incubator with Yu Li [l...@apache.org], Kent Yao [
> y...@apache.org]
> and Wenli Zhang [ovi...@apache.org] .
>
> We are open to hearing the feedback from the incubator.
>
> Best,
> Shaofeng Shi.
>
> [1] https://oap-project.github.io/gluten/
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>

Re: [DISCUSS] Gluten proposal

[Wenli Zhang]

+1
I am delighted to see the proposal for Gluten and honored to be a mentor for 
this innovative project. I look forward to contributing my insights and helping 
guide Gluten through the incubation process. Thank you for this opportunity, 
and I am ready to support the project’s success.

Best regards,
Wenli Zhang

On 2023/12/22 12:58:24 ShaoFeng Shi wrote:
> Hi IPMC members,
> 
> I would like to propose a new project to the ASF incubator - Gluten.
> 
> Gluten[1] is a middle layer responsible for offloading Apache Spark SQL
> queries to native engines. This project aims to address the CPU
> computational bottleneck to offload SparkSQL operators to native engines in
> data loading scenarios based on Apache Spark.
> 
> Here is the proposal -
> https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
> 
> I would be the Champion of the project. I will mentor and help the project
> through the incubator with Yu Li [l...@apache.org], Kent Yao [y...@apache.org]
> and Wenli Zhang [ovi...@apache.org] .
> 
> We are open to hearing the feedback from the incubator.
> 
> Best,
> Shaofeng Shi.
> 
> [1] https://oap-project.github.io/gluten/
> 
> Best regards,
> 
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofeng...@apache.org
> 
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> 

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [DISCUSS] Gluten proposal

[ShaoFeng Shi]

Hi Rakesh,

Thanks for your comment. The Trino plugin is under PoC, not ready for
widely use at this moment, so it is staged in another git repository:
https://github.com/oap-project/Gluten-Trino
Flink support is still in early discussion stage, need more collaboration
in the future.

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofeng...@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org




Rakesh Radhakrishnan  于2023年12月26日周二 11:24写道：

> Thanks Shaofeng and team for the efforts!
>
> Its really interesting to see that Flink and Trino projects can get the
> benefits of Gluten and boost their performance. Much appreciated, if you
> could share more on this integration, perf_numbers, pullrequest_reference,
> jira_reference etc. Will Flink and Trino already have a plugin mechanism in
> place to seamlessly integrate with Gluten ?
>
> Thanks,
> Rakesh
>
> On Sat, Dec 23, 2023 at 6:13 PM ShaoFeng Shi 
> wrote:
>
> > Hi Enrico,
> >
> > This is a good question. Actually I asked the same to the team when they
> > find me for coaching.
> >
> > As the proposal mentioned, Gluten is a middle layer between the JVM-based
> > SQL engine and native libraries. It connects the two sides, and build
> > common services like unified plan transformation, seamless native
> > integration, clear JNI interfaces, etc. Spark is the first engine it
> > supports now, but the framework and mechanism can also work for other
> > engines like Trino, Flink SQL in the future (the proposal also mentioned
> > this). Underlying Gluten, it can integrate with different libararies,
> like
> > Velox, Clickhouse and Arrow (not ready yet). So, its mission and scope is
> > different, that's why we put it as a new project.
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC,
> > Apache Incubator PMC,
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
> >
> >
> >
> > Enrico Olivelli  于2023年12月22日周五 22:43写道：
> >
> > > Shaofeng,
> > >
> > > Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi
> > >  ha scritto:
> > > >
> > > > Hi IPMC members,
> > > >
> > > > I would like to propose a new project to the ASF incubator - Gluten.
> > > >
> > > > Gluten[1] is a middle layer responsible for offloading Apache Spark
> SQL
> > > > queries to native engines. This project aims to address the CPU
> > > > computational bottleneck to offload SparkSQL operators to native
> > engines
> > > in
> > > > data loading scenarios based on Apache Spark.
> > >
> > > Have you considered making this project a subproject of Apache Spark ?
> > >
> > > Enrico
> > >
> > > >
> > > > Here is the proposal -
> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
> > > >
> > > > I would be the Champion of the project. I will mentor and help the
> > > project
> > > > through the incubator with Yu Li [l...@apache.org], Kent Yao [
> > > y...@apache.org]
> > > > and Wenli Zhang [ovi...@apache.org] .
> > > >
> > > > We are open to hearing the feedback from the incubator.
> > > >
> > > > Best,
> > > > Shaofeng Shi.
> > > >
> > > > [1] https://oap-project.github.io/gluten/
> > > >
> > > > Best regards,
> > > >
> > > > Shaofeng Shi 史少锋
> > > > Apache Kylin PMC,
> > > > Apache Incubator PMC,
> > > > Email: shaofeng...@apache.org
> > > >
> > > > Apache Kylin FAQ:
> > https://kylin.apache.org/docs/gettingstarted/faq.html
> > > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > >
> > > -
> > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > For additional commands, e-mail: general-h...@incubator.apache.org
> > >
> > >
> >
>

Re: [DISCUSS] Gluten proposal

[Rakesh Radhakrishnan]

Thanks Shaofeng and team for the efforts!

Its really interesting to see that Flink and Trino projects can get the
benefits of Gluten and boost their performance. Much appreciated, if you
could share more on this integration, perf_numbers, pullrequest_reference,
jira_reference etc. Will Flink and Trino already have a plugin mechanism in
place to seamlessly integrate with Gluten ?

Thanks,
Rakesh

On Sat, Dec 23, 2023 at 6:13 PM ShaoFeng Shi  wrote:

> Hi Enrico,
>
> This is a good question. Actually I asked the same to the team when they
> find me for coaching.
>
> As the proposal mentioned, Gluten is a middle layer between the JVM-based
> SQL engine and native libraries. It connects the two sides, and build
> common services like unified plan transformation, seamless native
> integration, clear JNI interfaces, etc. Spark is the first engine it
> supports now, but the framework and mechanism can also work for other
> engines like Trino, Flink SQL in the future (the proposal also mentioned
> this). Underlying Gluten, it can integrate with different libararies, like
> Velox, Clickhouse and Arrow (not ready yet). So, its mission and scope is
> different, that's why we put it as a new project.
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>
>
>
> Enrico Olivelli  于2023年12月22日周五 22:43写道：
>
> > Shaofeng,
> >
> > Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi
> >  ha scritto:
> > >
> > > Hi IPMC members,
> > >
> > > I would like to propose a new project to the ASF incubator - Gluten.
> > >
> > > Gluten[1] is a middle layer responsible for offloading Apache Spark SQL
> > > queries to native engines. This project aims to address the CPU
> > > computational bottleneck to offload SparkSQL operators to native
> engines
> > in
> > > data loading scenarios based on Apache Spark.
> >
> > Have you considered making this project a subproject of Apache Spark ?
> >
> > Enrico
> >
> > >
> > > Here is the proposal -
> > > https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
> > >
> > > I would be the Champion of the project. I will mentor and help the
> > project
> > > through the incubator with Yu Li [l...@apache.org], Kent Yao [
> > y...@apache.org]
> > > and Wenli Zhang [ovi...@apache.org] .
> > >
> > > We are open to hearing the feedback from the incubator.
> > >
> > > Best,
> > > Shaofeng Shi.
> > >
> > > [1] https://oap-project.github.io/gluten/
> > >
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > > Apache Kylin PMC,
> > > Apache Incubator PMC,
> > > Email: shaofeng...@apache.org
> > >
> > > Apache Kylin FAQ:
> https://kylin.apache.org/docs/gettingstarted/faq.html
> > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
> > -
> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > For additional commands, e-mail: general-h...@incubator.apache.org
> >
> >
>

Re: [DISCUSS] Gluten proposal

[Chunen Ni]

I've been keeping an eye on Gluten project since its starting up. And now, I'm 
so glad to see itsshowing up in the incubating discussion list.

The aim of Gluten project is simple yet significant, with the high performance 
of native libraries, to speed up Spark Sql queries. Its architecture is neat 
and full of scalability. 

Wish to see Gluten help more users to improve their Spark Sql efficiency.

On 2023/12/22 12:58:24 ShaoFeng Shi wrote:
> Hi IPMC members,
> 
> I would like to propose a new project to the ASF incubator - Gluten.
> 
> Gluten[1] is a middle layer responsible for offloading Apache Spark SQL
> queries to native engines. This project aims to address the CPU
> computational bottleneck to offload SparkSQL operators to native engines in
> data loading scenarios based on Apache Spark.
> 
> Here is the proposal -
> https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
> 
> I would be the Champion of the project. I will mentor and help the project
> through the incubator with Yu Li [l...@apache.org], Kent Yao [y...@apache.org]
> and Wenli Zhang [ovi...@apache.org] .
> 
> We are open to hearing the feedback from the incubator.
> 
> Best,
> Shaofeng Shi.
> 
> [1] https://oap-project.github.io/gluten/
> 
> Best regards,
> 
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofeng...@apache.org
> 
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> 

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [DISCUSS] Gluten proposal

[David Zollo]

+1, Glad to see a better performance project Gluten can enter into the
incubator.  I'm very interested in high performance projects.
I hope Gluten brings more possibilities for computational acceleration.


Best Regards

---
Apache DolphinScheduler PMC Chair & Apache SeaTunnel PMC member
David
Twitter: @WorkflowEasy 
Linkedin: https://www.linkedin.com/in/davidzollo

---


On Sun, Dec 24, 2023 at 10:15 AM ShaoFeng Shi 
wrote:

> Hello Felix, so far we have 4 mentors; I will contact you if need more.
> Thank you!
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>
>
>
> Felix Cheung  于2023年12月24日周日 06:00写道：
>
> > Very interesting. Let me know if I can help as mentor
> >
> >
> >
> > On Sat, Dec 23, 2023 at 7:30 AM tison  wrote:
> >
> > > Thanks for driving this proposal Shaofeng :D
> > >
> > > +1 to support this project entering the Incuabtor. It's in good shape
> > > to develop an OSS project with a diverse group. Hopefully, the
> > > Incubator can help it evaluate whether The Apache Way fits in its
> > > community and growth.
> > >
> > > One thing to be noticed is that you're not listed on the mentor list.
> > > By definition the champion helps the candidate go through the
> > > proposing stage and is not always a mentor. If you're going to be the
> > > mentor, you should add yourself to the mentor list also. But it's OK
> > > that you can serve as a mentor.
> > >
> > > Best,
> > > tison.
> > >
> > > ShaoFeng Shi  于2023年12月23日周六 20:43写道：
> > > >
> > > > Hi Enrico,
> > > >
> > > > This is a good question. Actually I asked the same to the team when
> > they
> > > > find me for coaching.
> > > >
> > > > As the proposal mentioned, Gluten is a middle layer between the
> > JVM-based
> > > > SQL engine and native libraries. It connects the two sides, and build
> > > > common services like unified plan transformation, seamless native
> > > > integration, clear JNI interfaces, etc. Spark is the first engine it
> > > > supports now, but the framework and mechanism can also work for other
> > > > engines like Trino, Flink SQL in the future (the proposal also
> > mentioned
> > > > this). Underlying Gluten, it can integrate with different libararies,
> > > like
> > > > Velox, Clickhouse and Arrow (not ready yet). So, its mission and
> scope
> > is
> > > > different, that's why we put it as a new project.
> > > >
> > > > Best regards,
> > > >
> > > > Shaofeng Shi 史少锋
> > > > Apache Kylin PMC,
> > > > Apache Incubator PMC,
> > > > Email: shaofeng...@apache.org
> > > >
> > > > Apache Kylin FAQ:
> > https://kylin.apache.org/docs/gettingstarted/faq.html
> > > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > > >
> > > >
> > > >
> > > >
> > > > Enrico Olivelli  于2023年12月22日周五 22:43写道：
> > > >
> > > > > Shaofeng,
> > > > >
> > > > > Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi
> > > > >  ha scritto:
> > > > > >
> > > > > > Hi IPMC members,
> > > > > >
> > > > > > I would like to propose a new project to the ASF incubator -
> > Gluten.
> > > > > >
> > > > > > Gluten[1] is a middle layer responsible for offloading Apache
> Spark
> > > SQL
> > > > > > queries to native engines. This project aims to address the CPU
> > > > > > computational bottleneck to offload SparkSQL operators to native
> > > engines
> > > > > in
> > > > > > data loading scenarios based on Apache Spark.
> > > > >
> > > > > Have you considered making this project a subproject of Apache
> Spark
> > ?
> > > > >
> > > > > Enrico
> > > > >
> > > > > >
> > > > > > Here is the proposal -
> > > > > >
> > https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
> > > > > >
> > > > > > I would be the Champion of the project. I will mentor and help
> the
> > > > > project
> > > > > > through the incubator with Yu Li [l...@apache.org], Kent Yao [
> > > > > y...@apache.org]
> > > > > > and Wenli Zhang [ovi...@apache.org] .
> > > > > >
> > > > > > We are open to hearing the feedback from the incubator.
> > > > > >
> > > > > > Best,
> > > > > > Shaofeng Shi.
> > > > > >
> > > > > > [1] https://oap-project.github.io/gluten/
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Shaofeng Shi 史少锋
> > > > > > Apache Kylin PMC,
> > > > > > Apache Incubator PMC,
> > > > > > Email: shaofeng...@apache.org
> > > > > >
> > > > > > Apache Kylin FAQ:
> > > https://kylin.apache.org/docs/gettingstarted/faq.html
> > > > > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > > > >
> > > > >
> -
> > > > > To 

Re: [DISCUSS] Gluten proposal

[ShaoFeng Shi]

Hello Felix, so far we have 4 mentors; I will contact you if need more.
Thank you!

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofeng...@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org




Felix Cheung  于2023年12月24日周日 06:00写道：

> Very interesting. Let me know if I can help as mentor
>
>
>
> On Sat, Dec 23, 2023 at 7:30 AM tison  wrote:
>
> > Thanks for driving this proposal Shaofeng :D
> >
> > +1 to support this project entering the Incuabtor. It's in good shape
> > to develop an OSS project with a diverse group. Hopefully, the
> > Incubator can help it evaluate whether The Apache Way fits in its
> > community and growth.
> >
> > One thing to be noticed is that you're not listed on the mentor list.
> > By definition the champion helps the candidate go through the
> > proposing stage and is not always a mentor. If you're going to be the
> > mentor, you should add yourself to the mentor list also. But it's OK
> > that you can serve as a mentor.
> >
> > Best,
> > tison.
> >
> > ShaoFeng Shi  于2023年12月23日周六 20:43写道：
> > >
> > > Hi Enrico,
> > >
> > > This is a good question. Actually I asked the same to the team when
> they
> > > find me for coaching.
> > >
> > > As the proposal mentioned, Gluten is a middle layer between the
> JVM-based
> > > SQL engine and native libraries. It connects the two sides, and build
> > > common services like unified plan transformation, seamless native
> > > integration, clear JNI interfaces, etc. Spark is the first engine it
> > > supports now, but the framework and mechanism can also work for other
> > > engines like Trino, Flink SQL in the future (the proposal also
> mentioned
> > > this). Underlying Gluten, it can integrate with different libararies,
> > like
> > > Velox, Clickhouse and Arrow (not ready yet). So, its mission and scope
> is
> > > different, that's why we put it as a new project.
> > >
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > > Apache Kylin PMC,
> > > Apache Incubator PMC,
> > > Email: shaofeng...@apache.org
> > >
> > > Apache Kylin FAQ:
> https://kylin.apache.org/docs/gettingstarted/faq.html
> > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > >
> > >
> > >
> > >
> > > Enrico Olivelli  于2023年12月22日周五 22:43写道：
> > >
> > > > Shaofeng,
> > > >
> > > > Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi
> > > >  ha scritto:
> > > > >
> > > > > Hi IPMC members,
> > > > >
> > > > > I would like to propose a new project to the ASF incubator -
> Gluten.
> > > > >
> > > > > Gluten[1] is a middle layer responsible for offloading Apache Spark
> > SQL
> > > > > queries to native engines. This project aims to address the CPU
> > > > > computational bottleneck to offload SparkSQL operators to native
> > engines
> > > > in
> > > > > data loading scenarios based on Apache Spark.
> > > >
> > > > Have you considered making this project a subproject of Apache Spark
> ?
> > > >
> > > > Enrico
> > > >
> > > > >
> > > > > Here is the proposal -
> > > > >
> https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
> > > > >
> > > > > I would be the Champion of the project. I will mentor and help the
> > > > project
> > > > > through the incubator with Yu Li [l...@apache.org], Kent Yao [
> > > > y...@apache.org]
> > > > > and Wenli Zhang [ovi...@apache.org] .
> > > > >
> > > > > We are open to hearing the feedback from the incubator.
> > > > >
> > > > > Best,
> > > > > Shaofeng Shi.
> > > > >
> > > > > [1] https://oap-project.github.io/gluten/
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Shaofeng Shi 史少锋
> > > > > Apache Kylin PMC,
> > > > > Apache Incubator PMC,
> > > > > Email: shaofeng...@apache.org
> > > > >
> > > > > Apache Kylin FAQ:
> > https://kylin.apache.org/docs/gettingstarted/faq.html
> > > > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > > >
> > > > -
> > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > > For additional commands, e-mail: general-h...@incubator.apache.org
> > > >
> > > >
> >
> > -
> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > For additional commands, e-mail: general-h...@incubator.apache.org
> >
> >
>

Re: [DISCUSS] Gluten proposal

[ShaoFeng Shi]

Hi tison,

I added myself into the mentor list just now, I will be there. Thanks for
your suggestion!

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofeng...@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org




tison  于2023年12月23日周六 23:30写道：

> Thanks for driving this proposal Shaofeng :D
>
> +1 to support this project entering the Incuabtor. It's in good shape
> to develop an OSS project with a diverse group. Hopefully, the
> Incubator can help it evaluate whether The Apache Way fits in its
> community and growth.
>
> One thing to be noticed is that you're not listed on the mentor list.
> By definition the champion helps the candidate go through the
> proposing stage and is not always a mentor. If you're going to be the
> mentor, you should add yourself to the mentor list also. But it's OK
> that you can serve as a mentor.
>
> Best,
> tison.
>
> ShaoFeng Shi  于2023年12月23日周六 20:43写道：
> >
> > Hi Enrico,
> >
> > This is a good question. Actually I asked the same to the team when they
> > find me for coaching.
> >
> > As the proposal mentioned, Gluten is a middle layer between the JVM-based
> > SQL engine and native libraries. It connects the two sides, and build
> > common services like unified plan transformation, seamless native
> > integration, clear JNI interfaces, etc. Spark is the first engine it
> > supports now, but the framework and mechanism can also work for other
> > engines like Trino, Flink SQL in the future (the proposal also mentioned
> > this). Underlying Gluten, it can integrate with different libararies,
> like
> > Velox, Clickhouse and Arrow (not ready yet). So, its mission and scope is
> > different, that's why we put it as a new project.
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC,
> > Apache Incubator PMC,
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
> >
> >
> >
> > Enrico Olivelli  于2023年12月22日周五 22:43写道：
> >
> > > Shaofeng,
> > >
> > > Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi
> > >  ha scritto:
> > > >
> > > > Hi IPMC members,
> > > >
> > > > I would like to propose a new project to the ASF incubator - Gluten.
> > > >
> > > > Gluten[1] is a middle layer responsible for offloading Apache Spark
> SQL
> > > > queries to native engines. This project aims to address the CPU
> > > > computational bottleneck to offload SparkSQL operators to native
> engines
> > > in
> > > > data loading scenarios based on Apache Spark.
> > >
> > > Have you considered making this project a subproject of Apache Spark ?
> > >
> > > Enrico
> > >
> > > >
> > > > Here is the proposal -
> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
> > > >
> > > > I would be the Champion of the project. I will mentor and help the
> > > project
> > > > through the incubator with Yu Li [l...@apache.org], Kent Yao [
> > > y...@apache.org]
> > > > and Wenli Zhang [ovi...@apache.org] .
> > > >
> > > > We are open to hearing the feedback from the incubator.
> > > >
> > > > Best,
> > > > Shaofeng Shi.
> > > >
> > > > [1] https://oap-project.github.io/gluten/
> > > >
> > > > Best regards,
> > > >
> > > > Shaofeng Shi 史少锋
> > > > Apache Kylin PMC,
> > > > Apache Incubator PMC,
> > > > Email: shaofeng...@apache.org
> > > >
> > > > Apache Kylin FAQ:
> https://kylin.apache.org/docs/gettingstarted/faq.html
> > > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > >
> > > -
> > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > For additional commands, e-mail: general-h...@incubator.apache.org
> > >
> > >
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSS] Gluten proposal

[Felix Cheung]

Very interesting. Let me know if I can help as mentor



On Sat, Dec 23, 2023 at 7:30 AM tison  wrote:

> Thanks for driving this proposal Shaofeng :D
>
> +1 to support this project entering the Incuabtor. It's in good shape
> to develop an OSS project with a diverse group. Hopefully, the
> Incubator can help it evaluate whether The Apache Way fits in its
> community and growth.
>
> One thing to be noticed is that you're not listed on the mentor list.
> By definition the champion helps the candidate go through the
> proposing stage and is not always a mentor. If you're going to be the
> mentor, you should add yourself to the mentor list also. But it's OK
> that you can serve as a mentor.
>
> Best,
> tison.
>
> ShaoFeng Shi  于2023年12月23日周六 20:43写道：
> >
> > Hi Enrico,
> >
> > This is a good question. Actually I asked the same to the team when they
> > find me for coaching.
> >
> > As the proposal mentioned, Gluten is a middle layer between the JVM-based
> > SQL engine and native libraries. It connects the two sides, and build
> > common services like unified plan transformation, seamless native
> > integration, clear JNI interfaces, etc. Spark is the first engine it
> > supports now, but the framework and mechanism can also work for other
> > engines like Trino, Flink SQL in the future (the proposal also mentioned
> > this). Underlying Gluten, it can integrate with different libararies,
> like
> > Velox, Clickhouse and Arrow (not ready yet). So, its mission and scope is
> > different, that's why we put it as a new project.
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC,
> > Apache Incubator PMC,
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
> >
> >
> >
> > Enrico Olivelli  于2023年12月22日周五 22:43写道：
> >
> > > Shaofeng,
> > >
> > > Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi
> > >  ha scritto:
> > > >
> > > > Hi IPMC members,
> > > >
> > > > I would like to propose a new project to the ASF incubator - Gluten.
> > > >
> > > > Gluten[1] is a middle layer responsible for offloading Apache Spark
> SQL
> > > > queries to native engines. This project aims to address the CPU
> > > > computational bottleneck to offload SparkSQL operators to native
> engines
> > > in
> > > > data loading scenarios based on Apache Spark.
> > >
> > > Have you considered making this project a subproject of Apache Spark ?
> > >
> > > Enrico
> > >
> > > >
> > > > Here is the proposal -
> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
> > > >
> > > > I would be the Champion of the project. I will mentor and help the
> > > project
> > > > through the incubator with Yu Li [l...@apache.org], Kent Yao [
> > > y...@apache.org]
> > > > and Wenli Zhang [ovi...@apache.org] .
> > > >
> > > > We are open to hearing the feedback from the incubator.
> > > >
> > > > Best,
> > > > Shaofeng Shi.
> > > >
> > > > [1] https://oap-project.github.io/gluten/
> > > >
> > > > Best regards,
> > > >
> > > > Shaofeng Shi 史少锋
> > > > Apache Kylin PMC,
> > > > Apache Incubator PMC,
> > > > Email: shaofeng...@apache.org
> > > >
> > > > Apache Kylin FAQ:
> https://kylin.apache.org/docs/gettingstarted/faq.html
> > > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > >
> > > -
> > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > For additional commands, e-mail: general-h...@incubator.apache.org
> > >
> > >
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSS] Gluten proposal

[tison]

Thanks for driving this proposal Shaofeng :D

+1 to support this project entering the Incuabtor. It's in good shape
to develop an OSS project with a diverse group. Hopefully, the
Incubator can help it evaluate whether The Apache Way fits in its
community and growth.

One thing to be noticed is that you're not listed on the mentor list.
By definition the champion helps the candidate go through the
proposing stage and is not always a mentor. If you're going to be the
mentor, you should add yourself to the mentor list also. But it's OK
that you can serve as a mentor.

Best,
tison.

ShaoFeng Shi  于2023年12月23日周六 20:43写道：
>
> Hi Enrico,
>
> This is a good question. Actually I asked the same to the team when they
> find me for coaching.
>
> As the proposal mentioned, Gluten is a middle layer between the JVM-based
> SQL engine and native libraries. It connects the two sides, and build
> common services like unified plan transformation, seamless native
> integration, clear JNI interfaces, etc. Spark is the first engine it
> supports now, but the framework and mechanism can also work for other
> engines like Trino, Flink SQL in the future (the proposal also mentioned
> this). Underlying Gluten, it can integrate with different libararies, like
> Velox, Clickhouse and Arrow (not ready yet). So, its mission and scope is
> different, that's why we put it as a new project.
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>
>
>
> Enrico Olivelli  于2023年12月22日周五 22:43写道：
>
> > Shaofeng,
> >
> > Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi
> >  ha scritto:
> > >
> > > Hi IPMC members,
> > >
> > > I would like to propose a new project to the ASF incubator - Gluten.
> > >
> > > Gluten[1] is a middle layer responsible for offloading Apache Spark SQL
> > > queries to native engines. This project aims to address the CPU
> > > computational bottleneck to offload SparkSQL operators to native engines
> > in
> > > data loading scenarios based on Apache Spark.
> >
> > Have you considered making this project a subproject of Apache Spark ?
> >
> > Enrico
> >
> > >
> > > Here is the proposal -
> > > https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
> > >
> > > I would be the Champion of the project. I will mentor and help the
> > project
> > > through the incubator with Yu Li [l...@apache.org], Kent Yao [
> > y...@apache.org]
> > > and Wenli Zhang [ovi...@apache.org] .
> > >
> > > We are open to hearing the feedback from the incubator.
> > >
> > > Best,
> > > Shaofeng Shi.
> > >
> > > [1] https://oap-project.github.io/gluten/
> > >
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > > Apache Kylin PMC,
> > > Apache Incubator PMC,
> > > Email: shaofeng...@apache.org
> > >
> > > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
> > -
> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > For additional commands, e-mail: general-h...@incubator.apache.org
> >
> >

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [DISCUSS] Gluten proposal

[ShaoFeng Shi]

Hi Enrico,

This is a good question. Actually I asked the same to the team when they
find me for coaching.

As the proposal mentioned, Gluten is a middle layer between the JVM-based
SQL engine and native libraries. It connects the two sides, and build
common services like unified plan transformation, seamless native
integration, clear JNI interfaces, etc. Spark is the first engine it
supports now, but the framework and mechanism can also work for other
engines like Trino, Flink SQL in the future (the proposal also mentioned
this). Underlying Gluten, it can integrate with different libararies, like
Velox, Clickhouse and Arrow (not ready yet). So, its mission and scope is
different, that's why we put it as a new project.

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofeng...@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org




Enrico Olivelli  于2023年12月22日周五 22:43写道：

> Shaofeng,
>
> Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi
>  ha scritto:
> >
> > Hi IPMC members,
> >
> > I would like to propose a new project to the ASF incubator - Gluten.
> >
> > Gluten[1] is a middle layer responsible for offloading Apache Spark SQL
> > queries to native engines. This project aims to address the CPU
> > computational bottleneck to offload SparkSQL operators to native engines
> in
> > data loading scenarios based on Apache Spark.
>
> Have you considered making this project a subproject of Apache Spark ?
>
> Enrico
>
> >
> > Here is the proposal -
> > https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
> >
> > I would be the Champion of the project. I will mentor and help the
> project
> > through the incubator with Yu Li [l...@apache.org], Kent Yao [
> y...@apache.org]
> > and Wenli Zhang [ovi...@apache.org] .
> >
> > We are open to hearing the feedback from the incubator.
> >
> > Best,
> > Shaofeng Shi.
> >
> > [1] https://oap-project.github.io/gluten/
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC,
> > Apache Incubator PMC,
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSS] Gluten proposal

[Enrico Olivelli]

Shaofeng,

Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi
 ha scritto:
>
> Hi IPMC members,
>
> I would like to propose a new project to the ASF incubator - Gluten.
>
> Gluten[1] is a middle layer responsible for offloading Apache Spark SQL
> queries to native engines. This project aims to address the CPU
> computational bottleneck to offload SparkSQL operators to native engines in
> data loading scenarios based on Apache Spark.

Have you considered making this project a subproject of Apache Spark ?

Enrico

>
> Here is the proposal -
> https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal
>
> I would be the Champion of the project. I will mentor and help the project
> through the incubator with Yu Li [l...@apache.org], Kent Yao [y...@apache.org]
> and Wenli Zhang [ovi...@apache.org] .
>
> We are open to hearing the feedback from the incubator.
>
> Best,
> Shaofeng Shi.
>
> [1] https://oap-project.github.io/gluten/
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org