Re: Fw: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11
Hi all,Blow is our working repohttps://github.com/spark-thriftserver/spark-thriftserver
Hope for your good suggestions.
angers.zhu
angers@gmail.com
签名由
网易邮箱大师
定制
On 12/30/2019 14:38，Sandeep Katta wrote：
+1On Mon, 30 Dec 2019 at 10:24, Gengliang <ltn...@gmail.com> wrote:+1On Sun, Dec 29, 2019 at 8:33 PM Wenchen Fan <cloud0...@gmail.com> wrote:+1 for the new thrift server to get rid of the Hive dependencies!On Mon, Dec 23, 2019 at 7:55 PM Yuming Wang <wgy...@gmail.com> wrote:I'm +1 for this SPIP for these two reasons:1. The current thriftserver has some issues that are not easy to solve, such as: SPARK-28636.2. The difference between the version of ORC we are using and the built-in Hive is using is getting bigger and bigger. We can't ensure that there will be no compatibility issues in the future. If thriftserver does not depend on Hive, it will be much easier to upgrade the built-in Hive in the future.On Sat, Dec 21, 2019 at 9:28 PM angers.zhu <angers@gmail.com> wrote:
Hi all, I have complete a Design doc about how to use and config this new thrift server, and some design detail about change and impersonation. Hope for your suggestions and ideas.SPIP DOC : https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.x97c6tj78zo0Design DOC : https://docs.google.com/document/d/1UKE9QTtHqSZBq0V_vEn54PlWaWPiRAKf_JvcT0skaSo/edit#heading=h.q1ed5q1ldh14Thrift server about configurations https://docs.google.com/document/d/1uI35qJmQO4FKE6pr0h3zetZqww-uI8QsQjxaYY_qb1s/edit?usp=drive_web=110963191229426834922
Best Regards
angers.zhu
angers@gmail.com
签名由
网易邮箱大师
定制
- Forwarded Message -
From：
angers.zhu
Date：
12/18/2019 22:29
To：
dev-ow...@spark.apache.org
Subject：
Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11
Add spark-dev group access privilege to google.
angers.zhu
angers@gmail.com
签名由
网易邮箱大师
定制
On 12/18/2019 22:02，Sandeep Katta wrote：
I couldn't access the doc, please give permission to the spark-dev groupOn Wed, 18 Dec 2019 at 18:05, angers.zhu <angers@gmail.com> wrote:
With the development of Spark and Hive，in current sql/hive-thriftserver module, we need to do a lot of work to solve code conflicts for different built-in hive versions.It's an annoying and unending work in current ways. And these issues have limited our ability and convenience to develop new features for Spark’s thrift server. We suppose to implemen
Re: Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11
+1 On Mon, 30 Dec 2019 at 10:24, Gengliang wrote: > +1 > > On Sun, Dec 29, 2019 at 8:33 PM Wenchen Fan wrote: > >> +1 for the new thrift server to get rid of the Hive dependencies! >> >> On Mon, Dec 23, 2019 at 7:55 PM Yuming Wang wrote: >> >>> I'm +1 for this SPIP for these two reasons: >>> >>> 1. The current thriftserver has some issues that are not easy to solve, >>> such as: SPARK-28636 <https://issues.apache.org/jira/browse/SPARK-28636> >>> . >>> 2. The difference between the version of ORC we are using and the >>> built-in Hive is using is getting bigger and bigger. We can't ensure that >>> there will be no compatibility issues in the future. If thriftserver does >>> not depend on Hive, it will be much easier to upgrade the built-in Hive in >>> the future. >>> >>> On Sat, Dec 21, 2019 at 9:28 PM angers.zhu wrote: >>> >>>> Hi all, >>>> >>>> I have complete a Design doc about how to use and config this new >>>> thrift server, and some design detail about change and impersonation. >>>> >>>> Hope for your suggestions and ideas. >>>> >>>> SPIP DOC : >>>> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.x97c6tj78zo0 >>>> Design DOC : >>>> https://docs.google.com/document/d/1UKE9QTtHqSZBq0V_vEn54PlWaWPiRAKf_JvcT0skaSo/edit#heading=h.q1ed5q1ldh14 >>>> Thrift server about configurations >>>> https://docs.google.com/document/d/1uI35qJmQO4FKE6pr0h3zetZqww-uI8QsQjxaYY_qb1s/edit?usp=drive_web=110963191229426834922 >>>> >>>> Best Regards >>>> angers.zhu >>>> angers@gmail.com >>>> >>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1=angers.zhu=angers.zhu%40gmail.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png=%5B%22angers.zhu%40gmail.com%22%5D> >>>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 >>>> >>>> - Forwarded Message - >>>> From： angers.zhu >>>> Date： 12/18/2019 22:29 >>>> To： dev-ow...@spark.apache.org >>>> >>>> Subject： Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based >>>> on protocol v11 >>>> >>>> Add spark-dev group access privilege to google. >>>> angers.zhu >>>> angers@gmail.com >>>> >>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1=angers.zhu=angers.zhu%40gmail.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png=%5B%22angers.zhu%40gmail.com%22%5D> >>>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 >>>> >>>> On 12/18/2019 22:02，Sandeep Katta >>>> wrote： >>>> >>>> I couldn't access the doc, please give permission to the spark-dev group >>>> >>>> On Wed, 18 Dec 2019 at 18:05, angers.zhu wrote: >>>> >>>>> With the development of Spark and Hive，in current >>>>> sql/hive-thriftserver module, >>>>> >>>>> we need to do a lot of work to solve code conflicts for different >>>>> built-in hive versions. >>>>> >>>>> It's an annoying and unending work in current ways. And these issues >>>>> have limited >>>>> >>>>> our ability and convenience to develop new features for Spark’s thrift >>>>> server. >>>>> >>>>> We suppose to implement a new thrift server and JDBC driver based >>>>> on Hive’s latest v11 >>>>> >>>>> TCLService.thrift thrift protocol. Finally, the new thrift server >>>>> have below feature: >>>>> >>>>>1. >>>>> >>>>>Build new module spark-service as spark’s thrift server >>>>>2. >>>>> >>>>>Don't need as much reflection and inherited code as >>>>>`hive-thriftser` modules >>>>>3. >>>>> >>>>>Support all functions current `sql/hive-thriftserver` support >>>>>4. >>>>> >>>>>Use all code maintained by spark itself, won’t depend on Hive >>>>>5. >>>>> >>>>>Support origin functions use spark’s own way, won't limited by >>>>>
Re: Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11
+1 On Sun, Dec 29, 2019 at 8:33 PM Wenchen Fan wrote: > +1 for the new thrift server to get rid of the Hive dependencies! > > On Mon, Dec 23, 2019 at 7:55 PM Yuming Wang wrote: > >> I'm +1 for this SPIP for these two reasons: >> >> 1. The current thriftserver has some issues that are not easy to solve, >> such as: SPARK-28636 <https://issues.apache.org/jira/browse/SPARK-28636>. >> 2. The difference between the version of ORC we are using and the >> built-in Hive is using is getting bigger and bigger. We can't ensure that >> there will be no compatibility issues in the future. If thriftserver does >> not depend on Hive, it will be much easier to upgrade the built-in Hive in >> the future. >> >> On Sat, Dec 21, 2019 at 9:28 PM angers.zhu wrote: >> >>> Hi all, >>> >>> I have complete a Design doc about how to use and config this new thrift >>> server, and some design detail about change and impersonation. >>> >>> Hope for your suggestions and ideas. >>> >>> SPIP DOC : >>> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.x97c6tj78zo0 >>> Design DOC : >>> https://docs.google.com/document/d/1UKE9QTtHqSZBq0V_vEn54PlWaWPiRAKf_JvcT0skaSo/edit#heading=h.q1ed5q1ldh14 >>> Thrift server about configurations >>> https://docs.google.com/document/d/1uI35qJmQO4FKE6pr0h3zetZqww-uI8QsQjxaYY_qb1s/edit?usp=drive_web=110963191229426834922 >>> >>> Best Regards >>> angers.zhu >>> angers@gmail.com >>> >>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1=angers.zhu=angers.zhu%40gmail.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png=%5B%22angers.zhu%40gmail.com%22%5D> >>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 >>> >>> - Forwarded Message - >>> From： angers.zhu >>> Date： 12/18/2019 22:29 >>> To： dev-ow...@spark.apache.org >>> >>> Subject： Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based >>> on protocol v11 >>> >>> Add spark-dev group access privilege to google. >>> angers.zhu >>> angers@gmail.com >>> >>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1=angers.zhu=angers.zhu%40gmail.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png=%5B%22angers.zhu%40gmail.com%22%5D> >>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 >>> >>> On 12/18/2019 22:02，Sandeep Katta >>> wrote： >>> >>> I couldn't access the doc, please give permission to the spark-dev group >>> >>> On Wed, 18 Dec 2019 at 18:05, angers.zhu wrote: >>> >>>> With the development of Spark and Hive，in current sql/hive-thriftserver >>>> module, >>>> >>>> we need to do a lot of work to solve code conflicts for different >>>> built-in hive versions. >>>> >>>> It's an annoying and unending work in current ways. And these issues >>>> have limited >>>> >>>> our ability and convenience to develop new features for Spark’s thrift >>>> server. >>>> >>>> We suppose to implement a new thrift server and JDBC driver based >>>> on Hive’s latest v11 >>>> >>>> TCLService.thrift thrift protocol. Finally, the new thrift server have >>>> below feature: >>>> >>>>1. >>>> >>>>Build new module spark-service as spark’s thrift server >>>>2. >>>> >>>>Don't need as much reflection and inherited code as >>>>`hive-thriftser` modules >>>>3. >>>> >>>>Support all functions current `sql/hive-thriftserver` support >>>>4. >>>> >>>>Use all code maintained by spark itself, won’t depend on Hive >>>>5. >>>> >>>>Support origin functions use spark’s own way, won't limited by >>>>Hive's code >>>>6. >>>> >>>>Support running without hive metastore or with hive metastore >>>>7. >>>> >>>>Support user impersonation by Multi-tenant splited hive >>>>authentication and DFS authentication >>>>8. >>>> >>>>Support session hook for with spark’s own code >>>>9. >>>> >>>>Add a new jdbc driver spark-jdbc, with spark’s own connection url >>>>“jdbc:spark::/” >>>>10. >>>> >>>>Support both hive-jdbc and spark-jdbc client, then we can support >>>>most clients and BI platform >>>> >>>> >>>> >>>> https://issues.apache.org/jira/browse/SPARK-29018 >>>> >>>> Google Doc: >>>> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit# >>>> >>>> [ ] +1: Accept the proposal as an official SPIP >>>> [ ] +0 >>>> [ ] -1: I don't think this is a good idea because ... >>>> >>>> I'll start with my +1 >>>> angers.zhu >>>> angers@gmail.com >>>> >>>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1=angers.zhu=angers.zhu%40gmail.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png=%5B%22angers.zhu%40gmail.com%22%5D> >>>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 >>>> >>>>
Re: Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11
+1 for the new thrift server to get rid of the Hive dependencies! On Mon, Dec 23, 2019 at 7:55 PM Yuming Wang wrote: > I'm +1 for this SPIP for these two reasons: > > 1. The current thriftserver has some issues that are not easy to solve, > such as: SPARK-28636 <https://issues.apache.org/jira/browse/SPARK-28636>. > 2. The difference between the version of ORC we are using and the built-in > Hive is using is getting bigger and bigger. We can't ensure that there will > be no compatibility issues in the future. If thriftserver does not depend > on Hive, it will be much easier to upgrade the built-in Hive in the future. > > On Sat, Dec 21, 2019 at 9:28 PM angers.zhu wrote: > >> Hi all, >> >> I have complete a Design doc about how to use and config this new thrift >> server, and some design detail about change and impersonation. >> >> Hope for your suggestions and ideas. >> >> SPIP DOC : >> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.x97c6tj78zo0 >> Design DOC : >> https://docs.google.com/document/d/1UKE9QTtHqSZBq0V_vEn54PlWaWPiRAKf_JvcT0skaSo/edit#heading=h.q1ed5q1ldh14 >> Thrift server about configurations >> https://docs.google.com/document/d/1uI35qJmQO4FKE6pr0h3zetZqww-uI8QsQjxaYY_qb1s/edit?usp=drive_web=110963191229426834922 >> >> Best Regards >> angers.zhu >> angers@gmail.com >> >> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1=angers.zhu=angers.zhu%40gmail.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png=%5B%22angers.zhu%40gmail.com%22%5D> >> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 >> >> ----- Forwarded Message --------- >> From： angers.zhu >> Date： 12/18/2019 22:29 >> To： dev-ow...@spark.apache.org >> >> Subject： Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based >> on protocol v11 >> >> Add spark-dev group access privilege to google. >> angers.zhu >> angers@gmail.com >> >> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1=angers.zhu=angers.zhu%40gmail.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png=%5B%22angers.zhu%40gmail.com%22%5D> >> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 >> >> On 12/18/2019 22:02，Sandeep Katta >> wrote： >> >> I couldn't access the doc, please give permission to the spark-dev group >> >> On Wed, 18 Dec 2019 at 18:05, angers.zhu wrote: >> >>> With the development of Spark and Hive，in current sql/hive-thriftserver >>> module, >>> >>> we need to do a lot of work to solve code conflicts for different >>> built-in hive versions. >>> >>> It's an annoying and unending work in current ways. And these issues >>> have limited >>> >>> our ability and convenience to develop new features for Spark’s thrift >>> server. >>> >>> We suppose to implement a new thrift server and JDBC driver based on >>> Hive’s >>> latest v11 >>> >>> TCLService.thrift thrift protocol. Finally, the new thrift server have >>> below feature: >>> >>>1. >>> >>>Build new module spark-service as spark’s thrift server >>>2. >>> >>>Don't need as much reflection and inherited code as `hive-thriftser` >>>modules >>>3. >>> >>>Support all functions current `sql/hive-thriftserver` support >>>4. >>> >>>Use all code maintained by spark itself, won’t depend on Hive >>>5. >>> >>>Support origin functions use spark’s own way, won't limited by >>>Hive's code >>>6. >>> >>>Support running without hive metastore or with hive metastore >>>7. >>> >>>Support user impersonation by Multi-tenant splited hive >>>authentication and DFS authentication >>>8. >>> >>>Support session hook for with spark’s own code >>>9. >>> >>>Add a new jdbc driver spark-jdbc, with spark’s own connection url >>>“jdbc:spark::/” >>>10. >>> >>>Support both hive-jdbc and spark-jdbc client, then we can support >>>most clients and BI platform >>> >>> >>> >>> https://issues.apache.org/jira/browse/SPARK-29018 >>> >>> Google Doc: >>> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit# >>> >>> [ ] +1: Accept the proposal as an official SPIP >>> [ ] +0 >>> [ ] -1: I don't think this is a good idea because ... >>> >>> I'll start with my +1 >>> angers.zhu >>> angers@gmail.com >>> >>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1=angers.zhu=angers.zhu%40gmail.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png=%5B%22angers.zhu%40gmail.com%22%5D> >>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 >>> >>>
Re: Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11
I'm +1 for this SPIP for these two reasons: 1. The current thriftserver has some issues that are not easy to solve, such as: SPARK-28636 <https://issues.apache.org/jira/browse/SPARK-28636>. 2. The difference between the version of ORC we are using and the built-in Hive is using is getting bigger and bigger. We can't ensure that there will be no compatibility issues in the future. If thriftserver does not depend on Hive, it will be much easier to upgrade the built-in Hive in the future. On Sat, Dec 21, 2019 at 9:28 PM angers.zhu wrote: > Hi all, > > I have complete a Design doc about how to use and config this new thrift > server, and some design detail about change and impersonation. > > Hope for your suggestions and ideas. > > SPIP DOC : > https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.x97c6tj78zo0 > Design DOC : > https://docs.google.com/document/d/1UKE9QTtHqSZBq0V_vEn54PlWaWPiRAKf_JvcT0skaSo/edit#heading=h.q1ed5q1ldh14 > Thrift server about configurations > https://docs.google.com/document/d/1uI35qJmQO4FKE6pr0h3zetZqww-uI8QsQjxaYY_qb1s/edit?usp=drive_web=110963191229426834922 > > Best Regards > angers.zhu > angers@gmail.com > > <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1=angers.zhu=angers.zhu%40gmail.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png=%5B%22angers.zhu%40gmail.com%22%5D> > 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 > > - Forwarded Message - > From： angers.zhu > Date： 12/18/2019 22:29 > To： dev-ow...@spark.apache.org > > Subject： Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on > protocol v11 > > Add spark-dev group access privilege to google. > angers.zhu > angers@gmail.com > > <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1=angers.zhu=angers.zhu%40gmail.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png=%5B%22angers.zhu%40gmail.com%22%5D> > 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 > > On 12/18/2019 22:02，Sandeep Katta > wrote： > > I couldn't access the doc, please give permission to the spark-dev group > > On Wed, 18 Dec 2019 at 18:05, angers.zhu wrote: > >> With the development of Spark and Hive，in current sql/hive-thriftserver >> module, >> >> we need to do a lot of work to solve code conflicts for different >> built-in hive versions. >> >> It's an annoying and unending work in current ways. And these issues have >> limited >> >> our ability and convenience to develop new features for Spark’s thrift >> server. >> >> We suppose to implement a new thrift server and JDBC driver based on >> Hive’s >> latest v11 >> >> TCLService.thrift thrift protocol. Finally, the new thrift server have >> below feature: >> >>1. >> >>Build new module spark-service as spark’s thrift server >>2. >> >>Don't need as much reflection and inherited code as `hive-thriftser` >>modules >>3. >> >>Support all functions current `sql/hive-thriftserver` support >>4. >> >>Use all code maintained by spark itself, won’t depend on Hive >>5. >> >>Support origin functions use spark’s own way, won't limited by Hive's >>code >>6. >> >>Support running without hive metastore or with hive metastore >>7. >> >>Support user impersonation by Multi-tenant splited hive >>authentication and DFS authentication >>8. >> >>Support session hook for with spark’s own code >>9. >> >>Add a new jdbc driver spark-jdbc, with spark’s own connection url >>“jdbc:spark::/” >>10. >> >>Support both hive-jdbc and spark-jdbc client, then we can support >>most clients and BI platform >> >> >> >> https://issues.apache.org/jira/browse/SPARK-29018 >> >> Google Doc: >> https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit# >> >> [ ] +1: Accept the proposal as an official SPIP >> [ ] +0 >> [ ] -1: I don't think this is a good idea because ... >> >> I'll start with my +1 >> angers.zhu >> angers@gmail.com >> >> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1=angers.zhu=angers.zhu%40gmail.com=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png=%5B%22angers.zhu%40gmail.com%22%5D> >> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 >> >>
Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11
Hi all, I have complete a Design doc about how to use and config this new thrift server, and some design detail about change and impersonation. Hope for your suggestions and ideas.SPIP DOC : https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#heading=h.x97c6tj78zo0Design DOC : https://docs.google.com/document/d/1UKE9QTtHqSZBq0V_vEn54PlWaWPiRAKf_JvcT0skaSo/edit#heading=h.q1ed5q1ldh14Thrift server about configurations https://docs.google.com/document/d/1uI35qJmQO4FKE6pr0h3zetZqww-uI8QsQjxaYY_qb1s/edit?usp=drive_web=110963191229426834922 Best Regards angers.zhu angers@gmail.com 签名由 网易邮箱大师 定制 - Forwarded Message - From： angers.zhu Date： 12/18/2019 22:29 To： dev-ow...@spark.apache.org Subject： Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11 Add spark-dev group access privilege to google. angers.zhu angers@gmail.com 签名由 网易邮箱大师 定制 On 12/18/2019 22:02，Sandeep Katta wrote： I couldn't access the doc, please give permission to the spark-dev groupOn Wed, 18 Dec 2019 at 18:05, angers.zhu <angers@gmail.com> wrote: With the development of Spark and Hive，in current sql/hive-thriftserver module, we need to do a lot of work to solve code conflicts for different built-in hive versions.It's an annoying and unending work in current ways. And these issues have limited our ability and convenience to develop new features for Spark’s thrift server. We suppose to implement a new thrift server and JDBC driver based on Hive’s latest v11 TCLService.thrift thrift protocol. Finally, the new thrift server have below feature:Build new module spark-service as spark’s thrift server Don't need as much reflection and inherited code as `hive-thriftser` modulesSupport all functions current `sql/hive-thriftserver` supportUse all code maintained by spark itself, won’t depend on HiveSupport origin functions use spark’s own way, won't limited by Hive's codeSupport running without hive metastore or with hive metastoreSupport user impersonation by Multi-tenant splited hive authentication and DFS authenticationSupport session hook for with spark’s own codeAdd a new jdbc driver spark-jdbc, with spark’s own connection url “jdbc:spark::/”Support both hive-jdbc and spark-jdbc client, then we can support most clients and BI platform https://issues.apache.org/jira/browse/SPARK-29018 Google Doc: https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#[ ] +1: Accept the proposal as an official SPIP[ ] +0[ ] -1: I don't think this is a good idea because ...I'll start with my +1 angers.zhu angers@gma
[VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11
With the development of Spark and Hive，in current sql/hive-thriftserver module, we need to do a lot of work to solve code conflicts for different built-in hive versions.It's an annoying and unending work in current ways. And these issues have limited our ability and convenience to develop new features for Spark’s thrift server. We suppose to implement a new thrift server and JDBC driver based on Hive’s latest v11 TCLService.thrift thrift protocol. Finally, the new thrift server have below feature:Build new module spark-service as spark’s thrift server Don't need as much reflection and inherited code as `hive-thriftser` modulesSupport all functions current `sql/hive-thriftserver` supportUse all code maintained by spark itself, won’t depend on HiveSupport origin functions use spark’s own way, won't limited by Hive's codeSupport running without hive metastore or with hive metastoreSupport user impersonation by Multi-tenant splited hive authentication and DFS authenticationSupport session hook for with spark’s own codeAdd a new jdbc driver spark-jdbc, with spark’s own connection url “jdbc:spark::/”Support both hive-jdbc and spark-jdbc client, then we can support most clients and BI platform https://issues.apache.org/jira/browse/SPARK-29018 Google Doc: https://docs.google.com/document/d/1ug4K5e2okF5Q2Pzi3qJiUILwwqkn0fVQaQ-Q95HEcJQ/edit#[ ] +1: Accept the proposal as an official SPIP[ ] +0[ ] -1: I don't think this is a good idea because ...I'll start with my +1 angers.zhu angers@gmail.com 签名由 网易邮箱大师 定制
