Hi Rohan and team, The work sounds exciting thanks for considering contributing back to the community.
The design document didn't arrive cause attachements are not allowed in many Apache lists. Maybe as a first step it would be nice to share a link to a Google doc where people can add comments and possibly provide some feedback on it. Then, I guess it makes sense to put in the wiki under the respective section (design documents) and/or upload it in the JIRA case. I am not very familiar with the area so not sure if I can help much pushing this forward but I am definitely interested to learn more about this work. Best, Stamatis On Tue, Aug 23, 2022, 2:05 AM Cameron Moberg <cjmob...@google.com.invalid> wrote: > > *Sending on behalf of Rohan where policies don't allow sending outside of > our domain for interns:* > Hello - > > During my internship I’ve been working on gRPC native support in the > standalone hive metastore as it comes with a variety of benefits. As a > proof of concept, my team, Dataproc Metastore on GCP currently uses a > client side proxy to translate Thrift requests to gRPC coupled with a > server side proxy to translate the gRPC requests back to Thrift. The > process is repeated in reverse to deliver the server response to the > client. While this approach has been successful, native gRPC support has > several cloud-centric advantages over the current configuration: > > - enables streaming support > - allows for native integrations in Hive ecosystem for various query > engines like Impala, Spark SQL, and Trino to take advantage of streaming > (eventually) > - has support for custom interceptors for more fine-grained control > over the server action > - built on HTTP/2 protocol > > I’ve opened a PR here <https://github.com/apache/hive/pull/3534> (just > fyi, no rush), some background – this proto3 definition has been refactored > to take a MethodNameRequest and MethodNameResponse to stop any future > backwards incompatibilities. Unfortunately, the other metastore.proto which > has SplitInfo uses a `required` field setting, which makes upgrading it not > feasible since moving away from `required` will change the SerDe of proto, > potentially a breaking change depending on clients. > While this is the last week of my internship my hosts cjmob...@google.com > and hchinch...@google.com will continue to develop in this area with > further implementation building on the proto. > > Attached is the full design doc, I’m not sure how I’m supposed to share > documents like this, so I can reupload this somewhere or convert to the > wiki. > > Comments are of course appreciated! > > Thank you, > Rohan Sonecha >
