I'm +1 on separating out the metastore. It recognizes the reality that a lot of different projects use the Hive Metastore and opening up the community is a great move.
..Owen On Fri, Jun 30, 2017 at 1:30 PM, Xuefu Zhang <xu...@uber.com> wrote: > +1, sounds like a good idea! > > On Fri, Jun 30, 2017 at 1:24 PM, Harsha <h...@harsha.io> wrote: > > > Thanks for the proposal Alan. I am +1 on separating the Hive Metastore. > > This is a great opportunity for building a Metastore to not only address > > schemas for the data at rest but also for the data in motion. We have a > > SchemaRegistry (http://github.com/hortonworks/registry) project that > > allows users to register schemas for data in motion and integrates with > > Kafka, Kinesis, Evenhubs and other messaging queues. This will provide > > us with opportunity to integrate our apis with Hive Metastore and > > provide with one project that is truly a single metastore that can hold > > all schemas. > > > > Thanks, > > Harsha > > > > On Fri, Jun 30, 2017, at 01:18 PM, Sergio Pena wrote: > > > Great, thanks Alan for putting all this in the email. > > > +1 > > > > > > Allowing other components to continue to use the Metastore without the > > > need > > > to use Hive dependencies is a big plus for them. I agree with > everything > > > you mention on the email. > > > > > > - Sergio > > > > > > On Fri, Jun 30, 2017 at 1:49 PM, Julian Hyde <jh...@apache.org> wrote: > > > > > > > +1 > > > > > > > > As a Calcite PMC member, I am very pleased to see this change. > Calcite > > > > reads metadata from a variety of sources (including JDBC databases, > > NoSQL > > > > databases such as Cassandra and Druid, and streaming systems), and if > > more > > > > of those sources choose to store their metadata in the metastore it > > will > > > > make our lives easier. > > > > > > > > Hive’s metastore has established a position as the place to go for > > > > metadata in the Hadoop ecosystem. Not all metadata is relational, or > > > > processed by Hive, so there are other parties using the metastore who > > > > justifiably would like to influence its direction. Opening up the > > metastore > > > > will help retain and extend this position. > > > > > > > > Julian > > > > > > > > > > > > On 2017-06-30 10:00 (-0700), "Dimitris ts...@apache.org> wrote: > > > > > > > > > > > > > > > On 2017-06-30 07:56 (-0700), Alan Gates <al...@gmail.com> wrote: > > > > > > > A few of us have been talking and come to the conclussion that it > > > > would be> > > > > > > a good thing to split out the Hive metastore into its own Apache > > > > project.> > > > > > > Below and in the linked wiki page we explain what we see as the > > > > advantages> > > > > > > to this and how we would go about it.> > > > > > > > > > > > > > Hive’s metastore has long been used by other projects in the > > Hadoop> > > > > > > ecosystem to store and access metadata. Apache Impala, Apache > > Spark,> > > > > > > Apache Drill, Presto, and other systems all use Hive’s metastore. > > > > Some,> > > > > > > like Impala and Presto can use it as their own metadata system > with > > > > the> > > > > > > rest of Hive not present.> > > > > > > > > > > > > > This sharing is excellent for the ecosystem. Together with HDFS > it > > > > allows> > > > > > > users to use the tool of their choice while still accessing the > > same > > > > shared> > > > > > > data. But having this shared metadata inside the Hive project > > limits > > > > the> > > > > > > ability of other projects to contribute to the metastore. It > also > > > > makes it> > > > > > > harder for new systems that have similar but not identical > > metadata> > > > > > > requirements (for example, stream processing systems on top of > > Apache> > > > > > > Kafka) to use Hive’s metastore. This difficulty for other > systems > > > > comes> > > > > > > out in two ways. One, it is hard for non-Hive community members > > to> > > > > > > participate in the project. Second, it adds operational cost > since > > > > users> > > > > > > are forced to deploy all of the Hive jars just to get the > > metastore to > > > > work.> > > > > > > > > > > > > > Therefore we propose to split Hive’s metastore out into a > separate > > > > Apache> > > > > > > project. This new project will continue to support the same > Thrift > > > > API as> > > > > > > the current metastore. It will continue to focus on being a > high> > > > > > > performance, fault tolerant, large scale, operational metastore > for > > > > SQL> > > > > > > engines and other systems that want to store schema information > > about > > > > their> > > > > > > data.> > > > > > > > > > > > > > By making it a separate project we will enable other projects to > > join > > > > us in> > > > > > > innovating on the metastore. It will simplify operations for > > non-Hive> > > > > > > users that want to use the metastore as they will no longer need > to > > > > install> > > > > > > Hive just to get the metastore. And it will attract new projects > > that> > > > > > > might otherwise feel the need to solve their metadata problems on > > > > their own.> > > > > > > > > > > > > > Any Hive PMC member or committer will be welcome to join the new > > > > project at> > > > > > > the same level. We propose this project go straight to a top > > level> > > > > > > project. Given that the initial PMC will be formed from > > experienced > > > > Hive> > > > > > > PMC members we do not believe incubation will be necessary. > (Note > > > > that the> > > > > > > Apache board will need to approve this.)> > > > > > > > > > > > > > Obviously there a many details involved in a proposal like this. > > > > Rather> > > > > > > than make this a ten page email we have filled out many of the > > details > > > > in a> > > > > > > wiki page:> > > > > > > https://cwiki.apache.org/confluence/display/Hive/ > > > > Metastore+TLP+Proposal> > > > > > > > > > > > > > Yongzhi Chen> > > > > > > Vihang Karajgaonkar> > > > > > > Sergio Pena> > > > > > > Sahil Takiar> > > > > > > Aihua Xu> > > > > > > Gunther Hagleitner> > > > > > > Thejas Nair> > > > > > > Alan Gates> > > > > > > > > > > > > > > > > > +1 (from Apache Impala's (incubating) perspective)> > > > > > > > > > > Dimitris> > > > > > > > > > > > > > > > Thanks, > > Harsha > > >
