+1, sounds like a good idea! On Fri, Jun 30, 2017 at 1:24 PM, Harsha <h...@harsha.io> wrote:
> Thanks for the proposal Alan. I am +1 on separating the Hive Metastore. > This is a great opportunity for building a Metastore to not only address > schemas for the data at rest but also for the data in motion. We have a > SchemaRegistry (http://github.com/hortonworks/registry) project that > allows users to register schemas for data in motion and integrates with > Kafka, Kinesis, Evenhubs and other messaging queues. This will provide > us with opportunity to integrate our apis with Hive Metastore and > provide with one project that is truly a single metastore that can hold > all schemas. > > Thanks, > Harsha > > On Fri, Jun 30, 2017, at 01:18 PM, Sergio Pena wrote: > > Great, thanks Alan for putting all this in the email. > > +1 > > > > Allowing other components to continue to use the Metastore without the > > need > > to use Hive dependencies is a big plus for them. I agree with everything > > you mention on the email. > > > > - Sergio > > > > On Fri, Jun 30, 2017 at 1:49 PM, Julian Hyde <jh...@apache.org> wrote: > > > > > +1 > > > > > > As a Calcite PMC member, I am very pleased to see this change. Calcite > > > reads metadata from a variety of sources (including JDBC databases, > NoSQL > > > databases such as Cassandra and Druid, and streaming systems), and if > more > > > of those sources choose to store their metadata in the metastore it > will > > > make our lives easier. > > > > > > Hive’s metastore has established a position as the place to go for > > > metadata in the Hadoop ecosystem. Not all metadata is relational, or > > > processed by Hive, so there are other parties using the metastore who > > > justifiably would like to influence its direction. Opening up the > metastore > > > will help retain and extend this position. > > > > > > Julian > > > > > > > > > On 2017-06-30 10:00 (-0700), "Dimitris ts...@apache.org> wrote: > > > > > > > > > > > > On 2017-06-30 07:56 (-0700), Alan Gates <al...@gmail.com> wrote: > > > > > > A few of us have been talking and come to the conclussion that it > > > would be> > > > > > a good thing to split out the Hive metastore into its own Apache > > > project.> > > > > > Below and in the linked wiki page we explain what we see as the > > > advantages> > > > > > to this and how we would go about it.> > > > > > > > > > > > Hive’s metastore has long been used by other projects in the > Hadoop> > > > > > ecosystem to store and access metadata. Apache Impala, Apache > Spark,> > > > > > Apache Drill, Presto, and other systems all use Hive’s metastore. > > > Some,> > > > > > like Impala and Presto can use it as their own metadata system with > > > the> > > > > > rest of Hive not present.> > > > > > > > > > > > This sharing is excellent for the ecosystem. Together with HDFS it > > > allows> > > > > > users to use the tool of their choice while still accessing the > same > > > shared> > > > > > data. But having this shared metadata inside the Hive project > limits > > > the> > > > > > ability of other projects to contribute to the metastore. It also > > > makes it> > > > > > harder for new systems that have similar but not identical > metadata> > > > > > requirements (for example, stream processing systems on top of > Apache> > > > > > Kafka) to use Hive’s metastore. This difficulty for other systems > > > comes> > > > > > out in two ways. One, it is hard for non-Hive community members > to> > > > > > participate in the project. Second, it adds operational cost since > > > users> > > > > > are forced to deploy all of the Hive jars just to get the > metastore to > > > work.> > > > > > > > > > > > Therefore we propose to split Hive’s metastore out into a separate > > > Apache> > > > > > project. This new project will continue to support the same Thrift > > > API as> > > > > > the current metastore. It will continue to focus on being a high> > > > > > performance, fault tolerant, large scale, operational metastore for > > > SQL> > > > > > engines and other systems that want to store schema information > about > > > their> > > > > > data.> > > > > > > > > > > > By making it a separate project we will enable other projects to > join > > > us in> > > > > > innovating on the metastore. It will simplify operations for > non-Hive> > > > > > users that want to use the metastore as they will no longer need to > > > install> > > > > > Hive just to get the metastore. And it will attract new projects > that> > > > > > might otherwise feel the need to solve their metadata problems on > > > their own.> > > > > > > > > > > > Any Hive PMC member or committer will be welcome to join the new > > > project at> > > > > > the same level. We propose this project go straight to a top > level> > > > > > project. Given that the initial PMC will be formed from > experienced > > > Hive> > > > > > PMC members we do not believe incubation will be necessary. (Note > > > that the> > > > > > Apache board will need to approve this.)> > > > > > > > > > > > Obviously there a many details involved in a proposal like this. > > > Rather> > > > > > than make this a ten page email we have filled out many of the > details > > > in a> > > > > > wiki page:> > > > > > https://cwiki.apache.org/confluence/display/Hive/ > > > Metastore+TLP+Proposal> > > > > > > > > > > > Yongzhi Chen> > > > > > Vihang Karajgaonkar> > > > > > Sergio Pena> > > > > > Sahil Takiar> > > > > > Aihua Xu> > > > > > Gunther Hagleitner> > > > > > Thejas Nair> > > > > > Alan Gates> > > > > > > > > > > > > > > +1 (from Apache Impala's (incubating) perspective)> > > > > > > > > Dimitris> > > > > > > > > > > Thanks, > Harsha >
