Re: Proposal: File based metastore

Have you looked at https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport? Why does this not suit your use case? Regards, Johannes Sent from my iPhone > On Feb 23, 2018, at 3:40 PM, Alexander Kolbasov wrote: > > Would it be useful to have a tool

Re: Proposal: File based metastore

Would it be useful to have a tool that can save database(s), table(s) and partition(s) metadata in a file and then import this file in another metastore? These files can be stored together with data files or elsewhere. This would allow for targeted exchange of metadata between multiple HMS

Re: Proposal: File based metastore

On Tue, Jan 30, 2018 at 1:16 PM, Ryan Blue wrote: > Thanks, Owen. > > I agree, Iceberg addresses a lot of the problems that you're hitting here. > It doesn't quite go as far as moving all metadata into the file system. You > can do that in HDFS and implementations that support

Re: Proposal: File based metastore

Hi Ryan, Is Hive support on the iceberg roadmap? Presumably its MetastoreClientFactory and storage API provide an integration point? Or is there perhaps some architectural detail that makes this impractical? I’m thinking not just of the ability to support Hive, but also the range of tooling that

Re: Proposal: File based metastore

Thanks, Owen. I agree, Iceberg addresses a lot of the problems that you're hitting here. It doesn't quite go as far as moving all metadata into the file system. You can do that in HDFS and implementations that support atomic rename, but not in S3 (Iceberg has an implementation of the HDFS one

Re: Proposal: File based metastore

On Mon, Jan 29, 2018 at 12:44 PM, Owen O'Malley wrote: > > > On Jan 29, 2018, at 9:29 AM, Edward Capriolo > wrote: > > > > On Mon, Jan 29, 2018 at 12:10 PM, Owen O'Malley > wrote: > >> You should really look at what the

Re: Proposal: File based metastore

> On Jan 29, 2018, at 9:29 AM, Edward Capriolo wrote: > > > > On Mon, Jan 29, 2018 at 12:10 PM, Owen O'Malley > wrote: > You should really look at what the Netflix guys are doing on Iceberg. > >

Re: Proposal: File based metastore

On Mon, Jan 29, 2018 at 12:10 PM, Owen O'Malley wrote: > You should really look at what the Netflix guys are doing on Iceberg. > > https://github.com/Netflix/iceberg > > They have put a lot of thought into how to efficiently handle tabular data > in S3. They put all of

Re: Proposal: File based metastore

You should really look at what the Netflix guys are doing on Iceberg. https://github.com/Netflix/iceberg They have put a lot of thought into how to efficiently handle tabular data in S3. They put all of the metadata in S3 except for a single link to the name of the table's root metadata file.