Alan Gates commented on PIG-823:
In response to Matei's comment:
The intent is not that this is Pig metadata, but that it be grid wide metadata.
We don't want to put it directly in HDFS by extending the namenode, since the
namenode is already heavily loaded and a central contention point in the
system. We also want it to remain optional, as many users will not need it.
The vision is that this will be a separate module that Hadoop users can choose
to install and use with their system, along with other modules they use, such
as Pig, Hive, Chuckwa, etc.
The Pig team is volunteering to put it in our contrib for now because Pig is
interested in it and willing to devote the resources to help it get started.
> Hadoop Metadata Service
> Key: PIG-823
> URL: https://issues.apache.org/jira/browse/PIG-823
> Project: Pig
> Issue Type: New Feature
> Reporter: Olga Natkovich
> This JIRA is created to track development of a metadata system for Hadoop.
> The goal of the system is to allow users and applications to register data
> stored on HDFS, search for the data available on HDFS, and associate metadata
> such as schema, statistics, etc. with a particular data unit or a data set
> stored on HDFS. The initial goal is to provide a fairly generic, low level
> abstraction that any user or application on HDFS can use to store an retrieve
> metadata. Over time a higher level abstractions closely tied to particular
> applications or tools can be developed.
> Over time, it would make sense for the metadata service to become a
> subproject within Hadoop. For now, the proposal is to make it a contrib to
> Pig since Pig SQL is likely to be the first user of the system.
This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.