[
https://issues.apache.org/jira/browse/PIG-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714547#action_12714547
]
Jeff Hammerbacher edited comment on PIG-823 at 5/29/09 11:48 AM:
-----------------------------------------------------------------
It's an open source project and easily extensible. There are many extensions to
the service within Facebook to support more general information. Why not try to
add the desired "lower level" metadata to the existing service as a patch to
Hive, since it's already got pluggable backends and a server implementation
already defined? Also, could you better define what "close to HDFS" means?
There's a lot of HDFS metadata stored in the NameNode. Also, the initial
implementation of the metadata repository for Hive stored data in HDFS, but it
was found to be quite useful to have a separate service for metadata. Perhaps
you could learn from their experiences?
was (Author: hammer):
It's an open source project and easily extensible. There are many
extensions to the service within Facebook to support more general information.
Why not try to add them to the existing service, since it's already got
pluggable backends and a server implementation already defined?
> Hadoop Metadata Service
> -----------------------
>
> Key: PIG-823
> URL: https://issues.apache.org/jira/browse/PIG-823
> Project: Pig
> Issue Type: New Feature
> Reporter: Olga Natkovich
>
> This JIRA is created to track development of a metadata system for Hadoop.
> The goal of the system is to allow users and applications to register data
> stored on HDFS, search for the data available on HDFS, and associate metadata
> such as schema, statistics, etc. with a particular data unit or a data set
> stored on HDFS. The initial goal is to provide a fairly generic, low level
> abstraction that any user or application on HDFS can use to store an retrieve
> metadata. Over time a higher level abstractions closely tied to particular
> applications or tools can be developed.
> Over time, it would make sense for the metadata service to become a
> subproject within Hadoop. For now, the proposal is to make it a contrib to
> Pig since Pig SQL is likely to be the first user of the system.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
