Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification.
The following page has been changed by AlanGates: http://wiki.apache.org/pig/MetadataInterfaceProposal New page: = Proposed Design for Pig Metadata Interface = With the introduction of SQL, Pig needs to be able to communicate with external metadata services. These communications will includes such operations as creating, altering, and dropping databases, tables, etc. It will also include metadata queries, such as requests to show available tables, etc. DDL operations of these sorts will be beyond the scope of the proposed metadata interfaces for load and storage functions. However, Pig should not be tightly tied to a single metadata implementation. It should be able to work with Owl, Hive's metastore, or any other metadata source that is added to Hadoop. To this end this document proposes an interface for operating with metadata systems. Different metadata connectors can then be implemented, one for each metadata system. == Interface == This interface will allow users to find information about tables, databases, etc. in the metadata store. For each call, it will pass the portion of the syntax tree relavant to the operation to the metadata connector. These structures will be versioned. {{{ /** * An interface to encapsulate DDL operations. */ interface MetadataDDL { void createTable(CreateTable ct) throws IOException; void alterTable(AlterTable at) throws IOException; // includes add and drop partition void dropTable(DropTable dt) throws IOException; SQLTable[] showTables(Database db) throws IOException; // info returned in SQLTable includes info on partitions void createDatabase(CreateDatabase cd) throws IOException; void alterDatabase(AlterDatabase ad) throws IOException; void dropDatabase(DropDatabase dd) throws IOException; SQLDatabase[] showDatabases() throws IOException; } }}} == Accessing Global Metadata From SQL == Pig will be configured to work with one global metadata source for a given set of SQL operations. This configuration will be via Pig's configuration file. It will specify the URI of the server to use and the implementation of !MetadataDDL to use with this server. == Accessing Global Metadata from Pig Latin == Pig Latin will not support a call to metadata within the language itself. Instead, it will support the ability to invoke a SQL DDL command. This SQL will then be sent to the SQL parser and dispatched through the metadata service as before. {{{ A = load ... ... SQL {"create table myTable ..."}; store Z into 'myTable' using OwlStorage(); }}}
