Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Pig Wiki" for change 

The following page has been changed by AlanGates:

New page:
= Proposed Design for Pig Metadata Interface =
With the introduction of SQL, Pig needs to be able to communicate with external 
metadata services.  These communications will includes such
operations as creating, altering, and dropping databases, tables, etc.  It will 
also include metadata queries, such as requests to show available
tables, etc.  DDL operations of these sorts will be beyond the scope of the 
proposed metadata
interfaces for load and storage functions.  However, Pig should not be tightly 
tied to a single metadata implementation.  It should be able to
work with Owl, Hive's metastore, or any other metadata source that is added to 
Hadoop.  To this end this document proposes an interface for
operating with metadata systems.  Different metadata connectors can then be 
implemented, one for each metadata system.

== Interface ==

This interface will allow users to find information about tables, databases, 
etc. in the metadata store.  For each call, it will pass the portion
of the syntax tree relavant to the operation to the metadata connector.  These 
structures will be versioned.


     * An interface to encapsulate DDL operations.
    interface MetadataDDL {
        void createTable(CreateTable ct) throws IOException;
        void alterTable(AlterTable at) throws IOException;  // includes add and 
drop partition
        void dropTable(DropTable dt) throws IOException;
        SQLTable[] showTables(Database db) throws IOException;  // info 
returned in SQLTable includes info on partitions

        void createDatabase(CreateDatabase cd) throws IOException;
        void alterDatabase(AlterDatabase ad) throws IOException;
        void dropDatabase(DropDatabase dd) throws IOException;
        SQLDatabase[] showDatabases() throws IOException;


== Accessing Global Metadata From SQL ==
Pig will be configured to work with one global metadata source for a given set 
of SQL operations.  This configuration will be via Pig's
configuration file.  It will specify the URI of the server to use and the 
implementation of !MetadataDDL to use with this server.

== Accessing Global Metadata from Pig Latin ==
Pig Latin will not support a call to metadata within the language itself.  
Instead, it will support the ability to invoke a SQL DDL command.
This SQL will then be sent to the SQL parser and dispatched through the 
metadata service as before.

    A = load ...
    SQL {"create table myTable ..."};
    store Z into 'myTable' using OwlStorage();

Reply via email to