Added: tajo/site/docs/devel/_sources/functions/json_func.txt URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/_sources/functions/json_func.txt?rev=1674465&view=auto ============================================================================== --- tajo/site/docs/devel/_sources/functions/json_func.txt (added) +++ tajo/site/docs/devel/_sources/functions/json_func.txt Sat Apr 18 08:50:16 2015 @@ -0,0 +1,17 @@ +******************************* +JSON Functions +******************************* + +.. function:: json_extract_path_text (string json, string xpath) + + Extracts JSON string from a JSON string based on json path specified and returns JSON string pointed to by xPath + + :param string: + :param string: + :rtype: text + :example: + + .. code-block:: sql + + json_extract_path_text('{"test" : {"key" : "tajo"}}','$.test.key'); + > tajo
Added: tajo/site/docs/devel/_sources/hive_integration.txt URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/_sources/hive_integration.txt?rev=1674465&view=auto ============================================================================== --- tajo/site/docs/devel/_sources/hive_integration.txt (added) +++ tajo/site/docs/devel/_sources/hive_integration.txt Sat Apr 18 08:50:16 2015 @@ -0,0 +1,42 @@ +************************************* +Hive Integration +************************************* + +Apache Tajo⢠catalog supports HiveCatalogStore to integrate with Apache Hiveâ¢. +This integration allows Tajo to access all tables used in Apache Hive. +Depending on your purpose, you can execute either SQL queries or HiveQL queries on the +same tables managed in Apache Hive. + +In order to use this feature, you need to build Tajo with a specified maven profile +and then add some configs into ``conf/tajo-env.sh`` and ``conf/catalog-site.xml``. +This section describes how to setup HiveMetaStore integration. +This instruction would take no more than five minutes. + +You need to set your Hive home directory to the environment variable ``HIVE_HOME`` in conf/tajo-env.sh as follows: :: + + export HIVE_HOME=/path/to/your/hive/directory + +If you need to use jdbc to connect HiveMetaStore, you have to prepare MySQL jdbc driver. +Next, you should set the path of MySQL JDBC driver jar file to the environment variable HIVE_JDBC_DRIVER_DIR in conf/tajo-env.sh as follows: :: + + export HIVE_JDBC_DRIVER_DIR==/path/to/your/mysql_jdbc_driver/mysql-connector-java-x.x.x-bin.jar + +Finally, you should specify HiveCatalogStore as Tajo catalog driver class in ``conf/catalog-site.xml`` as follows: :: + + <property> + <name>tajo.catalog.store.class</name> + <value>org.apache.tajo.catalog.store.HiveCatalogStore</value> + </property> + +.. note:: + + Hive stores a list of partitions for each table in its metastore. If new partitions are + directly added to HDFS, HiveMetastore will not able aware of these partitions unless the user + ``ALTER TABLE table_name ADD PARTITION`` commands on each of the newly added partitions or + ``MSCK REPAIR TABLE table_name`` command. + + But current tajo doesn't provide ``ADD PARTITION`` command and hive doesn't provide an api for + responding to ``MSK REPAIR TABLE`` command. Thus, if you insert data to hive partitioned + table and you want to scan the updated partitions through Tajo, you must run following command on hive :: + + $ MSCK REPAIR TABLE [table_name]; Added: tajo/site/docs/devel/_sources/table_management/text.txt URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/_sources/table_management/text.txt?rev=1674465&view=auto ============================================================================== --- tajo/site/docs/devel/_sources/table_management/text.txt (added) +++ tajo/site/docs/devel/_sources/table_management/text.txt Sat Apr 18 08:50:16 2015 @@ -0,0 +1,115 @@ +************************************* +TEXT +************************************* + +A character-separated values plain-text file represents a tabular data set consisting of rows and columns. +Each row is a plan-text line. A line is usually broken by a character line feed ``\n`` or carriage-return ``\r``. +The line feed ``\n`` is the default delimiter in Tajo. Each record consists of multiple fields, separated by +some other character or string, most commonly a literal vertical bar ``|``, comma ``,`` or tab ``\t``. +The vertical bar is used as the default field delimiter in Tajo. + +========================================= +How to Create a TEXT Table ? +========================================= + +If you are not familiar with the ``CREATE TABLE`` statement, please refer to the Data Definition Language :doc:`/sql_language/ddl`. + +In order to specify a certain file format for your table, you need to use the ``USING`` clause in your ``CREATE TABLE`` +statement. The below is an example statement for creating a table using *TEXT* format. + +.. code-block:: sql + + CREATE TABLE + table1 ( + id int, + name text, + score float, + type text + ) USING TEXT; + +========================================= +Physical Properties +========================================= + +Some table storage formats provide parameters for enabling or disabling features and adjusting physical parameters. +The ``WITH`` clause in the CREATE TABLE statement allows users to set those parameters. + +*TEXT* format provides the following physical properties. + +* ``text.delimiter``: delimiter character. ``|`` or ``\u0001`` is usually used, and the default field delimiter is ``|``. +* ``text.null``: ``NULL`` character. The default ``NULL`` character is an empty string ``''``. Hive's default ``NULL`` character is ``'\\N'``. +* ``compression.codec``: Compression codec. You can enable compression feature and set specified compression algorithm. The compression algorithm used to compress files. The compression codec name should be the fully qualified class name inherited from `org.apache.hadoop.io.compress.CompressionCodec <https://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/compress/CompressionCodec.html>`_. By default, compression is disabled. +* ``text.serde``: custom (De)serializer class. ``org.apache.tajo.storage.text.CSVLineSerDe`` is the default (De)serializer class. +* ``timezone``: the time zone that the table uses for writting. When table rows are read or written, ```timestamp``` and ```time``` column values are adjusted by this timezone if it is set. Time zone can be an abbreviation form like 'PST' or 'DST'. Also, it accepts an offset-based form like 'UTC+9' or a location-based form like 'Asia/Seoul'. +* ``text.error-tolerance.max-num``: the maximum number of permissible parsing errors. This value should be an integer value. By default, ``text.error-tolerance.max-num`` is ``0``. According to the value, parsing errors will be handled in different ways. + * If ``text.error-tolerance.max-num < 0``, all parsing errors are ignored. + * If ``text.error-tolerance.max-num == 0``, any parsing error is not allowed. If any error occurs, the query will be failed. (default) + * If ``text.error-tolerance.max-num > 0``, the given number of parsing errors in each task will be pemissible. + +The following example is to set a custom field delimiter, ``NULL`` character, and compression codec: + +.. code-block:: sql + + CREATE TABLE table1 ( + id int, + name text, + score float, + type text + ) USING TEXT WITH('text.delimiter'='\u0001', + 'text.null'='\\N', + 'compression.codec'='org.apache.hadoop.io.compress.SnappyCodec'); + +.. warning:: + + Be careful when using ``\n`` as the field delimiter because *TEXT* format tables use ``\n`` as the line delimiter. + At the moment, Tajo does not provide a way to specify the line delimiter. + +========================================= +Custom (De)serializer +========================================= + +The *TEXT* format not only provides reading and writing interfaces for text data but also allows users to process custom +plan-text file formats with user-defined (De)serializer classes. +For example, with custom (de)serializers, Tajo can process JSON file formats or any specialized plan-text file formats. + +In order to specify a custom (De)serializer, set a physical property ``text.serde``. +The property value should be a fully qualified class name. + +For example: + +.. code-block:: sql + + CREATE TABLE table1 ( + id int, + name text, + score float, + type text + ) USING TEXT WITH ('text.serde'='org.my.storage.CustomSerializerDeserializer') + + +========================================= +Null Value Handling Issues +========================================= +In default, ``NULL`` character in *TEXT* format is an empty string ``''``. +In other words, an empty field is basically recognized as a ``NULL`` value in Tajo. +If a field domain is ``TEXT``, an empty field is recognized as a string value ``''`` instead of ``NULL`` value. +Besides, You can also use your own ``NULL`` character by specifying a physical property ``text.null``. + +========================================= +Compatibility Issues with Apache Hive⢠+========================================= + +*TEXT* tables generated in Tajo can be processed directly by Apache Hive⢠without further processing. +In this section, we explain some compatibility issue for users who use both Hive and Tajo. + +If you set a custom field delimiter, the *TEXT* tables cannot be directly used in Hive. +In order to specify the custom field delimiter in Hive, you need to use ``ROW FORMAT DELIMITED FIELDS TERMINATED BY`` +clause in a Hive's ``CREATE TABLE`` statement as follows: + +.. code-block:: sql + + CREATE TABLE table1 (id int, name string, score float, type string) + ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' + STORED AS TEXT + +To the best of our knowledge, there is not way to specify a custom ``NULL`` character in Hive. Added: tajo/site/docs/devel/functions/json_func.html URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/functions/json_func.html?rev=1674465&view=auto ============================================================================== --- tajo/site/docs/devel/functions/json_func.html (added) +++ tajo/site/docs/devel/functions/json_func.html Sat Apr 18 08:50:16 2015 @@ -0,0 +1,294 @@ + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>JSON Functions — Apache Tajo 0.11.0 documentation</title> + + + + + + + <link href='https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700' rel='stylesheet' type='text/css'> + + + + + + + + + + <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> + + + + <link rel="top" title="Apache Tajo 0.11.0 documentation" href="../index.html"/> + <link rel="up" title="Functions" href="../functions.html"/> + <link rel="next" title="Table Management" href="../table_management.html"/> + <link rel="prev" title="Network Functions and Operators" href="network_func_and_operators.html"/> + + + <script src="https://cdnjs.cloudflare.com/ajax/libs/modernizr/2.6.2/modernizr.min.js";></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-nav-search"> + <a href="../index.html" class="fa fa-home"> Apache Tajo</a> + <div role="search"> + <form id ="rtd-search-form" class="wy-form" action="../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + <ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="../introduction.html">Introduction</a></li> +<li class="toctree-l1"><a class="reference internal" href="../getting_started.html">Getting Started</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../getting_started.html#prerequisites">Prerequisites</a></li> +<li class="toctree-l2"><a class="reference internal" href="../getting_started.html#dowload-and-unpack-the-source-code">Dowload and unpack the source code</a></li> +<li class="toctree-l2"><a class="reference internal" href="../getting_started.html#build-source-code">Build source code</a></li> +<li class="toctree-l2"><a class="reference internal" href="../getting_started.html#setting-up-a-local-tajo-cluster">Setting up a local Tajo cluster</a></li> +<li class="toctree-l2"><a class="reference internal" href="../getting_started.html#first-query-execution">First query execution</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../configuration/preliminary.html">Preliminary</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/cluster_setup.html">Cluster Setup</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/tajo_master_configuration.html">Tajo Master Configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/worker_configuration.html">Worker Configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/catalog_configuration.html">Catalog Configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/ha_configuration.html">High Availability for TajoMaster</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/service_config_defaults.html">Cluster Service Configuration Defaults</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/tajo-site-xml.html">The tajo-site.xml File</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/catalog-site-xml.html">The catalog-site.xml File</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../tsql.html">Tajo Shell (TSQL)</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../tsql/meta_command.html">Meta Commands</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/dfs_command.html">Executing HDFS commands</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/variables.html">Session Variables</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/admin_command.html">Administration Commands</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/intro.html">Introducing to TSQL</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/single_command.html">Executing a single command</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/execute_file.html">Executing Queries from Files</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/background_command.html">Executing as background process</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../sql_language.html">SQL Language</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/data_model.html">Data Model</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/ddl.html">Data Definition Language</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/insert.html">INSERT (OVERWRITE) INTO</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/queries.html">Queries</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/sql_expression.html">SQL Expressions</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/predicates.html">Predicates</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../time_zone.html">Time Zone</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../time_zone.html#server-cluster-time-zone">Server Cluster Time Zone</a></li> +<li class="toctree-l2"><a class="reference internal" href="../time_zone.html#table-time-zone">Table Time Zone</a></li> +<li class="toctree-l2"><a class="reference internal" href="../time_zone.html#client-time-zone">Client Time Zone</a></li> +<li class="toctree-l2"><a class="reference internal" href="../time_zone.html#time-zone-id">Time Zone ID</a></li> +<li class="toctree-l2"><a class="reference internal" href="../time_zone.html#examples-of-time-zone">Examples of Time Zone</a></li> +</ul> +</li> +<li class="toctree-l1 current"><a class="reference internal" href="../functions.html">Functions</a><ul class="current"> +<li class="toctree-l2 current"><a class="reference internal" href="../functions.html#built-in-functions">Built-in Functions</a></li> +<li class="toctree-l2"><a class="reference internal" href="../functions.html#python-user-defined-functions">Python User-defined Functions</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../table_management.html">Table Management</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../table_management/table_overview.html">Overview of Tajo Tables</a></li> +<li class="toctree-l2"><a class="reference internal" href="../table_management/file_formats.html">File Formats</a></li> +<li class="toctree-l2"><a class="reference internal" href="../table_management/compression.html">Compression</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../table_partitioning.html">Table Partitioning</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../partitioning/intro_to_partitioning.html">Introduction to Partitioning</a></li> +<li class="toctree-l2"><a class="reference internal" href="../partitioning/column_partitioning.html">Column Partitioning</a></li> +<li class="toctree-l2"><a class="reference internal" href="../partitioning/range_partitioning.html">Range Partitioning</a></li> +<li class="toctree-l2"><a class="reference internal" href="../partitioning/hash_partitioning.html">Hash Partitioning</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../index_overview.html">Index (Experimental Feature)</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../index/types.html">Index Types</a></li> +<li class="toctree-l2"><a class="reference internal" href="../index/how_to_use.html">How to use index?</a></li> +<li class="toctree-l2"><a class="reference internal" href="../index/future_work.html">Future Works</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../backup_and_restore.html">Backup and Restore</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../backup_and_restore/catalog.html">Backup and Restore Catalog</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../hive_integration.html">Hive Integration</a></li> +<li class="toctree-l1"><a class="reference internal" href="../hbase_integration.html">HBase Integration</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../hbase_integration.html#create-table">CREATE TABLE</a></li> +<li class="toctree-l2"><a class="reference internal" href="../hbase_integration.html#drop-table">DROP TABLE</a></li> +<li class="toctree-l2"><a class="reference internal" href="../hbase_integration.html#insert-overwrite-into">INSERT (OVERWRITE) INTO</a></li> +<li class="toctree-l2"><a class="reference internal" href="../hbase_integration.html#usage">Usage</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../swift_integration.html">OpenStack Swift Integration</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../swift_integration.html#swift-configuration">Swift configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="../swift_integration.html#hadoop-configurations">Hadoop configurations</a></li> +<li class="toctree-l2"><a class="reference internal" href="../swift_integration.html#tajo-configuration">Tajo configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="../swift_integration.html#querying-on-swift">Querying on Swift</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../jdbc_driver.html">Tajo JDBC Driver</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../jdbc_driver.html#how-to-get-jdbc-driver">How to get JDBC driver</a></li> +<li class="toctree-l2"><a class="reference internal" href="../jdbc_driver.html#setting-the-classpath">Setting the CLASSPATH</a></li> +<li class="toctree-l2"><a class="reference internal" href="../jdbc_driver.html#an-example-jdbc-client">An Example JDBC Client</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../tajo_client_api.html">Tajo Client API</a></li> +<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li> +</ul> + + + </div> + + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="../index.html">Apache Tajo</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + <div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="../index.html">Docs</a> »</li> + + <li><a href="../functions.html">Functions</a> »</li> + + <li>JSON Functions</li> + <li class="wy-breadcrumbs-aside"> + + <a href="../_sources/functions/json_func.txt" rel="nofollow"> View page source</a> + + </li> + </ul> + <hr/> +</div> + <div role="main"> + + <div class="section" id="json-functions"> +<h1>JSON Functions<a class="headerlink" href="#json-functions" title="Permalink to this headline">¶</a></h1> +<dl class="function"> +<dt id="json_extract_path_text"> +<tt class="descname">json_extract_path_text</tt><big>(</big><em>string json</em>, <em>string xpath</em><big>)</big><a class="headerlink" href="#json_extract_path_text" title="Permalink to this definition">¶</a></dt> +<dd><p>Extracts JSON string from a JSON string based on json path specified and returns JSON string pointed to by xPath</p> +<table class="docutils field-list" frame="void" rules="none"> +<col class="field-name" /> +<col class="field-body" /> +<tbody valign="top"> +<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> +<li><strong>string</strong> – </li> +<li><strong>string</strong> – </li> +</ul> +</td> +</tr> +<tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body"><p class="first">text</p> +</td> +</tr> +<tr class="field-odd field"><th class="field-name">Example:</th><td class="field-body"></td> +</tr> +</tbody> +</table> +<div class="highlight-sql"><div class="highlight"><pre><span class="n">json_extract_path_text</span><span class="p">(</span><span class="s1">'{"test" : {"key" : "tajo"}}'</span><span class="p">,</span><span class="s1">'$.test.key'</span><span class="p">);</span> +<span class="o">></span> <span class="n">tajo</span> +</pre></div> +</div> +</dd></dl> + +</div> + + + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="../table_management.html" class="btn btn-neutral float-right" title="Table Management"/>Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="network_func_and_operators.html" class="btn btn-neutral" title="Network Functions and Operators"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2014, Apache Tajo Team. + </p> + </div> + + <a href="https://github.com/snide/sphinx_rtd_theme";>Sphinx theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a> +</footer> + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'../', + VERSION:'0.11.0', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="../_static/jquery.js"></script> + <script type="text/javascript" src="../_static/underscore.js"></script> + <script type="text/javascript" src="../_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="../_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file Added: tajo/site/docs/devel/hive_integration.html URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/hive_integration.html?rev=1674465&view=auto ============================================================================== --- tajo/site/docs/devel/hive_integration.html (added) +++ tajo/site/docs/devel/hive_integration.html Sat Apr 18 08:50:16 2015 @@ -0,0 +1,301 @@ + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>Hive Integration — Apache Tajo 0.11.0 documentation</title> + + + + + + + <link href='https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700' rel='stylesheet' type='text/css'> + + + + + + + + + + <link rel="stylesheet" href="_static/css/theme.css" type="text/css" /> + + + + <link rel="top" title="Apache Tajo 0.11.0 documentation" href="index.html"/> + <link rel="next" title="HBase Integration" href="hbase_integration.html"/> + <link rel="prev" title="Backup and Restore Catalog" href="backup_and_restore/catalog.html"/> + + + <script src="https://cdnjs.cloudflare.com/ajax/libs/modernizr/2.6.2/modernizr.min.js";></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-nav-search"> + <a href="index.html" class="fa fa-home"> Apache Tajo</a> + <div role="search"> + <form id ="rtd-search-form" class="wy-form" action="search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + <ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="introduction.html">Introduction</a></li> +<li class="toctree-l1"><a class="reference internal" href="getting_started.html">Getting Started</a><ul> +<li class="toctree-l2"><a class="reference internal" href="getting_started.html#prerequisites">Prerequisites</a></li> +<li class="toctree-l2"><a class="reference internal" href="getting_started.html#dowload-and-unpack-the-source-code">Dowload and unpack the source code</a></li> +<li class="toctree-l2"><a class="reference internal" href="getting_started.html#build-source-code">Build source code</a></li> +<li class="toctree-l2"><a class="reference internal" href="getting_started.html#setting-up-a-local-tajo-cluster">Setting up a local Tajo cluster</a></li> +<li class="toctree-l2"><a class="reference internal" href="getting_started.html#first-query-execution">First query execution</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a><ul> +<li class="toctree-l2"><a class="reference internal" href="configuration/preliminary.html">Preliminary</a></li> +<li class="toctree-l2"><a class="reference internal" href="configuration/cluster_setup.html">Cluster Setup</a></li> +<li class="toctree-l2"><a class="reference internal" href="configuration/tajo_master_configuration.html">Tajo Master Configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="configuration/worker_configuration.html">Worker Configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="configuration/catalog_configuration.html">Catalog Configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="configuration/ha_configuration.html">High Availability for TajoMaster</a></li> +<li class="toctree-l2"><a class="reference internal" href="configuration/service_config_defaults.html">Cluster Service Configuration Defaults</a></li> +<li class="toctree-l2"><a class="reference internal" href="configuration/tajo-site-xml.html">The tajo-site.xml File</a></li> +<li class="toctree-l2"><a class="reference internal" href="configuration/catalog-site-xml.html">The catalog-site.xml File</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="tsql.html">Tajo Shell (TSQL)</a><ul> +<li class="toctree-l2"><a class="reference internal" href="tsql/meta_command.html">Meta Commands</a></li> +<li class="toctree-l2"><a class="reference internal" href="tsql/dfs_command.html">Executing HDFS commands</a></li> +<li class="toctree-l2"><a class="reference internal" href="tsql/variables.html">Session Variables</a></li> +<li class="toctree-l2"><a class="reference internal" href="tsql/admin_command.html">Administration Commands</a></li> +<li class="toctree-l2"><a class="reference internal" href="tsql/intro.html">Introducing to TSQL</a></li> +<li class="toctree-l2"><a class="reference internal" href="tsql/single_command.html">Executing a single command</a></li> +<li class="toctree-l2"><a class="reference internal" href="tsql/execute_file.html">Executing Queries from Files</a></li> +<li class="toctree-l2"><a class="reference internal" href="tsql/background_command.html">Executing as background process</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="sql_language.html">SQL Language</a><ul> +<li class="toctree-l2"><a class="reference internal" href="sql_language/data_model.html">Data Model</a></li> +<li class="toctree-l2"><a class="reference internal" href="sql_language/ddl.html">Data Definition Language</a></li> +<li class="toctree-l2"><a class="reference internal" href="sql_language/insert.html">INSERT (OVERWRITE) INTO</a></li> +<li class="toctree-l2"><a class="reference internal" href="sql_language/queries.html">Queries</a></li> +<li class="toctree-l2"><a class="reference internal" href="sql_language/sql_expression.html">SQL Expressions</a></li> +<li class="toctree-l2"><a class="reference internal" href="sql_language/predicates.html">Predicates</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="time_zone.html">Time Zone</a><ul> +<li class="toctree-l2"><a class="reference internal" href="time_zone.html#server-cluster-time-zone">Server Cluster Time Zone</a></li> +<li class="toctree-l2"><a class="reference internal" href="time_zone.html#table-time-zone">Table Time Zone</a></li> +<li class="toctree-l2"><a class="reference internal" href="time_zone.html#client-time-zone">Client Time Zone</a></li> +<li class="toctree-l2"><a class="reference internal" href="time_zone.html#time-zone-id">Time Zone ID</a></li> +<li class="toctree-l2"><a class="reference internal" href="time_zone.html#examples-of-time-zone">Examples of Time Zone</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="functions.html">Functions</a><ul> +<li class="toctree-l2"><a class="reference internal" href="functions.html#built-in-functions">Built-in Functions</a></li> +<li class="toctree-l2"><a class="reference internal" href="functions.html#python-user-defined-functions">Python User-defined Functions</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="table_management.html">Table Management</a><ul> +<li class="toctree-l2"><a class="reference internal" href="table_management/table_overview.html">Overview of Tajo Tables</a></li> +<li class="toctree-l2"><a class="reference internal" href="table_management/file_formats.html">File Formats</a></li> +<li class="toctree-l2"><a class="reference internal" href="table_management/compression.html">Compression</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="table_partitioning.html">Table Partitioning</a><ul> +<li class="toctree-l2"><a class="reference internal" href="partitioning/intro_to_partitioning.html">Introduction to Partitioning</a></li> +<li class="toctree-l2"><a class="reference internal" href="partitioning/column_partitioning.html">Column Partitioning</a></li> +<li class="toctree-l2"><a class="reference internal" href="partitioning/range_partitioning.html">Range Partitioning</a></li> +<li class="toctree-l2"><a class="reference internal" href="partitioning/hash_partitioning.html">Hash Partitioning</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="index_overview.html">Index (Experimental Feature)</a><ul> +<li class="toctree-l2"><a class="reference internal" href="index/types.html">Index Types</a></li> +<li class="toctree-l2"><a class="reference internal" href="index/how_to_use.html">How to use index?</a></li> +<li class="toctree-l2"><a class="reference internal" href="index/future_work.html">Future Works</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="backup_and_restore.html">Backup and Restore</a><ul> +<li class="toctree-l2"><a class="reference internal" href="backup_and_restore/catalog.html">Backup and Restore Catalog</a></li> +</ul> +</li> +<li class="toctree-l1 current"><a class="current reference internal" href="">Hive Integration</a></li> +<li class="toctree-l1"><a class="reference internal" href="hbase_integration.html">HBase Integration</a><ul> +<li class="toctree-l2"><a class="reference internal" href="hbase_integration.html#create-table">CREATE TABLE</a></li> +<li class="toctree-l2"><a class="reference internal" href="hbase_integration.html#drop-table">DROP TABLE</a></li> +<li class="toctree-l2"><a class="reference internal" href="hbase_integration.html#insert-overwrite-into">INSERT (OVERWRITE) INTO</a></li> +<li class="toctree-l2"><a class="reference internal" href="hbase_integration.html#usage">Usage</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="swift_integration.html">OpenStack Swift Integration</a><ul> +<li class="toctree-l2"><a class="reference internal" href="swift_integration.html#swift-configuration">Swift configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="swift_integration.html#hadoop-configurations">Hadoop configurations</a></li> +<li class="toctree-l2"><a class="reference internal" href="swift_integration.html#tajo-configuration">Tajo configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="swift_integration.html#querying-on-swift">Querying on Swift</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="jdbc_driver.html">Tajo JDBC Driver</a><ul> +<li class="toctree-l2"><a class="reference internal" href="jdbc_driver.html#how-to-get-jdbc-driver">How to get JDBC driver</a></li> +<li class="toctree-l2"><a class="reference internal" href="jdbc_driver.html#setting-the-classpath">Setting the CLASSPATH</a></li> +<li class="toctree-l2"><a class="reference internal" href="jdbc_driver.html#an-example-jdbc-client">An Example JDBC Client</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="tajo_client_api.html">Tajo Client API</a></li> +<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li> +</ul> + + + </div> + + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="index.html">Apache Tajo</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + <div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="index.html">Docs</a> »</li> + + <li>Hive Integration</li> + <li class="wy-breadcrumbs-aside"> + + <a href="_sources/hive_integration.txt" rel="nofollow"> View page source</a> + + </li> + </ul> + <hr/> +</div> + <div role="main"> + + <div class="section" id="hive-integration"> +<h1>Hive Integration<a class="headerlink" href="#hive-integration" title="Permalink to this headline">¶</a></h1> +<p>Apache Tajo⢠catalog supports HiveCatalogStore to integrate with Apache Hiveâ¢. +This integration allows Tajo to access all tables used in Apache Hive. +Depending on your purpose, you can execute either SQL queries or HiveQL queries on the +same tables managed in Apache Hive.</p> +<p>In order to use this feature, you need to build Tajo with a specified maven profile +and then add some configs into <tt class="docutils literal"><span class="pre">conf/tajo-env.sh</span></tt> and <tt class="docutils literal"><span class="pre">conf/catalog-site.xml</span></tt>. +This section describes how to setup HiveMetaStore integration. +This instruction would take no more than five minutes.</p> +<p>You need to set your Hive home directory to the environment variable <tt class="docutils literal"><span class="pre">HIVE_HOME</span></tt> in conf/tajo-env.sh as follows:</p> +<div class="highlight-python"><div class="highlight"><pre>export HIVE_HOME=/path/to/your/hive/directory +</pre></div> +</div> +<p>If you need to use jdbc to connect HiveMetaStore, you have to prepare MySQL jdbc driver. +Next, you should set the path of MySQL JDBC driver jar file to the environment variable HIVE_JDBC_DRIVER_DIR in conf/tajo-env.sh as follows:</p> +<div class="highlight-python"><div class="highlight"><pre>export HIVE_JDBC_DRIVER_DIR==/path/to/your/mysql_jdbc_driver/mysql-connector-java-x.x.x-bin.jar +</pre></div> +</div> +<p>Finally, you should specify HiveCatalogStore as Tajo catalog driver class in <tt class="docutils literal"><span class="pre">conf/catalog-site.xml</span></tt> as follows:</p> +<div class="highlight-python"><div class="highlight"><pre><property> + <name>tajo.catalog.store.class</name> + <value>org.apache.tajo.catalog.store.HiveCatalogStore</value> +</property> +</pre></div> +</div> +<div class="admonition note"> +<p class="first admonition-title">Note</p> +<p>Hive stores a list of partitions for each table in its metastore. If new partitions are +directly added to HDFS, HiveMetastore will not able aware of these partitions unless the user +<tt class="docutils literal"><span class="pre">ALTER</span> <span class="pre">TABLE</span> <span class="pre">table_name</span> <span class="pre">ADD</span> <span class="pre">PARTITION</span></tt> commands on each of the newly added partitions or +<tt class="docutils literal"><span class="pre">MSCK</span> <span class="pre">REPAIR</span> <span class="pre">TABLE</span> <span class="pre">table_name</span></tt> command.</p> +<p>But current tajo doesn’t provide <tt class="docutils literal"><span class="pre">ADD</span> <span class="pre">PARTITION</span></tt> command and hive doesn’t provide an api for +responding to <tt class="docutils literal"><span class="pre">MSK</span> <span class="pre">REPAIR</span> <span class="pre">TABLE</span></tt> command. Thus, if you insert data to hive partitioned +table and you want to scan the updated partitions through Tajo, you must run following command on hive</p> +<div class="last highlight-python"><div class="highlight"><pre>$ MSCK REPAIR TABLE [table_name]; +</pre></div> +</div> +</div> +</div> + + + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="hbase_integration.html" class="btn btn-neutral float-right" title="HBase Integration"/>Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="backup_and_restore/catalog.html" class="btn btn-neutral" title="Backup and Restore Catalog"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2014, Apache Tajo Team. + </p> + </div> + + <a href="https://github.com/snide/sphinx_rtd_theme";>Sphinx theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a> +</footer> + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'./', + VERSION:'0.11.0', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file Added: tajo/site/docs/devel/table_management/text.html URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/table_management/text.html?rev=1674465&view=auto ============================================================================== --- tajo/site/docs/devel/table_management/text.html (added) +++ tajo/site/docs/devel/table_management/text.html Sat Apr 18 08:50:16 2015 @@ -0,0 +1,360 @@ + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>TEXT — Apache Tajo 0.11.0 documentation</title> + + + + + + + <link href='https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700' rel='stylesheet' type='text/css'> + + + + + + + + + + <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> + + + + <link rel="top" title="Apache Tajo 0.11.0 documentation" href="../index.html"/> + <link rel="up" title="File Formats" href="file_formats.html"/> + <link rel="next" title="RCFile" href="rcfile.html"/> + <link rel="prev" title="File Formats" href="file_formats.html"/> + + + <script src="https://cdnjs.cloudflare.com/ajax/libs/modernizr/2.6.2/modernizr.min.js";></script> + +</head> + +<body class="wy-body-for-nav" role="document"> + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-nav-search"> + <a href="../index.html" class="fa fa-home"> Apache Tajo</a> + <div role="search"> + <form id ="rtd-search-form" class="wy-form" action="../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + <ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="../introduction.html">Introduction</a></li> +<li class="toctree-l1"><a class="reference internal" href="../getting_started.html">Getting Started</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../getting_started.html#prerequisites">Prerequisites</a></li> +<li class="toctree-l2"><a class="reference internal" href="../getting_started.html#dowload-and-unpack-the-source-code">Dowload and unpack the source code</a></li> +<li class="toctree-l2"><a class="reference internal" href="../getting_started.html#build-source-code">Build source code</a></li> +<li class="toctree-l2"><a class="reference internal" href="../getting_started.html#setting-up-a-local-tajo-cluster">Setting up a local Tajo cluster</a></li> +<li class="toctree-l2"><a class="reference internal" href="../getting_started.html#first-query-execution">First query execution</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../configuration/preliminary.html">Preliminary</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/cluster_setup.html">Cluster Setup</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/tajo_master_configuration.html">Tajo Master Configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/worker_configuration.html">Worker Configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/catalog_configuration.html">Catalog Configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/ha_configuration.html">High Availability for TajoMaster</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/service_config_defaults.html">Cluster Service Configuration Defaults</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/tajo-site-xml.html">The tajo-site.xml File</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/catalog-site-xml.html">The catalog-site.xml File</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../tsql.html">Tajo Shell (TSQL)</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../tsql/meta_command.html">Meta Commands</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/dfs_command.html">Executing HDFS commands</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/variables.html">Session Variables</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/admin_command.html">Administration Commands</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/intro.html">Introducing to TSQL</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/single_command.html">Executing a single command</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/execute_file.html">Executing Queries from Files</a></li> +<li class="toctree-l2"><a class="reference internal" href="../tsql/background_command.html">Executing as background process</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../sql_language.html">SQL Language</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/data_model.html">Data Model</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/ddl.html">Data Definition Language</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/insert.html">INSERT (OVERWRITE) INTO</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/queries.html">Queries</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/sql_expression.html">SQL Expressions</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/predicates.html">Predicates</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../time_zone.html">Time Zone</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../time_zone.html#server-cluster-time-zone">Server Cluster Time Zone</a></li> +<li class="toctree-l2"><a class="reference internal" href="../time_zone.html#table-time-zone">Table Time Zone</a></li> +<li class="toctree-l2"><a class="reference internal" href="../time_zone.html#client-time-zone">Client Time Zone</a></li> +<li class="toctree-l2"><a class="reference internal" href="../time_zone.html#time-zone-id">Time Zone ID</a></li> +<li class="toctree-l2"><a class="reference internal" href="../time_zone.html#examples-of-time-zone">Examples of Time Zone</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../functions.html">Functions</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../functions.html#built-in-functions">Built-in Functions</a></li> +<li class="toctree-l2"><a class="reference internal" href="../functions.html#python-user-defined-functions">Python User-defined Functions</a></li> +</ul> +</li> +<li class="toctree-l1 current"><a class="reference internal" href="../table_management.html">Table Management</a><ul class="current"> +<li class="toctree-l2"><a class="reference internal" href="table_overview.html">Overview of Tajo Tables</a></li> +<li class="toctree-l2 current"><a class="reference internal" href="file_formats.html">File Formats</a></li> +<li class="toctree-l2"><a class="reference internal" href="compression.html">Compression</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../table_partitioning.html">Table Partitioning</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../partitioning/intro_to_partitioning.html">Introduction to Partitioning</a></li> +<li class="toctree-l2"><a class="reference internal" href="../partitioning/column_partitioning.html">Column Partitioning</a></li> +<li class="toctree-l2"><a class="reference internal" href="../partitioning/range_partitioning.html">Range Partitioning</a></li> +<li class="toctree-l2"><a class="reference internal" href="../partitioning/hash_partitioning.html">Hash Partitioning</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../index_overview.html">Index (Experimental Feature)</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../index/types.html">Index Types</a></li> +<li class="toctree-l2"><a class="reference internal" href="../index/how_to_use.html">How to use index?</a></li> +<li class="toctree-l2"><a class="reference internal" href="../index/future_work.html">Future Works</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../backup_and_restore.html">Backup and Restore</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../backup_and_restore/catalog.html">Backup and Restore Catalog</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../hive_integration.html">Hive Integration</a></li> +<li class="toctree-l1"><a class="reference internal" href="../hbase_integration.html">HBase Integration</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../hbase_integration.html#create-table">CREATE TABLE</a></li> +<li class="toctree-l2"><a class="reference internal" href="../hbase_integration.html#drop-table">DROP TABLE</a></li> +<li class="toctree-l2"><a class="reference internal" href="../hbase_integration.html#insert-overwrite-into">INSERT (OVERWRITE) INTO</a></li> +<li class="toctree-l2"><a class="reference internal" href="../hbase_integration.html#usage">Usage</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../swift_integration.html">OpenStack Swift Integration</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../swift_integration.html#swift-configuration">Swift configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="../swift_integration.html#hadoop-configurations">Hadoop configurations</a></li> +<li class="toctree-l2"><a class="reference internal" href="../swift_integration.html#tajo-configuration">Tajo configuration</a></li> +<li class="toctree-l2"><a class="reference internal" href="../swift_integration.html#querying-on-swift">Querying on Swift</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../jdbc_driver.html">Tajo JDBC Driver</a><ul> +<li class="toctree-l2"><a class="reference internal" href="../jdbc_driver.html#how-to-get-jdbc-driver">How to get JDBC driver</a></li> +<li class="toctree-l2"><a class="reference internal" href="../jdbc_driver.html#setting-the-classpath">Setting the CLASSPATH</a></li> +<li class="toctree-l2"><a class="reference internal" href="../jdbc_driver.html#an-example-jdbc-client">An Example JDBC Client</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../tajo_client_api.html">Tajo Client API</a></li> +<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li> +</ul> + + + </div> + + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="../index.html">Apache Tajo</a> + </nav> + + + + <div class="wy-nav-content"> + <div class="rst-content"> + <div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> + <li><a href="../index.html">Docs</a> »</li> + + <li><a href="../table_management.html">Table Management</a> »</li> + + <li><a href="file_formats.html">File Formats</a> »</li> + + <li>TEXT</li> + <li class="wy-breadcrumbs-aside"> + + <a href="../_sources/table_management/text.txt" rel="nofollow"> View page source</a> + + </li> + </ul> + <hr/> +</div> + <div role="main"> + + <div class="section" id="text"> +<h1>TEXT<a class="headerlink" href="#text" title="Permalink to this headline">¶</a></h1> +<p>A character-separated values plain-text file represents a tabular data set consisting of rows and columns. +Each row is a plan-text line. A line is usually broken by a character line feed <tt class="docutils literal"><span class="pre">\n</span></tt> or carriage-return <tt class="docutils literal"><span class="pre">\r</span></tt>. +The line feed <tt class="docutils literal"><span class="pre">\n</span></tt> is the default delimiter in Tajo. Each record consists of multiple fields, separated by +some other character or string, most commonly a literal vertical bar <tt class="docutils literal"><span class="pre">|</span></tt>, comma <tt class="docutils literal"><span class="pre">,</span></tt> or tab <tt class="docutils literal"><span class="pre">\t</span></tt>. +The vertical bar is used as the default field delimiter in Tajo.</p> +<div class="section" id="how-to-create-a-text-table"> +<h2>How to Create a TEXT Table ?<a class="headerlink" href="#how-to-create-a-text-table" title="Permalink to this headline">¶</a></h2> +<p>If you are not familiar with the <tt class="docutils literal"><span class="pre">CREATE</span> <span class="pre">TABLE</span></tt> statement, please refer to the Data Definition Language <a class="reference internal" href="../sql_language/ddl.html"><em>Data Definition Language</em></a>.</p> +<p>In order to specify a certain file format for your table, you need to use the <tt class="docutils literal"><span class="pre">USING</span></tt> clause in your <tt class="docutils literal"><span class="pre">CREATE</span> <span class="pre">TABLE</span></tt> +statement. The below is an example statement for creating a table using <em>TEXT</em> format.</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="k">CREATE</span> <span class="k">TABLE</span> + <span class="n">table1</span> <span class="p">(</span> + <span class="n">id</span> <span class="nb">int</span><span class="p">,</span> + <span class="n">name</span> <span class="nb">text</span><span class="p">,</span> + <span class="n">score</span> <span class="nb">float</span><span class="p">,</span> + <span class="k">type</span> <span class="nb">text</span> + <span class="p">)</span> <span class="k">USING</span> <span class="nb">TEXT</span><span class="p">;</span> +</pre></div> +</div> +</div> +<div class="section" id="physical-properties"> +<h2>Physical Properties<a class="headerlink" href="#physical-properties" title="Permalink to this headline">¶</a></h2> +<p>Some table storage formats provide parameters for enabling or disabling features and adjusting physical parameters. +The <tt class="docutils literal"><span class="pre">WITH</span></tt> clause in the CREATE TABLE statement allows users to set those parameters.</p> +<p><em>TEXT</em> format provides the following physical properties.</p> +<ul class="simple"> +<li><tt class="docutils literal"><span class="pre">text.delimiter</span></tt>: delimiter character. <tt class="docutils literal"><span class="pre">|</span></tt> or <tt class="docutils literal"><span class="pre">\u0001</span></tt> is usually used, and the default field delimiter is <tt class="docutils literal"><span class="pre">|</span></tt>.</li> +<li><tt class="docutils literal"><span class="pre">text.null</span></tt>: <tt class="docutils literal"><span class="pre">NULL</span></tt> character. The default <tt class="docutils literal"><span class="pre">NULL</span></tt> character is an empty string <tt class="docutils literal"><span class="pre">''</span></tt>. Hive’s default <tt class="docutils literal"><span class="pre">NULL</span></tt> character is <tt class="docutils literal"><span class="pre">'\\N'</span></tt>.</li> +<li><tt class="docutils literal"><span class="pre">compression.codec</span></tt>: Compression codec. You can enable compression feature and set specified compression algorithm. The compression algorithm used to compress files. The compression codec name should be the fully qualified class name inherited from <a class="reference external" href="https://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/compress/CompressionCodec.html";>org.apache.hadoop.io.compress.CompressionCodec</a>. By default, compression is disabled.</li> +<li><tt class="docutils literal"><span class="pre">text.serde</span></tt>: custom (De)serializer class. <tt class="docutils literal"><span class="pre">org.apache.tajo.storage.text.CSVLineSerDe</span></tt> is the default (De)serializer class.</li> +<li><tt class="docutils literal"><span class="pre">timezone</span></tt>: the time zone that the table uses for writting. When table rows are read or written, <tt class="docutils literal"><span class="pre">`timestamp`</span></tt> and <tt class="docutils literal"><span class="pre">`time`</span></tt> column values are adjusted by this timezone if it is set. Time zone can be an abbreviation form like ‘PST’ or ‘DST’. Also, it accepts an offset-based form like ‘UTC+9’ or a location-based form like ‘Asia/Seoul’.</li> +<li><tt class="docutils literal"><span class="pre">text.error-tolerance.max-num</span></tt>: the maximum number of permissible parsing errors. This value should be an integer value. By default, <tt class="docutils literal"><span class="pre">text.error-tolerance.max-num</span></tt> is <tt class="docutils literal"><span class="pre">0</span></tt>. According to the value, parsing errors will be handled in different ways. +* If <tt class="docutils literal"><span class="pre">text.error-tolerance.max-num</span> <span class="pre"><</span> <span class="pre">0</span></tt>, all parsing errors are ignored. +* If <tt class="docutils literal"><span class="pre">text.error-tolerance.max-num</span> <span class="pre">==</span> <span class="pre">0</span></tt>, any parsing error is not allowed. If any error occurs, the query will be failed. (default) +* If <tt class="docutils literal"><span class="pre">text.error-tolerance.max-num</span> <span class="pre">></span> <span class="pre">0</span></tt>, the given number of parsing errors in each task will be pemissible.</li> +</ul> +<p>The following example is to set a custom field delimiter, <tt class="docutils literal"><span class="pre">NULL</span></tt> character, and compression codec:</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">table1</span> <span class="p">(</span> + <span class="n">id</span> <span class="nb">int</span><span class="p">,</span> + <span class="n">name</span> <span class="nb">text</span><span class="p">,</span> + <span class="n">score</span> <span class="nb">float</span><span class="p">,</span> + <span class="k">type</span> <span class="nb">text</span> +<span class="p">)</span> <span class="k">USING</span> <span class="nb">TEXT</span> <span class="k">WITH</span><span class="p">(</span><span class="s1">'text.delimiter'</span><span class="o">=</span><span class="s1">'\u0001'</span><span class="p">,</span> + <span class="s1">'text.null'</span><span class="o">=</span><span class="s1">'\\N'</span><span class="p">,</span> + <span class="s1">'compression.codec'</span><span class="o">=</span><span class="s1">'org.apache.hadoop.io.compress.SnappyCodec'</span><span class="p">);</span> +</pre></div> +</div> +<div class="admonition warning"> +<p class="first admonition-title">Warning</p> +<p class="last">Be careful when using <tt class="docutils literal"><span class="pre">\n</span></tt> as the field delimiter because <em>TEXT</em> format tables use <tt class="docutils literal"><span class="pre">\n</span></tt> as the line delimiter. +At the moment, Tajo does not provide a way to specify the line delimiter.</p> +</div> +</div> +<div class="section" id="custom-de-serializer"> +<h2>Custom (De)serializer<a class="headerlink" href="#custom-de-serializer" title="Permalink to this headline">¶</a></h2> +<p>The <em>TEXT</em> format not only provides reading and writing interfaces for text data but also allows users to process custom +plan-text file formats with user-defined (De)serializer classes. +For example, with custom (de)serializers, Tajo can process JSON file formats or any specialized plan-text file formats.</p> +<p>In order to specify a custom (De)serializer, set a physical property <tt class="docutils literal"><span class="pre">text.serde</span></tt>. +The property value should be a fully qualified class name.</p> +<p>For example:</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">table1</span> <span class="p">(</span> + <span class="n">id</span> <span class="nb">int</span><span class="p">,</span> + <span class="n">name</span> <span class="nb">text</span><span class="p">,</span> + <span class="n">score</span> <span class="nb">float</span><span class="p">,</span> + <span class="k">type</span> <span class="nb">text</span> +<span class="p">)</span> <span class="k">USING</span> <span class="nb">TEXT</span> <span class="k">WITH</span> <span class="p">(</span><span class="s1">'text.serde'</span><span class="o">=</span><span class="s1">'org.my.storage.CustomSerializerDeserializer'</span><span class="p">)</span> +</pre></div> +</div> +</div> +<div class="section" id="null-value-handling-issues"> +<h2>Null Value Handling Issues<a class="headerlink" href="#null-value-handling-issues" title="Permalink to this headline">¶</a></h2> +<p>In default, <tt class="docutils literal"><span class="pre">NULL</span></tt> character in <em>TEXT</em> format is an empty string <tt class="docutils literal"><span class="pre">''</span></tt>. +In other words, an empty field is basically recognized as a <tt class="docutils literal"><span class="pre">NULL</span></tt> value in Tajo. +If a field domain is <tt class="docutils literal"><span class="pre">TEXT</span></tt>, an empty field is recognized as a string value <tt class="docutils literal"><span class="pre">''</span></tt> instead of <tt class="docutils literal"><span class="pre">NULL</span></tt> value. +Besides, You can also use your own <tt class="docutils literal"><span class="pre">NULL</span></tt> character by specifying a physical property <tt class="docutils literal"><span class="pre">text.null</span></tt>.</p> +</div> +<div class="section" id="compatibility-issues-with-apache-hive"> +<h2>Compatibility Issues with Apache Hiveâ¢<a class="headerlink" href="#compatibility-issues-with-apache-hive" title="Permalink to this headline">¶</a></h2> +<p><em>TEXT</em> tables generated in Tajo can be processed directly by Apache Hive⢠without further processing. +In this section, we explain some compatibility issue for users who use both Hive and Tajo.</p> +<p>If you set a custom field delimiter, the <em>TEXT</em> tables cannot be directly used in Hive. +In order to specify the custom field delimiter in Hive, you need to use <tt class="docutils literal"><span class="pre">ROW</span> <span class="pre">FORMAT</span> <span class="pre">DELIMITED</span> <span class="pre">FIELDS</span> <span class="pre">TERMINATED</span> <span class="pre">BY</span></tt> +clause in a Hive’s <tt class="docutils literal"><span class="pre">CREATE</span> <span class="pre">TABLE</span></tt> statement as follows:</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">table1</span> <span class="p">(</span><span class="n">id</span> <span class="nb">int</span><span class="p">,</span> <span class="n">name</span> <span class="n">string</span><span class="p">,</span> <span class="n">score</span> <span class="nb">float</span><span class="p">,</span> <span class="k">type</span> <span class="n">string</span><span class="p">)</span> +<span class="k">ROW</span> <span class="n">FORMAT</span> <span class="n">DELIMITED</span> <span class="n">FIELDS</span> <span class="n">TERMINATED</span> <span class="k">BY</span> <span class="s1">'|'</span> +<span class="n">STORED</span> <span class="k">AS</span> <span class="nb">TEXT</span> +</pre></div> +</div> +<p>To the best of our knowledge, there is not way to specify a custom <tt class="docutils literal"><span class="pre">NULL</span></tt> character in Hive.</p> +</div> +</div> + + + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="rcfile.html" class="btn btn-neutral float-right" title="RCFile"/>Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="file_formats.html" class="btn btn-neutral" title="File Formats"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + © Copyright 2014, Apache Tajo Team. + </p> + </div> + + <a href="https://github.com/snide/sphinx_rtd_theme";>Sphinx theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a> +</footer> + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'../', + VERSION:'0.11.0', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true + }; + </script> + <script type="text/javascript" src="../_static/jquery.js"></script> + <script type="text/javascript" src="../_static/underscore.js"></script> + <script type="text/javascript" src="../_static/doctools.js"></script> + + + + + + <script type="text/javascript" src="../_static/js/theme.js"></script> + + + + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + </script> + + +</body> +</html> \ No newline at end of file
