Added: tajo/site/docs/current/_sources/table_management/text.txt URL: http://svn.apache.org/viewvc/tajo/site/docs/current/_sources/table_management/text.txt?rev=1688350&view=auto ============================================================================== --- tajo/site/docs/current/_sources/table_management/text.txt (added) +++ tajo/site/docs/current/_sources/table_management/text.txt Tue Jun 30 04:16:52 2015 @@ -0,0 +1,115 @@ +************************************* +TEXT +************************************* + +A character-separated values plain-text file represents a tabular data set consisting of rows and columns. +Each row is a plan-text line. A line is usually broken by a character line feed ``\n`` or carriage-return ``\r``. +The line feed ``\n`` is the default delimiter in Tajo. Each record consists of multiple fields, separated by +some other character or string, most commonly a literal vertical bar ``|``, comma ``,`` or tab ``\t``. +The vertical bar is used as the default field delimiter in Tajo. + +========================================= +How to Create a TEXT Table ? +========================================= + +If you are not familiar with the ``CREATE TABLE`` statement, please refer to the Data Definition Language :doc:`/sql_language/ddl`. + +In order to specify a certain file format for your table, you need to use the ``USING`` clause in your ``CREATE TABLE`` +statement. The below is an example statement for creating a table using *TEXT* format. + +.. code-block:: sql + + CREATE TABLE + table1 ( + id int, + name text, + score float, + type text + ) USING TEXT; + +========================================= +Physical Properties +========================================= + +Some table storage formats provide parameters for enabling or disabling features and adjusting physical parameters. +The ``WITH`` clause in the CREATE TABLE statement allows users to set those parameters. + +*TEXT* format provides the following physical properties. + +* ``text.delimiter``: delimiter character. ``|`` or ``\u0001`` is usually used, and the default field delimiter is ``|``. +* ``text.null``: ``NULL`` character. The default ``NULL`` character is an empty string ``''``. Hive's default ``NULL`` character is ``'\\N'``. +* ``compression.codec``: Compression codec. You can enable compression feature and set specified compression algorithm. The compression algorithm used to compress files. The compression codec name should be the fully qualified class name inherited from `org.apache.hadoop.io.compress.CompressionCodec <https://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/compress/CompressionCodec.html>`_. By default, compression is disabled. +* ``text.serde``: custom (De)serializer class. ``org.apache.tajo.storage.text.CSVLineSerDe`` is the default (De)serializer class. +* ``timezone``: the time zone that the table uses for writting. When table rows are read or written, ```timestamp``` and ```time``` column values are adjusted by this timezone if it is set. Time zone can be an abbreviation form like 'PST' or 'DST'. Also, it accepts an offset-based form like 'UTC+9' or a location-based form like 'Asia/Seoul'. +* ``text.error-tolerance.max-num``: the maximum number of permissible parsing errors. This value should be an integer value. By default, ``text.error-tolerance.max-num`` is ``0``. According to the value, parsing errors will be handled in different ways. + * If ``text.error-tolerance.max-num < 0``, all parsing errors are ignored. + * If ``text.error-tolerance.max-num == 0``, any parsing error is not allowed. If any error occurs, the query will be failed. (default) + * If ``text.error-tolerance.max-num > 0``, the given number of parsing errors in each task will be pemissible. + +The following example is to set a custom field delimiter, ``NULL`` character, and compression codec: + +.. code-block:: sql + + CREATE TABLE table1 ( + id int, + name text, + score float, + type text + ) USING TEXT WITH('text.delimiter'='\u0001', + 'text.null'='\\N', + 'compression.codec'='org.apache.hadoop.io.compress.SnappyCodec'); + +.. warning:: + + Be careful when using ``\n`` as the field delimiter because *TEXT* format tables use ``\n`` as the line delimiter. + At the moment, Tajo does not provide a way to specify the line delimiter. + +========================================= +Custom (De)serializer +========================================= + +The *TEXT* format not only provides reading and writing interfaces for text data but also allows users to process custom +plan-text file formats with user-defined (De)serializer classes. +For example, with custom (de)serializers, Tajo can process JSON file formats or any specialized plan-text file formats. + +In order to specify a custom (De)serializer, set a physical property ``text.serde``. +The property value should be a fully qualified class name. + +For example: + +.. code-block:: sql + + CREATE TABLE table1 ( + id int, + name text, + score float, + type text + ) USING TEXT WITH ('text.serde'='org.my.storage.CustomSerializerDeserializer') + + +========================================= +Null Value Handling Issues +========================================= +In default, ``NULL`` character in *TEXT* format is an empty string ``''``. +In other words, an empty field is basically recognized as a ``NULL`` value in Tajo. +If a field domain is ``TEXT``, an empty field is recognized as a string value ``''`` instead of ``NULL`` value. +Besides, You can also use your own ``NULL`` character by specifying a physical property ``text.null``. + +========================================= +Compatibility Issues with Apache Hive⢠+========================================= + +*TEXT* tables generated in Tajo can be processed directly by Apache Hive⢠without further processing. +In this section, we explain some compatibility issue for users who use both Hive and Tajo. + +If you set a custom field delimiter, the *TEXT* tables cannot be directly used in Hive. +In order to specify the custom field delimiter in Hive, you need to use ``ROW FORMAT DELIMITED FIELDS TERMINATED BY`` +clause in a Hive's ``CREATE TABLE`` statement as follows: + +.. code-block:: sql + + CREATE TABLE table1 (id int, name string, score float, type string) + ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' + STORED AS TEXT + +To the best of our knowledge, there is not way to specify a custom ``NULL`` character in Hive.
Modified: tajo/site/docs/current/_static/basic.css URL: http://svn.apache.org/viewvc/tajo/site/docs/current/_static/basic.css?rev=1688350&r1=1688349&r2=1688350&view=diff ============================================================================== --- tajo/site/docs/current/_static/basic.css (original) +++ tajo/site/docs/current/_static/basic.css Tue Jun 30 04:16:52 2015 @@ -4,7 +4,7 @@ * * Sphinx stylesheet -- basic theme. * - * :copyright: Copyright 2007-2014 by the Sphinx team, see AUTHORS. + * :copyright: Copyright 2007-2015 by the Sphinx team, see AUTHORS. * :license: BSD, see LICENSE for details. * */ @@ -197,7 +197,10 @@ h3:hover > a.headerlink, h4:hover > a.headerlink, h5:hover > a.headerlink, h6:hover > a.headerlink, -dt:hover > a.headerlink { +dt:hover > a.headerlink, +caption:hover > a.headerlink, +p.caption:hover > a.headerlink, +div.code-block-caption:hover > a.headerlink { visibility: visible; } @@ -314,6 +317,13 @@ table.docutils { border-collapse: collapse; } +table caption span.caption-number { + font-style: italic; +} + +table caption span.caption-text { +} + table.docutils td, table.docutils th { padding: 1px 8px 1px 5px; border-top: 0; @@ -344,6 +354,25 @@ table.citation td { border-bottom: none; } +/* -- figures --------------------------------------------------------------- */ + +div.figure { + margin: 0.5em; + padding: 0.5em; +} + +div.figure p.caption { + padding: 0.3em; +} + +div.figure p.caption span.caption-number { + font-style: italic; +} + +div.figure p.caption span.caption-text { +} + + /* -- other body styles ----------------------------------------------------- */ ol.arabic { @@ -406,6 +435,10 @@ dl.glossary dt { font-size: 1.3em; } +.sig-paren { + font-size: larger; +} + .versionmodified { font-style: italic; } @@ -471,22 +504,51 @@ table.highlighttable td { padding: 0 0.5em 0 0.5em; } -tt.descname { +div.code-block-caption { + padding: 2px 5px; + font-size: small; +} + +div.code-block-caption code { + background-color: transparent; +} + +div.code-block-caption + div > div.highlight > pre { + margin-top: 0; +} + +div.code-block-caption span.caption-number { + padding: 0.1em 0.3em; + font-style: italic; +} + +div.code-block-caption span.caption-text { +} + +div.literal-block-wrapper { + padding: 1em 1em 0; +} + +div.literal-block-wrapper div.highlight { + margin: 0; +} + +code.descname { background-color: transparent; font-weight: bold; font-size: 1.2em; } -tt.descclassname { +code.descclassname { background-color: transparent; } -tt.xref, a tt { +code.xref, a code { background-color: transparent; font-weight: bold; } -h1 tt, h2 tt, h3 tt, h4 tt, h5 tt, h6 tt { +h1 code, h2 code, h3 code, h4 code, h5 code, h6 code { background-color: transparent; } Modified: tajo/site/docs/current/_static/doctools.js URL: http://svn.apache.org/viewvc/tajo/site/docs/current/_static/doctools.js?rev=1688350&r1=1688349&r2=1688350&view=diff ============================================================================== --- tajo/site/docs/current/_static/doctools.js (original) +++ tajo/site/docs/current/_static/doctools.js Tue Jun 30 04:16:52 2015 @@ -4,7 +4,7 @@ * * Sphinx JavaScript utilities for all documentation. * - * :copyright: Copyright 2007-2014 by the Sphinx team, see AUTHORS. + * :copyright: Copyright 2007-2015 by the Sphinx team, see AUTHORS. * :license: BSD, see LICENSE for details. * */ @@ -91,6 +91,30 @@ jQuery.fn.highlightText = function(text, }); }; +/* + * backward compatibility for jQuery.browser + * This will be supported until firefox bug is fixed. + */ +if (!jQuery.browser) { + jQuery.uaMatch = function(ua) { + ua = ua.toLowerCase(); + + var match = /(chrome)[ \/]([\w.]+)/.exec(ua) || + /(webkit)[ \/]([\w.]+)/.exec(ua) || + /(opera)(?:.*version|)[ \/]([\w.]+)/.exec(ua) || + /(msie) ([\w.]+)/.exec(ua) || + ua.indexOf("compatible") < 0 && /(mozilla)(?:.*? rv:([\w.]+)|)/.exec(ua) || + []; + + return { + browser: match[ 1 ] || "", + version: match[ 2 ] || "0" + }; + }; + jQuery.browser = {}; + jQuery.browser[jQuery.uaMatch(navigator.userAgent).browser] = true; +} + /** * Small JavaScript module for the documentation. */ @@ -152,9 +176,10 @@ var Documentation = { /** * workaround a firefox stupidity + * see: https://bugzilla.mozilla.org/show_bug.cgi?id=645075 */ fixFirefoxAnchorBug : function() { - if (document.location.hash && $.browser.mozilla) + if (document.location.hash) window.setTimeout(function() { document.location.href += ''; }, 10); Modified: tajo/site/docs/current/_static/down-pressed.png URL: http://svn.apache.org/viewvc/tajo/site/docs/current/_static/down-pressed.png?rev=1688350&r1=1688349&r2=1688350&view=diff ============================================================================== Binary files - no diff available. Modified: tajo/site/docs/current/_static/down.png URL: http://svn.apache.org/viewvc/tajo/site/docs/current/_static/down.png?rev=1688350&r1=1688349&r2=1688350&view=diff ============================================================================== Binary files - no diff available. Modified: tajo/site/docs/current/_static/file.png URL: http://svn.apache.org/viewvc/tajo/site/docs/current/_static/file.png?rev=1688350&r1=1688349&r2=1688350&view=diff ============================================================================== Binary files - no diff available.
