Modified: tajo/site/docs/devel/index/how_to_use.html URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/index/how_to_use.html?rev=1728396&r1=1728395&r2=1728396&view=diff ============================================================================== --- tajo/site/docs/devel/index/how_to_use.html (original) +++ tajo/site/docs/devel/index/how_to_use.html Thu Feb 4 00:37:56 2016 @@ -78,6 +78,7 @@ <li class="toctree-l2"><a class="reference internal" href="../configuration/service_config_defaults.html">Cluster Service Configuration Defaults</a></li> <li class="toctree-l2"><a class="reference internal" href="../configuration/tajo-site-xml.html">The tajo-site.xml File</a></li> <li class="toctree-l2"><a class="reference internal" href="../configuration/catalog-site-xml.html">The catalog-site.xml File</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/storage-site-json.html">The storage-site.json File</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="../tsql.html">Tajo Shell (TSQL)</a><ul> @@ -100,6 +101,7 @@ <li class="toctree-l2"><a class="reference internal" href="../sql_language/joins.html">Joins</a></li> <li class="toctree-l2"><a class="reference internal" href="../sql_language/sql_expression.html">SQL Expressions</a></li> <li class="toctree-l2"><a class="reference internal" href="../sql_language/predicates.html">Predicates</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/explain.html">EXPLAIN</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="../time_zone.html">Time Zone</a><ul> @@ -211,11 +213,12 @@ <h1>How to use index?<a class="headerlink" href="#how-to-use-index" title="Permalink to this headline">¶</a></h1> <div class="section" id="create-index"> <h2>1. Create index<a class="headerlink" href="#create-index" title="Permalink to this headline">¶</a></h2> -<p>The first step for utilizing index is index creation. You can create index using SQL (<a class="reference internal" href="../sql_language/ddl.html"><em>Data Definition Language</em></a>) or Tajo API (<a class="reference internal" href="../tajo_client_api.html"><em>Tajo Client API</em></a>). For example, you can create a BST index on the lineitem table by submitting the following SQL to Tajo.</p> +<p>The first step for utilizing index is to create an index. You can create an index using SQL (<a class="reference internal" href="../sql_language/ddl.html"><em>Data Definition Language</em></a>) or Tajo API (<a class="reference internal" href="../tajo_client_api.html"><em>Tajo Client API</em></a>). +For example, the following SQL statement will create a BST index on the lineitem table.</p> <div class="highlight-sql"><div class="highlight"><pre><span class="k">default</span><span class="o">></span> <span class="k">create</span> <span class="k">index</span> <span class="n">l_orderkey_idx</span> <span class="k">on</span> <span class="n">lineitem</span> <span class="p">(</span><span class="n">l_orderkey</span><span class="p">);</span> </pre></div> </div> -<p>If the index is created successfully, you can see the information about that index as follows:</p> +<p>If the index is created successfully, you can see the index information as follows:</p> <div class="highlight-python"><div class="highlight"><pre>default> \d lineitem table name: default.lineitem @@ -253,19 +256,26 @@ Indexes: </div> <div class="section" id="enable-disable-index-scans"> <h2>2. Enable/disable index scans<a class="headerlink" href="#enable-disable-index-scans" title="Permalink to this headline">¶</a></h2> -<p>When an index is successfully created, you must enable the index scan feature as follows:</p> +<p>Reading data using index is disabled by default. +So, exploiting the created index, you need a further step, enabling ‘index scan’ as following:</p> <div class="highlight-sql"><div class="highlight"><pre>default> \set INDEX_ENABLED true </pre></div> </div> -<p>If you don’t want to use the index scan feature anymore, you can simply disable it as follows:</p> +<p>If you don’t want to use index scan anymore, you can simply disable it as follows:</p> <div class="highlight-sql"><div class="highlight"><pre>default> \set INDEX_ENABLED false </pre></div> </div> <div class="admonition note"> <p class="first admonition-title">Note</p> -<p class="last">Once the index scan feature is enabled, Tajo currently always performs the index scan regardless of its efficiency. You should set this option when the expected number of retrieved tuples is sufficiently small.</p> +<p class="last">Once index scan is enabled, Tajo will perform ‘index scan’ if possible. In some cases, it may cause performance +degradation. If you always want to get better performance, you should either enable or disable ‘index scan’ +according to selectivity. Usually, the performance gain of index will increase when the selectivity is low.</p> </div> </div> +<div class="section" id="index-backup-and-restore"> +<h2>3. Index backup and restore<a class="headerlink" href="#index-backup-and-restore" title="Permalink to this headline">¶</a></h2> +<p>Tajo currently provides only the catalog backup and restore for index. Please refer to <a class="reference internal" href="../backup_and_restore/catalog.html"><em>Backup and Restore Catalog</em></a> for more information about catalog backup and restore.</p> +</div> </div> @@ -286,7 +296,7 @@ Indexes: <div role="contentinfo"> <p> - © Copyright 2014, Apache Tajo Team. + © Copyright 2015, Apache Tajo Team. </p> </div>
Modified: tajo/site/docs/devel/index/types.html URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/index/types.html?rev=1728396&r1=1728395&r2=1728396&view=diff ============================================================================== --- tajo/site/docs/devel/index/types.html (original) +++ tajo/site/docs/devel/index/types.html Thu Feb 4 00:37:56 2016 @@ -78,6 +78,7 @@ <li class="toctree-l2"><a class="reference internal" href="../configuration/service_config_defaults.html">Cluster Service Configuration Defaults</a></li> <li class="toctree-l2"><a class="reference internal" href="../configuration/tajo-site-xml.html">The tajo-site.xml File</a></li> <li class="toctree-l2"><a class="reference internal" href="../configuration/catalog-site-xml.html">The catalog-site.xml File</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/storage-site-json.html">The storage-site.json File</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="../tsql.html">Tajo Shell (TSQL)</a><ul> @@ -100,6 +101,7 @@ <li class="toctree-l2"><a class="reference internal" href="../sql_language/joins.html">Joins</a></li> <li class="toctree-l2"><a class="reference internal" href="../sql_language/sql_expression.html">SQL Expressions</a></li> <li class="toctree-l2"><a class="reference internal" href="../sql_language/predicates.html">Predicates</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/explain.html">EXPLAIN</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="../time_zone.html">Time Zone</a><ul> @@ -209,8 +211,8 @@ <div class="section" id="index-types"> <h1>Index Types<a class="headerlink" href="#index-types" title="Permalink to this headline">¶</a></h1> -<p>Currently, Tajo supports only one type of index, <code class="docutils literal"><span class="pre">TWO_LEVEL_BIN_TREE</span></code>, shortly <code class="docutils literal"><span class="pre">BST</span></code>. The BST index is a kind of binary search tree which is extended to be permanently stored on disk. It consists of two levels of nodes; a leaf node indexes the keys with the positions of data in an HDFS block and a root node indexes the keys with the leaf node indices.</p> -<p>When an index scan is started, the query engine first reads the root node and finds the search key. If it finds a leaf node corresponding to the search key, it subsequently finds the search key in that leaf node. Finally, it directly reads a tuple corresponding to the search key from HDFS.</p> +<p>Currently, Tajo supports only one type of index, <code class="docutils literal"><span class="pre">TWO_LEVEL_BIN_TREE</span></code>, shortly <code class="docutils literal"><span class="pre">BST</span></code>. The BST index is a kind of binary search tree which is extended to be permanently stored on disk. It consists of two levels of nodes; a leaf node indexes the keys with the offsets to data stored on HDFS, and a root node indexes the keys with the offsets to the leaf nodes.</p> +<p>When an index scan is started, the query engine first reads the root node and finds the search key. If it successfully finds a leaf node corresponding to the search key, it subsequently finds the search key in that leaf node. Finally, it directly reads a tuple corresponding to the search key from HDFS.</p> </div> @@ -231,7 +233,7 @@ <div role="contentinfo"> <p> - © Copyright 2014, Apache Tajo Team. + © Copyright 2015, Apache Tajo Team. </p> </div> Modified: tajo/site/docs/devel/index_overview.html URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/index_overview.html?rev=1728396&r1=1728395&r2=1728396&view=diff ============================================================================== --- tajo/site/docs/devel/index_overview.html (original) +++ tajo/site/docs/devel/index_overview.html Thu Feb 4 00:37:56 2016 @@ -77,6 +77,7 @@ <li class="toctree-l2"><a class="reference internal" href="configuration/service_config_defaults.html">Cluster Service Configuration Defaults</a></li> <li class="toctree-l2"><a class="reference internal" href="configuration/tajo-site-xml.html">The tajo-site.xml File</a></li> <li class="toctree-l2"><a class="reference internal" href="configuration/catalog-site-xml.html">The catalog-site.xml File</a></li> +<li class="toctree-l2"><a class="reference internal" href="configuration/storage-site-json.html">The storage-site.json File</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="tsql.html">Tajo Shell (TSQL)</a><ul> @@ -99,6 +100,7 @@ <li class="toctree-l2"><a class="reference internal" href="sql_language/joins.html">Joins</a></li> <li class="toctree-l2"><a class="reference internal" href="sql_language/sql_expression.html">SQL Expressions</a></li> <li class="toctree-l2"><a class="reference internal" href="sql_language/predicates.html">Predicates</a></li> +<li class="toctree-l2"><a class="reference internal" href="sql_language/explain.html">EXPLAIN</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="time_zone.html">Time Zone</a><ul> @@ -207,11 +209,6 @@ <div class="section" id="index-experimental-feature"> <h1>Index (Experimental Feature)<a class="headerlink" href="#index-experimental-feature" title="Permalink to this headline">¶</a></h1> <p>An index is a data structure that is used for efficient query processing. Using an index, the Tajo query engine can directly retrieve search values.</p> -<p>This is still an experimental feature. In order to use indexes, you must check out the source code of the <code class="docutils literal"><span class="pre">index_support</span></code> branch:</p> -<div class="highlight-python"><div class="highlight"><pre>git clone -b index_support https://git-wip-us.apache.org/repos/asf/tajo.git tajo-index -</pre></div> -</div> -<p>For the source code build, please refer to <a class="reference internal" href="getting_started.html"><em>Getting Started</em></a>.</p> <p>The following sections describe the supported index types, the query execution with an index, and the future works.</p> <div class="toctree-wrapper compound"> <ul> @@ -240,7 +237,7 @@ <div role="contentinfo"> <p> - © Copyright 2014, Apache Tajo Team. + © Copyright 2015, Apache Tajo Team. </p> </div> Modified: tajo/site/docs/devel/introduction.html URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/introduction.html?rev=1728396&r1=1728395&r2=1728396&view=diff ============================================================================== --- tajo/site/docs/devel/introduction.html (original) +++ tajo/site/docs/devel/introduction.html Thu Feb 4 00:37:56 2016 @@ -77,6 +77,7 @@ <li class="toctree-l2"><a class="reference internal" href="configuration/service_config_defaults.html">Cluster Service Configuration Defaults</a></li> <li class="toctree-l2"><a class="reference internal" href="configuration/tajo-site-xml.html">The tajo-site.xml File</a></li> <li class="toctree-l2"><a class="reference internal" href="configuration/catalog-site-xml.html">The catalog-site.xml File</a></li> +<li class="toctree-l2"><a class="reference internal" href="configuration/storage-site-json.html">The storage-site.json File</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="tsql.html">Tajo Shell (TSQL)</a><ul> @@ -99,6 +100,7 @@ <li class="toctree-l2"><a class="reference internal" href="sql_language/joins.html">Joins</a></li> <li class="toctree-l2"><a class="reference internal" href="sql_language/sql_expression.html">SQL Expressions</a></li> <li class="toctree-l2"><a class="reference internal" href="sql_language/predicates.html">Predicates</a></li> +<li class="toctree-l2"><a class="reference internal" href="sql_language/explain.html">EXPLAIN</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="time_zone.html">Time Zone</a><ul> @@ -235,7 +237,7 @@ queries in order to avoid the worst quer <div role="contentinfo"> <p> - © Copyright 2014, Apache Tajo Team. + © Copyright 2015, Apache Tajo Team. </p> </div> Modified: tajo/site/docs/devel/jdbc_driver.html URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/jdbc_driver.html?rev=1728396&r1=1728395&r2=1728396&view=diff ============================================================================== --- tajo/site/docs/devel/jdbc_driver.html (original) +++ tajo/site/docs/devel/jdbc_driver.html Thu Feb 4 00:37:56 2016 @@ -77,6 +77,7 @@ <li class="toctree-l2"><a class="reference internal" href="configuration/service_config_defaults.html">Cluster Service Configuration Defaults</a></li> <li class="toctree-l2"><a class="reference internal" href="configuration/tajo-site-xml.html">The tajo-site.xml File</a></li> <li class="toctree-l2"><a class="reference internal" href="configuration/catalog-site-xml.html">The catalog-site.xml File</a></li> +<li class="toctree-l2"><a class="reference internal" href="configuration/storage-site-json.html">The storage-site.json File</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="tsql.html">Tajo Shell (TSQL)</a><ul> @@ -99,6 +100,7 @@ <li class="toctree-l2"><a class="reference internal" href="sql_language/joins.html">Joins</a></li> <li class="toctree-l2"><a class="reference internal" href="sql_language/sql_expression.html">SQL Expressions</a></li> <li class="toctree-l2"><a class="reference internal" href="sql_language/predicates.html">Predicates</a></li> +<li class="toctree-l2"><a class="reference internal" href="sql_language/explain.html">EXPLAIN</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="time_zone.html">Time Zone</a><ul> @@ -222,9 +224,9 @@ In this section, we explain how to get J <div class="section" id="from-building-source-code"> <h3>From Building Source Code<a class="headerlink" href="#from-building-source-code" title="Permalink to this headline">¶</a></h3> <p>You can build Tajo from the source code and then get JAR files as follows:</p> -<div class="highlight-bash"><div class="highlight"><pre><span class="nv">$ </span>tar xzvf tajo-x.y.z-src.tar.gz -<span class="nv">$ </span>mvn clean package -DskipTests -Pdist -Dtar -<span class="nv">$ </span>ls -l tajo-dist/target/tajo-x.y.z/share/jdbc-dist/tajo-jdbc-x.y.z.jar +<div class="highlight-bash"><div class="highlight"><pre>$ tar xzvf tajo-x.y.z-src.tar.gz +$ mvn clean package -DskipTests -Pdist -Dtar +$ ls -l tajo-dist/target/tajo-x.y.z/share/jdbc-dist/tajo-jdbc-x.y.z.jar </pre></div> </div> </div> @@ -232,7 +234,7 @@ In this section, we explain how to get J <div class="section" id="setting-the-classpath"> <h2>Setting the CLASSPATH<a class="headerlink" href="#setting-the-classpath" title="Permalink to this headline">¶</a></h2> <p>In order to use the JDBC driver, you should add <code class="docutils literal"><span class="pre">tajo-jdbc-x.y.z.jar</span></code> in your <code class="docutils literal"><span class="pre">CLASSPATH</span></code>.</p> -<div class="highlight-bash"><div class="highlight"><pre><span class="nv">CLASSPATH</span><span class="o">=</span>path/to/tajo-jdbc-x.y.z.jar:<span class="nv">$CLASSPATH</span> +<div class="highlight-bash"><div class="highlight"><pre><span class="nv">CLASSPATH</span><span class="o">=</span>path/to/tajo-jdbc-x.y.z.jar:<span class="nv">$C</span>LASSPATH </pre></div> </div> </div> @@ -351,7 +353,7 @@ All tables are contained in <code class= <div role="contentinfo"> <p> - © Copyright 2014, Apache Tajo Team. + © Copyright 2015, Apache Tajo Team. </p> </div> Modified: tajo/site/docs/devel/partitioning/column_partitioning.html URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/partitioning/column_partitioning.html?rev=1728396&r1=1728395&r2=1728396&view=diff ============================================================================== --- tajo/site/docs/devel/partitioning/column_partitioning.html (original) +++ tajo/site/docs/devel/partitioning/column_partitioning.html Thu Feb 4 00:37:56 2016 @@ -78,6 +78,7 @@ <li class="toctree-l2"><a class="reference internal" href="../configuration/service_config_defaults.html">Cluster Service Configuration Defaults</a></li> <li class="toctree-l2"><a class="reference internal" href="../configuration/tajo-site-xml.html">The tajo-site.xml File</a></li> <li class="toctree-l2"><a class="reference internal" href="../configuration/catalog-site-xml.html">The catalog-site.xml File</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/storage-site-json.html">The storage-site.json File</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="../tsql.html">Tajo Shell (TSQL)</a><ul> @@ -100,6 +101,7 @@ <li class="toctree-l2"><a class="reference internal" href="../sql_language/joins.html">Joins</a></li> <li class="toctree-l2"><a class="reference internal" href="../sql_language/sql_expression.html">SQL Expressions</a></li> <li class="toctree-l2"><a class="reference internal" href="../sql_language/predicates.html">Predicates</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/explain.html">EXPLAIN</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="../time_zone.html">Time Zone</a><ul> @@ -214,26 +216,49 @@ <h2>How to Create a Column Partitioned Table<a class="headerlink" href="#how-to-create-a-column-partitioned-table" title="Permalink to this headline">¶</a></h2> <p>You can create a partitioned table by using the <code class="docutils literal"><span class="pre">PARTITION</span> <span class="pre">BY</span></code> clause. For a column partitioned table, you should use the <code class="docutils literal"><span class="pre">PARTITION</span> <span class="pre">BY</span> <span class="pre">COLUMN</span></code> clause with partition keys.</p> -<p>For example, assume there is a table <code class="docutils literal"><span class="pre">orders</span></code> composed of the following schema.</p> -<div class="highlight-python"><div class="highlight"><pre>id INT, -item_name TEXT, -price FLOAT -</pre></div> -</div> -<p>Also, assume that you want to use <code class="docutils literal"><span class="pre">order_date</span> <span class="pre">TEXT</span></code> and <code class="docutils literal"><span class="pre">ship_date</span> <span class="pre">TEXT</span></code> as the partition keys. -Then, you should create a table as follows:</p> -<div class="highlight-sql"><div class="highlight"><pre><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">orders</span> <span class="p">(</span> - <span class="n">id</span> <span class="nb">INT</span><span class="p">,</span> - <span class="n">item_name</span> <span class="nb">TEXT</span><span class="p">,</span> - <span class="n">price</span> -<span class="p">)</span> <span class="n">PARTITION</span> <span class="k">BY</span> <span class="k">COLUMN</span> <span class="p">(</span><span class="n">order_date</span> <span class="nb">TEXT</span><span class="p">,</span> <span class="n">ship_date</span> <span class="nb">TEXT</span><span class="p">);</span> +<p>For example, assume a table with the following schema.</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="n">id</span> <span class="nb">INT</span><span class="p">,</span> +<span class="n">name</span> <span class="nb">TEXT</span><span class="p">,</span> +<span class="n">gender</span> <span class="nb">char</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> +<span class="n">grade</span> <span class="nb">TEXT</span><span class="p">,</span> +<span class="n">country</span> <span class="nb">TEXT</span><span class="p">,</span> +<span class="n">city</span> <span class="nb">TEXT</span><span class="p">,</span> +<span class="n">phone</span> <span class="nb">TEXT</span> +<span class="p">);</span> </pre></div> </div> +<p>If you want to make country as partitioned column, your Tajo definition would be this:</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">student</span> <span class="p">(</span> + <span class="n">id</span> <span class="nb">INT</span><span class="p">,</span> + <span class="n">name</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">gender</span> <span class="nb">char</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> + <span class="n">grade</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">city</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">phone</span> <span class="nb">TEXT</span> +<span class="p">)</span> <span class="n">PARTITION</span> <span class="k">BY</span> <span class="k">COLUMN</span> <span class="p">(</span><span class="n">country</span> <span class="nb">TEXT</span><span class="p">);</span> +</pre></div> +</div> +<p>Let us assume you want to use more partition columns and parquet file format. Here’s an example statement to create a table:</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">student</span> <span class="p">(</span> + <span class="n">id</span> <span class="nb">INT</span><span class="p">,</span> + <span class="n">name</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">gender</span> <span class="nb">char</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> + <span class="n">grade</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">phone</span> <span class="nb">TEXT</span> +<span class="p">)</span> <span class="k">USING</span> <span class="n">PARQUET</span> +<span class="n">PARTITION</span> <span class="k">BY</span> <span class="k">COLUMN</span> <span class="p">(</span><span class="n">country</span> <span class="nb">TEXT</span><span class="p">,</span> <span class="n">city</span> <span class="nb">TEXT</span><span class="p">);</span> +</pre></div> +</div> +<p>The statement above creates the student table with id, name, grade, etc. The table is also partitioned and data is stored in parquet files.</p> +<p>You might have noticed that while the partitioning key columns are a part of the table DDL, theyâre only listed in the <code class="docutils literal"><span class="pre">PARTITION</span> <span class="pre">BY</span></code> clause. In Tajo, as data is written to disk, each partition of data will be automatically split out into different folders, e.g. country=USA/city=NEWYORK. During a read operation, Tajo will use the folder structure to quickly locate the right partitions and also return the partitioning columns as columns in the result set.</p> </div> -<div class="section" id="partition-pruning-on-column-partitioned-tables"> -<h2>Partition Pruning on Column Partitioned Tables<a class="headerlink" href="#partition-pruning-on-column-partitioned-tables" title="Permalink to this headline">¶</a></h2> -<p>The following predicates in the <code class="docutils literal"><span class="pre">WHERE</span></code> clause can be used to prune unqualified column partitions without processing -during query planning phase.</p> +<div class="section" id="querying-partitioned-tables"> +<h2>Querying Partitioned Tables<a class="headerlink" href="#querying-partitioned-tables" title="Permalink to this headline">¶</a></h2> +<p>If a table created using the <code class="docutils literal"><span class="pre">PARTITION</span> <span class="pre">BY</span></code> clause, a query can do partition pruning and scan only a fraction of the table relevant to the partitions specified by the query. Tajo currently does partition pruning if the partition predicates are specified in the WHERE clause. For example, if table student is partitioned on column country and column city, the following query retrieves rows in <code class="docutils literal"><span class="pre">country=KOREA\city=SEOUL</span></code> directory.</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">student</span> <span class="k">WHERE</span> <span class="n">country</span> <span class="o">=</span> <span class="s1">'KOREA'</span> <span class="k">AND</span> <span class="n">city</span> <span class="o">=</span> <span class="s1">'SEOUL'</span><span class="p">;</span> +</pre></div> +</div> +<p>The following predicates in the <code class="docutils literal"><span class="pre">WHERE</span></code> clause can be used to prune column partitions during query planning phase.</p> <ul class="simple"> <li><code class="docutils literal"><span class="pre">=</span></code></li> <li><code class="docutils literal"><span class="pre"><></span></code></li> @@ -245,10 +270,134 @@ during query planning phase.</p> <li>IN list predicates</li> </ul> </div> +<div class="section" id="add-data-to-partition-table"> +<h2>Add data to Partition Table<a class="headerlink" href="#add-data-to-partition-table" title="Permalink to this headline">¶</a></h2> +<p>Tajo provides a very useful feature of dynamic partitioning. You don’t need to use any syntax with both <code class="docutils literal"><span class="pre">INSERT</span> <span class="pre">INTO</span> <span class="pre">...</span> <span class="pre">SELECT</span></code> and <code class="docutils literal"><span class="pre">Create</span> <span class="pre">Table</span> <span class="pre">As</span> <span class="pre">Select(CTAS)</span></code> statments for dynamic partitioning. Tajo will automatically filter the data, create directories, move filtered data to appropriate directory and create partition over it.</p> +<p>For example, assume there are both <code class="docutils literal"><span class="pre">student_source</span></code> and <code class="docutils literal"><span class="pre">student</span></code> tables composed of the following schema.</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">student_source</span> <span class="p">(</span> + <span class="n">id</span> <span class="nb">INT</span><span class="p">,</span> + <span class="n">name</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">gender</span> <span class="nb">char</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> + <span class="n">grade</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">country</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">city</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">phone</span> <span class="nb">TEXT</span> +<span class="p">);</span> + +<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">student</span> <span class="p">(</span> + <span class="n">id</span> <span class="nb">INT</span><span class="p">,</span> + <span class="n">name</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">gender</span> <span class="nb">char</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> + <span class="n">grade</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">phone</span> <span class="nb">TEXT</span> +<span class="p">)</span> <span class="n">PARTITION</span> <span class="k">BY</span> <span class="k">COLUMN</span> <span class="p">(</span><span class="n">country</span> <span class="nb">TEXT</span><span class="p">,</span> <span class="n">city</span> <span class="nb">TEXT</span><span class="p">);</span> +</pre></div> +</div> +<div class="section" id="how-to-insert-dynamically-to-partition-table"> +<h3>How to INSERT dynamically to partition table<a class="headerlink" href="#how-to-insert-dynamically-to-partition-table" title="Permalink to this headline">¶</a></h3> +<p>If you want to load an entire country or an entire city in one fell swoop:</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="k">INSERT</span> <span class="n">OVERWRITE</span> <span class="k">INTO</span> <span class="n">student</span> +<span class="k">SELECT</span> <span class="n">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">gender</span><span class="p">,</span> <span class="n">grade</span><span class="p">,</span> <span class="n">phone</span><span class="p">,</span> <span class="n">country</span><span class="p">,</span> <span class="n">city</span> +<span class="k">FROM</span> <span class="n">student_source</span><span class="p">;</span> +</pre></div> +</div> +</div> +<div class="section" id="how-to-ctas-dynamically-to-partition-table"> +<h3>How to CTAS dynamically to partition table<a class="headerlink" href="#how-to-ctas-dynamically-to-partition-table" title="Permalink to this headline">¶</a></h3> +<p>when a partition table is created:</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="k">DROP</span> <span class="k">TABLE</span> <span class="n">if</span> <span class="k">exists</span> <span class="n">student</span><span class="p">;</span> + +<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">student</span> <span class="p">(</span> + <span class="n">id</span> <span class="nb">INT</span><span class="p">,</span> + <span class="n">name</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">gender</span> <span class="nb">char</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> + <span class="n">grade</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">phone</span> <span class="nb">TEXT</span> +<span class="p">)</span> <span class="n">PARTITION</span> <span class="k">BY</span> <span class="k">COLUMN</span> <span class="p">(</span><span class="n">country</span> <span class="nb">TEXT</span><span class="p">,</span> <span class="n">city</span> <span class="nb">TEXT</span><span class="p">)</span> +<span class="k">AS</span> <span class="k">SELECT</span> <span class="n">id</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">gender</span><span class="p">,</span> <span class="n">grade</span><span class="p">,</span> <span class="n">phone</span><span class="p">,</span> <span class="n">country</span><span class="p">,</span> <span class="n">city</span> +<span class="k">FROM</span> <span class="n">student_source</span><span class="p">;</span> +</pre></div> +</div> +<div class="admonition note"> +<p class="first admonition-title">Note</p> +<p class="last">When loading data into a partition, itâs necessary to include the partition columns as the last columns in the query. The column names in the source query donât need to match the partition column names.</p> +</div> +</div> +</div> <div class="section" id="compatibility-issues-with-apache-hive"> <h2>Compatibility Issues with Apache Hiveâ¢<a class="headerlink" href="#compatibility-issues-with-apache-hive" title="Permalink to this headline">¶</a></h2> -<p>If partitioned tables of Hive are created as external tables in Tajo, Tajo can process the Hive partitioned tables directly. -There haven’t known compatibility issues yet.</p> +<p>If partitioned tables of Hive are created as external tables in Tajo, Tajo can process the Hive partitioned tables directly.</p> +<div class="section" id="how-to-create-partition-table"> +<h3>How to create partition table<a class="headerlink" href="#how-to-create-partition-table" title="Permalink to this headline">¶</a></h3> +<p>If you create a partition table as follows in Tajo:</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="k">default</span><span class="o">></span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">student</span> <span class="p">(</span> + <span class="n">id</span> <span class="nb">INT</span><span class="p">,</span> + <span class="n">name</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">gender</span> <span class="nb">char</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> + <span class="n">grade</span> <span class="nb">TEXT</span><span class="p">,</span> + <span class="n">phone</span> <span class="nb">TEXT</span> +<span class="p">)</span> <span class="n">PARTITION</span> <span class="k">BY</span> <span class="k">COLUMN</span> <span class="p">(</span><span class="n">country</span> <span class="nb">TEXT</span><span class="p">,</span> <span class="n">city</span> <span class="nb">TEXT</span><span class="p">);</span> +</pre></div> +</div> +<p>And then you can get table information in Hive:</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="n">hive</span><span class="o">></span> <span class="k">desc</span> <span class="n">student</span><span class="p">;</span> +<span class="n">OK</span> +<span class="n">id</span> <span class="nb">int</span> +<span class="n">name</span> <span class="n">string</span> +<span class="n">gender</span> <span class="nb">char</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> +<span class="n">grade</span> <span class="n">string</span> +<span class="n">phone</span> <span class="n">string</span> +<span class="n">country</span> <span class="n">string</span> +<span class="n">city</span> <span class="n">string</span> + +<span class="o">#</span> <span class="n">Partition</span> <span class="n">Information</span> +<span class="o">#</span> <span class="n">col_name</span> <span class="n">data_type</span> <span class="k">comment</span> + +<span class="n">country</span> <span class="n">string</span> +<span class="n">city</span> <span class="n">string</span> +</pre></div> +</div> +<p>Or as you create the table in Hive:</p> +<div class="highlight-sql"><div class="highlight"><pre><span class="n">hive</span> <span class="o">></span> <span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">student</span> <span class="p">(</span> + <span class="n">id</span> <span class="nb">int</span><span class="p">,</span> + <span class="n">name</span> <span class="n">string</span><span class="p">,</span> + <span class="n">gender</span> <span class="nb">char</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> + <span class="n">grade</span> <span class="n">string</span><span class="p">,</span> + <span class="n">phone</span> <span class="n">string</span> +<span class="p">)</span> <span class="n">PARTITIONED</span> <span class="k">BY</span> <span class="p">(</span><span class="n">country</span> <span class="n">string</span><span class="p">,</span> <span class="n">city</span> <span class="n">string</span><span class="p">)</span> +<span class="k">ROW</span> <span class="n">FORMAT</span> <span class="n">DELIMITED</span> + <span class="n">FIELDS</span> <span class="n">TERMINATED</span> <span class="k">BY</span> <span class="s1">'|'</span> <span class="p">;</span> +</pre></div> +</div> +<p>You will see table information in Tajo:</p> +<div class="highlight-sql"><div class="highlight"><pre>default> \d student; +table name: default.student +table uri: hdfs://your_hdfs_namespace/user/hive/warehouse/student +store type: TEXT +number of rows: 0 +volume: 0 B +Options: + 'text.null'='\\N' + 'transient_lastDdlTime'='1438756422' + 'text.delimiter'='|' + +schema: +id INT4 +name TEXT +gender CHAR(1) +grade TEXT +phone TEXT + +Partitions: +type:COLUMN +columns::default.student.country (TEXT), default.student.city (TEXT) +</pre></div> +</div> +</div> +<div class="section" id="how-to-add-data-to-partition-table"> +<h3>How to add data to partition table<a class="headerlink" href="#how-to-add-data-to-partition-table" title="Permalink to this headline">¶</a></h3> +<p>In Tajo, you can add data dynamically to partition table of Hive with both <code class="docutils literal"><span class="pre">INSERT</span> <span class="pre">INTO</span> <span class="pre">...</span> <span class="pre">SELECT</span></code> and <code class="docutils literal"><span class="pre">Create</span> <span class="pre">Table</span> <span class="pre">As</span> <span class="pre">Select</span> <span class="pre">(CTAS)</span></code> statments. Tajo will automatically filter the data to HiveMetastore, create directories and move filtered data to appropriate directory on the distributed file system.</p> +</div> </div> </div> @@ -270,7 +419,7 @@ There haven’t known compatibility <div role="contentinfo"> <p> - © Copyright 2014, Apache Tajo Team. + © Copyright 2015, Apache Tajo Team. </p> </div> Modified: tajo/site/docs/devel/partitioning/hash_partitioning.html URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/partitioning/hash_partitioning.html?rev=1728396&r1=1728395&r2=1728396&view=diff ============================================================================== --- tajo/site/docs/devel/partitioning/hash_partitioning.html (original) +++ tajo/site/docs/devel/partitioning/hash_partitioning.html Thu Feb 4 00:37:56 2016 @@ -78,6 +78,7 @@ <li class="toctree-l2"><a class="reference internal" href="../configuration/service_config_defaults.html">Cluster Service Configuration Defaults</a></li> <li class="toctree-l2"><a class="reference internal" href="../configuration/tajo-site-xml.html">The tajo-site.xml File</a></li> <li class="toctree-l2"><a class="reference internal" href="../configuration/catalog-site-xml.html">The catalog-site.xml File</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/storage-site-json.html">The storage-site.json File</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="../tsql.html">Tajo Shell (TSQL)</a><ul> @@ -100,6 +101,7 @@ <li class="toctree-l2"><a class="reference internal" href="../sql_language/joins.html">Joins</a></li> <li class="toctree-l2"><a class="reference internal" href="../sql_language/sql_expression.html">SQL Expressions</a></li> <li class="toctree-l2"><a class="reference internal" href="../sql_language/predicates.html">Predicates</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/explain.html">EXPLAIN</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="../time_zone.html">Time Zone</a><ul> @@ -232,7 +234,7 @@ <div role="contentinfo"> <p> - © Copyright 2014, Apache Tajo Team. + © Copyright 2015, Apache Tajo Team. </p> </div> Modified: tajo/site/docs/devel/partitioning/intro_to_partitioning.html URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/partitioning/intro_to_partitioning.html?rev=1728396&r1=1728395&r2=1728396&view=diff ============================================================================== --- tajo/site/docs/devel/partitioning/intro_to_partitioning.html (original) +++ tajo/site/docs/devel/partitioning/intro_to_partitioning.html Thu Feb 4 00:37:56 2016 @@ -78,6 +78,7 @@ <li class="toctree-l2"><a class="reference internal" href="../configuration/service_config_defaults.html">Cluster Service Configuration Defaults</a></li> <li class="toctree-l2"><a class="reference internal" href="../configuration/tajo-site-xml.html">The tajo-site.xml File</a></li> <li class="toctree-l2"><a class="reference internal" href="../configuration/catalog-site-xml.html">The catalog-site.xml File</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/storage-site-json.html">The storage-site.json File</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="../tsql.html">Tajo Shell (TSQL)</a><ul> @@ -100,6 +101,7 @@ <li class="toctree-l2"><a class="reference internal" href="../sql_language/joins.html">Joins</a></li> <li class="toctree-l2"><a class="reference internal" href="../sql_language/sql_expression.html">SQL Expressions</a></li> <li class="toctree-l2"><a class="reference internal" href="../sql_language/predicates.html">Predicates</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/explain.html">EXPLAIN</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="../time_zone.html">Time Zone</a><ul> @@ -243,7 +245,7 @@ Currently, Apache Tajo only provides Apa <div role="contentinfo"> <p> - © Copyright 2014, Apache Tajo Team. + © Copyright 2015, Apache Tajo Team. </p> </div> Modified: tajo/site/docs/devel/partitioning/range_partitioning.html URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/partitioning/range_partitioning.html?rev=1728396&r1=1728395&r2=1728396&view=diff ============================================================================== --- tajo/site/docs/devel/partitioning/range_partitioning.html (original) +++ tajo/site/docs/devel/partitioning/range_partitioning.html Thu Feb 4 00:37:56 2016 @@ -78,6 +78,7 @@ <li class="toctree-l2"><a class="reference internal" href="../configuration/service_config_defaults.html">Cluster Service Configuration Defaults</a></li> <li class="toctree-l2"><a class="reference internal" href="../configuration/tajo-site-xml.html">The tajo-site.xml File</a></li> <li class="toctree-l2"><a class="reference internal" href="../configuration/catalog-site-xml.html">The catalog-site.xml File</a></li> +<li class="toctree-l2"><a class="reference internal" href="../configuration/storage-site-json.html">The storage-site.json File</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="../tsql.html">Tajo Shell (TSQL)</a><ul> @@ -100,6 +101,7 @@ <li class="toctree-l2"><a class="reference internal" href="../sql_language/joins.html">Joins</a></li> <li class="toctree-l2"><a class="reference internal" href="../sql_language/sql_expression.html">SQL Expressions</a></li> <li class="toctree-l2"><a class="reference internal" href="../sql_language/predicates.html">Predicates</a></li> +<li class="toctree-l2"><a class="reference internal" href="../sql_language/explain.html">EXPLAIN</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="../time_zone.html">Time Zone</a><ul> @@ -232,7 +234,7 @@ <div role="contentinfo"> <p> - © Copyright 2014, Apache Tajo Team. + © Copyright 2015, Apache Tajo Team. </p> </div> Modified: tajo/site/docs/devel/search.html URL: http://svn.apache.org/viewvc/tajo/site/docs/devel/search.html?rev=1728396&r1=1728395&r2=1728396&view=diff ============================================================================== --- tajo/site/docs/devel/search.html (original) +++ tajo/site/docs/devel/search.html Thu Feb 4 00:37:56 2016 @@ -75,6 +75,7 @@ <li class="toctree-l2"><a class="reference internal" href="configuration/service_config_defaults.html">Cluster Service Configuration Defaults</a></li> <li class="toctree-l2"><a class="reference internal" href="configuration/tajo-site-xml.html">The tajo-site.xml File</a></li> <li class="toctree-l2"><a class="reference internal" href="configuration/catalog-site-xml.html">The catalog-site.xml File</a></li> +<li class="toctree-l2"><a class="reference internal" href="configuration/storage-site-json.html">The storage-site.json File</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="tsql.html">Tajo Shell (TSQL)</a><ul> @@ -97,6 +98,7 @@ <li class="toctree-l2"><a class="reference internal" href="sql_language/joins.html">Joins</a></li> <li class="toctree-l2"><a class="reference internal" href="sql_language/sql_expression.html">SQL Expressions</a></li> <li class="toctree-l2"><a class="reference internal" href="sql_language/predicates.html">Predicates</a></li> +<li class="toctree-l2"><a class="reference internal" href="sql_language/explain.html">EXPLAIN</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="time_zone.html">Time Zone</a><ul> @@ -222,7 +224,7 @@ <div role="contentinfo"> <p> - © Copyright 2014, Apache Tajo Team. + © Copyright 2015, Apache Tajo Team. </p> </div>
