http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/content/file-structure-of-carbondata.html ---------------------------------------------------------------------- diff --git a/content/file-structure-of-carbondata.html b/content/file-structure-of-carbondata.html index c14ea6d..bd2be65 100644 --- a/content/file-structure-of-carbondata.html +++ b/content/file-structure-of-carbondata.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -249,7 +257,7 @@ <p>The file directory structure is as below:</p> <p><a href="../docs/images/2-1_1.png?raw=true" target="_blank" rel="noopener noreferrer"><img src="https://github.com/apache/carbondata/blob/master/docs/images/2-1_1.png?raw=true"; alt="File Directory Structure" style="max-width:100%;"></a></p> <ol> -<li>ModifiedTime.htmlt records the timestamp of the metadata with the modification time attribute of the file. When the drop table and create table are used, the modification time of the file is updated.This is common to all databases and hence is kept in parallel to databases</li> +<li>ModifiedTime.htmlt records the timestamp of the metadata with the modification time attribute of the file. When the drop table and create table are used, the modification time of the file is updated. This is common to all databases and hence is kept in parallel to databases</li> <li>The <strong>default</strong> is the database name and contains the user tables.default is used when user doesn't specify any database name;else user configured database name will be the directory name. user_table is the table name.</li> <li>Metadata directory stores schema files, tablestatus and dictionary files (including .dict, .dictmeta and .sortindex). There are three types of metadata data information files.</li> <li>data and index files are stored under directory named <strong>Fact</strong>. The Fact directory has a Part0 partition directory, where 0 is the partition number.</li>
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/content/how-to-contribute-to-apache-carbondata.html ---------------------------------------------------------------------- diff --git a/content/how-to-contribute-to-apache-carbondata.html b/content/how-to-contribute-to-apache-carbondata.html index 122b763..392814c 100644 --- a/content/how-to-contribute-to-apache-carbondata.html +++ b/content/how-to-contribute-to-apache-carbondata.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -238,7 +246,7 @@ create it. Please discuss your proposal with a committer or the component lead i alternatively, on the developer mailing list(<a href="mailto:[email protected]";>[email protected]</a>).</p> <p>If there?s an existing JIRA issue for your intended contribution, please comment about your intended work. Once the work is understood, a committer will assign the issue to you. -(If you don?t have a JIRA role yet, you?ll be added to the ?contributor? role.) If an issue is +(If you don?t have a JIRA role yet, you?ll be added to the "contributor" role.) If an issue is currently assigned, please check with the current assignee before reassigning.</p> <p>For moderate or large contributions, you should not start coding or writing a design doc unless there is a corresponding JIRA issue assigned to you for that work. Simple changes, @@ -334,7 +342,7 @@ When you make a revision, always push it in a new commit.</p> Please make sure those tests pass,the contribution cannot be merged otherwise.</p> <h4> <a id="lgtm" class="anchor" href="#lgtm" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>LGTM</h4> -<p>Once the reviewer is happy with the change, they?ll respond with an LGTM (?looks good to me!?). +<p>Once the reviewer is happy with the change, they?ll respond with an LGTM ("looks good to me!"). At this point, the committer will take over, possibly make some additional touch ups, and merge your changes into the codebase.</p> <p>In the case both the author and the reviewer are committers, either can merge the pull request. http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/content/index.html ---------------------------------------------------------------------- diff --git a/content/index.html b/content/index.html index f059d16..5c5b663 100644 --- a/content/index.html +++ b/content/index.html @@ -54,6 +54,9 @@ class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -310,6 +313,13 @@ </h4> <div class="linkblock"> <div class="block-row"> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a> + <span class="release-date">Oct 2018</span> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Apache+CarbonData+1.5.0+Release"; + class="whatsnew" target="_blank">what's new</a> + </div> + <div class="block-row"> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a> <span class="release-date">Aug 2018</span> @@ -468,7 +478,7 @@ to do is:</p> <ol class="orderlist"> <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; target="_blank">Download</a>the latest release. </li> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/content/installation-guide.html ---------------------------------------------------------------------- diff --git a/content/installation-guide.html b/content/installation-guide.html deleted file mode 100644 index 2e7fab6..0000000 --- a/content/installation-guide.html +++ /dev/null @@ -1,455 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'> - <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> - <title>CarbonData</title> - <style> - - </style> - <!-- Bootstrap --> - - <link rel="stylesheet" href="css/bootstrap.min.css"> - <link href="css/style.css" rel="stylesheet"> - <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> - <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> - <!--[if lt IE 9]> - <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script> - <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script> - <![endif]--> - <script src="js/jquery.min.js"></script> - <script src="js/bootstrap.min.js"></script> - - -</head> -<body> -<header> - <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper"> - <div class="container"> - <div class="navbar-header"> - <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse" - class="navbar-toggle collapsed" type="button"> - <span class="sr-only">Toggle navigation</span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - <a href="index.html" class="logo"> - <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/> - </a> - </div> - <div class="navbar-collapse collapse cd_navcontnt" id="navbar"> - <ul class="nav navbar-nav navbar-right navlist-custom"> - <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a> - </li> - <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true" - aria-expanded="false"> Download <span class="caret"></span></a> - <ul class="dropdown-menu"> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; - target="_blank">Apache CarbonData 1.4.1</a></li> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/"; - target="_blank">Apache CarbonData 1.4.0</a></li> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; - target="_blank">Apache CarbonData 1.3.1</a></li> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> - <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; - target="_blank">Release Archive</a></li> - </ul> - </li> - <li><a href="mainpage.html" class="active">Documentation</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" - aria-expanded="false">Community <span class="caret"></span></a> - <ul class="dropdown-menu"> - <li> - <a href="https://github.com/apache/carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"; - target="_blank">Contributing to CarbonData</a></li> - <li> - <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"; - target="_blank">Release Guide</a></li> - <li> - <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"; - target="_blank">Project PMC and Committers</a></li> - <li> - <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; - target="_blank">CarbonData Meetups</a></li> - <li><a href="security.html">Apache CarbonData Security</a></li> - <li><a href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache - Jira</a></li> - <li><a href="videogallery.html">CarbonData Videos </a></li> - </ul> - </li> - <li class="dropdown"> - <a href="http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" - data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> - <ul class="dropdown-menu"> - <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> - <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> - <li><a href="http://www.apache.org/foundation/sponsorship.html"; - target="_blank">Sponsorship</a></li> - <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> - </ul> - </li> - - <li class="dropdown"> - <a href="http://www.apache.org/"; class="hidden-lg hidden-md hidden-sm dropdown-toggle" - data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> - <ul class="dropdown-menu"> - <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> - <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> - <li><a href="http://www.apache.org/foundation/sponsorship.html"; - target="_blank">Sponsorship</a></li> - <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> - </ul> - </li> - - <li> - <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a> - - </li> - - </ul> - </div><!--/.nav-collapse --> - <div id="search-box"> - <form method="get" action="http://www.google.com/search"; target="_blank"> - <div class="search-block"> - <table border="0" cellpadding="0" width="100%"> - <tr> - <td style="width:80%"> - <input type="text" name="q" size=" 5" maxlength="255" value="" - class="search-input" placeholder="Search...." required/> - </td> - <td style="width:20%"> - <input type="submit" value="Search"/></td> - </tr> - <tr> - <td align="left" style="font-size:75%" colspan="2"> - <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/> - <span style=" position: relative; top: -3px;"> Only search for CarbonData</span> - </td> - </tr> - </table> - </div> - </form> - </div> - </div> - </nav> -</header> <!-- end Header part --> - -<div class="fixed-padding"></div> <!-- top padding with fixde header --> - -<section><!-- Dashboard nav --> - <div class="container-fluid q"> - <div class="col-sm-12 col-md-12 maindashboard"> - <div class="row"> - <section> - <div style="padding:10px 15px;"> - <div id="viewpage" name="viewpage"> - <div class="row"> - <div class="col-sm-12 col-md-12"> - <div> -<h1> -<a id="installation-guide" class="anchor" href="#installation-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installation Guide</h1> -<p>This tutorial guides you through the installation and configuration of CarbonData in the following two modes :</p> -<ul> -<li><a href="#installing-and-configuring-carbondata-on-standalone-spark-cluster">Installing and Configuring CarbonData on Standalone Spark Cluster</a></li> -<li><a href="#installing-and-configuring-carbondata-on-spark-on-yarn-cluster">Installing and Configuring CarbonData on Spark on YARN Cluster</a></li> -</ul> -<p>followed by :</p> -<ul> -<li><a href="#query-execution-using-carbondata-thrift-server">Query Execution using CarbonData Thrift Server</a></li> -</ul> -<h2> -<a id="installing-and-configuring-carbondata-on-standalone-spark-cluster" class="anchor" href="#installing-and-configuring-carbondata-on-standalone-spark-cluster" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing and Configuring CarbonData on Standalone Spark Cluster</h2> -<h3> -<a id="prerequisites" class="anchor" href="#prerequisites" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prerequisites</h3> -<ul> -<li> -<p>Hadoop HDFS and Yarn should be installed and running.</p> -</li> -<li> -<p>Spark should be installed and running on all the cluster nodes.</p> -</li> -<li> -<p>CarbonData user should have permission to access HDFS.</p> -</li> -</ul> -<h3> -<a id="procedure" class="anchor" href="#procedure" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Procedure</h3> -<ol> -<li> -<p><a href="https://github.com/apache/carbondata/blob/master/build/README.md"; target=_blank>Build the CarbonData</a> project and get the assembly jar from <code>./assembly/target/scala-2.1x/carbondata_xxx.jar</code>.</p> -</li> -<li> -<p>Copy <code>./assembly/target/scala-2.1x/carbondata_xxx.jar</code> to <code>$SPARK_HOME/carbonlib</code> folder.</p> -<p><strong>NOTE</strong>: Create the carbonlib folder if it does not exist inside <code>$SPARK_HOME</code> path.</p> -</li> -<li> -<p>Add the carbonlib folder path in the Spark classpath. (Edit <code>$SPARK_HOME/conf/spark-env.sh</code> file and modify the value of <code>SPARK_CLASSPATH</code> by appending <code>$SPARK_HOME/carbonlib/*</code> to the existing value)</p> -</li> -<li> -<p>Copy the <code>./conf/carbon.properties.template</code> file from CarbonData repository to <code>$SPARK_HOME/conf/</code> folder and rename the file to <code>carbon.properties</code>.</p> -</li> -<li> -<p>Repeat Step 2 to Step 5 in all the nodes of the cluster.</p> -</li> -<li> -<p>In Spark node[master], configure the properties mentioned in the following table in <code>$SPARK_HOME/conf/spark-defaults.conf</code> file.</p> -</li> -</ol> -<table> -<thead> -<tr> -<th>Property</th> -<th>Value</th> -<th>Description</th> -</tr> -</thead> -<tbody> -<tr> -<td>spark.driver.extraJavaOptions</td> -<td><code>-Dcarbon.properties.filepath = $SPARK_HOME/conf/carbon.properties</code></td> -<td>A string of extra JVM options to pass to the driver. For instance, GC settings or other logging.</td> -</tr> -<tr> -<td>spark.executor.extraJavaOptions</td> -<td><code>-Dcarbon.properties.filepath = $SPARK_HOME/conf/carbon.properties</code></td> -<td>A string of extra JVM options to pass to executors. For instance, GC settings or other logging. <strong>NOTE</strong>: You can enter multiple values separated by space.</td> -</tr> -</tbody> -</table> -<ol start="7"> -<li>Add the following properties in <code>$SPARK_HOME/conf/carbon.properties</code> file:</li> -</ol> -<table> -<thead> -<tr> -<th>Property</th> -<th>Required</th> -<th>Description</th> -<th>Example</th> -<th>Remark</th> -</tr> -</thead> -<tbody> -<tr> -<td>carbon.storelocation</td> -<td>NO</td> -<td>Location where data CarbonData will create the store and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path.</td> -<td>hdfs://HOSTNAME:PORT/Opt/CarbonStore</td> -<td>Propose to set HDFS directory</td> -</tr> -</tbody> -</table> -<ol start="8"> -<li>Verify the installation. For example:</li> -</ol> -<pre><code>./spark-shell --master spark://HOSTNAME:PORT --total-executor-cores 2 ---executor-memory 2G -</code></pre> -<p><strong>NOTE</strong>: Make sure you have permissions for CarbonData JARs and files through which driver and executor will start.</p> -<p>To get started with CarbonData : <a href="quick-start-guide.html">Quick Start</a>, <a href="data-management-on-carbondata.html">Data Management on CarbonData</a></p> -<h2> -<a id="installing-and-configuring-carbondata-on-spark-on-yarn-cluster" class="anchor" href="#installing-and-configuring-carbondata-on-spark-on-yarn-cluster" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing and Configuring CarbonData on Spark on YARN Cluster</h2> -<p>This section provides the procedure to install CarbonData on "Spark on YARN" cluster.</p> -<h3> -<a id="prerequisites-1" class="anchor" href="#prerequisites-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prerequisites</h3> -<ul> -<li>Hadoop HDFS and Yarn should be installed and running.</li> -<li>Spark should be installed and running in all the clients.</li> -<li>CarbonData user should have permission to access HDFS.</li> -</ul> -<h3> -<a id="procedure-1" class="anchor" href="#procedure-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Procedure</h3> -<p>The following steps are only for Driver Nodes. (Driver nodes are the one which starts the spark context.)</p> -<ol> -<li> -<p><a href="https://github.com/apache/carbondata/blob/master/build/README.md"; target=_blank>Build the CarbonData</a> project and get the assembly jar from <code>./assembly/target/scala-2.1x/carbondata_xxx.jar</code> and copy to <code>$SPARK_HOME/carbonlib</code> folder.</p> -<p><strong>NOTE</strong>: Create the carbonlib folder if it does not exists inside <code>$SPARK_HOME</code> path.</p> -</li> -<li> -<p>Copy the <code>./conf/carbon.properties.template</code> file from CarbonData repository to <code>$SPARK_HOME/conf/</code> folder and rename the file to <code>carbon.properties</code>.</p> -</li> -<li> -<p>Create <code>tar.gz</code> file of carbonlib folder and move it inside the carbonlib folder.</p> -</li> -</ol> -<pre><code>cd $SPARK_HOME -tar -zcvf carbondata.tar.gz carbonlib/ -mv carbondata.tar.gz carbonlib/ -</code></pre> -<ol start="4"> -<li>Configure the properties mentioned in the following table in <code>$SPARK_HOME/conf/spark-defaults.conf</code> file.</li> -</ol> -<table> -<thead> -<tr> -<th>Property</th> -<th>Description</th> -<th>Value</th> -</tr> -</thead> -<tbody> -<tr> -<td>spark.master</td> -<td>Set this value to run the Spark in yarn cluster mode.</td> -<td>Set yarn-client to run the Spark in yarn cluster mode.</td> -</tr> -<tr> -<td>spark.yarn.dist.files</td> -<td>Comma-separated list of files to be placed in the working directory of each executor.</td> -<td><code>$SPARK_HOME/conf/carbon.properties</code></td> -</tr> -<tr> -<td>spark.yarn.dist.archives</td> -<td>Comma-separated list of archives to be extracted into the working directory of each executor.</td> -<td><code>$SPARK_HOME/carbonlib/carbondata.tar.gz</code></td> -</tr> -<tr> -<td>spark.executor.extraJavaOptions</td> -<td>A string of extra JVM options to pass to executors. For instance <strong>NOTE</strong>: You can enter multiple values separated by space.</td> -<td><code>-Dcarbon.properties.filepath = carbon.properties</code></td> -</tr> -<tr> -<td>spark.executor.extraClassPath</td> -<td>Extra classpath entries to prepend to the classpath of executors. <strong>NOTE</strong>: If SPARK_CLASSPATH is defined in spark-env.sh, then comment it and append the values in below parameter spark.driver.extraClassPath</td> -<td><code>carbondata.tar.gz/carbonlib/*</code></td> -</tr> -<tr> -<td>spark.driver.extraClassPath</td> -<td>Extra classpath entries to prepend to the classpath of the driver. <strong>NOTE</strong>: If SPARK_CLASSPATH is defined in spark-env.sh, then comment it and append the value in below parameter spark.driver.extraClassPath.</td> -<td><code>$SPARK_HOME/carbonlib/*</code></td> -</tr> -<tr> -<td>spark.driver.extraJavaOptions</td> -<td>A string of extra JVM options to pass to the driver. For instance, GC settings or other logging.</td> -<td><code>-Dcarbon.properties.filepath = $SPARK_HOME/conf/carbon.properties</code></td> -</tr> -</tbody> -</table> -<ol start="5"> -<li>Add the following properties in <code>$SPARK_HOME/conf/carbon.properties</code>:</li> -</ol> -<table> -<thead> -<tr> -<th>Property</th> -<th>Required</th> -<th>Description</th> -<th>Example</th> -<th>Default Value</th> -</tr> -</thead> -<tbody> -<tr> -<td>carbon.storelocation</td> -<td>NO</td> -<td>Location where CarbonData will create the store and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path.</td> -<td>hdfs://HOSTNAME:PORT/Opt/CarbonStore</td> -<td>Propose to set HDFS directory</td> -</tr> -</tbody> -</table> -<ol start="6"> -<li>Verify the installation.</li> -</ol> -<pre><code> ./bin/spark-shell --master yarn-client --driver-memory 1g - --executor-cores 2 --executor-memory 2G -</code></pre> -<p><strong>NOTE</strong>: Make sure you have permissions for CarbonData JARs and files through which driver and executor will start.</p> -<p>Getting started with CarbonData : <a href="quick-start-guide.html">Quick Start</a>, <a href="data-management-on-carbondata.html">Data Management on CarbonData</a></p> -<h2> -<a id="query-execution-using-carbondata-thrift-server" class="anchor" href="#query-execution-using-carbondata-thrift-server" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Query Execution Using CarbonData Thrift Server</h2> -<h3> -<a id="starting-carbondata-thrift-server" class="anchor" href="#starting-carbondata-thrift-server" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Starting CarbonData Thrift Server.</h3> -<p>a. cd <code>$SPARK_HOME</code></p> -<p>b. Run the following command to start the CarbonData thrift server.</p> -<pre><code>./bin/spark-submit ---class org.apache.carbondata.spark.thriftserver.CarbonThriftServer -$SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path> -</code></pre> -<table> -<thead> -<tr> -<th>Parameter</th> -<th>Description</th> -<th>Example</th> -</tr> -</thead> -<tbody> -<tr> -<td>CARBON_ASSEMBLY_JAR</td> -<td>CarbonData assembly jar name present in the <code>$SPARK_HOME/carbonlib/</code> folder.</td> -<td>carbondata_2.xx-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar</td> -</tr> -<tr> -<td>carbon_store_path</td> -<td>This is a parameter to the CarbonThriftServer class. This a HDFS path where CarbonData files will be kept. Strongly Recommended to put same as carbon.storelocation parameter of carbon.properties. If not specified then it takes spark.sql.warehouse.dir path.</td> -<td><code>hdfs://<host_name>:port/user/hive/warehouse/carbon.store</code></td> -</tr> -</tbody> -</table> -<p><strong>NOTE</strong>: From Spark 1.6, by default the Thrift server runs in multi-session mode. Which means each JDBC/ODBC connection owns a copy of their own SQL configuration and temporary function registry. Cached tables are still shared though. If you prefer to run the Thrift server in single-session mode and share all SQL configuration and temporary function registry, please set option <code>spark.sql.hive.thriftServer.singleSession</code> to <code>true</code>. You may either add this option to <code>spark-defaults.conf</code>, or pass it to <code>spark-submit.sh</code> via <code>--conf</code>:</p> -<pre><code>./bin/spark-submit ---conf spark.sql.hive.thriftServer.singleSession=true ---class org.apache.carbondata.spark.thriftserver.CarbonThriftServer -$SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path> -</code></pre> -<p><strong>But</strong> in single-session mode, if one user changes the database from one connection, the database of the other connections will be changed too.</p> -<p><strong>Examples</strong></p> -<ul> -<li>Start with default memory and executors.</li> -</ul> -<pre><code>./bin/spark-submit ---class org.apache.carbondata.spark.thriftserver.CarbonThriftServer -$SPARK_HOME/carbonlib -/carbondata_2.xx-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar -hdfs://<host_name>:port/user/hive/warehouse/carbon.store -</code></pre> -<ul> -<li>Start with Fixed executors and resources.</li> -</ul> -<pre><code>./bin/spark-submit ---class org.apache.carbondata.spark.thriftserver.CarbonThriftServer ---num-executors 3 --driver-memory 20g --executor-memory 250g ---executor-cores 32 -/srv/OSCON/BigData/HACluster/install/spark/sparkJdbc/lib -/carbondata_2.xx-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar -hdfs://<host_name>:port/user/hive/warehouse/carbon.store -</code></pre> -<h3> -<a id="connecting-to-carbondata-thrift-server-using-beeline" class="anchor" href="#connecting-to-carbondata-thrift-server-using-beeline" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Connecting to CarbonData Thrift Server Using Beeline.</h3> -<pre><code> cd $SPARK_HOME - ./sbin/start-thriftserver.sh - ./bin/beeline -u jdbc:hive2://<thriftserver_host>:port - - Example - ./bin/beeline -u jdbc:hive2://10.10.10.10:10000 -</code></pre> -</div> -</div> -</div> -</div> -<div class="doc-footer"> - <a href="#top" class="scroll-top">Top</a> -</div> -</div> -</section> -</div> -</div> -</div> -</section><!-- End systemblock part --> -<script src="js/custom.js"></script> -</body> -</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/content/introduction.html ---------------------------------------------------------------------- diff --git a/content/introduction.html b/content/introduction.html index 068d711..8f18870 100644 --- a/content/introduction.html +++ b/content/introduction.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -229,17 +237,15 @@ </ul> <h2> <a id="carbondata-features--functions" class="anchor" href="#carbondata-features--functions" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData Features & Functions</h2> -<p>CarbonData has rich set of featues to support various use cases in Big Data analytics.The below table lists the major features supported by CarbonData.</p> +<p>CarbonData has rich set of features to support various use cases in Big Data analytics. The below table lists the major features supported by CarbonData.</p> <h3> <a id="table-management" class="anchor" href="#table-management" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Table Management</h3> <ul> <li> <h5> <a id="ddl-create-alterdropctas" class="anchor" href="#ddl-create-alterdropctas" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DDL (Create, Alter,Drop,CTAS)</h5> +<p>CarbonData provides its own DDL to create and manage carbondata tables. These DDL conform to Hive,Spark SQL format and support additional properties and configuration to take advantages of CarbonData functionalities.</p> </li> -</ul> -<p>? CarbonData provides its own DDL to create and manage carbondata tables.These DDL conform to Hive,Spark SQL format and support additional properties and configuration to take advantages of CarbonData functionalities.</p> -<ul> <li> <h5> <a id="dmlloadinsert" class="anchor" href="#dmlloadinsert" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DML(Load,Insert)</h5> @@ -263,7 +269,7 @@ <li> <h5> <a id="compaction" class="anchor" href="#compaction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compaction</h5> -<p>CarbonData manages incremental loads as segments.Compaction help to compact the growing number of segments and also to improve query filter pruning.</p> +<p>CarbonData manages incremental loads as segments. Compaction helps to compact the growing number of segments and also to improve query filter pruning.</p> </li> <li> <h5> @@ -277,12 +283,12 @@ <li> <h5> <a id="pre-aggregate" class="anchor" href="#pre-aggregate" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Pre-Aggregate</h5> -<p>CarbonData has concept of datamaps to assist in pruning of data while querying so that performance is faster.Pre Aggregate tables are kind of datamaps which can improve the query performance by order of magnitude.CarbonData will automatically pre-aggregae the incremental data and re-write the query to automatically fetch from the most appropriate pre-aggregate table to serve the query faster.</p> +<p>CarbonData has concept of datamaps to assist in pruning of data while querying so that performance is faster.Pre Aggregate tables are kind of datamaps which can improve the query performance by order of magnitude.CarbonData will automatically pre-aggregate the incremental data and re-write the query to automatically fetch from the most appropriate pre-aggregate table to serve the query faster.</p> </li> <li> <h5> <a id="time-series" class="anchor" href="#time-series" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Time Series</h5> -<p>CarbonData has built in understanding of time order(Year, month,day,hour, minute,second).Time series is a pre-aggregate table which can automatically roll-up the data to the desired level during incremental load and serve the query from the most appropriate pre-aggregate table.</p> +<p>CarbonData has built in understanding of time order(Year, month,day,hour, minute,second). Time series is a pre-aggregate table which can automatically roll-up the data to the desired level during incremental load and serve the query from the most appropriate pre-aggregate table.</p> </li> <li> <h5> @@ -297,7 +303,7 @@ <li> <h5> <a id="mv-materialized-views" class="anchor" href="#mv-materialized-views" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>MV (Materialized Views)</h5> -<p>MVs are kind of pre-aggregate tables which can support efficent query re-write and processing.CarbonData provides MV which can rewrite query to fetch from any table(including non-carbondata tables).Typical usecase is to store the aggregated data of a non-carbondata fact table into carbondata and use mv to rewrite the query to fetch from carbondata.</p> +<p>MVs are kind of pre-aggregate tables which can support efficent query re-write and processing.CarbonData provides MV which can rewrite query to fetch from any table(including non-carbondata tables). Typical usecase is to store the aggregated data of a non-carbondata fact table into carbondata and use mv to rewrite the query to fetch from carbondata.</p> </li> </ul> <h3> @@ -315,12 +321,12 @@ <li> <h5> <a id="carbondata-writer" class="anchor" href="#carbondata-writer" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData writer</h5> -<p>CarbonData supports writing data from non-spark application using SDK.Users can use SDK to generate carbondata files from custom applications.Typical usecase is to write the streaming application plugged in to kafka and use carbondata as sink(target) table for storing.</p> +<p>CarbonData supports writing data from non-spark application using SDK.Users can use SDK to generate carbondata files from custom applications. Typical usecase is to write the streaming application plugged in to kafka and use carbondata as sink(target) table for storing.</p> </li> <li> <h5> <a id="carbondata-reader" class="anchor" href="#carbondata-reader" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CarbonData reader</h5> -<p>CarbonData supports reading of data from non-spark application using SDK.Users can use the SDK to read the carbondata files from their application and do custom processing.</p> +<p>CarbonData supports reading of data from non-spark application using SDK. Users can use the SDK to read the carbondata files from their application and do custom processing.</p> </li> </ul> <h3> @@ -329,7 +335,7 @@ <li> <h5> <a id="s3" class="anchor" href="#s3" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>S3</h5> -<p>CarbonData can write to S3, OBS or any cloud storage confirming to S3 protocol.CarbonData uses the HDFS api to write to cloud object stores.</p> +<p>CarbonData can write to S3, OBS or any cloud storage confirming to S3 protocol. CarbonData uses the HDFS api to write to cloud object stores.</p> </li> <li> <h5> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/content/language-manual.html ---------------------------------------------------------------------- diff --git a/content/language-manual.html b/content/language-manual.html index a0ea674..74c18f4 100644 --- a/content/language-manual.html +++ b/content/language-manual.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -236,11 +244,12 @@ <li>Data Manipulation Statements <ul> <li> -<a href="./dml-of-carbondata.html">DML:</a> <a href="./dml-of-carbondata.html#load-data">Load</a>, <a href="./ddl-of-carbondata.html#insert-overwrite">Insert</a>, <a href="./dml-of-carbondata.html#update">Update</a>, <a href="./dml-of-carbondata.html#delete">Delete</a> +<a href="./dml-of-carbondata.html">DML:</a> <a href="./dml-of-carbondata.html#load-data">Load</a>, <a href="./dml-of-carbondata.html#insert-data-into-carbondata-table">Insert</a>, <a href="./dml-of-carbondata.html#update">Update</a>, <a href="./dml-of-carbondata.html#delete">Delete</a> </li> <li><a href="./segment-management-on-carbondata.html">Segment Management</a></li> </ul> </li> +<li><a href="./carbon-as-spark-datasource-guide.html">CarbonData as Spark's Datasource</a></li> <li><a href="./configuration-parameters.html">Configuration Properties</a></li> </ul> <script> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/content/lucene-datamap-guide.html ---------------------------------------------------------------------- diff --git a/content/lucene-datamap-guide.html b/content/lucene-datamap-guide.html index b8164a2..357286a 100644 --- a/content/lucene-datamap-guide.html +++ b/content/lucene-datamap-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -239,7 +247,7 @@ ON TABLE main_table <h2> <a id="lucene-datamap-introduction" class="anchor" href="#lucene-datamap-introduction" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Lucene DataMap Introduction</h2> <p>Lucene is a high performance, full featured text search engine. Lucene is integrated to carbon as -an index datamap and managed along with main tables by CarbonData.User can create lucene datamap +an index datamap and managed along with main tables by CarbonData. User can create lucene datamap to improve query performance on string columns which has content of more length. So, user can search tokenized word or pattern of it using lucene query on text content.</p> <p>For instance, main table called <strong>datamap_test</strong> which is defined as:</p> @@ -281,7 +289,7 @@ value is compression, the index file size will be compressed.</p> Queries are to be made on main table. when a query with TEXT_MATCH('name:c10') or TEXT_MATCH_WITH_LIMIT('name:n10',10)[the second parameter represents the number of result to be returned, if user does not specify this value, all results will be returned without any limit] is -fired, two jobs are fired.The first job writes the temporary files in folder created at table level +fired, two jobs are fired. The first job writes the temporary files in folder created at table level which contains lucene's seach results and these files will be read in second job to give faster results. These temporary files will be cleared once the query finishes.</p> <p>User can verify whether a query can leverage Lucene datamap or not by executing <code>EXPLAIN</code> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/content/mainpage.html ---------------------------------------------------------------------- diff --git a/content/mainpage.html b/content/mainpage.html deleted file mode 100644 index d515853..0000000 --- a/content/mainpage.html +++ /dev/null @@ -1,214 +0,0 @@ -<!DOCTYPE html> -<html lang="en"> -<head> - <meta charset="utf-8"> - <meta http-equiv="X-UA-Compatible" content="IE=edge"> - <meta name="viewport" content="width=device-width, initial-scale=1"> - <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'> - <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> - <title>CarbonData</title> - <style> - - </style> - <!-- Bootstrap --> - - <link rel="stylesheet" href="css/bootstrap.min.css"> - <link href="css/style.css" rel="stylesheet"> - <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> - <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> - <!--[if lt IE 9]> - <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script> - <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script> - <![endif]--> - <script src="js/jquery.min.js"></script> - <script src="js/bootstrap.min.js"></script> - - -</head> -<body> -<header> - <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper"> - <div class="container"> - <div class="navbar-header"> - <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse" - class="navbar-toggle collapsed" type="button"> - <span class="sr-only">Toggle navigation</span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - <span class="icon-bar"></span> - </button> - <a href="index.html" class="logo"> - <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/> - </a> - </div> - <div class="navbar-collapse collapse cd_navcontnt" id="navbar"> - <ul class="nav navbar-nav navbar-right navlist-custom"> - <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a> - </li> - <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true" - aria-expanded="false"> Download <span class="caret"></span></a> - <ul class="dropdown-menu"> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; - target="_blank">Apache CarbonData 1.4.1</a></li> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/"; - target="_blank">Apache CarbonData 1.4.0</a></li> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; - target="_blank">Apache CarbonData 1.3.1</a></li> - <li> - <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; - target="_blank">Apache CarbonData 1.3.0</a></li> - <li> - <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; - target="_blank">Release Archive</a></li> - </ul> - </li> - <li><a href="mainpage.html" class="active">Documentation</a></li> - <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" - aria-expanded="false">Community <span class="caret"></span></a> - <ul class="dropdown-menu"> - <li> - <a href="https://github.com/apache/carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"; - target="_blank">Contributing to CarbonData</a></li> - <li> - <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"; - target="_blank">Release Guide</a></li> - <li> - <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"; - target="_blank">Project PMC and Committers</a></li> - <li> - <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; - target="_blank">CarbonData Meetups</a></li> - <li><a href="security.html">Apache CarbonData Security</a></li> - <li><a href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache - Jira</a></li> - <li><a href="videogallery.html">CarbonData Videos </a></li> - </ul> - </li> - <li class="dropdown"> - <a href="http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" - data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> - <ul class="dropdown-menu"> - <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> - <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> - <li><a href="http://www.apache.org/foundation/sponsorship.html"; - target="_blank">Sponsorship</a></li> - <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> - </ul> - </li> - - <li class="dropdown"> - <a href="http://www.apache.org/"; class="hidden-lg hidden-md hidden-sm dropdown-toggle" - data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> - <ul class="dropdown-menu"> - <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> - <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> - <li><a href="http://www.apache.org/foundation/sponsorship.html"; - target="_blank">Sponsorship</a></li> - <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> - </ul> - </li> - - <li> - <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a> - - </li> - - </ul> - </div><!--/.nav-collapse --> - <div id="search-box"> - <form method="get" action="http://www.google.com/search";> - <div class="search-block"> - <table border="0" cellpadding="0" width="100%"> - <tr> - <td style="width:80%"> - <input type="text" name="q" size=" 5" maxlength="255" value="" - class="search-input" placeholder="Search...." required/> - </td> - <td style="width:20%"> - <input type="submit" value="Search"/></td> - </tr> - <tr> - <td align="left" style="font-size:75%" colspan="2"> - <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/> - <span style=" position: relative; top: -3px;"> Only search for CarbonData</span> - </td> - </tr> - </table> - </div> - </form> - </div> - </div> - </nav> -</header> <!-- end Header part --> - -<div class="fixed-padding"></div> <!-- top padding with fixde header --> - -<section><!-- Dashboard nav --> - <div class="container-fluid q"> - <div class="col-sm-12 col-md-12 maindashboard"> - <div class="row"> - <section> - <div style="padding:10px 15px;"> - <div id="viewpage" name="viewpage"> - <div class="doc-heading"> - <h4 class="title">Documentation - <span class="title-underline"></span> - </h4> - </div> - - <div class="row"> - - <div class="col-sm-12 col-md-12"> - <span class="text-justify"> - Welcome to Apache CarbonData. Apache CarbonData is a new big data file format for faster interactive query using advanced columnar storage, index, compression and encoding techniques to improve computing efficiency, which helps in speeding up queries by an order of magnitude faster over PetaBytes of data. This user guide provides a detailed description about the CarbonData and its features. - Let's get started ! - </span> - <hr style="margin: 12px 0 8px"> - <div> - <ul class="sub-nav"> - <li><a href="quick-start-guide.html">Quick Start</a></li> - <li><a href="file-structure-of-carbondata.html">CarbonData File Structure</a></li> - <li><a href="supported-data-types-in-carbondata.html">Data Types</a></li> - <li><a href="data-management-on-carbondata.html">Data Management On CarbonData</a></li> - <li><a href="installation-guide.html">Installation Guide</a></li> - <li><a href="configuration-parameters.html">Configuring CarbonData</a></li> - <li><a href="streaming-guide.html">Streaming Guide</a></li> - <li><a href="sdk-guide.html">SDK Guide</a></li> - <li><a href="s3-guide.html">S3 Guide (Alpha Feature)</a></li> - <li><a href="datamap-developer-guide.html">DataMap Developer Guide</a></li> - <li><a href="datamap-management.html">CarbonData DataMap Management</a></li> - <li><a href="bloomfilter-datamap-guide.html">CarbonData BloomFilter DataMap (Alpha Feature)</a></li> - <li><a href="lucene-datamap-guide.html">CarbonData Lucene DataMap (Alpha Feature)</a></li> - <li><a href="preaggregate-datamap-guide.html">CarbonData Pre-aggregate DataMap</a></li> - <li><a href="timeseries-datamap-guide.html">CarbonData Timeseries DataMap</a></li> - <li><a href="faq.html">FAQs</a></li> - <li><a href="troubleshooting.html">Troubleshooting</a></li> - <li><a href="useful-tips-on-carbondata.html">Useful Tips</a></li> - - </ul> - </div> - </div> - </div> - </div> - <div class="doc-footer"> - <a href="#top" class="scroll-top">Top</a> - </div> - </div> - </section> - </div> - </div> - </div> -</section><!-- End systemblock part --> -</div> -</div> -</div> -</section><!-- End systemblock part --> -<script src="js/custom.js"></script> -</body> -</html> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/content/performance-tuning.html ---------------------------------------------------------------------- diff --git a/content/performance-tuning.html b/content/performance-tuning.html index 480911c..c63462f 100644 --- a/content/performance-tuning.html +++ b/content/performance-tuning.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -381,7 +389,7 @@ You can configure CarbonData by tuning following properties in carbon.properties <tbody> <tr> <td>carbon.number.of.cores.while.loading</td> -<td>Default: 2.This value should be >= 2</td> +<td>Default: 2. This value should be >= 2</td> <td>Specifies the number of cores used for data processing during data loading in CarbonData.</td> </tr> <tr> @@ -447,7 +455,7 @@ scenarios. After the completion of POC, some of the configurations impacting the <td>spark/carbonlib/carbon.properties</td> <td>Data loading and Querying</td> <td>For minor compaction, specifies the number of segments to be merged in stage 1 and number of compacted segments to be merged in stage 2.</td> -<td>Each CarbonData load will create one segment, if every load is small in size it will generate many small file over a period of time impacting the query performance. Configuring this parameter will merge the small segment to one big segment which will sort the data and improve the performance. For Example in one telecommunication scenario, the performance improves about 2 times after minor compaction.</td> +<td>Each CarbonData load will create one segment, if every load is small in size it will generate many small files over a period of time impacting the query performance. Configuring this parameter will merge the small segment to one big segment which will sort the data and improve the performance. For Example in one telecommunication scenario, the performance improves about 2 times after minor compaction.</td> </tr> <tr> <td>spark.sql.shuffle.partitions</td> @@ -489,21 +497,21 @@ scenarios. After the completion of POC, some of the configurations impacting the <td>spark/carbonlib/carbon.properties</td> <td>Data loading</td> <td>Specify the name of compressor to compress the intermediate sort temporary files during sort procedure in data loading.</td> -<td>The optional values are 'SNAPPY','GZIP','BZIP2','LZ4','ZSTD' and empty. By default, empty means that Carbondata will not compress the sort temp files. This parameter will be useful if you encounter disk bottleneck.</td> +<td>The optional values are 'SNAPPY','GZIP','BZIP2','LZ4','ZSTD', and empty. By default, empty means that Carbondata will not compress the sort temp files. This parameter will be useful if you encounter disk bottleneck.</td> </tr> <tr> <td>carbon.load.skewedDataOptimization.enabled</td> <td>spark/carbonlib/carbon.properties</td> <td>Data loading</td> <td>Whether to enable size based block allocation strategy for data loading.</td> -<td>When loading, carbondata will use file size based block allocation strategy for task distribution. It will make sure that all the executors process the same size of data -- It's useful if the size of your input data files varies widely, say 1MB~1GB.</td> +<td>When loading, carbondata will use file size based block allocation strategy for task distribution. It will make sure that all the executors process the same size of data -- It's useful if the size of your input data files varies widely, say 1MB to 1GB.</td> </tr> <tr> <td>carbon.load.min.size.enabled</td> <td>spark/carbonlib/carbon.properties</td> <td>Data loading</td> <td>Whether to enable node minumun input data size allocation strategy for data loading.</td> -<td>When loading, carbondata will use node minumun input data size allocation strategy for task distribution. It will make sure the node load the minimum amount of data -- It's useful if the size of your input data files very small, say 1MB~256MB,Avoid generating a large number of small files.</td> +<td>When loading, carbondata will use node minumun input data size allocation strategy for task distribution. It will make sure the nodes load the minimum amount of data -- It's useful if the size of your input data files very small, say 1MB to 256MB,Avoid generating a large number of small files.</td> </tr> </tbody> </table> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/content/preaggregate-datamap-guide.html ---------------------------------------------------------------------- diff --git a/content/preaggregate-datamap-guide.html b/content/preaggregate-datamap-guide.html index 6b0783e..c3d4a85 100644 --- a/content/preaggregate-datamap-guide.html +++ b/content/preaggregate-datamap-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -444,7 +452,7 @@ pre-aggregate tables. To further improve the query performance, compaction on pr can be triggered to merge the segments and files in the pre-aggregate tables.</p> <h2> <a id="data-management-with-pre-aggregate-tables" class="anchor" href="#data-management-with-pre-aggregate-tables" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Data Management with pre-aggregate tables</h2> -<p>In current implementation, data consistence need to be maintained for both main table and pre-aggregate +<p>In current implementation, data consistency needs to be maintained for both main table and pre-aggregate tables. Once there is pre-aggregate table created on the main table, following command on the main table is not supported:</p> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/content/quick-start-guide.html ---------------------------------------------------------------------- diff --git a/content/quick-start-guide.html b/content/quick-start-guide.html index f3e8a8f..8e599d3 100644 --- a/content/quick-start-guide.html +++ b/content/quick-start-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -212,7 +220,7 @@ <div> <h1> <a id="quick-start" class="anchor" href="#quick-start" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick Start</h1> -<p>This tutorial provides a quick introduction to using CarbonData.To follow along with this guide, first download a packaged release of CarbonData from the <a href="https://dist.apache.org/repos/dist/release/carbondata/"; target=_blank rel="nofollow">CarbonData website</a>.Alternatively it can be created following <a href="https://github.com/apache/carbondata/tree/master/build"; target=_blank>Building CarbonData</a> steps.</p> +<p>This tutorial provides a quick introduction to using CarbonData. To follow along with this guide, first download a packaged release of CarbonData from the <a href="https://dist.apache.org/repos/dist/release/carbondata/"; target=_blank rel="nofollow">CarbonData website</a>.Alternatively it can be created following <a href="https://github.com/apache/carbondata/tree/master/build"; target=_blank>Building CarbonData</a> steps.</p> <h2> <a id="prerequisites" class="anchor" href="#prerequisites" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prerequisites</h2> <ul> @@ -233,7 +241,7 @@ EOF </ul> <h2> <a id="integration" class="anchor" href="#integration" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Integration</h2> -<p>CarbonData can be integrated with Spark and Presto Execution Engines.The below documentation guides on Installing and Configuring with these execution engines.</p> +<p>CarbonData can be integrated with Spark and Presto Execution Engines. The below documentation guides on Installing and Configuring with these execution engines.</p> <h3> <a id="spark" class="anchor" href="#spark" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Spark</h3> <p><a href="#installing-and-configuring-carbondata-to-run-locally-with-spark-shell">Installing and Configuring CarbonData to run locally with Spark Shell</a></p> @@ -555,26 +563,26 @@ hdfs://<host_name>:port/user/hive/warehouse/carbon.store </code></pre> <h2> <a id="installing-and-configuring-carbondata-on-presto" class="anchor" href="#installing-and-configuring-carbondata-on-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing and Configuring CarbonData on Presto</h2> -<p><strong>NOTE:</strong> <strong>CarbonData tables cannot be created nor loaded from Presto.User need to create CarbonData Table and load data into it +<p><strong>NOTE:</strong> <strong>CarbonData tables cannot be created nor loaded from Presto. User need to create CarbonData Table and load data into it either with <a href="#installing-and-configuring-carbondata-to-run-locally-with-spark-shell">Spark</a> or <a href="./sdk-guide.html">SDK</a>. Once the table is created,it can be queried from Presto.</strong></p> <h3> <a id="installing-presto" class="anchor" href="#installing-presto" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing Presto</h3> <ol> <li> -<p>Download the 0.187 version of Presto using: -<code>wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.187/presto-server-0.187.tar.gz</code></p> +<p>Download the 0.210 version of Presto using: +<code>wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.210/presto-server-0.210.tar.gz</code></p> </li> <li> -<p>Extract Presto tar file: <code>tar zxvf presto-server-0.187.tar.gz</code>.</p> +<p>Extract Presto tar file: <code>tar zxvf presto-server-0.210.tar.gz</code>.</p> </li> <li> <p>Download the Presto CLI for the coordinator and name it presto.</p> </li> </ol> -<pre><code> wget https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.187/presto-cli-0.187-executable.jar +<pre><code> wget https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.210/presto-cli-0.210-executable.jar - mv presto-cli-0.187-executable.jar presto + mv presto-cli-0.210-executable.jar presto chmod +x presto </code></pre> @@ -582,7 +590,7 @@ Once the table is created,it can be queried from Presto.</strong></p> <a id="create-configuration-files" class="anchor" href="#create-configuration-files" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create Configuration Files</h3> <ol> <li> -<p>Create <code>etc</code> folder in presto-server-0.187 directory.</p> +<p>Create <code>etc</code> folder in presto-server-0.210 directory.</p> </li> <li> <p>Create <code>config.properties</code>, <code>jvm.config</code>, <code>log.properties</code>, and <code>node.properties</code> files.</p> @@ -624,10 +632,15 @@ node.data-dir=/home/ubuntu/data <pre><code>coordinator=true node-scheduler.include-coordinator=false http-server.http.port=8086 -query.max-memory=50GB -query.max-memory-per-node=2GB +query.max-memory=5GB +query.max-total-memory-per-node=5GB +query.max-memory-per-node=3GB +memory.heap-headroom-per-node=1GB discovery-server.enabled=true -discovery.uri=<coordinator_ip>:8086 +discovery.uri=http://localhost:8086 +task.max-worker-threads=4 +optimizer.dictionary-aggregation=true +optimizer.optimize-hash-generation = false </code></pre> <p>The options <code>node-scheduler.include-coordinator=false</code> and <code>coordinator=true</code> indicate that the node is the coordinator and tells the coordinator not to do any of the computation work itself and to use the workers.</p> <p><strong>Note</strong>: It is recommended to set <code>query.max-memory-per-node</code> to half of the JVM config max memory, though the workload is highly concurrent, lower value for <code>query.max-memory-per-node</code> is to be used.</p> @@ -640,7 +653,7 @@ Then, <code>query.max-memory=<30GB * number of nodes></code>.</p> <a id="contents-of-your-configproperties-1" class="anchor" href="#contents-of-your-configproperties-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Contents of your config.properties</h5> <pre><code>coordinator=false http-server.http.port=8086 -query.max-memory=50GB +query.max-memory=5GB query.max-memory-per-node=2GB discovery.uri=<coordinator_ip>:8086 </code></pre> @@ -663,10 +676,10 @@ discovery.uri=<coordinator_ip>:8086 </ol> <h3> <a id="start-presto-server-on-all-nodes" class="anchor" href="#start-presto-server-on-all-nodes" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Start Presto Server on all nodes</h3> -<pre><code>./presto-server-0.187/bin/launcher start +<pre><code>./presto-server-0.210/bin/launcher start </code></pre> <p>To run it as a background process.</p> -<pre><code>./presto-server-0.187/bin/launcher run +<pre><code>./presto-server-0.210/bin/launcher run </code></pre> <p>To run it in foreground.</p> <h3> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/content/release-guide.html ---------------------------------------------------------------------- diff --git a/content/release-guide.html b/content/release-guide.html index cb47540..891d897 100644 --- a/content/release-guide.html +++ b/content/release-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6ad7599a/content/s3-guide.html ---------------------------------------------------------------------- diff --git a/content/s3-guide.html b/content/s3-guide.html index 57af913..62c51c3 100644 --- a/content/s3-guide.html +++ b/content/s3-guide.html @@ -52,6 +52,9 @@ aria-expanded="false"> Download <span class="caret"></span></a> <ul class="dropdown-menu"> <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.5.0/"; + target="_blank">Apache CarbonData 1.5.0</a></li> + <li> <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; target="_blank">Apache CarbonData 1.4.1</a></li> <li> @@ -179,7 +182,12 @@ <a class="nav__item nav__sub__item" href="./timeseries-datamap-guide.html">Time Series</a> </div> - <a class="b-nav__api nav__item" href="./sdk-guide.html">API</a> + <div class="nav__item nav__item__with__subs"> + <a class="b-nav__api nav__item nav__sub__anchor" href="./sdk-guide.html">API</a> + <a class="nav__item nav__sub__item" href="./sdk-guide.html">Java SDK</a> + <a class="nav__item nav__sub__item" href="./CSDK-guide.html">C++ SDK</a> + </div> + <a class="b-nav__perf nav__item" href="./performance-tuning.html">Performance Tuning</a> <a class="b-nav__s3 nav__item" href="./s3-guide.html">S3 Storage</a> <a class="b-nav__faq nav__item" href="./faq.html">FAQ</a> @@ -211,7 +219,7 @@ <div class="col-sm-12 col-md-12"> <div> <h1> -<a id="s3-guide-alpha-feature-141" class="anchor" href="#s3-guide-alpha-feature-141" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>S3 Guide (Alpha Feature 1.4.1)</h1> +<a id="s3-guide" class="anchor" href="#s3-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>S3 Guide</h1> <p>Object storage is the recommended storage format in cloud as it can support storing large data files. S3 APIs are widely used for accessing object stores. This can be used to store or retrieve data on Amazon cloud, Huawei Cloud(OBS) or on any other object
![https://github.com/apache/carbondata/blob/master/docs/images/2-1_1.png?raw=true"](https://github.com/apache/carbondata/blob/master/docs/images/2-1_1.png?raw=true"){kind=link}