http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6f8949f1/content/installation-guide.html ---------------------------------------------------------------------- diff --git a/content/installation-guide.html b/content/installation-guide.html new file mode 100644 index 0000000..2e7fab6 --- /dev/null +++ b/content/installation-guide.html @@ -0,0 +1,455 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'> + <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> + <title>CarbonData</title> + <style> + + </style> + <!-- Bootstrap --> + + <link rel="stylesheet" href="css/bootstrap.min.css"> + <link href="css/style.css" rel="stylesheet"> + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script> + <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script> + <![endif]--> + <script src="js/jquery.min.js"></script> + <script src="js/bootstrap.min.js"></script> + + +</head> +<body> +<header> + <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper"> + <div class="container"> + <div class="navbar-header"> + <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse" + class="navbar-toggle collapsed" type="button"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a href="index.html" class="logo"> + <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/> + </a> + </div> + <div class="navbar-collapse collapse cd_navcontnt" id="navbar"> + <ul class="nav navbar-nav navbar-right navlist-custom"> + <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a> + </li> + <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false"> Download <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; + target="_blank">Apache CarbonData 1.4.1</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/"; + target="_blank">Apache CarbonData 1.4.0</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; + target="_blank">Apache CarbonData 1.3.1</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; + target="_blank">Apache CarbonData 1.3.0</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; + target="_blank">Release Archive</a></li> + </ul> + </li> + <li><a href="mainpage.html" class="active">Documentation</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false">Community <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://github.com/apache/carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"; + target="_blank">Contributing to CarbonData</a></li> + <li> + <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"; + target="_blank">Release Guide</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"; + target="_blank">Project PMC and Committers</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; + target="_blank">CarbonData Meetups</a></li> + <li><a href="security.html">Apache CarbonData Security</a></li> + <li><a href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache + Jira</a></li> + <li><a href="videogallery.html">CarbonData Videos </a></li> + </ul> + </li> + <li class="dropdown"> + <a href="http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="http://www.apache.org/"; class="hidden-lg hidden-md hidden-sm dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li> + <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a> + + </li> + + </ul> + </div><!--/.nav-collapse --> + <div id="search-box"> + <form method="get" action="http://www.google.com/search"; target="_blank"> + <div class="search-block"> + <table border="0" cellpadding="0" width="100%"> + <tr> + <td style="width:80%"> + <input type="text" name="q" size=" 5" maxlength="255" value="" + class="search-input" placeholder="Search...." required/> + </td> + <td style="width:20%"> + <input type="submit" value="Search"/></td> + </tr> + <tr> + <td align="left" style="font-size:75%" colspan="2"> + <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/> + <span style=" position: relative; top: -3px;"> Only search for CarbonData</span> + </td> + </tr> + </table> + </div> + </form> + </div> + </div> + </nav> +</header> <!-- end Header part --> + +<div class="fixed-padding"></div> <!-- top padding with fixde header --> + +<section><!-- Dashboard nav --> + <div class="container-fluid q"> + <div class="col-sm-12 col-md-12 maindashboard"> + <div class="row"> + <section> + <div style="padding:10px 15px;"> + <div id="viewpage" name="viewpage"> + <div class="row"> + <div class="col-sm-12 col-md-12"> + <div> +<h1> +<a id="installation-guide" class="anchor" href="#installation-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installation Guide</h1> +<p>This tutorial guides you through the installation and configuration of CarbonData in the following two modes :</p> +<ul> +<li><a href="#installing-and-configuring-carbondata-on-standalone-spark-cluster">Installing and Configuring CarbonData on Standalone Spark Cluster</a></li> +<li><a href="#installing-and-configuring-carbondata-on-spark-on-yarn-cluster">Installing and Configuring CarbonData on Spark on YARN Cluster</a></li> +</ul> +<p>followed by :</p> +<ul> +<li><a href="#query-execution-using-carbondata-thrift-server">Query Execution using CarbonData Thrift Server</a></li> +</ul> +<h2> +<a id="installing-and-configuring-carbondata-on-standalone-spark-cluster" class="anchor" href="#installing-and-configuring-carbondata-on-standalone-spark-cluster" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing and Configuring CarbonData on Standalone Spark Cluster</h2> +<h3> +<a id="prerequisites" class="anchor" href="#prerequisites" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prerequisites</h3> +<ul> +<li> +<p>Hadoop HDFS and Yarn should be installed and running.</p> +</li> +<li> +<p>Spark should be installed and running on all the cluster nodes.</p> +</li> +<li> +<p>CarbonData user should have permission to access HDFS.</p> +</li> +</ul> +<h3> +<a id="procedure" class="anchor" href="#procedure" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Procedure</h3> +<ol> +<li> +<p><a href="https://github.com/apache/carbondata/blob/master/build/README.md"; target=_blank>Build the CarbonData</a> project and get the assembly jar from <code>./assembly/target/scala-2.1x/carbondata_xxx.jar</code>.</p> +</li> +<li> +<p>Copy <code>./assembly/target/scala-2.1x/carbondata_xxx.jar</code> to <code>$SPARK_HOME/carbonlib</code> folder.</p> +<p><strong>NOTE</strong>: Create the carbonlib folder if it does not exist inside <code>$SPARK_HOME</code> path.</p> +</li> +<li> +<p>Add the carbonlib folder path in the Spark classpath. (Edit <code>$SPARK_HOME/conf/spark-env.sh</code> file and modify the value of <code>SPARK_CLASSPATH</code> by appending <code>$SPARK_HOME/carbonlib/*</code> to the existing value)</p> +</li> +<li> +<p>Copy the <code>./conf/carbon.properties.template</code> file from CarbonData repository to <code>$SPARK_HOME/conf/</code> folder and rename the file to <code>carbon.properties</code>.</p> +</li> +<li> +<p>Repeat Step 2 to Step 5 in all the nodes of the cluster.</p> +</li> +<li> +<p>In Spark node[master], configure the properties mentioned in the following table in <code>$SPARK_HOME/conf/spark-defaults.conf</code> file.</p> +</li> +</ol> +<table> +<thead> +<tr> +<th>Property</th> +<th>Value</th> +<th>Description</th> +</tr> +</thead> +<tbody> +<tr> +<td>spark.driver.extraJavaOptions</td> +<td><code>-Dcarbon.properties.filepath = $SPARK_HOME/conf/carbon.properties</code></td> +<td>A string of extra JVM options to pass to the driver. For instance, GC settings or other logging.</td> +</tr> +<tr> +<td>spark.executor.extraJavaOptions</td> +<td><code>-Dcarbon.properties.filepath = $SPARK_HOME/conf/carbon.properties</code></td> +<td>A string of extra JVM options to pass to executors. For instance, GC settings or other logging. <strong>NOTE</strong>: You can enter multiple values separated by space.</td> +</tr> +</tbody> +</table> +<ol start="7"> +<li>Add the following properties in <code>$SPARK_HOME/conf/carbon.properties</code> file:</li> +</ol> +<table> +<thead> +<tr> +<th>Property</th> +<th>Required</th> +<th>Description</th> +<th>Example</th> +<th>Remark</th> +</tr> +</thead> +<tbody> +<tr> +<td>carbon.storelocation</td> +<td>NO</td> +<td>Location where data CarbonData will create the store and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path.</td> +<td>hdfs://HOSTNAME:PORT/Opt/CarbonStore</td> +<td>Propose to set HDFS directory</td> +</tr> +</tbody> +</table> +<ol start="8"> +<li>Verify the installation. For example:</li> +</ol> +<pre><code>./spark-shell --master spark://HOSTNAME:PORT --total-executor-cores 2 +--executor-memory 2G +</code></pre> +<p><strong>NOTE</strong>: Make sure you have permissions for CarbonData JARs and files through which driver and executor will start.</p> +<p>To get started with CarbonData : <a href="quick-start-guide.html">Quick Start</a>, <a href="data-management-on-carbondata.html">Data Management on CarbonData</a></p> +<h2> +<a id="installing-and-configuring-carbondata-on-spark-on-yarn-cluster" class="anchor" href="#installing-and-configuring-carbondata-on-spark-on-yarn-cluster" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing and Configuring CarbonData on Spark on YARN Cluster</h2> +<p>This section provides the procedure to install CarbonData on "Spark on YARN" cluster.</p> +<h3> +<a id="prerequisites-1" class="anchor" href="#prerequisites-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prerequisites</h3> +<ul> +<li>Hadoop HDFS and Yarn should be installed and running.</li> +<li>Spark should be installed and running in all the clients.</li> +<li>CarbonData user should have permission to access HDFS.</li> +</ul> +<h3> +<a id="procedure-1" class="anchor" href="#procedure-1" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Procedure</h3> +<p>The following steps are only for Driver Nodes. (Driver nodes are the one which starts the spark context.)</p> +<ol> +<li> +<p><a href="https://github.com/apache/carbondata/blob/master/build/README.md"; target=_blank>Build the CarbonData</a> project and get the assembly jar from <code>./assembly/target/scala-2.1x/carbondata_xxx.jar</code> and copy to <code>$SPARK_HOME/carbonlib</code> folder.</p> +<p><strong>NOTE</strong>: Create the carbonlib folder if it does not exists inside <code>$SPARK_HOME</code> path.</p> +</li> +<li> +<p>Copy the <code>./conf/carbon.properties.template</code> file from CarbonData repository to <code>$SPARK_HOME/conf/</code> folder and rename the file to <code>carbon.properties</code>.</p> +</li> +<li> +<p>Create <code>tar.gz</code> file of carbonlib folder and move it inside the carbonlib folder.</p> +</li> +</ol> +<pre><code>cd $SPARK_HOME +tar -zcvf carbondata.tar.gz carbonlib/ +mv carbondata.tar.gz carbonlib/ +</code></pre> +<ol start="4"> +<li>Configure the properties mentioned in the following table in <code>$SPARK_HOME/conf/spark-defaults.conf</code> file.</li> +</ol> +<table> +<thead> +<tr> +<th>Property</th> +<th>Description</th> +<th>Value</th> +</tr> +</thead> +<tbody> +<tr> +<td>spark.master</td> +<td>Set this value to run the Spark in yarn cluster mode.</td> +<td>Set yarn-client to run the Spark in yarn cluster mode.</td> +</tr> +<tr> +<td>spark.yarn.dist.files</td> +<td>Comma-separated list of files to be placed in the working directory of each executor.</td> +<td><code>$SPARK_HOME/conf/carbon.properties</code></td> +</tr> +<tr> +<td>spark.yarn.dist.archives</td> +<td>Comma-separated list of archives to be extracted into the working directory of each executor.</td> +<td><code>$SPARK_HOME/carbonlib/carbondata.tar.gz</code></td> +</tr> +<tr> +<td>spark.executor.extraJavaOptions</td> +<td>A string of extra JVM options to pass to executors. For instance <strong>NOTE</strong>: You can enter multiple values separated by space.</td> +<td><code>-Dcarbon.properties.filepath = carbon.properties</code></td> +</tr> +<tr> +<td>spark.executor.extraClassPath</td> +<td>Extra classpath entries to prepend to the classpath of executors. <strong>NOTE</strong>: If SPARK_CLASSPATH is defined in spark-env.sh, then comment it and append the values in below parameter spark.driver.extraClassPath</td> +<td><code>carbondata.tar.gz/carbonlib/*</code></td> +</tr> +<tr> +<td>spark.driver.extraClassPath</td> +<td>Extra classpath entries to prepend to the classpath of the driver. <strong>NOTE</strong>: If SPARK_CLASSPATH is defined in spark-env.sh, then comment it and append the value in below parameter spark.driver.extraClassPath.</td> +<td><code>$SPARK_HOME/carbonlib/*</code></td> +</tr> +<tr> +<td>spark.driver.extraJavaOptions</td> +<td>A string of extra JVM options to pass to the driver. For instance, GC settings or other logging.</td> +<td><code>-Dcarbon.properties.filepath = $SPARK_HOME/conf/carbon.properties</code></td> +</tr> +</tbody> +</table> +<ol start="5"> +<li>Add the following properties in <code>$SPARK_HOME/conf/carbon.properties</code>:</li> +</ol> +<table> +<thead> +<tr> +<th>Property</th> +<th>Required</th> +<th>Description</th> +<th>Example</th> +<th>Default Value</th> +</tr> +</thead> +<tbody> +<tr> +<td>carbon.storelocation</td> +<td>NO</td> +<td>Location where CarbonData will create the store and write the data in its own format. If not specified then it takes spark.sql.warehouse.dir path.</td> +<td>hdfs://HOSTNAME:PORT/Opt/CarbonStore</td> +<td>Propose to set HDFS directory</td> +</tr> +</tbody> +</table> +<ol start="6"> +<li>Verify the installation.</li> +</ol> +<pre><code> ./bin/spark-shell --master yarn-client --driver-memory 1g + --executor-cores 2 --executor-memory 2G +</code></pre> +<p><strong>NOTE</strong>: Make sure you have permissions for CarbonData JARs and files through which driver and executor will start.</p> +<p>Getting started with CarbonData : <a href="quick-start-guide.html">Quick Start</a>, <a href="data-management-on-carbondata.html">Data Management on CarbonData</a></p> +<h2> +<a id="query-execution-using-carbondata-thrift-server" class="anchor" href="#query-execution-using-carbondata-thrift-server" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Query Execution Using CarbonData Thrift Server</h2> +<h3> +<a id="starting-carbondata-thrift-server" class="anchor" href="#starting-carbondata-thrift-server" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Starting CarbonData Thrift Server.</h3> +<p>a. cd <code>$SPARK_HOME</code></p> +<p>b. Run the following command to start the CarbonData thrift server.</p> +<pre><code>./bin/spark-submit +--class org.apache.carbondata.spark.thriftserver.CarbonThriftServer +$SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path> +</code></pre> +<table> +<thead> +<tr> +<th>Parameter</th> +<th>Description</th> +<th>Example</th> +</tr> +</thead> +<tbody> +<tr> +<td>CARBON_ASSEMBLY_JAR</td> +<td>CarbonData assembly jar name present in the <code>$SPARK_HOME/carbonlib/</code> folder.</td> +<td>carbondata_2.xx-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar</td> +</tr> +<tr> +<td>carbon_store_path</td> +<td>This is a parameter to the CarbonThriftServer class. This a HDFS path where CarbonData files will be kept. Strongly Recommended to put same as carbon.storelocation parameter of carbon.properties. If not specified then it takes spark.sql.warehouse.dir path.</td> +<td><code>hdfs://<host_name>:port/user/hive/warehouse/carbon.store</code></td> +</tr> +</tbody> +</table> +<p><strong>NOTE</strong>: From Spark 1.6, by default the Thrift server runs in multi-session mode. Which means each JDBC/ODBC connection owns a copy of their own SQL configuration and temporary function registry. Cached tables are still shared though. If you prefer to run the Thrift server in single-session mode and share all SQL configuration and temporary function registry, please set option <code>spark.sql.hive.thriftServer.singleSession</code> to <code>true</code>. You may either add this option to <code>spark-defaults.conf</code>, or pass it to <code>spark-submit.sh</code> via <code>--conf</code>:</p> +<pre><code>./bin/spark-submit +--conf spark.sql.hive.thriftServer.singleSession=true +--class org.apache.carbondata.spark.thriftserver.CarbonThriftServer +$SPARK_HOME/carbonlib/$CARBON_ASSEMBLY_JAR <carbon_store_path> +</code></pre> +<p><strong>But</strong> in single-session mode, if one user changes the database from one connection, the database of the other connections will be changed too.</p> +<p><strong>Examples</strong></p> +<ul> +<li>Start with default memory and executors.</li> +</ul> +<pre><code>./bin/spark-submit +--class org.apache.carbondata.spark.thriftserver.CarbonThriftServer +$SPARK_HOME/carbonlib +/carbondata_2.xx-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar +hdfs://<host_name>:port/user/hive/warehouse/carbon.store +</code></pre> +<ul> +<li>Start with Fixed executors and resources.</li> +</ul> +<pre><code>./bin/spark-submit +--class org.apache.carbondata.spark.thriftserver.CarbonThriftServer +--num-executors 3 --driver-memory 20g --executor-memory 250g +--executor-cores 32 +/srv/OSCON/BigData/HACluster/install/spark/sparkJdbc/lib +/carbondata_2.xx-x.x.x-SNAPSHOT-shade-hadoop2.7.2.jar +hdfs://<host_name>:port/user/hive/warehouse/carbon.store +</code></pre> +<h3> +<a id="connecting-to-carbondata-thrift-server-using-beeline" class="anchor" href="#connecting-to-carbondata-thrift-server-using-beeline" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Connecting to CarbonData Thrift Server Using Beeline.</h3> +<pre><code> cd $SPARK_HOME + ./sbin/start-thriftserver.sh + ./bin/beeline -u jdbc:hive2://<thriftserver_host>:port + + Example + ./bin/beeline -u jdbc:hive2://10.10.10.10:10000 +</code></pre> +</div> +</div> +</div> +</div> +<div class="doc-footer"> + <a href="#top" class="scroll-top">Top</a> +</div> +</div> +</section> +</div> +</div> +</div> +</section><!-- End systemblock part --> +<script src="js/custom.js"></script> +</body> +</html> \ No newline at end of file
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6f8949f1/content/mainpage.html ---------------------------------------------------------------------- diff --git a/content/mainpage.html b/content/mainpage.html new file mode 100644 index 0000000..d515853 --- /dev/null +++ b/content/mainpage.html @@ -0,0 +1,214 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'> + <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> + <title>CarbonData</title> + <style> + + </style> + <!-- Bootstrap --> + + <link rel="stylesheet" href="css/bootstrap.min.css"> + <link href="css/style.css" rel="stylesheet"> + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script> + <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script> + <![endif]--> + <script src="js/jquery.min.js"></script> + <script src="js/bootstrap.min.js"></script> + + +</head> +<body> +<header> + <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper"> + <div class="container"> + <div class="navbar-header"> + <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse" + class="navbar-toggle collapsed" type="button"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a href="index.html" class="logo"> + <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/> + </a> + </div> + <div class="navbar-collapse collapse cd_navcontnt" id="navbar"> + <ul class="nav navbar-nav navbar-right navlist-custom"> + <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a> + </li> + <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false"> Download <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; + target="_blank">Apache CarbonData 1.4.1</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/"; + target="_blank">Apache CarbonData 1.4.0</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; + target="_blank">Apache CarbonData 1.3.1</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; + target="_blank">Apache CarbonData 1.3.0</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; + target="_blank">Release Archive</a></li> + </ul> + </li> + <li><a href="mainpage.html" class="active">Documentation</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false">Community <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://github.com/apache/carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"; + target="_blank">Contributing to CarbonData</a></li> + <li> + <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"; + target="_blank">Release Guide</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"; + target="_blank">Project PMC and Committers</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; + target="_blank">CarbonData Meetups</a></li> + <li><a href="security.html">Apache CarbonData Security</a></li> + <li><a href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache + Jira</a></li> + <li><a href="videogallery.html">CarbonData Videos </a></li> + </ul> + </li> + <li class="dropdown"> + <a href="http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="http://www.apache.org/"; class="hidden-lg hidden-md hidden-sm dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li> + <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a> + + </li> + + </ul> + </div><!--/.nav-collapse --> + <div id="search-box"> + <form method="get" action="http://www.google.com/search";> + <div class="search-block"> + <table border="0" cellpadding="0" width="100%"> + <tr> + <td style="width:80%"> + <input type="text" name="q" size=" 5" maxlength="255" value="" + class="search-input" placeholder="Search...." required/> + </td> + <td style="width:20%"> + <input type="submit" value="Search"/></td> + </tr> + <tr> + <td align="left" style="font-size:75%" colspan="2"> + <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/> + <span style=" position: relative; top: -3px;"> Only search for CarbonData</span> + </td> + </tr> + </table> + </div> + </form> + </div> + </div> + </nav> +</header> <!-- end Header part --> + +<div class="fixed-padding"></div> <!-- top padding with fixde header --> + +<section><!-- Dashboard nav --> + <div class="container-fluid q"> + <div class="col-sm-12 col-md-12 maindashboard"> + <div class="row"> + <section> + <div style="padding:10px 15px;"> + <div id="viewpage" name="viewpage"> + <div class="doc-heading"> + <h4 class="title">Documentation + <span class="title-underline"></span> + </h4> + </div> + + <div class="row"> + + <div class="col-sm-12 col-md-12"> + <span class="text-justify"> + Welcome to Apache CarbonData. Apache CarbonData is a new big data file format for faster interactive query using advanced columnar storage, index, compression and encoding techniques to improve computing efficiency, which helps in speeding up queries by an order of magnitude faster over PetaBytes of data. This user guide provides a detailed description about the CarbonData and its features. + Let's get started ! + </span> + <hr style="margin: 12px 0 8px"> + <div> + <ul class="sub-nav"> + <li><a href="quick-start-guide.html">Quick Start</a></li> + <li><a href="file-structure-of-carbondata.html">CarbonData File Structure</a></li> + <li><a href="supported-data-types-in-carbondata.html">Data Types</a></li> + <li><a href="data-management-on-carbondata.html">Data Management On CarbonData</a></li> + <li><a href="installation-guide.html">Installation Guide</a></li> + <li><a href="configuration-parameters.html">Configuring CarbonData</a></li> + <li><a href="streaming-guide.html">Streaming Guide</a></li> + <li><a href="sdk-guide.html">SDK Guide</a></li> + <li><a href="s3-guide.html">S3 Guide (Alpha Feature)</a></li> + <li><a href="datamap-developer-guide.html">DataMap Developer Guide</a></li> + <li><a href="datamap-management.html">CarbonData DataMap Management</a></li> + <li><a href="bloomfilter-datamap-guide.html">CarbonData BloomFilter DataMap (Alpha Feature)</a></li> + <li><a href="lucene-datamap-guide.html">CarbonData Lucene DataMap (Alpha Feature)</a></li> + <li><a href="preaggregate-datamap-guide.html">CarbonData Pre-aggregate DataMap</a></li> + <li><a href="timeseries-datamap-guide.html">CarbonData Timeseries DataMap</a></li> + <li><a href="faq.html">FAQs</a></li> + <li><a href="troubleshooting.html">Troubleshooting</a></li> + <li><a href="useful-tips-on-carbondata.html">Useful Tips</a></li> + + </ul> + </div> + </div> + </div> + </div> + <div class="doc-footer"> + <a href="#top" class="scroll-top">Top</a> + </div> + </div> + </section> + </div> + </div> + </div> +</section><!-- End systemblock part --> +</div> +</div> +</div> +</section><!-- End systemblock part --> +<script src="js/custom.js"></script> +</body> +</html> http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6f8949f1/content/sdk-writer-guide.html ---------------------------------------------------------------------- diff --git a/content/sdk-writer-guide.html b/content/sdk-writer-guide.html new file mode 100644 index 0000000..36bb9ad --- /dev/null +++ b/content/sdk-writer-guide.html @@ -0,0 +1,549 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'> + <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> + <title>CarbonData</title> + <style> + + </style> + <!-- Bootstrap --> + + <link rel="stylesheet" href="css/bootstrap.min.css"> + <link href="css/style.css" rel="stylesheet"> + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script> + <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script> + <![endif]--> + <script src="js/jquery.min.js"></script> + <script src="js/bootstrap.min.js"></script> + + +</head> +<body> +<header> + <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper"> + <div class="container"> + <div class="navbar-header"> + <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse" + class="navbar-toggle collapsed" type="button"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a href="index.html" class="logo"> + <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/> + </a> + </div> + <div class="navbar-collapse collapse cd_navcontnt" id="navbar"> + <ul class="nav navbar-nav navbar-right navlist-custom"> + <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a> + </li> + <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false"> Download <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/"; + target="_blank">Apache CarbonData 1.4.0</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; + target="_blank">Apache CarbonData 1.3.1</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; + target="_blank">Apache CarbonData 1.3.0</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.2.0/"; + target="_blank">Apache CarbonData 1.2.0</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.1/"; + target="_blank">Apache CarbonData 1.1.1</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.1.0/"; + target="_blank">Apache CarbonData 1.1.0</a></li> + <li> + <a href="http://archive.apache.org/dist/incubator/carbondata/1.0.0-incubating/"; + target="_blank">Apache CarbonData 1.0.0</a></li> + <li> + <a href="http://archive.apache.org/dist/incubator/carbondata/0.2.0-incubating/"; + target="_blank">Apache CarbonData 0.2.0</a></li> + <li> + <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.1-incubating/"; + target="_blank">Apache CarbonData 0.1.1</a></li> + <li> + <a href="http://archive.apache.org/dist/incubator/carbondata/0.1.0-incubating/"; + target="_blank">Apache CarbonData 0.1.0</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; + target="_blank">Release Archive</a></li> + </ul> + </li> + <li><a href="mainpage.html" class="active">Documentation</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false">Community <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://github.com/apache/carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"; + target="_blank">Contributing to CarbonData</a></li> + <li> + <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"; + target="_blank">Release Guide</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"; + target="_blank">Project PMC and Committers</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; + target="_blank">CarbonData Meetups</a></li> + <li><a href="security.html">Apache CarbonData Security</a></li> + <li><a href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache + Jira</a></li> + <li><a href="videogallery.html">CarbonData Videos </a></li> + </ul> + </li> + <li class="dropdown"> + <a href="http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="http://www.apache.org/"; class="hidden-lg hidden-md hidden-sm dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li> + <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a> + + </li> + + </ul> + </div><!--/.nav-collapse --> + <div id="search-box"> + <form method="get" action="http://www.google.com/search"; target="_blank"> + <div class="search-block"> + <table border="0" cellpadding="0" width="100%"> + <tr> + <td style="width:80%"> + <input type="text" name="q" size=" 5" maxlength="255" value="" + class="search-input" placeholder="Search...." required/> + </td> + <td style="width:20%"> + <input type="submit" value="Search"/></td> + </tr> + <tr> + <td align="left" style="font-size:75%" colspan="2"> + <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/> + <span style=" position: relative; top: -3px;"> Only search for CarbonData</span> + </td> + </tr> + </table> + </div> + </form> + </div> + </div> + </nav> +</header> <!-- end Header part --> + +<div class="fixed-padding"></div> <!-- top padding with fixde header --> + +<section><!-- Dashboard nav --> + <div class="container-fluid q"> + <div class="col-sm-12 col-md-12 maindashboard"> + <div class="row"> + <section> + <div style="padding:10px 15px;"> + <div id="viewpage" name="viewpage"> + <div class="row"> + <div class="col-sm-12 col-md-12"> + <div><h1> +<a id="sdk-writer-guide" class="anchor" href="#sdk-writer-guide" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SDK Writer Guide</h1> +<p>In the carbon jars package, there exist a carbondata-store-sdk-x.x.x-SNAPSHOT.jar. +This SDK writer, writes carbondata file and carbonindex file at a given path. +External client can make use of this writer to convert other format data or live data to create carbondata and index files. +These SDK writer output contains just a carbondata and carbonindex files. No metadata folder will be present.</p> +<h2> +<a id="quick-example" class="anchor" href="#quick-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Quick example</h2> +<h3> +<a id="example-with-csv-format" class="anchor" href="#example-with-csv-format" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Example with csv format</h3> +<div class="highlight highlight-source-java"><pre> <span class="pl-k">import</span> <span class="pl-smi">java.io.IOException</span>; + + <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException</span>; + <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.core.metadata.datatype.DataTypes</span>; + <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.CarbonWriter</span>; + <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.CarbonWriterBuilder</span>; + <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.Field</span>; + <span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.Schema</span>; + + <span class="pl-k">public</span> <span class="pl-k">class</span> <span class="pl-en">TestSdk</span> { + + <span class="pl-k">public</span> <span class="pl-k">static</span> <span class="pl-k">void</span> <span class="pl-en">main</span>(<span class="pl-k">String</span>[] <span class="pl-v">args</span>) <span class="pl-k">throws</span> <span class="pl-smi">IOException</span>, <span class="pl-smi">InvalidLoadOptionException</span> { + testSdkWriter(); + } + + <span class="pl-k">public</span> <span class="pl-k">static</span> <span class="pl-k">void</span> <span class="pl-en">testSdkWriter</span>() <span class="pl-k">throws</span> <span class="pl-smi">IOException</span>, <span class="pl-smi">InvalidLoadOptionException</span> { + <span class="pl-smi">String</span> path <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>/home/root1/Documents/ab/temp<span class="pl-pds">"</span></span>; + + <span class="pl-k">Field</span>[] fields <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-smi">Field</span>[<span class="pl-c1">2</span>]; + fields[<span class="pl-c1">0</span>] <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-smi">Field</span>(<span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>, <span class="pl-smi">DataTypes</span><span class="pl-c1"><span class="pl-k">.</span>STRING</span>); + fields[<span class="pl-c1">1</span>] <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-smi">Field</span>(<span class="pl-s"><span class="pl-pds">"</span>age<span class="pl-pds">"</span></span>, <span class="pl-smi">DataTypes</span><span class="pl-c1"><span class="pl-k">.</span>INT</span>); + + <span class="pl-smi">Schema</span> schema <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-smi">Schema</span>(fields); + + <span class="pl-smi">CarbonWriterBuilder</span> builder <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder()<span class="pl-k">.</span>withSchema(schema)<span class="pl-k">.</span>outputPath(path); + + <span class="pl-smi">CarbonWriter</span> writer <span class="pl-k">=</span> builder<span class="pl-k">.</span>buildWriterForCSVInput(); + + <span class="pl-k">int</span> rows <span class="pl-k">=</span> <span class="pl-c1">5</span>; + <span class="pl-k">for</span> (<span class="pl-k">int</span> i <span class="pl-k">=</span> <span class="pl-c1">0</span>; i <span class="pl-k"><</span> rows; i<span class="pl-k">++</span>) { + writer<span class="pl-k">.</span>write(<span class="pl-k">new</span> <span class="pl-smi">String</span>[] { <span class="pl-s"><span class="pl-pds">"</span>robot<span class="pl-pds">"</span></span> <span class="pl-k">+</span> (i <span class="pl-k">%</span> <span class="pl-c1">10</span>), <span class="pl-smi">String</span><span class="pl-k">.</span>valueOf(i) }); + } + writer<span class="pl-k">.</span>close(); + } + }</pre></div> +<h3> +<a id="example-with-avro-format" class="anchor" href="#example-with-avro-format" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Example with Avro format</h3> +<div class="highlight highlight-source-java"><pre><span class="pl-k">import</span> <span class="pl-smi">java.io.IOException</span>; + +<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException</span>; +<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.core.metadata.datatype.DataTypes</span>; +<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.AvroCarbonWriter</span>; +<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.CarbonWriter</span>; +<span class="pl-k">import</span> <span class="pl-smi">org.apache.carbondata.sdk.file.Field</span>; + +<span class="pl-k">import</span> <span class="pl-smi">org.apache.avro.generic.GenericData</span>; +<span class="pl-k">import</span> <span class="pl-smi">org.apache.commons.lang.CharEncoding</span>; + +<span class="pl-k">import</span> <span class="pl-smi">tech.allegro.schema.json2avro.converter.JsonAvroConverter</span>; + +<span class="pl-k">public</span> <span class="pl-k">class</span> <span class="pl-en">TestSdkAvro</span> { + + <span class="pl-k">public</span> <span class="pl-k">static</span> <span class="pl-k">void</span> <span class="pl-en">main</span>(<span class="pl-k">String</span>[] <span class="pl-v">args</span>) <span class="pl-k">throws</span> <span class="pl-smi">IOException</span>, <span class="pl-smi">InvalidLoadOptionException</span> { + testSdkWriter(); + } + + + <span class="pl-k">public</span> <span class="pl-k">static</span> <span class="pl-k">void</span> <span class="pl-en">testSdkWriter</span>() <span class="pl-k">throws</span> <span class="pl-smi">IOException</span>, <span class="pl-smi">InvalidLoadOptionException</span> { + <span class="pl-smi">String</span> path <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>./AvroCarbonWriterSuiteWriteFiles<span class="pl-pds">"</span></span>; + <span class="pl-c"><span class="pl-c">//</span> Avro schema</span> + <span class="pl-smi">String</span> avroSchema <span class="pl-k">=</span> + <span class="pl-s"><span class="pl-pds">"</span>{<span class="pl-pds">"</span></span> <span class="pl-k">+</span> + <span class="pl-s"><span class="pl-pds">"</span> <span class="pl-cce">\"</span>type<span class="pl-cce">\"</span> : <span class="pl-cce">\"</span>record<span class="pl-cce">\"</span>,<span class="pl-pds">"</span></span> <span class="pl-k">+</span> + <span class="pl-s"><span class="pl-pds">"</span> <span class="pl-cce">\"</span>name<span class="pl-cce">\"</span> : <span class="pl-cce">\"</span>Acme<span class="pl-cce">\"</span>,<span class="pl-pds">"</span></span> <span class="pl-k">+</span> + <span class="pl-s"><span class="pl-pds">"</span> <span class="pl-cce">\"</span>fields<span class="pl-cce">\"</span> : [<span class="pl-pds">"</span></span> + <span class="pl-k">+</span> <span class="pl-s"><span class="pl-pds">"</span>{ <span class="pl-cce">\"</span>name<span class="pl-cce">\"</span> : <span class="pl-cce">\"</span>fname<span class="pl-cce">\"</span>, <span class="pl-cce">\"</span>type<span class="pl-cce">\"</span> : <span class="pl-cce">\"</span>string<span class="pl-cce">\"</span> },<span class="pl-pds">"</span></span> + <span class="pl-k">+</span> <span class="pl-s"><span class="pl-pds">"</span>{ <span class="pl-cce">\"</span>name<span class="pl-cce">\"</span> : <span class="pl-cce">\"</span>age<span class="pl-cce">\"</span>, <span class="pl-cce">\"</span>type<span class="pl-cce">\"</span> : <span class="pl-cce">\"</span>int<span class="pl-cce">\"</span> }]<span class="pl-pds">"</span></span> <span class="pl-k">+</span> + <span class="pl-s"><span class="pl-pds">"</span>}<span class="pl-pds">"</span></span>; + + <span class="pl-smi">String</span> json <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>{<span class="pl-cce">\"</span>fname<span class="pl-cce">\"</span>:<span class="pl-cce">\"</span>bob<span class="pl-cce">\"</span>, <span class="pl-cce">\"</span>age<span class="pl-cce">\"</span>:10}<span class="pl-pds">"</span></span>; + + <span class="pl-c"><span class="pl-c">//</span> conversion to GenericData.Record</span> + <span class="pl-smi">JsonAvroConverter</span> converter <span class="pl-k">=</span> <span class="pl-k">new</span> <span class="pl-smi">JsonAvroConverter</span>(); + <span class="pl-smi">GenericData</span><span class="pl-k">.</span><span class="pl-smi">Record</span> record <span class="pl-k">=</span> converter<span class="pl-k">.</span>convertToGenericDataRecord( + json<span class="pl-k">.</span>getBytes(<span class="pl-smi">CharEncoding</span><span class="pl-c1"><span class="pl-k">.</span>UTF_8</span>), <span class="pl-k">new</span> <span class="pl-smi">org.apache.avro<span class="pl-k">.</span>Schema</span>.<span class="pl-smi">Parser</span>()<span class="pl-k">.</span>parse(avroSchema)); + + <span class="pl-c"><span class="pl-c">//</span> prepare carbon schema from avro schema </span> + <span class="pl-smi">org.apache.carbondata.sdk.file<span class="pl-k">.</span>Schema</span> carbonSchema <span class="pl-k">=</span> + <span class="pl-smi">AvroCarbonWriter</span><span class="pl-k">.</span>getCarbonSchemaFromAvroSchema(avroSchema); + + <span class="pl-k">try</span> { + <span class="pl-smi">CarbonWriter</span> writer <span class="pl-k">=</span> <span class="pl-smi">CarbonWriter</span><span class="pl-k">.</span>builder() + .withSchema(carbonSchema) + .outputPath(path) + .buildWriterForAvroInput(); + + <span class="pl-k">for</span> (<span class="pl-k">int</span> i <span class="pl-k">=</span> <span class="pl-c1">0</span>; i <span class="pl-k"><</span> <span class="pl-c1">100</span>; i<span class="pl-k">++</span>) { + writer<span class="pl-k">.</span>write(record); + } + writer<span class="pl-k">.</span>close(); + } <span class="pl-k">catch</span> (<span class="pl-smi">Exception</span> e) { + e<span class="pl-k">.</span>printStackTrace(); + } + } +}</pre></div> +<h2> +<a id="datatypes-mapping" class="anchor" href="#datatypes-mapping" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Datatypes Mapping</h2> +<p>Each of SQL data types are mapped into data types of SDK. Following are the mapping:</p> +<table> +<thead> +<tr> +<th>SQL DataTypes</th> +<th>Mapped SDK DataTypes</th> +</tr> +</thead> +<tbody> +<tr> +<td>BOOLEAN</td> +<td>DataTypes.BOOLEAN</td> +</tr> +<tr> +<td>SMALLINT</td> +<td>DataTypes.SHORT</td> +</tr> +<tr> +<td>INTEGER</td> +<td>DataTypes.INT</td> +</tr> +<tr> +<td>BIGINT</td> +<td>DataTypes.LONG</td> +</tr> +<tr> +<td>DOUBLE</td> +<td>DataTypes.DOUBLE</td> +</tr> +<tr> +<td>VARCHAR</td> +<td>DataTypes.STRING</td> +</tr> +<tr> +<td>DATE</td> +<td>DataTypes.DATE</td> +</tr> +<tr> +<td>TIMESTAMP</td> +<td>DataTypes.TIMESTAMP</td> +</tr> +<tr> +<td>STRING</td> +<td>DataTypes.STRING</td> +</tr> +<tr> +<td>DECIMAL</td> +<td>DataTypes.createDecimalType(precision, scale)</td> +</tr> +</tbody> +</table> +<h2> +<a id="api-list" class="anchor" href="#api-list" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>API List</h2> +<h3> +<a id="class-orgapachecarbondatasdkfilecarbonwriterbuilder" class="anchor" href="#class-orgapachecarbondatasdkfilecarbonwriterbuilder" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.CarbonWriterBuilder</h3> +<pre><code>/** +* prepares the builder with the schema provided +* @param schema is instance of Schema +* This method must be called when building CarbonWriterBuilder +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder withSchema(Schema schema); +</code></pre> +<pre><code>/** +* Sets the output path of the writer builder +* @param path is the absolute path where output files are written +* This method must be called when building CarbonWriterBuilder +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder outputPath(String path); +</code></pre> +<pre><code>/** +* If set false, writes the carbondata and carbonindex files in a flat folder structure +* @param isTransactionalTable is a boolelan value +* if set to false, then writes the carbondata and carbonindex files +* in a flat folder structure. +* if set to true, then writes the carbondata and carbonindex files +* in segment folder structure.. +* By default set to false. +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder isTransactionalTable(boolean isTransactionalTable); +</code></pre> +<pre><code>/** +* to set the timestamp in the carbondata and carbonindex index files +* @param UUID is a timestamp to be used in the carbondata and carbonindex index files. +* By default set to zero. +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder uniqueIdentifier(long UUID); +</code></pre> +<pre><code>/** +* To set the carbondata file size in MB between 1MB-2048MB +* @param blockSize is size in MB between 1MB to 2048 MB +* default value is 1024 MB +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder withBlockSize(int blockSize); +</code></pre> +<pre><code>/** +* To set the blocklet size of carbondata file +* @param blockletSize is blocklet size in MB +* default value is 64 MB +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder withBlockletSize(int blockletSize); +</code></pre> +<pre><code>/** +* sets the list of columns that needs to be in sorted order +* @param sortColumns is a string array of columns that needs to be sorted. +* If it is null or by default all dimensions are selected for sorting +* If it is empty array, no columns are sorted +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder sortBy(String[] sortColumns); +</code></pre> +<pre><code>/** +* If set, create a schema file in metadata folder. +* @param persist is a boolean value, If set to true, creates a schema file in metadata folder. +* By default set to false. will not create metadata folder +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder persistSchemaFile(boolean persist); +</code></pre> +<pre><code>/** +* sets the taskNo for the writer. SDKs concurrently running +* will set taskNo in order to avoid conflicts in file's name during write. +* @param taskNo is the TaskNo user wants to specify. +* by default it is system time in nano seconds. +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder taskNo(String taskNo); +</code></pre> +<pre><code>/** +* To support the load options for sdk writer +* @param options key,value pair of load options. +* supported keys values are +* a. bad_records_logger_enable -- true (write into separate logs), false +* b. bad_records_action -- FAIL, FORCE, IGNORE, REDIRECT +* c. bad_record_path -- path +* d. dateformat -- same as JAVA SimpleDateFormat +* e. timestampformat -- same as JAVA SimpleDateFormat +* f. complex_delimiter_level_1 -- value to Split the complexTypeData +* g. complex_delimiter_level_2 -- value to Split the nested complexTypeData +* h. quotechar +* i. escapechar +* +* Default values are as follows. +* +* a. bad_records_logger_enable -- "false" +* b. bad_records_action -- "FAIL" +* c. bad_record_path -- "" +* d. dateformat -- "" , uses from carbon.properties file +* e. timestampformat -- "", uses from carbon.properties file +* f. complex_delimiter_level_1 -- "$" +* g. complex_delimiter_level_2 -- ":" +* h. quotechar -- "\"" +* i. escapechar -- "\\" +* +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder withLoadOptions(Map<String, String> options); +</code></pre> +<pre><code>/** +* Build a {@link CarbonWriter}, which accepts row in CSV format object +* @return CSVCarbonWriter +* @throws IOException +* @throws InvalidLoadOptionException +*/ +public CarbonWriter buildWriterForCSVInput() throws IOException, InvalidLoadOptionException; +</code></pre> +<pre><code>/** +* Build a {@link CarbonWriter}, which accepts Avro format object +* @return AvroCarbonWriter +* @throws IOException +* @throws InvalidLoadOptionException +*/ +public CarbonWriter buildWriterForAvroInput() throws IOException, InvalidLoadOptionException; +</code></pre> +<h3> +<a id="class-orgapachecarbondatasdkfilecarbonwriter" class="anchor" href="#class-orgapachecarbondatasdkfilecarbonwriter" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.CarbonWriter</h3> +<pre><code>/** +* Write an object to the file, the format of the object depends on the implementation +* If AvroCarbonWriter, object is of type org.apache.avro.generic.GenericData.Record +* If CSVCarbonWriter, object is of type String[] +* Note: This API is not thread safe +* @param object +* @throws IOException +*/ +public abstract void write(Object object) throws IOException; +</code></pre> +<pre><code>/** +* Flush and close the writer +*/ +public abstract void close() throws IOException; +</code></pre> +<pre><code>/** +* Create a {@link CarbonWriterBuilder} to build a {@link CarbonWriter} +*/ +public static CarbonWriterBuilder builder() { +return new CarbonWriterBuilder(); +} +</code></pre> +<h3> +<a id="class-orgapachecarbondatasdkfilefield" class="anchor" href="#class-orgapachecarbondatasdkfilefield" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.Field</h3> +<pre><code>/** +* Field Constructor +* @param name name of the field +* @param type datatype of field, specified in strings. +*/ +public Field(String name, String type); +</code></pre> +<pre><code>/** +* Field constructor +* @param name name of the field +* @param type datatype of the field of class DataType +*/ +public Field(String name, DataType type); +</code></pre> +<h3> +<a id="class-orgapachecarbondatasdkfileschema" class="anchor" href="#class-orgapachecarbondatasdkfileschema" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.Schema</h3> +<pre><code>/** +* construct a schema with fields +* @param fields +*/ +public Schema(Field[] fields); +</code></pre> +<pre><code>/** +* Create a Schema using JSON string, for example: +* [ +* {"name":"string"}, +* {"age":"int"} +* ] +* @param json specified as string +* @return Schema +*/ +public static Schema parseJson(String json); +</code></pre> +<h3> +<a id="class-orgapachecarbondatasdkfileavrocarbonwriter" class="anchor" href="#class-orgapachecarbondatasdkfileavrocarbonwriter" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Class org.apache.carbondata.sdk.file.AvroCarbonWriter</h3> +<pre><code>/** +* converts avro schema to carbon schema, required by carbonWriter +* +* @param avroSchemaString json formatted avro schema as string +* @return carbon sdk schema +*/ +public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(String avroSchemaString); +</code></pre> +</div> +</div> +</div> +</div> +<div class="doc-footer"> + <a href="#top" class="scroll-top">Top</a> +</div> +</div> +</section> +</div> +</div> +</div> +</section><!-- End systemblock part --> +<script src="js/custom.js"></script> +</body> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/6f8949f1/content/troubleshooting.html ---------------------------------------------------------------------- diff --git a/content/troubleshooting.html b/content/troubleshooting.html new file mode 100644 index 0000000..c668dc9 --- /dev/null +++ b/content/troubleshooting.html @@ -0,0 +1,366 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <link href='images/favicon.ico' rel='shortcut icon' type='image/x-icon'> + <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> + <title>CarbonData</title> + <style> + + </style> + <!-- Bootstrap --> + + <link rel="stylesheet" href="css/bootstrap.min.css"> + <link href="css/style.css" rel="stylesheet"> + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js";></script> + <script src="https://oss.maxcdn.scom/respond/1.4.2/respond.min.js";></script> + <![endif]--> + <script src="js/jquery.min.js"></script> + <script src="js/bootstrap.min.js"></script> + + +</head> +<body> +<header> + <nav class="navbar navbar-default navbar-custom cd-navbar-wrapper"> + <div class="container"> + <div class="navbar-header"> + <button aria-controls="navbar" aria-expanded="false" data-target="#navbar" data-toggle="collapse" + class="navbar-toggle collapsed" type="button"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <a href="index.html" class="logo"> + <img src="images/CarbonDataLogo.png" alt="CarbonData logo" title="CarbocnData logo"/> + </a> + </div> + <div class="navbar-collapse collapse cd_navcontnt" id="navbar"> + <ul class="nav navbar-nav navbar-right navlist-custom"> + <li><a href="index.html" class="hidden-xs"><i class="fa fa-home" aria-hidden="true"></i> </a> + </li> + <li><a href="index.html" class="hidden-lg hidden-md hidden-sm">Home</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle " data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false"> Download <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.1/"; + target="_blank">Apache CarbonData 1.4.1</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.4.0/"; + target="_blank">Apache CarbonData 1.4.0</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.1/"; + target="_blank">Apache CarbonData 1.3.1</a></li> + <li> + <a href="https://dist.apache.org/repos/dist/release/carbondata/1.3.0/"; + target="_blank">Apache CarbonData 1.3.0</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/Releases"; + target="_blank">Release Archive</a></li> + </ul> + </li> + <li><a href="mainpage.html" class="active">Documentation</a></li> + <li class="dropdown"> + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" + aria-expanded="false">Community <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li> + <a href="https://github.com/apache/carbondata/blob/master/docs/How-to-contribute-to-Apache-CarbonData.md"; + target="_blank">Contributing to CarbonData</a></li> + <li> + <a href="https://github.com/apache/carbondata/blob/master/docs/release-guide.md"; + target="_blank">Release Guide</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/display/CARBONDATA/PMC+and+Committers+member+list"; + target="_blank">Project PMC and Committers</a></li> + <li> + <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66850609"; + target="_blank">CarbonData Meetups</a></li> + <li><a href="security.html">Apache CarbonData Security</a></li> + <li><a href="https://issues.apache.org/jira/browse/CARBONDATA"; target="_blank">Apache + Jira</a></li> + <li><a href="videogallery.html">CarbonData Videos </a></li> + </ul> + </li> + <li class="dropdown"> + <a href="http://www.apache.org/"; class="apache_link hidden-xs dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li class="dropdown"> + <a href="http://www.apache.org/"; class="hidden-lg hidden-md hidden-sm dropdown-toggle" + data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Apache</a> + <ul class="dropdown-menu"> + <li><a href="http://www.apache.org/"; target="_blank">Apache Homepage</a></li> + <li><a href="http://www.apache.org/licenses/"; target="_blank">License</a></li> + <li><a href="http://www.apache.org/foundation/sponsorship.html"; + target="_blank">Sponsorship</a></li> + <li><a href="http://www.apache.org/foundation/thanks.html"; target="_blank">Thanks</a></li> + </ul> + </li> + + <li> + <a href="#" id="search-icon"><i class="fa fa-search" aria-hidden="true"></i></a> + + </li> + + </ul> + </div><!--/.nav-collapse --> + <div id="search-box"> + <form method="get" action="http://www.google.com/search"; target="_blank"> + <div class="search-block"> + <table border="0" cellpadding="0" width="100%"> + <tr> + <td style="width:80%"> + <input type="text" name="q" size=" 5" maxlength="255" value="" + class="search-input" placeholder="Search...." required/> + </td> + <td style="width:20%"> + <input type="submit" value="Search"/></td> + </tr> + <tr> + <td align="left" style="font-size:75%" colspan="2"> + <input type="checkbox" name="sitesearch" value="carbondata.apache.org" checked/> + <span style=" position: relative; top: -3px;"> Only search for CarbonData</span> + </td> + </tr> + </table> + </div> + </form> + </div> + </div> + </nav> +</header> <!-- end Header part --> + +<div class="fixed-padding"></div> <!-- top padding with fixde header --> + +<section><!-- Dashboard nav --> + <div class="container-fluid q"> + <div class="col-sm-12 col-md-12 maindashboard"> + <div class="row"> + <section> + <div style="padding:10px 15px;"> + <div id="viewpage" name="viewpage"> + <div class="row"> + <div class="col-sm-12 col-md-12"> + <div> +<h1> +<a id="troubleshooting" class="anchor" href="#troubleshooting" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Troubleshooting</h1> +<p>This tutorial is designed to provide troubleshooting for end users and developers +who are building, deploying, and using CarbonData.</p> +<h2> +<a id="when-loading-data-gets-tablestatuslock-issues" class="anchor" href="#when-loading-data-gets-tablestatuslock-issues" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>When loading data, gets tablestatus.lock issues:</h2> +<p><strong>Symptom</strong></p> +<pre><code>17/11/11 16:48:13 ERROR LocalFileLock: main hdfs:/localhost:9000/carbon/store/default/hdfstable/tablestatus.lock (No such file or directory) +java.io.FileNotFoundException: hdfs:/localhost:9000/carbon/store/default/hdfstable/tablestatus.lock (No such file or directory) + at java.io.FileOutputStream.open0(Native Method) + at java.io.FileOutputStream.open(FileOutputStream.java:270) + at java.io.FileOutputStream.<init>(FileOutputStream.java:213) + at java.io.FileOutputStream.<init>(FileOutputStream.java:101) +</code></pre> +<p><strong>Possible Cause</strong> +If you use <code><hdfs path></code> as store path when creating carbonsession, may get the errors,because the default is LOCALLOCK.</p> +<p><strong>Procedure</strong> +Before creating carbonsession, sets as below:</p> +<pre><code>import org.apache.carbondata.core.util.CarbonProperties +import org.apache.carbondata.core.constants.CarbonCommonConstants +CarbonProperties.getInstance().addProperty(CarbonCommonConstants.LOCK_TYPE, "HDFSLOCK") +</code></pre> +<h2> +<a id="failed-to-load-thrift-libraries" class="anchor" href="#failed-to-load-thrift-libraries" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to load thrift libraries</h2> +<p><strong>Symptom</strong></p> +<p>Thrift throws following exception :</p> +<pre><code>thrift: error while loading shared libraries: +libthriftc.so.0: cannot open shared object file: No such file or directory +</code></pre> +<p><strong>Possible Cause</strong></p> +<p>The complete path to the directory containing the libraries is not configured correctly.</p> +<p><strong>Procedure</strong></p> +<p>Follow the Apache thrift docs at <a href="https://thrift.apache.org/docs/install"; target=_blank rel="nofollow">https://thrift.apache.org/docs/install</a> to install thrift correctly.</p> +<h2> +<a id="failed-to-launch-the-spark-shell" class="anchor" href="#failed-to-launch-the-spark-shell" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to launch the Spark Shell</h2> +<p><strong>Symptom</strong></p> +<p>The shell prompts the following error :</p> +<pre><code>org.apache.spark.sql.CarbonContext$$anon$$apache$spark$sql$catalyst$analysis +$OverrideCatalog$_setter_$org$apache$spark$sql$catalyst$analysis +$OverrideCatalog$$overrides_$e +</code></pre> +<p><strong>Possible Cause</strong></p> +<p>The Spark Version and the selected Spark Profile do not match.</p> +<p><strong>Procedure</strong></p> +<ol> +<li> +<p>Ensure your spark version and selected profile for spark are correct.</p> +</li> +<li> +<p>Use the following command :</p> +</li> +</ol> +<pre><code>"mvn -Pspark-2.1 -Dspark.version {yourSparkVersion} clean package" +</code></pre> +<p>Note : Refrain from using "mvn clean package" without specifying the profile.</p> +<h2> +<a id="failed-to-execute-load-query-on-cluster" class="anchor" href="#failed-to-execute-load-query-on-cluster" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to execute load query on cluster.</h2> +<p><strong>Symptom</strong></p> +<p>Load query failed with the following exception:</p> +<pre><code>Dictionary file is locked for updation. +</code></pre> +<p><strong>Possible Cause</strong></p> +<p>The carbon.properties file is not identical in all the nodes of the cluster.</p> +<p><strong>Procedure</strong></p> +<p>Follow the steps to ensure the carbon.properties file is consistent across all the nodes:</p> +<ol> +<li> +<p>Copy the carbon.properties file from the master node to all the other nodes in the cluster. +For example, you can use ssh to copy this file to all the nodes.</p> +</li> +<li> +<p>For the changes to take effect, restart the Spark cluster.</p> +</li> +</ol> +<h2> +<a id="failed-to-execute-insert-query-on-cluster" class="anchor" href="#failed-to-execute-insert-query-on-cluster" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to execute insert query on cluster.</h2> +<p><strong>Symptom</strong></p> +<p>Load query failed with the following exception:</p> +<pre><code>Dictionary file is locked for updation. +</code></pre> +<p><strong>Possible Cause</strong></p> +<p>The carbon.properties file is not identical in all the nodes of the cluster.</p> +<p><strong>Procedure</strong></p> +<p>Follow the steps to ensure the carbon.properties file is consistent across all the nodes:</p> +<ol> +<li> +<p>Copy the carbon.properties file from the master node to all the other nodes in the cluster. +For example, you can use scp to copy this file to all the nodes.</p> +</li> +<li> +<p>For the changes to take effect, restart the Spark cluster.</p> +</li> +</ol> +<h2> +<a id="failed-to-connect-to-hiveuser-with-thrift" class="anchor" href="#failed-to-connect-to-hiveuser-with-thrift" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to connect to hiveuser with thrift</h2> +<p><strong>Symptom</strong></p> +<p>We get the following exception :</p> +<pre><code>Cannot connect to hiveuser. +</code></pre> +<p><strong>Possible Cause</strong></p> +<p>The external process does not have permission to access.</p> +<p><strong>Procedure</strong></p> +<p>Ensure that the Hiveuser in mysql must allow its access to the external processes.</p> +<h2> +<a id="failed-to-read-the-metastore-db-during-table-creation" class="anchor" href="#failed-to-read-the-metastore-db-during-table-creation" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to read the metastore db during table creation.</h2> +<p><strong>Symptom</strong></p> +<p>We get the following exception on trying to connect :</p> +<pre><code>Cannot read the metastore db +</code></pre> +<p><strong>Possible Cause</strong></p> +<p>The metastore db is dysfunctional.</p> +<p><strong>Procedure</strong></p> +<p>Remove the metastore db from the carbon.metastore in the Spark Directory.</p> +<h2> +<a id="failed-to-load-data-on-the-cluster" class="anchor" href="#failed-to-load-data-on-the-cluster" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to load data on the cluster</h2> +<p><strong>Symptom</strong></p> +<p>Data loading fails with the following exception :</p> +<pre><code>Data Load failure exception +</code></pre> +<p><strong>Possible Cause</strong></p> +<p>The following issue can cause the failure :</p> +<ol> +<li> +<p>The core-site.xml, hive-site.xml, yarn-site and carbon.properties are not consistent across all nodes of the cluster.</p> +</li> +<li> +<p>Path to hdfs ddl is not configured correctly in the carbon.properties.</p> +</li> +</ol> +<p><strong>Procedure</strong></p> +<p>Follow the steps to ensure the following configuration files are consistent across all the nodes:</p> +<ol> +<li> +<p>Copy the core-site.xml, hive-site.xml, yarn-site,carbon.properties files from the master node to all the other nodes in the cluster. +For example, you can use scp to copy this file to all the nodes.</p> +<p>Note : Set the path to hdfs ddl in carbon.properties in the master node.</p> +</li> +<li> +<p>For the changes to take effect, restart the Spark cluster.</p> +</li> +</ol> +<h2> +<a id="failed-to-insert-data-on-the-cluster" class="anchor" href="#failed-to-insert-data-on-the-cluster" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to insert data on the cluster</h2> +<p><strong>Symptom</strong></p> +<p>Insertion fails with the following exception :</p> +<pre><code>Data Load failure exception +</code></pre> +<p><strong>Possible Cause</strong></p> +<p>The following issue can cause the failure :</p> +<ol> +<li> +<p>The core-site.xml, hive-site.xml, yarn-site and carbon.properties are not consistent across all nodes of the cluster.</p> +</li> +<li> +<p>Path to hdfs ddl is not configured correctly in the carbon.properties.</p> +</li> +</ol> +<p><strong>Procedure</strong></p> +<p>Follow the steps to ensure the following configuration files are consistent across all the nodes:</p> +<ol> +<li> +<p>Copy the core-site.xml, hive-site.xml, yarn-site,carbon.properties files from the master node to all the other nodes in the cluster. +For example, you can use scp to copy this file to all the nodes.</p> +<p>Note : Set the path to hdfs ddl in carbon.properties in the master node.</p> +</li> +<li> +<p>For the changes to take effect, restart the Spark cluster.</p> +</li> +</ol> +<h2> +<a id="failed-to-execute-concurrent-operationsloadinsertupdate-on-table-by-multiple-workers" class="anchor" href="#failed-to-execute-concurrent-operationsloadinsertupdate-on-table-by-multiple-workers" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to execute Concurrent Operations(Load,Insert,Update) on table by multiple workers.</h2> +<p><strong>Symptom</strong></p> +<p>Execution fails with the following exception :</p> +<pre><code>Table is locked for updation. +</code></pre> +<p><strong>Possible Cause</strong></p> +<p>Concurrency not supported.</p> +<p><strong>Procedure</strong></p> +<p>Worker must wait for the query execution to complete and the table to release the lock for another query execution to succeed.</p> +<h2> +<a id="failed-to-create-a-table-with-a-single-numeric-column" class="anchor" href="#failed-to-create-a-table-with-a-single-numeric-column" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failed to create a table with a single numeric column.</h2> +<p><strong>Symptom</strong></p> +<p>Execution fails with the following exception :</p> +<pre><code>Table creation fails. +</code></pre> +<p><strong>Possible Cause</strong></p> +<p>Behaviour not supported.</p> +<p><strong>Procedure</strong></p> +<p>A single column that can be considered as dimension is mandatory for table creation.</p> +</div> +</div> +</div> +</div> +<div class="doc-footer"> + <a href="#top" class="scroll-top">Top</a> +</div> +</div> +</section> +</div> +</div> +</div> +</section><!-- End systemblock part --> +<script src="js/custom.js"></script> +</body> +</html> \ No newline at end of file
