[2/2] incubator-carbondata git commit: [CARBONDATA-856] Alter Table - TABLE RENAME feature documentation This closes #756

[chenliang613]

[CARBONDATA-856] Alter Table - TABLE RENAME feature documentation This closes 
#756


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/914d61e3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/914d61e3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/914d61e3

Branch: refs/heads/master
Commit: 914d61e32f6bcb3b61e9da1957c95bfcee142fb4
Parents: cfefb74 b2950b4
Author: chenliang613 
Authored: Fri Apr 7 15:06:05 2017 +0530
Committer: chenliang613 
Committed: Fri Apr 7 15:06:05 2017 +0530

--
 docs/ddl-operation-on-carbondata.md | 29 +
 1 file changed, 29 insertions(+)
--

[1/2] incubator-carbondata git commit: [CARBONDATA-856] Alter Table - TABLE RENAME feature documentation

[chenliang613]

Repository: incubator-carbondata
Updated Branches:
  refs/heads/master cfefb7418 -> 914d61e32


[CARBONDATA-856] Alter Table - TABLE RENAME feature documentation

The documentation for Table rename as part of the alter table feature.

Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/b2950b44
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/b2950b44
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/b2950b44

Branch: refs/heads/master
Commit: b2950b44482bfdf51e3ae4879f081a3bb9f93e28
Parents: cfefb74
Author: srigopalmohanty 
Authored: Fri Apr 7 13:10:36 2017 +0530
Committer: chenliang613 
Committed: Fri Apr 7 15:02:50 2017 +0530

--
 docs/ddl-operation-on-carbondata.md | 29 +
 1 file changed, 29 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/b2950b44/docs/ddl-operation-on-carbondata.md
--
diff --git a/docs/ddl-operation-on-carbondata.md 
b/docs/ddl-operation-on-carbondata.md
index 215cff8..392bb39 100644
--- a/docs/ddl-operation-on-carbondata.md
+++ b/docs/ddl-operation-on-carbondata.md
@@ -28,6 +28,7 @@ The following DDL operations are supported in CarbonData :
 * [DROP TABLE](#drop-table)
 * [COMPACTION](#compaction)
 * [BUCKETING](#bucketing)
+* [TABLE RENAME](#table-rename)
 
 ## CREATE TABLE
   This command can be used to create a CarbonData table by specifying the list 
of fields along with the table properties.
@@ -229,3 +230,31 @@ of columns is used.
   'BUCKETCOLUMNS'='productName')
  ```
 
+## TABLE RENAME
+  This command is used to rename the existing table.
+
+### Syntax
+```
+   ALTER TABLE [db_name.]table_name RENAME TO new_table_name;
+```
+
+### Parameter Description
+
+| Parameter | Description | 
+|---|---|
+| db_name | Name of the database. If this parameter is left unspecified, the 
current database is selected. | 
+| table_name | Name of the existing table.|
+|new_table_name | New table name for the existing table. | 
+
+### Usage Guidelines
+Following conditions must be met for successful rename operation:
+* Queries running in parallel which requires the formation of path using the 
table name for reading carbon store files might fail during this operation.
+* Secondary index table rename is not permitted.
+
+### Example:
+```
+ALTER TABLE carbon RENAME TO carbondata;
+
+   ALTER TABLE test_db.carbon RENAME TO test_db.carbondata;
+```
+

[3/4] incubator-carbondata git commit: sort columns

[jackylk]

sort columns


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/fcf74017
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/fcf74017
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/fcf74017

Branch: refs/heads/12-dev
Commit: fcf740175a71f04a72e8c493b10a13250fce7523
Parents: 914d61e
Author: QiangCai 
Authored: Thu Mar 2 17:48:54 2017 +0800
Committer: QiangCai 
Committed: Fri Apr 7 16:01:16 2017 +0530

--
 .../core/constants/CarbonCommonConstants.java   |   1 +
 .../core/datastore/block/SegmentProperties.java |  42 +++
 .../impl/VariableLengthDimensionDataChunk.java  |  58 +++-
 .../impl/btree/BTreeDataRefNodeFinder.java  |  24 +-
 .../core/keygenerator/mdkey/Bits.java   |   6 +-
 .../ThriftWrapperSchemaConverterImpl.java   |  18 +
 .../core/metadata/schema/table/CarbonTable.java |  24 ++
 .../schema/table/column/ColumnSchema.java   |  10 +
 .../impl/DictionaryBasedResultCollector.java|  12 +-
 ...structureBasedDictionaryResultCollector.java |   4 +-
 .../scan/filter/FilterExpressionProcessor.java  |   4 +-
 .../carbondata/core/scan/filter/FilterUtil.java |  98 --
 ...velRangeLessThanEqualFilterExecuterImpl.java |   2 +-
 .../RowLevelRangeLessThanFiterExecuterImpl.java |   2 +-
 .../visitor/NoDictionaryTypeVisitor.java|   3 +-
 .../visitor/RangeNoDictionaryTypeVisitor.java   |   3 +-
 .../AbstractDetailQueryResultIterator.java  |   4 +-
 .../scan/result/vector/CarbonColumnVector.java  |   7 +
 .../util/AbstractDataFileFooterConverter.java   |   7 +
 .../apache/carbondata/core/util/ByteUtil.java   | 336 ++-
 .../apache/carbondata/core/util/CarbonUtil.java |  18 +
 .../carbondata/core/util/DataTypeUtil.java  |  70 
 .../impl/btree/BTreeBlockFinderTest.java|  10 +-
 .../core/scan/filter/FilterUtilTest.java|   7 +-
 .../carbondata/hadoop/CarbonInputFormat.java|   4 +-
 .../internal/index/impl/InMemoryBTreeIndex.java |   4 +-
 .../impl/DictionaryDecodeReadSupport.java   |   3 +
 .../hadoop/util/BlockLevelTraverser.java|   4 +-
 .../hadoop/test/util/StoreCreator.java  |   5 +
 .../testsuite/sortcolumns/TestSortColumns.scala | 267 +++
 .../spark/sql/catalyst/CarbonDDLSqlParser.scala | 125 +++
 .../execution/command/carbonTableSchema.scala   |  66 ++--
 .../apache/spark/sql/TestCarbonSqlParser.scala  |  56 +---
 .../readsupport/SparkRowReadSupportImpl.java|  13 +-
 .../vectorreader/ColumnarVectorWrapper.java |  13 +
 .../org/apache/spark/sql/TableCreator.scala | 145 
 .../merger/CompactionResultSortProcessor.java   |  10 +-
 .../merger/RowResultMergerProcessor.java|  14 +-
 .../newflow/CarbonDataLoadConfiguration.java|  24 ++
 .../newflow/DataLoadProcessBuilder.java |  24 +-
 .../impl/NonDictionaryFieldConverterImpl.java   |  21 +-
 .../sort/impl/ParallelReadMergeSorterImpl.java  |   3 +-
 ...arallelReadMergeSorterWithBucketingImpl.java |   3 +-
 .../sort/unsafe/UnsafeCarbonRowPage.java|  11 +-
 .../newflow/sort/unsafe/UnsafeSortDataRows.java |  11 +-
 .../unsafe/comparator/UnsafeRowComparator.java  |  10 +-
 .../UnsafeRowComparatorForNormalDIms.java   |   8 +-
 .../holder/UnsafeFinalMergePageHolder.java  |   4 +-
 .../unsafe/holder/UnsafeInmemoryHolder.java |   5 +-
 .../holder/UnsafeSortTempFileChunkHolder.java   |   2 +-
 .../UnsafeSingleThreadFinalSortFilesMerger.java |   4 +-
 .../newflow/steps/NoSortProcessorStepImpl.java  | 153 +
 .../sortdata/IntermediateFileMerger.java|   3 +-
 .../sortdata/NewRowComparator.java  |  12 +-
 .../sortdata/NewRowComparatorForNormalDims.java |  10 +-
 .../sortandgroupby/sortdata/RowComparator.java  |  10 +-
 .../sortdata/RowComparatorForNormalDims.java|  10 +-
 .../sortandgroupby/sortdata/SortDataRows.java   |  13 +-
 .../sortandgroupby/sortdata/SortParameters.java |  43 +++
 .../sortdata/SortTempFileChunkHolder.java   |  11 +-
 .../store/CarbonFactDataHandlerColumnar.java|  45 ++-
 .../store/CarbonFactDataHandlerModel.java   |   7 +
 .../store/SingleThreadFinalSortFilesMerger.java |   8 +-
 63 files changed, 1601 insertions(+), 353 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/fcf74017/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
 
b/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
index b82d53c..6cee7f6 100644
--- 

[2/4] incubator-carbondata git commit: sort columns

[jackylk]

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/fcf74017/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortcolumns/TestSortColumns.scala
--
diff --git 
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortcolumns/TestSortColumns.scala
 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortcolumns/TestSortColumns.scala
new file mode 100644
index 000..88b0421
--- /dev/null
+++ 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortcolumns/TestSortColumns.scala
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.spark.testsuite.sortcolumns
+
+import org.apache.spark.sql.common.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+class TestSortColumns extends QueryTest with BeforeAndAfterAll {
+
+  override def beforeAll {
+dropTable
+
+sql("CREATE TABLE origintable1 (empno int, empname String, designation 
String, doj Timestamp, workgroupcategory int, workgroupcategoryname String, 
deptno int, deptname String, projectcode int, projectjoindate Timestamp, 
projectenddate Timestamp,attendance int,utilization int,salary int) STORED BY 
'org.apache.carbondata.format'")
+sql(s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE 
origintable1 OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '\"')""")
+  }
+
+  test("create table with no dictionary sort_columns") {
+sql("CREATE TABLE sorttable1 (empno int, empname String, designation 
String, doj Timestamp, workgroupcategory int, workgroupcategoryname String, 
deptno int, deptname String, projectcode int, projectjoindate Timestamp, 
projectenddate Timestamp,attendance int,utilization int,salary int) STORED BY 
'org.apache.carbondata.format' tblproperties('sort_columns'='empno')")
+sql(s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE 
sorttable1 OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '\"')""")
+checkAnswer(sql("select empno from sorttable1"), sql("select empno from 
sorttable1 order by empno"))
+  }
+
+  test("create table with dictionary sort_columns") {
+sql("CREATE TABLE sorttable2 (empno int, empname String, designation 
String, doj Timestamp, workgroupcategory int, workgroupcategoryname String, 
deptno int, deptname String, projectcode int, projectjoindate Timestamp, 
projectenddate Timestamp,attendance int,utilization int,salary int) STORED BY 
'org.apache.carbondata.format' tblproperties('sort_columns'='empname')")
+sql(s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE 
sorttable2 OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '\"')""")
+checkAnswer(sql("select empname from sorttable2"),sql("select empname from 
origintable1"))
+  }
+
+  test("create table with direct-dictioanry sort_columns") {
+sql("CREATE TABLE sorttable3 (empno int, empname String, designation 
String, doj Timestamp, workgroupcategory int, workgroupcategoryname String, 
deptno int, deptname String, projectcode int, projectjoindate Timestamp, 
projectenddate Timestamp,attendance int,utilization int,salary int) STORED BY 
'org.apache.carbondata.format' tblproperties('sort_columns'='doj')")
+sql(s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE 
sorttable3 OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '\"')""")
+checkAnswer(sql("select doj from sorttable3"), sql("select doj from 
sorttable3 order by doj"))
+  }
+
+  test("create table with multi-sort_columns and data loading with offheap 
safe") {
+try {
+  setLoadingProperties("true", "false", "false")
+  sql("CREATE TABLE sorttable4_offheap_safe (empno int, empname String, 
designation String, doj Timestamp, workgroupcategory int, workgroupcategoryname 
String, deptno int, deptname String, projectcode int, projectjoindate 
Timestamp, projectenddate Timestamp,attendance int,utilization int,salary int) 
STORED BY 'org.apache.carbondata.format' 
tblproperties('sort_columns'='workgroupcategory, 

[4/4] incubator-carbondata git commit: [CARBONDATA-782]support SORT_COLUMNS This closes #757

[jackylk]

[CARBONDATA-782]support SORT_COLUMNS This closes #757


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/f60f6b62
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/f60f6b62
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/f60f6b62

Branch: refs/heads/12-dev
Commit: f60f6b62ce022519ed2fbfaf9de381f9c73f1ca1
Parents: 914d61e fcf7401
Author: jackylk 
Authored: Fri Apr 7 20:10:45 2017 +0800
Committer: jackylk 
Committed: Fri Apr 7 20:10:45 2017 +0800

--
 .../core/constants/CarbonCommonConstants.java   |   1 +
 .../core/datastore/block/SegmentProperties.java |  42 +++
 .../impl/VariableLengthDimensionDataChunk.java  |  58 +++-
 .../impl/btree/BTreeDataRefNodeFinder.java  |  24 +-
 .../core/keygenerator/mdkey/Bits.java   |   6 +-
 .../ThriftWrapperSchemaConverterImpl.java   |  18 +
 .../core/metadata/schema/table/CarbonTable.java |  24 ++
 .../schema/table/column/ColumnSchema.java   |  10 +
 .../impl/DictionaryBasedResultCollector.java|  12 +-
 ...structureBasedDictionaryResultCollector.java |   4 +-
 .../scan/filter/FilterExpressionProcessor.java  |   4 +-
 .../carbondata/core/scan/filter/FilterUtil.java |  98 --
 ...velRangeLessThanEqualFilterExecuterImpl.java |   2 +-
 .../RowLevelRangeLessThanFiterExecuterImpl.java |   2 +-
 .../visitor/NoDictionaryTypeVisitor.java|   3 +-
 .../visitor/RangeNoDictionaryTypeVisitor.java   |   3 +-
 .../AbstractDetailQueryResultIterator.java  |   4 +-
 .../scan/result/vector/CarbonColumnVector.java  |   7 +
 .../util/AbstractDataFileFooterConverter.java   |   7 +
 .../apache/carbondata/core/util/ByteUtil.java   | 336 ++-
 .../apache/carbondata/core/util/CarbonUtil.java |  18 +
 .../carbondata/core/util/DataTypeUtil.java  |  70 
 .../impl/btree/BTreeBlockFinderTest.java|  10 +-
 .../core/scan/filter/FilterUtilTest.java|   7 +-
 .../carbondata/hadoop/CarbonInputFormat.java|   4 +-
 .../internal/index/impl/InMemoryBTreeIndex.java |   4 +-
 .../impl/DictionaryDecodeReadSupport.java   |   3 +
 .../hadoop/util/BlockLevelTraverser.java|   4 +-
 .../hadoop/test/util/StoreCreator.java  |   5 +
 .../testsuite/sortcolumns/TestSortColumns.scala | 267 +++
 .../spark/sql/catalyst/CarbonDDLSqlParser.scala | 125 +++
 .../execution/command/carbonTableSchema.scala   |  66 ++--
 .../apache/spark/sql/TestCarbonSqlParser.scala  |  56 +---
 .../readsupport/SparkRowReadSupportImpl.java|  13 +-
 .../vectorreader/ColumnarVectorWrapper.java |  13 +
 .../org/apache/spark/sql/TableCreator.scala | 145 
 .../merger/CompactionResultSortProcessor.java   |  10 +-
 .../merger/RowResultMergerProcessor.java|  14 +-
 .../newflow/CarbonDataLoadConfiguration.java|  24 ++
 .../newflow/DataLoadProcessBuilder.java |  24 +-
 .../impl/NonDictionaryFieldConverterImpl.java   |  21 +-
 .../sort/impl/ParallelReadMergeSorterImpl.java  |   3 +-
 ...arallelReadMergeSorterWithBucketingImpl.java |   3 +-
 .../sort/unsafe/UnsafeCarbonRowPage.java|  11 +-
 .../newflow/sort/unsafe/UnsafeSortDataRows.java |  11 +-
 .../unsafe/comparator/UnsafeRowComparator.java  |  10 +-
 .../UnsafeRowComparatorForNormalDIms.java   |   8 +-
 .../holder/UnsafeFinalMergePageHolder.java  |   4 +-
 .../unsafe/holder/UnsafeInmemoryHolder.java |   5 +-
 .../holder/UnsafeSortTempFileChunkHolder.java   |   2 +-
 .../UnsafeSingleThreadFinalSortFilesMerger.java |   4 +-
 .../newflow/steps/NoSortProcessorStepImpl.java  | 153 +
 .../sortdata/IntermediateFileMerger.java|   3 +-
 .../sortdata/NewRowComparator.java  |  12 +-
 .../sortdata/NewRowComparatorForNormalDims.java |  10 +-
 .../sortandgroupby/sortdata/RowComparator.java  |  10 +-
 .../sortdata/RowComparatorForNormalDims.java|  10 +-
 .../sortandgroupby/sortdata/SortDataRows.java   |  13 +-
 .../sortandgroupby/sortdata/SortParameters.java |  43 +++
 .../sortdata/SortTempFileChunkHolder.java   |  11 +-
 .../store/CarbonFactDataHandlerColumnar.java|  45 ++-
 .../store/CarbonFactDataHandlerModel.java   |   7 +
 .../store/SingleThreadFinalSortFilesMerger.java |   8 +-
 63 files changed, 1601 insertions(+), 353 deletions(-)
--

[1/4] incubator-carbondata git commit: sort columns

[jackylk]

Repository: incubator-carbondata
Updated Branches:
  refs/heads/12-dev 914d61e32 -> f60f6b62c


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/fcf74017/processing/src/main/java/org/apache/carbondata/processing/newflow/steps/NoSortProcessorStepImpl.java
--
diff --git 
a/processing/src/main/java/org/apache/carbondata/processing/newflow/steps/NoSortProcessorStepImpl.java
 
b/processing/src/main/java/org/apache/carbondata/processing/newflow/steps/NoSortProcessorStepImpl.java
new file mode 100644
index 000..bde89ed
--- /dev/null
+++ 
b/processing/src/main/java/org/apache/carbondata/processing/newflow/steps/NoSortProcessorStepImpl.java
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.processing.newflow.steps;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.util.DataTypeUtil;
+import org.apache.carbondata.processing.newflow.AbstractDataLoadProcessorStep;
+import org.apache.carbondata.processing.newflow.CarbonDataLoadConfiguration;
+import org.apache.carbondata.processing.newflow.DataField;
+import org.apache.carbondata.processing.newflow.row.CarbonRow;
+import org.apache.carbondata.processing.util.CarbonDataProcessorUtil;
+import org.apache.carbondata.processing.util.NonDictionaryUtil;
+
+/**
+ * if the table doesn't have sort_columns, just convert row format.
+ */
+public class NoSortProcessorStepImpl extends AbstractDataLoadProcessorStep {
+
+  private int dimensionCount;
+
+  private int dimensionWithComplexCount;
+
+  private int noDictCount;
+
+  private int measureCount;
+
+  private boolean[] isNoDictionaryDimensionColumn;
+
+  private char[] aggType;
+
+  public NoSortProcessorStepImpl(CarbonDataLoadConfiguration configuration,
+  AbstractDataLoadProcessorStep child) {
+super(configuration, child);
+this.dimensionWithComplexCount = configuration.getDimensionCount();
+this.noDictCount =
+configuration.getNoDictionaryCount() + 
configuration.getComplexDimensionCount();
+this.dimensionCount = configuration.getDimensionCount() - this.noDictCount;
+this.measureCount = configuration.getMeasureCount();
+this.isNoDictionaryDimensionColumn =
+
CarbonDataProcessorUtil.getNoDictionaryMapping(configuration.getDataFields());
+this.aggType = CarbonDataProcessorUtil
+.getAggType(configuration.getMeasureCount(), 
configuration.getMeasureFields());
+  }
+
+  @Override public DataField[] getOutput() {
+return child.getOutput();
+  }
+
+  @Override public void initialize() throws IOException {
+child.initialize();
+  }
+
+  /**
+   * convert input CarbonRow to output CarbonRow
+   * e.g. There is a table as following,
+   * the number of dictionary dimensions is a,
+   * the number of no-dictionary dimensions is b,
+   * the number of complex dimensions is c,
+   * the number of measures is d.
+   * input CarbonRow format:  the length of Object[] data is a+b+c+d, the 
number of all columns.
+   * 

+   * | Part | Object item| describe
 |
+   * 

+   * | Object[0 ~ a+b-1]| Integer, byte[], Integer, ...  | dict + no 
dict dimensions|
+   * 

+   * | Object[a+b ~ a+b+c-1]| byte[], byte[], ...| complex 
dimensions   |
+   * 

+   * | Object[a+b+c ~ a+b+c+d-1]| int, byte[], ...   | measures
 |
+   * 

+   * output CarbonRow format: the length of object[] data is 3.
+   * 

+   * | Part | Object item   

[30/49] incubator-carbondata git commit: Fixed order by limit with select * query

[jackylk]

Fixed order by limit with select * query


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/82b61d47
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/82b61d47
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/82b61d47

Branch: refs/heads/12-dev
Commit: 82b61d4799a0e2eea6064d4e997fa1524c7f7b1d
Parents: 68cbe15
Author: ravipesala 
Authored: Tue Apr 4 15:18:46 2017 +0530
Committer: kumarvishal 
Committed: Thu Apr 6 16:28:21 2017 +0530

--
 .../sortexpr/AllDataTypesTestCaseSort.scala | 18 +--
 .../spark/sql/CarbonCatalystOperators.scala | 19 +--
 .../spark/sql/CarbonDictionaryDecoder.scala | 33 +++-
 .../execution/CarbonLateDecodeStrategy.scala| 14 ++---
 .../sql/optimizer/CarbonLateDecodeRule.scala| 27 ++--
 5 files changed, 98 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/82b61d47/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortexpr/AllDataTypesTestCaseSort.scala
--
diff --git 
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortexpr/AllDataTypesTestCaseSort.scala
 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortexpr/AllDataTypesTestCaseSort.scala
index bdb470a..34d3cee 100644
--- 
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortexpr/AllDataTypesTestCaseSort.scala
+++ 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortexpr/AllDataTypesTestCaseSort.scala
@@ -27,6 +27,8 @@ import org.scalatest.BeforeAndAfterAll
 class AllDataTypesTestCaseSort extends QueryTest with BeforeAndAfterAll {
 
   override def beforeAll {
+sql("drop table if exists alldatatypestablesort")
+sql("drop table if exists alldatatypestablesort_hive")
 sql("CREATE TABLE alldatatypestablesort (empno int, empname String, 
designation String, doj Timestamp, workgroupcategory int, workgroupcategoryname 
String, deptno int, deptname String, projectcode int, projectjoindate 
Timestamp, projectenddate Timestamp,attendance int,utilization int,salary int) 
STORED BY 'org.apache.carbondata.format'")
 sql(s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE 
alldatatypestablesort OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '\"')""");
 
@@ -41,8 +43,20 @@ class AllDataTypesTestCaseSort extends QueryTest with 
BeforeAndAfterAll {
   sql("select empno,empname,utilization,count(salary),sum(empno) from 
alldatatypestablesort_hive where empname in ('arvind','ayushi') group by 
empno,empname,utilization order by empno"))
   }
 
+  test("select * from alldatatypestablesort order by empname limit 10") {
+sql("select * from alldatatypestablesort order by empname limit 
10").collect()
+  }
+
+  test("select * from alldatatypestablesort order by salary limit 2") {
+sql("select * from alldatatypestablesort order by salary limit 
2").collect()
+  }
+
+  test("select * from alldatatypestablesort where empname='arvind' order by 
salary limit 2") {
+sql("select * from alldatatypestablesort where empname='arvind' order by 
salary limit 2").collect()
+  }
+
   override def afterAll {
-sql("drop table alldatatypestablesort")
-sql("drop table alldatatypestablesort_hive")
+sql("drop table if exists alldatatypestablesort")
+sql("drop table if exists alldatatypestablesort_hive")
   }
 }
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/82b61d47/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonCatalystOperators.scala
--
diff --git 
a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonCatalystOperators.scala
 
b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonCatalystOperators.scala
index 4070088..9b1533e 100644
--- 
a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonCatalystOperators.scala
+++ 
b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonCatalystOperators.scala
@@ -20,6 +20,7 @@ package org.apache.spark.sql
 import org.apache.spark.sql.catalyst.TableIdentifier
 import org.apache.spark.sql.catalyst.expressions._
 import org.apache.spark.sql.catalyst.plans.logical.{UnaryNode, _}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
 import org.apache.spark.sql.hive.{HiveContext, HiveSessionCatalog}
 import org.apache.spark.sql.optimizer.CarbonDecoderRelation
 import org.apache.spark.sql.types.{StringType, TimestampType}
@@ -33,8 

[49/49] incubator-carbondata git commit: [CARBONDATA-856] Alter Table - TABLE RENAME feature documentation This closes #756

[jackylk]

[CARBONDATA-856] Alter Table - TABLE RENAME feature documentation This closes 
#756


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/914d61e3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/914d61e3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/914d61e3

Branch: refs/heads/12-dev
Commit: 914d61e32f6bcb3b61e9da1957c95bfcee142fb4
Parents: cfefb74 b2950b4
Author: chenliang613 
Authored: Fri Apr 7 15:06:05 2017 +0530
Committer: chenliang613 
Committed: Fri Apr 7 15:06:05 2017 +0530

--
 docs/ddl-operation-on-carbondata.md | 29 +
 1 file changed, 29 insertions(+)
--

[43/49] incubator-carbondata git commit: fixed ArrayIndexOutOfBoundException for ColumnarVectorWrapper

[jackylk]

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/7e2e0f3d/integration/spark-common-test/src/test/resources/restructure/data_2000.csv
--
diff --git 
a/integration/spark-common-test/src/test/resources/restructure/data_2000.csv 
b/integration/spark-common-test/src/test/resources/restructure/data_2000.csv
new file mode 100644
index 000..9db3ee4
--- /dev/null
+++ b/integration/spark-common-test/src/test/resources/restructure/data_2000.csv
@@ -0,0 +1,2013 @@
+,,,
+8999,,,
+,CUST_NAME_0,,
+,,ACTIVE_EMUI_VERSION_0,
+,,,1970-01-01 11:00:03
+1970-01-01 12:00:03,,,
+,1233720368578,,
+,,-223372036854,
+,,,12345678901.1234
+22345678901.1234,,,
+,11234567489.7976,,
+,,-11234567489.7976,
+,,,0
+9000,CUST_NAME_0,ACTIVE_EMUI_VERSION_0,1970-01-01 10:00:03,1970-01-01 
11:00:03,123372036854,-223372036854,12345678901.1234,22345678901.1234,11234567489.7976,-11234567489.7976,1
+9001,CUST_NAME_1,ACTIVE_EMUI_VERSION_1,1970-01-02 01:00:03,1970-01-02 
02:00:03,123372036855,-223372036853,12345678902.1234,22345678902.1234,11234567489.7976,-11234567489.7976,2
+9002,CUST_NAME_2,ACTIVE_EMUI_VERSION_2,1970-01-03 01:00:03,1970-01-03 
02:00:03,123372036856,-223372036852,12345678903.1234,22345678903.1234,11234567489.7976,-11234567489.7976,3
+9003,CUST_NAME_3,ACTIVE_EMUI_VERSION_3,1970-01-04 01:00:03,1970-01-04 
02:00:03,123372036857,-223372036851,12345678904.1234,22345678904.1234,11234567489.7976,-11234567489.7976,4
+9004,CUST_NAME_4,ACTIVE_EMUI_VERSION_4,1970-01-05 01:00:03,1970-01-05 
02:00:03,123372036858,-223372036850,12345678905.1234,22345678905.1234,11234567489.7976,-11234567489.7976,5
+9005,CUST_NAME_5,ACTIVE_EMUI_VERSION_5,1970-01-06 01:00:03,1970-01-06 
02:00:03,123372036859,-223372036849,12345678906.1234,22345678906.1234,11234567489.7976,-11234567489.7976,6
+9006,CUST_NAME_6,ACTIVE_EMUI_VERSION_6,1970-01-07 01:00:03,1970-01-07 
02:00:03,123372036860,-223372036848,12345678907.1234,22345678907.1234,11234567489.7976,-11234567489.7976,7
+9007,CUST_NAME_7,ACTIVE_EMUI_VERSION_7,1970-01-08 01:00:03,1970-01-08 
02:00:03,123372036861,-223372036847,12345678908.1234,22345678908.1234,11234567489.7976,-11234567489.7976,8
+9008,CUST_NAME_8,ACTIVE_EMUI_VERSION_8,1970-01-09 01:00:03,1970-01-09 
02:00:03,123372036862,-223372036846,12345678909.1234,22345678909.1234,11234567489.7976,-11234567489.7976,9
+9009,CUST_NAME_9,ACTIVE_EMUI_VERSION_9,1970-01-10 01:00:03,1970-01-10 
02:00:03,123372036863,-223372036845,12345678910.1234,22345678910.1234,11234567489.7976,-11234567489.7976,10
+9010,CUST_NAME_00010,ACTIVE_EMUI_VERSION_00010,1970-01-11 01:00:03,1970-01-11 
02:00:03,123372036864,-223372036844,12345678911.1234,22345678911.1234,11234567489.7976,-11234567489.7976,11
+9011,CUST_NAME_00011,ACTIVE_EMUI_VERSION_00011,1970-01-12 01:00:03,1970-01-12 
02:00:03,123372036865,-223372036843,12345678912.1234,22345678912.1234,11234567489.7976,-11234567489.7976,12
+9012,CUST_NAME_00012,ACTIVE_EMUI_VERSION_00012,1970-01-13 01:00:03,1970-01-13 
02:00:03,123372036866,-223372036842,12345678913.1234,22345678913.1234,11234567489.7976,-11234567489.7976,13
+9013,CUST_NAME_00013,ACTIVE_EMUI_VERSION_00013,1970-01-14 01:00:03,1970-01-14 
02:00:03,123372036867,-223372036841,12345678914.1234,22345678914.1234,11234567489.7976,-11234567489.7976,14
+9014,CUST_NAME_00014,ACTIVE_EMUI_VERSION_00014,1970-01-15 01:00:03,1970-01-15 
02:00:03,123372036868,-223372036840,12345678915.1234,22345678915.1234,11234567489.7976,-11234567489.7976,15
+9015,CUST_NAME_00015,ACTIVE_EMUI_VERSION_00015,1970-01-16 01:00:03,1970-01-16 
02:00:03,123372036869,-223372036839,12345678916.1234,22345678916.1234,11234567489.7976,-11234567489.7976,16
+9016,CUST_NAME_00016,ACTIVE_EMUI_VERSION_00016,1970-01-17 01:00:03,1970-01-17 
02:00:03,123372036870,-223372036838,12345678917.1234,22345678917.1234,11234567489.7976,-11234567489.7976,17
+9017,CUST_NAME_00017,ACTIVE_EMUI_VERSION_00017,1970-01-18 01:00:03,1970-01-18 
02:00:03,123372036871,-223372036837,12345678918.1234,22345678918.1234,11234567489.7976,-11234567489.7976,18
+9018,CUST_NAME_00018,ACTIVE_EMUI_VERSION_00018,1970-01-19 01:00:03,1970-01-19 
02:00:03,123372036872,-223372036836,12345678919.1234,22345678919.1234,11234567489.7976,-11234567489.7976,19
+9019,CUST_NAME_00019,ACTIVE_EMUI_VERSION_00019,1970-01-20 01:00:03,1970-01-20 
02:00:03,123372036873,-223372036835,12345678920.1234,22345678920.1234,11234567489.7976,-11234567489.7976,20
+9020,CUST_NAME_00020,ACTIVE_EMUI_VERSION_00020,1970-01-21 01:00:03,1970-01-21 
02:00:03,123372036874,-223372036834,12345678921.1234,22345678921.1234,11234567489.7976,-11234567489.7976,21
+9021,CUST_NAME_00021,ACTIVE_EMUI_VERSION_00021,1970-01-22 01:00:03,1970-01-22 

[25/49] incubator-carbondata git commit: Compaction lock should also be acquired during alter operation as alter and compaction on same table should not be allowed concurrently.

[jackylk]

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/cc59b247/processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
--
diff --git 
a/processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
 
b/processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
new file mode 100644
index 000..ee667c2
--- /dev/null
+++ 
b/processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
@@ -0,0 +1,1385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.processing.merger;
+
+import java.io.File;
+import java.io.IOException;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Calendar;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.filesystem.CarbonFile;
+import org.apache.carbondata.core.datastore.filesystem.CarbonFileFilter;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.locks.CarbonLockFactory;
+import org.apache.carbondata.core.locks.ICarbonLock;
+import org.apache.carbondata.core.locks.LockUsage;
+import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
+import org.apache.carbondata.core.metadata.CarbonTableIdentifier;
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
+import org.apache.carbondata.core.mutate.CarbonUpdateUtil;
+import org.apache.carbondata.core.mutate.DeleteDeltaBlockDetails;
+import org.apache.carbondata.core.mutate.SegmentUpdateDetails;
+import org.apache.carbondata.core.reader.CarbonDeleteFilesDataReader;
+import org.apache.carbondata.core.statusmanager.LoadMetadataDetails;
+import org.apache.carbondata.core.statusmanager.SegmentStatusManager;
+import org.apache.carbondata.core.statusmanager.SegmentUpdateStatusManager;
+import org.apache.carbondata.core.util.CarbonProperties;
+import org.apache.carbondata.core.util.path.CarbonStorePath;
+import org.apache.carbondata.core.util.path.CarbonTablePath;
+import org.apache.carbondata.core.writer.CarbonDeleteDeltaWriterImpl;
+import org.apache.carbondata.processing.model.CarbonLoadModel;
+
+/**
+ * utility class for load merging.
+ */
+public final class CarbonDataMergerUtil {
+  private static final LogService LOGGER =
+  LogServiceFactory.getLogService(CarbonDataMergerUtil.class.getName());
+
+  private CarbonDataMergerUtil() {
+
+  }
+
+  /**
+   * Returns the size of all the carbondata files present in the segment.
+   * @param carbonFile
+   * @return
+   */
+  private static long getSizeOfFactFileInLoad(CarbonFile carbonFile) {
+long factSize = 0;
+
+// carbon data file case.
+CarbonFile[] factFile = carbonFile.listFiles(new CarbonFileFilter() {
+
+  @Override public boolean accept(CarbonFile file) {
+return CarbonTablePath.isCarbonDataFile(file.getName());
+  }
+});
+
+for (CarbonFile fact : factFile) {
+  factSize += fact.getSize();
+}
+
+return factSize;
+  }
+
+  /**
+   * To check whether the merge property is enabled or not.
+   *
+   * @return
+   */
+
+  public static boolean checkIfAutoLoadMergingRequired() {
+// load merge is not supported as per new store format
+// moving the load merge check in early to avoid unnecessary load listing 
causing IOException
+// check whether carbons segment merging operation is enabled or not.
+// default will be false.
+String isLoadMergeEnabled = CarbonProperties.getInstance()
+.getProperty(CarbonCommonConstants.ENABLE_AUTO_LOAD_MERGE,
+CarbonCommonConstants.DEFAULT_ENABLE_AUTO_LOAD_MERGE);
+if (isLoadMergeEnabled.equalsIgnoreCase("false")) {
+  return false;
+}
+return true;
+  }
+
+  

[03/49] incubator-carbondata git commit: Changes done: 1. Clean up of folders created locally during data load and insert into operations. 2. Setting the load status properly for success, partial succ

[jackylk]

Changes done:
1. Clean up of folders created locally during data load and insert into 
operations.
2. Setting the load status properly for success, partial success and failure 
cases.
3. Printing load statistics in case of success and partial success.


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/487e41dd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/487e41dd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/487e41dd

Branch: refs/heads/12-dev
Commit: 487e41ddd13d9e19813cc1b9c4eda73376f1c8ba
Parents: ada081d
Author: manishgupta88 
Authored: Wed Apr 5 19:04:14 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 10:33:07 2017 +0530

--
 .../carbondata/spark/load/CarbonLoaderUtil.java | 30 
 .../spark/rdd/NewCarbonDataLoadRDD.scala| 30 
 2 files changed, 49 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/487e41dd/integration/spark-common/src/main/java/org/apache/carbondata/spark/load/CarbonLoaderUtil.java
--
diff --git 
a/integration/spark-common/src/main/java/org/apache/carbondata/spark/load/CarbonLoaderUtil.java
 
b/integration/spark-common/src/main/java/org/apache/carbondata/spark/load/CarbonLoaderUtil.java
index cc16398..95f0b10 100644
--- 
a/integration/spark-common/src/main/java/org/apache/carbondata/spark/load/CarbonLoaderUtil.java
+++ 
b/integration/spark-common/src/main/java/org/apache/carbondata/spark/load/CarbonLoaderUtil.java
@@ -36,6 +36,9 @@ import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
 
 import org.apache.carbondata.common.logging.LogService;
 import org.apache.carbondata.common.logging.LogServiceFactory;
@@ -171,13 +174,30 @@ public final class CarbonLoaderUtil {
   tempLocationKey = CarbonCommonConstants.COMPACTION_KEY_WORD + '_' + 
tempLocationKey;
 }
 // form local store location
-String localStoreLocation = CarbonProperties.getInstance()
+final String localStoreLocation = CarbonProperties.getInstance()
 .getProperty(tempLocationKey, 
CarbonCommonConstants.STORE_LOCATION_DEFAULT_VAL);
+// submit local folder clean up in another thread so that main thread 
execution is not blocked
+ExecutorService localFolderDeletionService = 
Executors.newFixedThreadPool(1);
 try {
-  CarbonUtil.deleteFoldersAndFiles(new 
File(localStoreLocation).getParentFile());
-  LOGGER.info("Deleted the local store location" + localStoreLocation);
-} catch (IOException | InterruptedException e) {
-  LOGGER.error(e, "Failed to delete local data load folder location");
+  localFolderDeletionService.submit(new Callable() {
+@Override public Void call() throws Exception {
+  try {
+long startTime = System.currentTimeMillis();
+File file = new File(localStoreLocation);
+CarbonUtil.deleteFoldersAndFiles(file.getParentFile());
+LOGGER.info(
+"Deleted the local store location" + localStoreLocation + " : 
TIme taken: " + (
+System.currentTimeMillis() - startTime));
+  } catch (IOException | InterruptedException e) {
+LOGGER.error(e, "Failed to delete local data load folder 
location");
+  }
+  return null;
+}
+  });
+} finally {
+  if (null != localFolderDeletionService) {
+localFolderDeletionService.shutdown();
+  }
 }
 
   }

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/487e41dd/integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
--
diff --git 
a/integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
 
b/integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
index 0690ba1..72ee90f 100644
--- 
a/integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
+++ 
b/integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
@@ -223,7 +223,7 @@ class NewCarbonDataLoadRDD[K, V](
 carbonLoadModel.getTableName + CarbonCommonConstants.UNDERSCORE + 
theSplit.index
   try {
 loadMetadataDetails.setPartitionCount(partitionID)
-
loadMetadataDetails.setLoadStatus(CarbonCommonConstants.STORE_LOADSTATUS_FAILURE)

[06/49] incubator-carbondata git commit: [CARBONDATA-863] Support creation and deletion of dictionary files through RDD during alter add and drop This closes #733

[jackylk]

[CARBONDATA-863] Support creation and deletion of dictionary files through RDD 
during alter add and drop This closes #733


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/65e67918
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/65e67918
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/65e67918

Branch: refs/heads/12-dev
Commit: 65e6791873c019370eed54d164b69db6941e034d
Parents: 4a7adfa b5ba4c6
Author: ravipesala 
Authored: Thu Apr 6 10:54:16 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 10:54:16 2017 +0530

--
 .../core/cache/dictionary/ManageDictionary.java |  85 +++---
 .../spark/rdd/AlterTableAddColumnRDD.scala  | 110 +++
 .../spark/rdd/AlterTableDropColumnRDD.scala |  96 
 .../execution/command/carbonTableSchema.scala   |  21 ++--
 .../execution/command/AlterTableCommands.scala  |  48 +---
 5 files changed, 286 insertions(+), 74 deletions(-)
--

[17/49] incubator-carbondata git commit: [CARBONDATA-814] bad record log file writing is not correct

[jackylk]

[CARBONDATA-814] bad record log file writing is not correct


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/3af2d650
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/3af2d650
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/3af2d650

Branch: refs/heads/12-dev
Commit: 3af2d650372b8e0e85b03133229c1b7c15e4dafc
Parents: 05b790a
Author: mohammadshahidkhan 
Authored: Fri Mar 24 10:33:02 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 14:56:37 2017 +0530

--
 .../core/scan/expression/RangeExpressionEvaluator.java   | 7 +++
 .../newflow/steps/DataConverterProcessorStepImpl.java| 8 
 .../processing/store/CarbonFactDataHandlerColumnar.java  | 5 -
 3 files changed, 15 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/3af2d650/core/src/main/java/org/apache/carbondata/core/scan/expression/RangeExpressionEvaluator.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/scan/expression/RangeExpressionEvaluator.java
 
b/core/src/main/java/org/apache/carbondata/core/scan/expression/RangeExpressionEvaluator.java
index 7cabf10..d857fd8 100644
--- 
a/core/src/main/java/org/apache/carbondata/core/scan/expression/RangeExpressionEvaluator.java
+++ 
b/core/src/main/java/org/apache/carbondata/core/scan/expression/RangeExpressionEvaluator.java
@@ -22,6 +22,8 @@ import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
 import org.apache.carbondata.core.constants.CarbonCommonConstants;
 import org.apache.carbondata.core.metadata.datatype.DataType;
 import 
org.apache.carbondata.core.scan.expression.conditional.GreaterThanEqualToExpression;
@@ -41,6 +43,8 @@ import static 
org.apache.carbondata.core.scan.filter.intf.ExpressionType.LESSTHA
 import static 
org.apache.carbondata.core.scan.filter.intf.ExpressionType.LESSTHAN_EQUALTO;
 
 public class RangeExpressionEvaluator {
+  private static final LogService LOG =
+  
LogServiceFactory.getLogService(RangeExpressionEvaluator.class.getName());
   private Expression expr;
   private Expression srcNode;
   private Expression srcParentNode;
@@ -160,6 +164,9 @@ public class RangeExpressionEvaluator {
 }
 
 if ((null != startMin) && (null != endMax)) {
+  LOG.info(
+  "GreaterThan and LessThan Filter Expression changed to Range 
Expression for column "
+  + colName);
   // the node can be converted to RANGE.
   Expression n1 = startMin.getCurrentExp();
   Expression n2 = endMax.getCurrentExp();

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/3af2d650/processing/src/main/java/org/apache/carbondata/processing/newflow/steps/DataConverterProcessorStepImpl.java
--
diff --git 
a/processing/src/main/java/org/apache/carbondata/processing/newflow/steps/DataConverterProcessorStepImpl.java
 
b/processing/src/main/java/org/apache/carbondata/processing/newflow/steps/DataConverterProcessorStepImpl.java
index cb6baf4..cc99469 100644
--- 
a/processing/src/main/java/org/apache/carbondata/processing/newflow/steps/DataConverterProcessorStepImpl.java
+++ 
b/processing/src/main/java/org/apache/carbondata/processing/newflow/steps/DataConverterProcessorStepImpl.java
@@ -37,6 +37,7 @@ import 
org.apache.carbondata.processing.newflow.converter.impl.RowConverterImpl;
 import org.apache.carbondata.processing.newflow.row.CarbonRow;
 import org.apache.carbondata.processing.newflow.row.CarbonRowBatch;
 import 
org.apache.carbondata.processing.surrogatekeysgenerator.csvbased.BadRecordsLogger;
+import org.apache.carbondata.processing.util.CarbonDataProcessorUtil;
 
 /**
  * Replace row data fields with dictionary values if column is configured 
dictionary encoded.
@@ -171,6 +172,13 @@ public class DataConverterProcessorStepImpl extends 
AbstractDataLoadProcessorSte
 if (!closed) {
   if (null != badRecordLogger) {
 badRecordLogger.closeStreams();
+// rename the bad record in progress to normal
+CarbonTableIdentifier identifier =
+configuration.getTableIdentifier().getCarbonTableIdentifier();
+CarbonDataProcessorUtil.renameBadRecordsFromInProgressToNormal(
+identifier.getDatabaseName() + File.separator + 
identifier.getTableName()
++ File.separator + configuration.getSegmentId() + 
File.separator + configuration
+.getTaskNo());
   

[13/49] incubator-carbondata git commit: create database DBNAME , here DBNAME should be case insensitive.

[jackylk]

create database DBNAME , here DBNAME should be case insensitive.

while creating the folder convert to lower case and create.


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/0f678331
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/0f678331
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/0f678331

Branch: refs/heads/12-dev
Commit: 0f67833121452c98e57ce9e263fd91bf1cb07557
Parents: 975a5f7
Author: ravikiran 
Authored: Thu Apr 6 11:06:08 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 14:06:23 2017 +0530

--
 .../src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/0f678331/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
--
diff --git 
a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
 
b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
index 3645f95..39d8798 100644
--- 
a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
+++ 
b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
@@ -657,7 +657,7 @@ class CarbonMetastore(conf: RuntimeConfig, val storePath: 
String) {
   }
 
   def createDatabaseDirectory(dbName: String) {
-val databasePath = storePath + File.separator + dbName
+val databasePath = storePath + File.separator + dbName.toLowerCase
 val fileType = FileFactory.getFileType(databasePath)
 FileFactory.mkdirs(databasePath, fileType)
   }

[29/49] incubator-carbondata git commit: [CARBONDATA-780] Alter table support for compaction through sort step. This closes #703

[jackylk]

[CARBONDATA-780] Alter table support for compaction through sort step. This 
closes #703


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/68cbe150
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/68cbe150
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/68cbe150

Branch: refs/heads/12-dev
Commit: 68cbe1508a4703eba3de1731e77b217e604c677d
Parents: 58cd933 cc59b24
Author: ravipesala 
Authored: Thu Apr 6 16:18:36 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 16:18:36 2017 +0530

--
 .../core/locks/CarbonLockFactory.java   |8 +
 .../DictionaryBasedVectorResultCollector.java   |   12 +-
 .../RestructureBasedRawResultCollector.java |2 -
 .../spark/compaction/CompactionCallable.java|   44 +
 .../carbondata/spark/load/CarbonLoaderUtil.java |   19 +-
 .../spark/merger/CarbonCompactionExecutor.java  |  211 ---
 .../spark/merger/CarbonCompactionUtil.java  |  354 -
 .../spark/merger/CarbonDataMergerUtil.java  | 1371 -
 .../merger/CarbonDataMergerUtilResult.java  |   33 -
 .../spark/merger/CompactionCallable.java|   44 -
 .../carbondata/spark/merger/CompactionType.java |   31 -
 .../spark/merger/NodeBlockRelation.java |   58 -
 .../spark/merger/NodeMultiBlockRelation.java|   57 -
 .../spark/merger/RowResultMerger.java   |  367 -
 .../carbondata/spark/merger/TableMeta.java  |   40 -
 .../spark/merger/TupleConversionAdapter.java|   67 -
 .../merger/exeception/SliceMergerException.java |   78 -
 .../spark/rdd/CarbonIUDMergerRDD.scala  |2 +-
 .../carbondata/spark/rdd/CarbonMergerRDD.scala  |   37 +-
 .../apache/carbondata/spark/rdd/Compactor.scala |3 +-
 .../spark/rdd/DataManagementFunc.scala  |3 +-
 .../execution/command/carbonTableSchema.scala   |2 +-
 .../spark/rdd/CarbonDataRDDFactory.scala|2 +-
 .../sql/CarbonDatasourceHadoopRelation.scala|2 +-
 .../spark/sql/CarbonDatasourceRelation.scala|2 +-
 .../sql/execution/command/IUDCommands.scala |   15 +-
 .../apache/spark/sql/hive/CarbonMetastore.scala |2 +-
 .../spark/rdd/CarbonDataRDDFactory.scala|2 +-
 .../sql/CarbonDatasourceHadoopRelation.scala|2 +-
 .../execution/command/AlterTableCommands.scala  |   82 +-
 .../apache/spark/sql/hive/CarbonMetastore.scala |2 +-
 .../org/apache/spark/util/AlterTableUtil.scala  |   82 +-
 .../org/apache/spark/util/Compaction.scala  |2 +-
 .../AlterTableValidationTestCase.scala  |2 +-
 .../merger/AbstractResultProcessor.java |   64 +
 .../merger/CarbonCompactionExecutor.java|  231 +++
 .../processing/merger/CarbonCompactionUtil.java |  383 +
 .../processing/merger/CarbonDataMergerUtil.java | 1385 ++
 .../merger/CarbonDataMergerUtilResult.java  |   33 +
 .../merger/CompactionResultSortProcessor.java   |  401 +
 .../processing/merger/CompactionType.java   |   31 +
 .../processing/merger/NodeBlockRelation.java|   58 +
 .../merger/NodeMultiBlockRelation.java  |   57 +
 .../merger/RowResultMergerProcessor.java|  233 +++
 .../carbondata/processing/merger/TableMeta.java |   40 +
 .../merger/TupleConversionAdapter.java  |   67 +
 .../merger/exeception/SliceMergerException.java |   78 +
 .../newflow/DataLoadProcessBuilder.java |4 +-
 .../steps/DataWriterProcessorStepImpl.java  |   37 +-
 .../sortandgroupby/sortdata/SortParameters.java |3 +-
 .../store/CarbonFactDataHandlerModel.java   |   72 +
 .../util/CarbonDataProcessorUtil.java   |   75 +
 52 files changed, 3427 insertions(+), 2865 deletions(-)
--

[10/49] incubator-carbondata git commit: [CARBONDATA-866] remove kettle from configuration parameter file This closes #743

[jackylk]

[CARBONDATA-866] remove kettle from configuration parameter file This closes 
#743


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/69edde1d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/69edde1d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/69edde1d

Branch: refs/heads/12-dev
Commit: 69edde1d0ffedbb62bce42134ae240478223ea6e
Parents: 81cebdb 0ef41ae
Author: chenliang613 
Authored: Thu Apr 6 12:29:11 2017 +0530
Committer: chenliang613 
Committed: Thu Apr 6 12:29:11 2017 +0530

--
 docs/configuration-parameters.md | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
--

[14/49] incubator-carbondata git commit: [CARBONDATA-875] in create database DDL , DBNAME should be case insensitive. This closes #742

[jackylk]

[CARBONDATA-875] in create database DDL , DBNAME should be case insensitive. 
This closes #742


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/bca8b88b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/bca8b88b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/bca8b88b

Branch: refs/heads/12-dev
Commit: bca8b88bba4a056655362ab77411b3a7e301eacf
Parents: 975a5f7 0f67833
Author: ravipesala 
Authored: Thu Apr 6 14:06:51 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 14:06:51 2017 +0530

--
 .../src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[07/49] incubator-carbondata git commit: Remove configurations for Kettle from installation-guide.md

[jackylk]

Remove configurations for Kettle from installation-guide.md


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/1fc38510
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/1fc38510
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/1fc38510

Branch: refs/heads/12-dev
Commit: 1fc38510c8ec78405c8040ea043e262637ca9ec1
Parents: 65e6791
Author: Manoj 
Authored: Wed Apr 5 17:42:50 2017 +0530
Committer: chenliang613 
Committed: Thu Apr 6 12:15:12 2017 +0530

--
 docs/installation-guide.md | 30 +-
 1 file changed, 9 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/1fc38510/docs/installation-guide.md
--
diff --git a/docs/installation-guide.md b/docs/installation-guide.md
index c5bf6df..2fb6766 100644
--- a/docs/installation-guide.md
+++ b/docs/installation-guide.md
@@ -50,29 +50,23 @@ followed by :
 
 4. Copy the `./conf/carbon.properties.template` file from CarbonData 
repository to `$SPARK_HOME/conf/` folder and rename the file to 
`carbon.properties`.
 
-5. Copy the `./processing/carbonplugins` folder from CarbonData repository to 
`$SPARK_HOME/carbonlib/` folder.
-
-**NOTE**: carbonplugins will contain .kettle folder.
-
-6. Repeat Step 2 to Step 5 in all the nodes of the cluster.
+5. Repeat Step 2 to Step 5 in all the nodes of the cluster.
 
-7. In Spark node[master], configure the properties mentioned in the following 
table in `$SPARK_HOME/conf/spark-defaults.conf` file.
+6. In Spark node[master], configure the properties mentioned in the following 
table in `$SPARK_HOME/conf/spark-defaults.conf` file.
 
| Property | Value | Description |

|-|-|-|
-   | carbon.kettle.home | `$SPARK_HOME/carbonlib/carbonplugins` | Path that 
will be used by CarbonData internally to create graph for loading the data |
| spark.driver.extraJavaOptions | 
`-Dcarbon.properties.filepath=$SPARK_HOME/conf/carbon.properties` | A string of 
extra JVM options to pass to the driver. For instance, GC settings or other 
logging. |
| spark.executor.extraJavaOptions | 
`-Dcarbon.properties.filepath=$SPARK_HOME/conf/carbon.properties` | A string of 
extra JVM options to pass to executors. For instance, GC settings or other 
logging. **NOTE**: You can enter multiple values separated by space. |
 
-8. Add the following properties in `$SPARK_HOME/conf/carbon.properties` file:
+7. Add the following properties in `$SPARK_HOME/conf/carbon.properties` file:
 
| Property | Required | Description 
   | Example
 | Remark  |

|--|--||-|-|
| carbon.storelocation | NO   | Location where data CarbonData will 
create the store and write the data in its own format. | 
hdfs://HOSTNAME:PORT/Opt/CarbonStore  | Propose to set HDFS directory |
-   | carbon.kettle.home   | YES  | Path that will be used by CarbonData 
internally to create graph for loading the data. | 
`$SPARK_HOME/carbonlib/carbonplugins` | |
 
 
-9. Verify the installation. For example:
+8. Verify the installation. For example:
 
```
./spark-shell --master spark://HOSTNAME:PORT --total-executor-cores 2
@@ -100,13 +94,9 @@ To get started with CarbonData : [Quick 
Start](quick-start-guide.md), [DDL Opera
 
 **NOTE**: Create the carbonlib folder if it does not exists inside 
`$SPARK_HOME` path.
 
-2. Copy the `./processing/carbonplugins` folder from CarbonData repository to 
`$SPARK_HOME/carbonlib/` folder.
-
-**NOTE**: carbonplugins will contain .kettle folder.
+2. Copy the `./conf/carbon.properties.template` file from CarbonData 
repository to `$SPARK_HOME/conf/` folder and rename the file to 
`carbon.properties`.
 
-3. Copy the `./conf/carbon.properties.template` file from CarbonData 
repository to `$SPARK_HOME/conf/` folder and rename the file to 
`carbon.properties`.
-
-4. Create `tar,gz` file of carbonlib folder and move it inside the carbonlib 
folder.
+3. Create `tar,gz` file of carbonlib folder and move it inside the carbonlib 
folder.
 
 ```
cd $SPARK_HOME
@@ -114,7 +104,7 @@ To get started with CarbonData : [Quick 

[31/49] incubator-carbondata git commit: [CARBONDATA-874] Fixed order by limit with select * query This closes #726

[jackylk]

[CARBONDATA-874] Fixed order by limit with select * query This closes #726


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/1b20c2df
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/1b20c2df
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/1b20c2df

Branch: refs/heads/12-dev
Commit: 1b20c2dfdc249916247cae765afe2851c0561448
Parents: 68cbe15 82b61d4
Author: kumarvishal 
Authored: Thu Apr 6 16:31:00 2017 +0530
Committer: kumarvishal 
Committed: Thu Apr 6 16:31:00 2017 +0530

--
 .../sortexpr/AllDataTypesTestCaseSort.scala | 18 +--
 .../spark/sql/CarbonCatalystOperators.scala | 19 +--
 .../spark/sql/CarbonDictionaryDecoder.scala | 33 +++-
 .../execution/CarbonLateDecodeStrategy.scala| 14 ++---
 .../sql/optimizer/CarbonLateDecodeRule.scala| 27 ++--
 5 files changed, 98 insertions(+), 13 deletions(-)
--

[32/49] incubator-carbondata git commit: changes in troubleshooting.md

[jackylk]

changes in troubleshooting.md


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/951c5074
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/951c5074
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/951c5074

Branch: refs/heads/12-dev
Commit: 951c50741947b300a2631f35e1511cfa7e1e8cf8
Parents: 1b20c2d
Author: vandana 
Authored: Thu Apr 6 13:12:05 2017 +0530
Committer: chenliang613 
Committed: Thu Apr 6 16:35:55 2017 +0530

--
 docs/troubleshooting.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/951c5074/docs/troubleshooting.md
--
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
index 9181d83..c9d2095 100644
--- a/docs/troubleshooting.md
+++ b/docs/troubleshooting.md
@@ -226,7 +226,7 @@ who are building, deploying, and using CarbonData.
 
   **Procedure**
 
-  Worker must wait for the query execution to complete and the table to 
release the lock for another query execution to succeed..
+  Worker must wait for the query execution to complete and the table to 
release the lock for another query execution to succeed.
 
 ## Failed to create a table with a single numeric column.
 
@@ -240,7 +240,7 @@ who are building, deploying, and using CarbonData.
 
   **Possible Cause**
 
-  Behavior not supported.
+  Behaviour not supported.
 
   **Procedure**
 

[41/49] incubator-carbondata git commit: [CARBONDATA-846] Added support to revert changes if query fails. This closes #727

[jackylk]

[CARBONDATA-846] Added support to revert changes if query fails. This closes 
#727


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/6f12fbb3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/6f12fbb3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/6f12fbb3

Branch: refs/heads/12-dev
Commit: 6f12fbb313c330cf5a5c4c3da0a0598f8b3e7ebf
Parents: bbade2a 2a4f09b
Author: ravipesala 
Authored: Thu Apr 6 17:59:13 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 17:59:13 2017 +0530

--
 .../ThriftWrapperSchemaConverterImpl.java   |   4 +-
 .../carbondata/core/util/DataTypeUtil.java  |  16 +-
 .../execution/command/carbonTableSchema.scala   |   2 +-
 .../execution/command/AlterTableCommands.scala  |  48 +++---
 .../apache/spark/sql/hive/CarbonMetastore.scala |  30 +++-
 .../org/apache/spark/util/AlterTableUtil.scala  | 163 ++-
 .../restructure/AlterTableRevertTestCase.scala  |  69 
 .../spark/sql/common/util/QueryTest.scala   |   3 +
 8 files changed, 303 insertions(+), 32 deletions(-)
--

[15/49] incubator-carbondata git commit: if no locktype is configured and store type is HDFS set HDFS lock as default

[jackylk]

if no locktype is configured and store type is HDFS set HDFS lock as default


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/41bfd4a5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/41bfd4a5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/41bfd4a5

Branch: refs/heads/12-dev
Commit: 41bfd4a571c1e63c82c927ee3d487b532a982600
Parents: bca8b88
Author: Manohar 
Authored: Wed Apr 5 20:03:30 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 14:09:57 2017 +0530

--
 .../apache/spark/sql/hive/CarbonMetastore.scala | 28 ++--
 .../apache/spark/sql/hive/CarbonMetastore.scala | 28 ++--
 2 files changed, 28 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/41bfd4a5/integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
--
diff --git 
a/integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
 
b/integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
index 9fcd185..d5cc540 100644
--- 
a/integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
+++ 
b/integration/spark/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
@@ -161,24 +161,24 @@ class CarbonMetastore(hiveContext: HiveContext, val 
storePath: String,
 val statistic = new QueryStatistic()
 // creating zookeeper instance once.
 // if zookeeper is configured as carbon lock type.
-val zookeeperUrl: String = 
hiveContext.getConf(CarbonCommonConstants.ZOOKEEPER_URL, null)
-if (zookeeperUrl != null) {
-  
CarbonProperties.getInstance.addProperty(CarbonCommonConstants.ZOOKEEPER_URL, 
zookeeperUrl)
-  ZookeeperInit.getInstance(zookeeperUrl)
-  LOGGER.info("Zookeeper url is configured. Taking the zookeeper as lock 
type.")
-  var configuredLockType = CarbonProperties.getInstance
-.getProperty(CarbonCommonConstants.LOCK_TYPE)
-  if (null == configuredLockType) {
-configuredLockType = CarbonCommonConstants.CARBON_LOCK_TYPE_ZOOKEEPER
-CarbonProperties.getInstance
-  .addProperty(CarbonCommonConstants.LOCK_TYPE,
-configuredLockType)
-  }
+val zookeeperurl = 
hiveContext.getConf(CarbonCommonConstants.ZOOKEEPER_URL, null)
+if (null != zookeeperurl) {
+  CarbonProperties.getInstance
+.addProperty(CarbonCommonConstants.ZOOKEEPER_URL, zookeeperurl)
 }
-
 if (metadataPath == null) {
   return null
 }
+// if no locktype is configured and store type is HDFS set HDFS lock as 
default
+if (null == CarbonProperties.getInstance
+  .getProperty(CarbonCommonConstants.LOCK_TYPE) &&
+FileType.HDFS == FileFactory.getFileType(metadataPath)) {
+  CarbonProperties.getInstance
+.addProperty(CarbonCommonConstants.LOCK_TYPE,
+  CarbonCommonConstants.CARBON_LOCK_TYPE_HDFS
+)
+  LOGGER.info("Default lock type HDFSLOCK is configured")
+}
 val fileType = FileFactory.getFileType(metadataPath)
 val metaDataBuffer = new ArrayBuffer[TableMeta]
 fillMetaData(metadataPath, fileType, metaDataBuffer)

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/41bfd4a5/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
--
diff --git 
a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
 
b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
index 39d8798..f7ea344 100644
--- 
a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
+++ 
b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetastore.scala
@@ -200,24 +200,24 @@ class CarbonMetastore(conf: RuntimeConfig, val storePath: 
String) {
 val statistic = new QueryStatistic()
 // creating zookeeper instance once.
 // if zookeeper is configured as carbon lock type.
-val zookeeperUrl: String = conf.get(CarbonCommonConstants.ZOOKEEPER_URL, 
null)
-if (zookeeperUrl != null) {
-  
CarbonProperties.getInstance.addProperty(CarbonCommonConstants.ZOOKEEPER_URL, 
zookeeperUrl)
-  ZookeeperInit.getInstance(zookeeperUrl)
-  LOGGER.info("Zookeeper url is configured. Taking the zookeeper as lock 
type.")
-  var configuredLockType = CarbonProperties.getInstance
-.getProperty(CarbonCommonConstants.LOCK_TYPE)
-  if (null == configuredLockType) {
-configuredLockType = CarbonCommonConstants.CARBON_LOCK_TYPE_ZOOKEEPER
-

[48/49] incubator-carbondata git commit: [CARBONDATA-856] Alter Table - TABLE RENAME feature documentation

[jackylk]

[CARBONDATA-856] Alter Table - TABLE RENAME feature documentation

The documentation for Table rename as part of the alter table feature.

Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/b2950b44
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/b2950b44
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/b2950b44

Branch: refs/heads/12-dev
Commit: b2950b44482bfdf51e3ae4879f081a3bb9f93e28
Parents: cfefb74
Author: srigopalmohanty 
Authored: Fri Apr 7 13:10:36 2017 +0530
Committer: chenliang613 
Committed: Fri Apr 7 15:02:50 2017 +0530

--
 docs/ddl-operation-on-carbondata.md | 29 +
 1 file changed, 29 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/b2950b44/docs/ddl-operation-on-carbondata.md
--
diff --git a/docs/ddl-operation-on-carbondata.md 
b/docs/ddl-operation-on-carbondata.md
index 215cff8..392bb39 100644
--- a/docs/ddl-operation-on-carbondata.md
+++ b/docs/ddl-operation-on-carbondata.md
@@ -28,6 +28,7 @@ The following DDL operations are supported in CarbonData :
 * [DROP TABLE](#drop-table)
 * [COMPACTION](#compaction)
 * [BUCKETING](#bucketing)
+* [TABLE RENAME](#table-rename)
 
 ## CREATE TABLE
   This command can be used to create a CarbonData table by specifying the list 
of fields along with the table properties.
@@ -229,3 +230,31 @@ of columns is used.
   'BUCKETCOLUMNS'='productName')
  ```
 
+## TABLE RENAME
+  This command is used to rename the existing table.
+
+### Syntax
+```
+   ALTER TABLE [db_name.]table_name RENAME TO new_table_name;
+```
+
+### Parameter Description
+
+| Parameter | Description | 
+|---|---|
+| db_name | Name of the database. If this parameter is left unspecified, the 
current database is selected. | 
+| table_name | Name of the existing table.|
+|new_table_name | New table name for the existing table. | 
+
+### Usage Guidelines
+Following conditions must be met for successful rename operation:
+* Queries running in parallel which requires the formation of path using the 
table name for reading carbon store files might fail during this operation.
+* Secondary index table rename is not permitted.
+
+### Example:
+```
+ALTER TABLE carbon RENAME TO carbondata;
+
+   ALTER TABLE test_db.carbon RENAME TO test_db.carbondata;
+```
+

[05/49] incubator-carbondata git commit: Changes done: 1. Support creation and deletion of dictionary files during alter add and drop columns through RDD to parallelize the task and increase the perfo

[jackylk]

Changes done:
1. Support creation and deletion of dictionary files during alter add and drop 
columns through RDD to parallelize the task and increase the performance
2. Support clean up of dictionary files in case any failure occurs during alter 
add columns operation


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/b5ba4c6e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/b5ba4c6e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/b5ba4c6e

Branch: refs/heads/12-dev
Commit: b5ba4c6ea2d864f099bd4112e2cd5260e615a0a8
Parents: 4a7adfa
Author: manishgupta88 
Authored: Tue Apr 4 19:59:18 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 10:53:51 2017 +0530

--
 .../core/cache/dictionary/ManageDictionary.java |  85 +++---
 .../spark/rdd/AlterTableAddColumnRDD.scala  | 110 +++
 .../spark/rdd/AlterTableDropColumnRDD.scala |  96 
 .../execution/command/carbonTableSchema.scala   |  21 ++--
 .../execution/command/AlterTableCommands.scala  |  48 +---
 5 files changed, 286 insertions(+), 74 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/b5ba4c6e/core/src/main/java/org/apache/carbondata/core/cache/dictionary/ManageDictionary.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/cache/dictionary/ManageDictionary.java
 
b/core/src/main/java/org/apache/carbondata/core/cache/dictionary/ManageDictionary.java
index 706bc20..0a38890 100644
--- 
a/core/src/main/java/org/apache/carbondata/core/cache/dictionary/ManageDictionary.java
+++ 
b/core/src/main/java/org/apache/carbondata/core/cache/dictionary/ManageDictionary.java
@@ -18,7 +18,6 @@
 package org.apache.carbondata.core.cache.dictionary;
 
 import java.io.IOException;
-import java.util.List;
 
 import org.apache.carbondata.common.logging.LogService;
 import org.apache.carbondata.common.logging.LogServiceFactory;
@@ -30,8 +29,7 @@ import 
org.apache.carbondata.core.datastore.filesystem.CarbonFileFilter;
 import org.apache.carbondata.core.datastore.impl.FileFactory;
 import org.apache.carbondata.core.metadata.CarbonTableIdentifier;
 import org.apache.carbondata.core.metadata.ColumnIdentifier;
-import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
-import org.apache.carbondata.core.metadata.schema.table.column.CarbonColumn;
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema;
 import org.apache.carbondata.core.util.path.CarbonStorePath;
 import org.apache.carbondata.core.util.path.CarbonTablePath;
 
@@ -51,62 +49,59 @@ public class ManageDictionary {
* This method will delete the dictionary files for the given column IDs and
* clear the dictionary cache
*
-   * @param dictionaryColumns
-   * @param carbonTable
+   * @param columnSchema
+   * @param carbonTableIdentifier
+   * @param storePath
*/
-  public static void deleteDictionaryFileAndCache(List 
dictionaryColumns,
-  CarbonTable carbonTable) {
-if (!dictionaryColumns.isEmpty()) {
-  CarbonTableIdentifier carbonTableIdentifier = 
carbonTable.getCarbonTableIdentifier();
-  CarbonTablePath carbonTablePath =
-  CarbonStorePath.getCarbonTablePath(carbonTable.getStorePath(), 
carbonTableIdentifier);
-  String metadataDirectoryPath = 
carbonTablePath.getMetadataDirectoryPath();
-  CarbonFile metadataDir = FileFactory
-  .getCarbonFile(metadataDirectoryPath, 
FileFactory.getFileType(metadataDirectoryPath));
-  for (final CarbonColumn column : dictionaryColumns) {
-// sort index file is created with dictionary size appended to it. So 
all the files
-// with a given column ID need to be listed
-CarbonFile[] listFiles = metadataDir.listFiles(new CarbonFileFilter() {
-  @Override public boolean accept(CarbonFile path) {
-if (path.getName().startsWith(column.getColumnId())) {
-  return true;
-}
-return false;
-  }
-});
-for (CarbonFile file : listFiles) {
-  // try catch is inside for loop because even if one deletion fails, 
other files
-  // still need to be deleted
-  try {
-FileFactory.deleteFile(file.getCanonicalPath(),
-FileFactory.getFileType(file.getCanonicalPath()));
-  } catch (IOException e) {
-LOGGER.error(
-"Failed to delete dictionary or sortIndex file for column " + 
column.getColName()
-+ "with column ID " + column.getColumnId());
-  }
+  public static void deleteDictionaryFileAndCache(final ColumnSchema 

[39/49] incubator-carbondata git commit: [CARBONDATA-862] Removed Use Kettle From Documentation This closes #731

[jackylk]

[CARBONDATA-862] Removed Use Kettle From Documentation This closes #731


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/bbade2aa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/bbade2aa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/bbade2aa

Branch: refs/heads/12-dev
Commit: bbade2aa855ada21777139c22088788bc8b397d1
Parents: 1624c5f 97d2789
Author: chenliang613 
Authored: Thu Apr 6 16:51:52 2017 +0530
Committer: chenliang613 
Committed: Thu Apr 6 16:51:52 2017 +0530

--
 docs/dml-operation-on-carbondata.md | 9 -
 1 file changed, 9 deletions(-)
--

[27/49] incubator-carbondata git commit: Compaction lock should also be acquired during alter operation as alter and compaction on same table should not be allowed concurrently.

[jackylk]

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/cc59b247/integration/spark-common/src/main/java/org/apache/carbondata/spark/merger/CarbonDataMergerUtil.java
--
diff --git 
a/integration/spark-common/src/main/java/org/apache/carbondata/spark/merger/CarbonDataMergerUtil.java
 
b/integration/spark-common/src/main/java/org/apache/carbondata/spark/merger/CarbonDataMergerUtil.java
deleted file mode 100644
index 14bed0a..000
--- 
a/integration/spark-common/src/main/java/org/apache/carbondata/spark/merger/CarbonDataMergerUtil.java
+++ /dev/null
@@ -1,1371 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.carbondata.spark.merger;
-
-import java.io.File;
-import java.io.IOException;
-import java.text.ParseException;
-import java.text.SimpleDateFormat;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Calendar;
-import java.util.Collections;
-import java.util.Comparator;
-import java.util.Date;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Set;
-
-import org.apache.carbondata.common.logging.LogService;
-import org.apache.carbondata.common.logging.LogServiceFactory;
-import org.apache.carbondata.core.constants.CarbonCommonConstants;
-import org.apache.carbondata.core.datastore.filesystem.CarbonFile;
-import org.apache.carbondata.core.datastore.filesystem.CarbonFileFilter;
-import org.apache.carbondata.core.datastore.impl.FileFactory;
-import org.apache.carbondata.core.locks.CarbonLockFactory;
-import org.apache.carbondata.core.locks.ICarbonLock;
-import org.apache.carbondata.core.locks.LockUsage;
-import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
-import org.apache.carbondata.core.metadata.CarbonTableIdentifier;
-import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
-import org.apache.carbondata.core.mutate.CarbonUpdateUtil;
-import org.apache.carbondata.core.mutate.DeleteDeltaBlockDetails;
-import org.apache.carbondata.core.mutate.SegmentUpdateDetails;
-import org.apache.carbondata.core.reader.CarbonDeleteFilesDataReader;
-import org.apache.carbondata.core.statusmanager.LoadMetadataDetails;
-import org.apache.carbondata.core.statusmanager.SegmentStatusManager;
-import org.apache.carbondata.core.statusmanager.SegmentUpdateStatusManager;
-import org.apache.carbondata.core.util.CarbonProperties;
-import org.apache.carbondata.core.util.path.CarbonStorePath;
-import org.apache.carbondata.core.util.path.CarbonTablePath;
-import org.apache.carbondata.core.writer.CarbonDeleteDeltaWriterImpl;
-import org.apache.carbondata.processing.model.CarbonLoadModel;
-import org.apache.carbondata.spark.load.CarbonLoaderUtil;
-
-/**
- * utility class for load merging.
- */
-public final class CarbonDataMergerUtil {
-  private static final LogService LOGGER =
-  LogServiceFactory.getLogService(CarbonDataMergerUtil.class.getName());
-
-  private CarbonDataMergerUtil() {
-
-  }
-
-  /**
-   * Returns the size of all the carbondata files present in the segment.
-   * @param carbonFile
-   * @return
-   */
-  private static long getSizeOfFactFileInLoad(CarbonFile carbonFile) {
-long factSize = 0;
-
-// carbon data file case.
-CarbonFile[] factFile = carbonFile.listFiles(new CarbonFileFilter() {
-
-  @Override public boolean accept(CarbonFile file) {
-return CarbonTablePath.isCarbonDataFile(file.getName());
-  }
-});
-
-for (CarbonFile fact : factFile) {
-  factSize += fact.getSize();
-}
-
-return factSize;
-  }
-
-  /**
-   * To check whether the merge property is enabled or not.
-   *
-   * @return
-   */
-
-  public static boolean checkIfAutoLoadMergingRequired() {
-// load merge is not supported as per new store format
-// moving the load merge check in early to avoid unnecessary load listing 
causing IOException
-// check whether carbons segment merging operation is enabled or not.
-// default will be false.
-String isLoadMergeEnabled = CarbonProperties.getInstance()
-.getProperty(CarbonCommonConstants.ENABLE_AUTO_LOAD_MERGE,
-CarbonCommonConstants.DEFAULT_ENABLE_AUTO_LOAD_MERGE);
-if 

[44/49] incubator-carbondata git commit: fixed ArrayIndexOutOfBoundException for ColumnarVectorWrapper

[jackylk]

fixed ArrayIndexOutOfBoundException for ColumnarVectorWrapper


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/7e2e0f3d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/7e2e0f3d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/7e2e0f3d

Branch: refs/heads/12-dev
Commit: 7e2e0f3dea340a82611c07fc4acd2d79d7f2c8a7
Parents: 6f12fbb
Author: kunal642 
Authored: Thu Apr 6 18:35:57 2017 +0530
Committer: kunal642 
Committed: Thu Apr 6 18:35:57 2017 +0530

--
 .../test/resources/restructure/data_2000.csv| 2013 ++
 .../vectorreader/ColumnarVectorWrapper.java |6 +-
 .../vectorreader/AddColumnTestCases.scala   |   21 +
 3 files changed, 2036 insertions(+), 4 deletions(-)
--

[19/49] incubator-carbondata git commit: fixed issue with meta lock not getting deleted for rename table

[jackylk]

fixed issue with meta lock not getting deleted for rename table


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/7ae1cd8f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/7ae1cd8f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/7ae1cd8f

Branch: refs/heads/12-dev
Commit: 7ae1cd8fe77a391cb5f5addec4e19fbff5b06820
Parents: 9554971
Author: kunal642 
Authored: Fri Mar 31 16:03:17 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 15:11:22 2017 +0530

--
 .../core/locks/AbstractCarbonLock.java  |  9 +
 .../carbondata/core/locks/HdfsFileLock.java | 36 +---
 .../carbondata/core/locks/ICarbonLock.java  |  8 +
 .../carbondata/core/locks/LocalFileLock.java|  5 ++-
 .../execution/command/AlterTableCommands.scala  | 15 ++--
 .../org/apache/spark/util/AlterTableUtil.scala  |  1 +
 .../AlterTableValidationTestCase.scala  |  8 +
 7 files changed, 60 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/7ae1cd8f/core/src/main/java/org/apache/carbondata/core/locks/AbstractCarbonLock.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/locks/AbstractCarbonLock.java 
b/core/src/main/java/org/apache/carbondata/core/locks/AbstractCarbonLock.java
index eeb04d5..e927a7e 100644
--- 
a/core/src/main/java/org/apache/carbondata/core/locks/AbstractCarbonLock.java
+++ 
b/core/src/main/java/org/apache/carbondata/core/locks/AbstractCarbonLock.java
@@ -18,6 +18,7 @@
 package org.apache.carbondata.core.locks;
 
 import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
 import org.apache.carbondata.core.util.CarbonProperties;
 
 /**
@@ -72,4 +73,12 @@ public abstract class AbstractCarbonLock implements 
ICarbonLock {
 
   }
 
+  public boolean releaseLockManually(String lockFile) {
+try {
+  return FileFactory.deleteFile(lockFile, 
FileFactory.getFileType(lockFile));
+} catch (Exception e) {
+  return false;
+}
+  }
+
 }

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/7ae1cd8f/core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java 
b/core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java
index aed73b3..985ced1 100644
--- a/core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java
+++ b/core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java
@@ -23,6 +23,7 @@ import java.io.IOException;
 import org.apache.carbondata.common.logging.LogService;
 import org.apache.carbondata.common.logging.LogServiceFactory;
 import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.filesystem.CarbonFile;
 import org.apache.carbondata.core.datastore.impl.FileFactory;
 import org.apache.carbondata.core.metadata.CarbonTableIdentifier;
 import org.apache.carbondata.core.util.CarbonProperties;
@@ -102,34 +103,31 @@ public class HdfsFileLock extends AbstractCarbonLock {
*/
   @Override
   public boolean unlock() {
+boolean status = false;
 if (null != dataOutputStream) {
   try {
 dataOutputStream.close();
+status = true;
   } catch (IOException e) {
-try {
-  if (!FileFactory.isFileExist(location, 
FileFactory.getFileType(location))) {
-return true;
-  }
-} catch (IOException e1) {
-  LOGGER.error("Exception in isFileExist of the lock file " + 
e1.getMessage());
-}
-LOGGER.error("Exception in unlocking of the lock file " + 
e.getMessage());
-return false;
+status = false;
   } finally {
-try {
-  if (FileFactory.isFileExist(location, 
FileFactory.getFileType(location))) {
-if (FileFactory.getCarbonFile(location, 
FileFactory.getFileType(location)).delete()) {
-  LOGGER.info("Deleted the lock file " + location);
-} else {
-  LOGGER.error("Not able to delete the lock file " + location);
-}
+CarbonFile carbonFile =
+FileFactory.getCarbonFile(location, 
FileFactory.getFileType(location));
+if (carbonFile.exists()) {
+  if (carbonFile.delete()) {
+LOGGER.info("Deleted the lock file " + location);
+  } else {
+LOGGER.error("Not able to delete the lock file " + location);
+status = false;
   }
-} 

[09/49] incubator-carbondata git commit: remove kettle from configuration file

[jackylk]

remove kettle from configuration file


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/0ef41ae2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/0ef41ae2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/0ef41ae2

Branch: refs/heads/12-dev
Commit: 0ef41ae2bc21f2b19f4b7d8efa34d32e4aca6b99
Parents: 81cebdb
Author: Vinod Rohilla 
Authored: Thu Apr 6 11:47:11 2017 +0530
Committer: chenliang613 
Committed: Thu Apr 6 12:28:32 2017 +0530

--
 docs/configuration-parameters.md | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/0ef41ae2/docs/configuration-parameters.md
--
diff --git a/docs/configuration-parameters.md b/docs/configuration-parameters.md
index 774734a..4f454e6 100644
--- a/docs/configuration-parameters.md
+++ b/docs/configuration-parameters.md
@@ -36,8 +36,7 @@ This section provides the details of all the configurations 
required for the Car
 | carbon.storelocation | /user/hive/warehouse/carbon.store | Location where 
CarbonData will create the store, and write the data in its own format. NOTE: 
Store location should be in HDFS. |
 | carbon.ddl.base.hdfs.url | hdfs://hacluster/opt/data | This property is used 
to configure the HDFS relative path, the path configured in 
carbon.ddl.base.hdfs.url will be appended to the HDFS path configured in 
fs.defaultFS. If this path is configured, then user need not pass the complete 
path while dataload. For example: If absolute path of the csv file is 
hdfs://10.18.101.155:54310/data/cnbc/2016/xyz.csv, the path 
"hdfs://10.18.101.155:54310" will come from property fs.defaultFS and user can 
configure the /data/cnbc/ as carbon.ddl.base.hdfs.url. Now while dataload user 
can specify the csv path as /2016/xyz.csv. |
 | carbon.badRecords.location | /opt/Carbon/Spark/badrecords | Path where the 
bad records are stored. |
-| carbon.kettle.home | $SPARK_HOME/carbonlib/carbonplugins | Configuration for 
loading the data with kettle. |
-| carbon.data.file.version | 2 | If this parameter value is set to 1, then 
CarbonData will support the data load which is in old format(0.x version). If 
the value is set to 2(1.x onwards version), then CarbonData will support the 
data load of new format only.|
+| carbon.data.file.version | 2 | If this parameter value is set to 1, then 
CarbonData will support the data load which is in old format(0.x version). If 
the value is set to 2(1.x onwards version), then CarbonData will support the 
data load of new format only.|
 
 ##  Performance Configuration
 This section provides the details of all the configurations required for 
CarbonData Performance Optimization.

[28/49] incubator-carbondata git commit: Compaction lock should also be acquired during alter operation as alter and compaction on same table should not be allowed concurrently.

[jackylk]

Compaction lock should also be acquired during alter operation as alter and 
compaction on same table should not be allowed concurrently.

Handling for compaction for restructure case. Handled to completely sort the 
data again if any restructured block is selected for compaction

Handled review comments


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/cc59b247
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/cc59b247
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/cc59b247

Branch: refs/heads/12-dev
Commit: cc59b247aff7a6fd5d164d83dfe01890cfcdb2e1
Parents: 0f0907a
Author: manishgupta88 
Authored: Wed Mar 15 19:24:05 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 16:17:24 2017 +0530

--
 .../core/locks/CarbonLockFactory.java   |8 +
 .../DictionaryBasedVectorResultCollector.java   |   12 +-
 .../RestructureBasedRawResultCollector.java |2 -
 .../spark/compaction/CompactionCallable.java|   44 +
 .../carbondata/spark/load/CarbonLoaderUtil.java |   19 +-
 .../spark/merger/CarbonCompactionExecutor.java  |  211 ---
 .../spark/merger/CarbonCompactionUtil.java  |  354 -
 .../spark/merger/CarbonDataMergerUtil.java  | 1371 -
 .../merger/CarbonDataMergerUtilResult.java  |   33 -
 .../spark/merger/CompactionCallable.java|   44 -
 .../merger/CompactionResultSortProcessor.java   |  563 ---
 .../carbondata/spark/merger/CompactionType.java |   31 -
 .../spark/merger/NodeBlockRelation.java |   58 -
 .../spark/merger/NodeMultiBlockRelation.java|   57 -
 .../spark/merger/RowResultMerger.java   |  367 -
 .../carbondata/spark/merger/TableMeta.java  |   40 -
 .../spark/merger/TupleConversionAdapter.java|   67 -
 .../merger/exeception/SliceMergerException.java |   78 -
 .../spark/rdd/CarbonIUDMergerRDD.scala  |2 +-
 .../carbondata/spark/rdd/CarbonMergerRDD.scala  |   37 +-
 .../apache/carbondata/spark/rdd/Compactor.scala |3 +-
 .../spark/rdd/DataManagementFunc.scala  |3 +-
 .../execution/command/carbonTableSchema.scala   |2 +-
 .../spark/rdd/CarbonDataRDDFactory.scala|2 +-
 .../sql/CarbonDatasourceHadoopRelation.scala|2 +-
 .../spark/sql/CarbonDatasourceRelation.scala|2 +-
 .../sql/execution/command/IUDCommands.scala |   15 +-
 .../apache/spark/sql/hive/CarbonMetastore.scala |2 +-
 .../spark/rdd/CarbonDataRDDFactory.scala|2 +-
 .../sql/CarbonDatasourceHadoopRelation.scala|2 +-
 .../execution/command/AlterTableCommands.scala  |   82 +-
 .../apache/spark/sql/hive/CarbonMetastore.scala |2 +-
 .../org/apache/spark/util/AlterTableUtil.scala  |   82 +-
 .../org/apache/spark/util/Compaction.scala  |2 +-
 .../AlterTableValidationTestCase.scala  |2 +-
 .../merger/AbstractResultProcessor.java |   64 +
 .../merger/CarbonCompactionExecutor.java|  231 +++
 .../processing/merger/CarbonCompactionUtil.java |  383 +
 .../processing/merger/CarbonDataMergerUtil.java | 1385 ++
 .../merger/CarbonDataMergerUtilResult.java  |   33 +
 .../merger/CompactionResultSortProcessor.java   |  401 +
 .../processing/merger/CompactionType.java   |   31 +
 .../processing/merger/NodeBlockRelation.java|   58 +
 .../merger/NodeMultiBlockRelation.java  |   57 +
 .../merger/RowResultMergerProcessor.java|  233 +++
 .../carbondata/processing/merger/TableMeta.java |   40 +
 .../merger/TupleConversionAdapter.java  |   67 +
 .../merger/exeception/SliceMergerException.java |   78 +
 .../newflow/DataLoadProcessBuilder.java |4 +-
 .../steps/DataWriterProcessorStepImpl.java  |   37 +-
 .../sortandgroupby/sortdata/SortParameters.java |3 +-
 .../store/CarbonFactDataHandlerModel.java   |   72 +
 .../util/CarbonDataProcessorUtil.java   |   75 +
 53 files changed, 3427 insertions(+), 3428 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/cc59b247/core/src/main/java/org/apache/carbondata/core/locks/CarbonLockFactory.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/locks/CarbonLockFactory.java 
b/core/src/main/java/org/apache/carbondata/core/locks/CarbonLockFactory.java
index 8cc494e..f9520ab 100644
--- a/core/src/main/java/org/apache/carbondata/core/locks/CarbonLockFactory.java
+++ b/core/src/main/java/org/apache/carbondata/core/locks/CarbonLockFactory.java
@@ -17,6 +17,8 @@
 
 package org.apache.carbondata.core.locks;
 
+import org.apache.carbondata.common.logging.LogService;
+import 

[04/49] incubator-carbondata git commit: [CARBONDATA-870] Folders and files not getting cleaned up created locally during data load operation. This closes #735

[jackylk]

[CARBONDATA-870] Folders and files not getting cleaned up created locally 
during data load operation. This closes #735


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/4a7adfa9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/4a7adfa9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/4a7adfa9

Branch: refs/heads/12-dev
Commit: 4a7adfa97ae62ea946a101f5ae9b2148e76f
Parents: ada081d 487e41d
Author: ravipesala 
Authored: Thu Apr 6 10:33:41 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 10:33:41 2017 +0530

--
 .../carbondata/spark/load/CarbonLoaderUtil.java | 30 
 .../spark/rdd/NewCarbonDataLoadRDD.scala| 30 
 2 files changed, 49 insertions(+), 11 deletions(-)
--

[42/49] incubator-carbondata git commit: fixed ArrayIndexOutOfBoundException for ColumnarVectorWrapper

[jackylk]

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/7e2e0f3d/integration/spark2/src/main/java/org/apache/carbondata/spark/vectorreader/ColumnarVectorWrapper.java
--
diff --git 
a/integration/spark2/src/main/java/org/apache/carbondata/spark/vectorreader/ColumnarVectorWrapper.java
 
b/integration/spark2/src/main/java/org/apache/carbondata/spark/vectorreader/ColumnarVectorWrapper.java
index 2a9c701..f94c0b2 100644
--- 
a/integration/spark2/src/main/java/org/apache/carbondata/spark/vectorreader/ColumnarVectorWrapper.java
+++ 
b/integration/spark2/src/main/java/org/apache/carbondata/spark/vectorreader/ColumnarVectorWrapper.java
@@ -60,8 +60,7 @@ class ColumnarVectorWrapper implements CarbonColumnVector {
 
   @Override public void putDecimals(int rowId, int count, Decimal value, int 
precision) {
 for (int i = 0; i < count; i++) {
-  rowId += i;
-  putDecimal(rowId, value, precision);
+  putDecimal(rowId++, value, precision);
 }
   }
 
@@ -79,8 +78,7 @@ class ColumnarVectorWrapper implements CarbonColumnVector {
 
   @Override public void putBytes(int rowId, int count, byte[] value) {
 for (int i = 0; i < count; i++) {
-  rowId += i;
-  putBytes(rowId, value);
+  putBytes(rowId++, value);
 }
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/7e2e0f3d/integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/vectorreader/AddColumnTestCases.scala
--
diff --git 
a/integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/vectorreader/AddColumnTestCases.scala
 
b/integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/vectorreader/AddColumnTestCases.scala
index 36c9d95..747af05 100644
--- 
a/integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/vectorreader/AddColumnTestCases.scala
+++ 
b/integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/vectorreader/AddColumnTestCases.scala
@@ -167,6 +167,27 @@ class AddColumnTestCases extends QueryTest with 
BeforeAndAfterAll {
 sql("drop table carbon_new")
   }
 
+  test("test to check if vector result collector is able to fetch large amount 
of data") {
+sql("DROP TABLE IF EXISTS carbon_new")
+sql(
+  """CREATE TABLE carbon_new (CUST_ID int,CUST_NAME 
String,ACTIVE_EMUI_VERSION string, DOB
+|timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 
bigint,DECIMAL_COLUMN1
+|decimal(30,10), DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, 
Double_COLUMN2
+|double,INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' 
TBLPROPERTIES
+|("TABLE_BLOCKSIZE"= "256 MB")""".stripMargin)
+sql("alter table carbon_new drop columns(CUST_NAME)")
+sql(s"LOAD DATA INPATH '$resourcesPath/restructure/data_2000.csv' into 
table " +
+"carbon_new OPTIONS('DELIMITER'=',' , 
'QUOTECHAR'='\"','BAD_RECORDS_ACTION'='FORCE'," +
+
"'FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1," +
+
"BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2," 
+
+"INTEGER_COLUMN1')")
+sql(
+  """alter table carbon_new add columns(CUST_NAME string) TBLPROPERTIES
+('DICTIONARY_EXCLUDE'='CUST_NAME', 
'DEFAULT.VALUE.CUST_NAME'='testuser')""")
+checkAnswer(sql("select distinct(CUST_NAME) from 
carbon_new"),Row("testuser"))
+  }
+
+
   override def afterAll {
 sql("DROP TABLE IF EXISTS addcolumntest")
 sql("drop table if exists hivetable")

[24/49] incubator-carbondata git commit: Compaction lock should also be acquired during alter operation as alter and compaction on same table should not be allowed concurrently.

[jackylk]

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/cc59b247/processing/src/main/java/org/apache/carbondata/processing/merger/TableMeta.java
--
diff --git 
a/processing/src/main/java/org/apache/carbondata/processing/merger/TableMeta.java
 
b/processing/src/main/java/org/apache/carbondata/processing/merger/TableMeta.java
new file mode 100644
index 000..b84d695
--- /dev/null
+++ 
b/processing/src/main/java/org/apache/carbondata/processing/merger/TableMeta.java
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.processing.merger;
+
+import java.io.Serializable;
+
+import org.apache.carbondata.core.metadata.CarbonTableIdentifier;
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
+
+public class TableMeta implements Serializable {
+
+  private static final long serialVersionUID = -174987469829431L;
+
+  public CarbonTableIdentifier carbonTableIdentifier;
+  public String storePath;
+  public CarbonTable carbonTable;
+
+  public TableMeta(CarbonTableIdentifier carbonTableIdentifier, String 
storePath,
+  CarbonTable carbonTable) {
+this.carbonTableIdentifier = carbonTableIdentifier;
+this.storePath = storePath;
+this.carbonTable = carbonTable;
+  }
+
+}

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/cc59b247/processing/src/main/java/org/apache/carbondata/processing/merger/TupleConversionAdapter.java
--
diff --git 
a/processing/src/main/java/org/apache/carbondata/processing/merger/TupleConversionAdapter.java
 
b/processing/src/main/java/org/apache/carbondata/processing/merger/TupleConversionAdapter.java
new file mode 100644
index 000..f508a87
--- /dev/null
+++ 
b/processing/src/main/java/org/apache/carbondata/processing/merger/TupleConversionAdapter.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.processing.merger;
+
+import org.apache.carbondata.core.datastore.block.SegmentProperties;
+import org.apache.carbondata.core.scan.wrappers.ByteArrayWrapper;
+
+/**
+ * This class will be used to convert the Result into the format used in data 
writer.
+ */
+class TupleConversionAdapter {
+
+  private final SegmentProperties segmentproperties;
+
+  private int noDictionaryPresentIndex;
+
+  private int measureCount;
+
+  private boolean isNoDictionaryPresent;
+
+  public TupleConversionAdapter(SegmentProperties segmentProperties) {
+this.measureCount = segmentProperties.getMeasures().size();
+this.isNoDictionaryPresent = 
segmentProperties.getNumberOfNoDictionaryDimension() > 0;
+if (isNoDictionaryPresent) {
+  noDictionaryPresentIndex++;
+}
+this.segmentproperties = segmentProperties;
+  }
+
+  /**
+   * Converting the raw result to the format understandable by the data writer.
+   * @param carbonTuple
+   * @return
+   */
+  public Object[] getObjectArray(Object[] carbonTuple) {
+Object[] row = new Object[measureCount + noDictionaryPresentIndex + 1];
+int index = 0;
+// put measures.
+
+for (int j = 1; j <= measureCount; j++) {
+  row[index++] = carbonTuple[j];
+}
+
+// put No dictionary byte []
+if (isNoDictionaryPresent) {
+  row[index++] = ((ByteArrayWrapper) carbonTuple[0]).getNoDictionaryKeys();
+}
+
+// put No Dictionary Dims
+ 

[40/49] incubator-carbondata git commit: added support to revert changes if query fails

[jackylk]

added support to revert changes if query fails


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/2a4f09b7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/2a4f09b7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/2a4f09b7

Branch: refs/heads/12-dev
Commit: 2a4f09b7ab125581c7caa2bf57513abc07ac3c7f
Parents: bbade2a
Author: kunal642 
Authored: Wed Mar 15 14:38:31 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 17:58:39 2017 +0530

--
 .../ThriftWrapperSchemaConverterImpl.java   |   4 +-
 .../carbondata/core/util/DataTypeUtil.java  |  16 +-
 .../execution/command/carbonTableSchema.scala   |   2 +-
 .../execution/command/AlterTableCommands.scala  |  48 +++---
 .../apache/spark/sql/hive/CarbonMetastore.scala |  30 +++-
 .../org/apache/spark/util/AlterTableUtil.scala  | 163 ++-
 .../restructure/AlterTableRevertTestCase.scala  |  69 
 .../spark/sql/common/util/QueryTest.scala   |   3 +
 8 files changed, 303 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/2a4f09b7/core/src/main/java/org/apache/carbondata/core/metadata/converter/ThriftWrapperSchemaConverterImpl.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/metadata/converter/ThriftWrapperSchemaConverterImpl.java
 
b/core/src/main/java/org/apache/carbondata/core/metadata/converter/ThriftWrapperSchemaConverterImpl.java
index 09ed368..974cc81 100644
--- 
a/core/src/main/java/org/apache/carbondata/core/metadata/converter/ThriftWrapperSchemaConverterImpl.java
+++ 
b/core/src/main/java/org/apache/carbondata/core/metadata/converter/ThriftWrapperSchemaConverterImpl.java
@@ -404,8 +404,10 @@ public class ThriftWrapperSchemaConverterImpl implements 
SchemaConverter {
   org.apache.carbondata.format.TableInfo externalTableInfo, String dbName, 
String tableName,
   String storePath) {
 TableInfo wrapperTableInfo = new TableInfo();
+List 
schemaEvolutionList =
+
externalTableInfo.getFact_table().getSchema_evolution().getSchema_evolution_history();
 wrapperTableInfo.setLastUpdatedTime(
-
externalTableInfo.getFact_table().getSchema_evolution().getSchema_evolution_history().get(0)
+schemaEvolutionList.get(schemaEvolutionList.size() - 1)
 .getTime_stamp());
 wrapperTableInfo.setDatabaseName(dbName);
 wrapperTableInfo.setTableUniqueName(dbName + "_" + tableName);

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/2a4f09b7/core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java 
b/core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java
index e437405..76df425 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java
@@ -573,10 +573,10 @@ public final class DataTypeUtil {
* Below method will be used to convert the data into byte[]
*
* @param data
-   * @param actualDataType actual data type
+   * @param ColumnSchema
* @return actual data in byte[]
*/
-  public static byte[] convertDataToBytesBasedOnDataType(String data, DataType 
actualDataType) {
+  public static byte[] convertDataToBytesBasedOnDataType(String data, 
ColumnSchema columnSchema) {
 if (null == data) {
   return null;
 } else if (CarbonCommonConstants.MEMBER_DEFAULT_VAL.equals(data)) {
@@ -585,7 +585,7 @@ public final class DataTypeUtil {
 }
 try {
   long parsedIntVal = 0;
-  switch (actualDataType) {
+  switch (columnSchema.getDataType()) {
 case INT:
   parsedIntVal = (long) Integer.parseInt(data);
   return String.valueOf(parsedIntVal)
@@ -602,13 +602,17 @@ public final class DataTypeUtil {
   
.getBytes(Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
 case DATE:
 case TIMESTAMP:
-  DirectDictionaryGenerator directDictionaryGenerator =
-  
DirectDictionaryKeyGeneratorFactory.getDirectDictionaryGenerator(actualDataType);
+  DirectDictionaryGenerator directDictionaryGenerator = 
DirectDictionaryKeyGeneratorFactory
+  .getDirectDictionaryGenerator(columnSchema.getDataType());
   int value = 
directDictionaryGenerator.generateDirectSurrogateKey(data);
   return String.valueOf(value)
   
.getBytes(Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
 case DECIMAL:
-  

[16/49] incubator-carbondata git commit: [CARBONDATA-871] If store type is HDFS then set HDFS lock as default. This closes #736

[jackylk]

[CARBONDATA-871] If store type is HDFS then set HDFS lock as default. This 
closes #736


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/05b790a3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/05b790a3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/05b790a3

Branch: refs/heads/12-dev
Commit: 05b790a3503942be9a80b80ec4859490321c4658
Parents: bca8b88 41bfd4a
Author: ravipesala 
Authored: Thu Apr 6 14:10:33 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 14:10:33 2017 +0530

--
 .../apache/spark/sql/hive/CarbonMetastore.scala | 28 ++--
 .../apache/spark/sql/hive/CarbonMetastore.scala | 28 ++--
 2 files changed, 28 insertions(+), 28 deletions(-)
--

[02/49] incubator-carbondata git commit: [CARBONDATA-845] Insert Select Into Same Table This closes #723

[jackylk]

[CARBONDATA-845] Insert Select Into Same Table This closes #723


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/ada081d8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/ada081d8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/ada081d8

Branch: refs/heads/12-dev
Commit: ada081d89be67ad28c31401acb71a84e3e97244a
Parents: 950a6d0 f527d3d
Author: ravipesala 
Authored: Thu Apr 6 10:18:50 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 10:18:50 2017 +0530

--
 .../InsertIntoCarbonTableTestCase.scala | 28 +
 .../spark/sql/hive/CarbonAnalysisRules.scala| 43 +++---
 .../spark/sql/CarbonCatalystOperators.scala | 19 +++
 .../sql/CarbonDatasourceHadoopRelation.scala| 13 +
 .../sql/execution/command/DDLStrategy.scala |  5 +-
 .../execution/command/carbonTableSchema.scala   |  3 +-
 .../spark/sql/hive/CarbonAnalysisRules.scala| 60 
 .../spark/sql/hive/CarbonSessionState.scala | 30 +-
 8 files changed, 163 insertions(+), 38 deletions(-)
--

[11/49] incubator-carbondata git commit: While executing drop table , invalidating the table from hive metastore by running the command externally

[jackylk]

While executing drop table , invalidating the table from hive metastore by 
running the command externally


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/df3dea90
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/df3dea90
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/df3dea90

Branch: refs/heads/12-dev
Commit: df3dea90b088e1de2725cbb5cef1c8f90209d9fe
Parents: 69edde1
Author: Manohar 
Authored: Wed Apr 5 20:58:48 2017 +0530
Committer: Manohar 
Committed: Thu Apr 6 12:49:46 2017 +0530

--
 .../carbondata/core/locks/HdfsFileLock.java | 28 
 .../spark/sql/hive/CarbonHiveMetadataUtil.scala |  3 ++-
 2 files changed, 19 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/df3dea90/core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java 
b/core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java
index b3d79e9..aed73b3 100644
--- a/core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java
+++ b/core/src/main/java/org/apache/carbondata/core/locks/HdfsFileLock.java
@@ -23,7 +23,6 @@ import java.io.IOException;
 import org.apache.carbondata.common.logging.LogService;
 import org.apache.carbondata.common.logging.LogServiceFactory;
 import org.apache.carbondata.core.constants.CarbonCommonConstants;
-import org.apache.carbondata.core.datastore.filesystem.CarbonFile;
 import org.apache.carbondata.core.datastore.impl.FileFactory;
 import org.apache.carbondata.core.metadata.CarbonTableIdentifier;
 import org.apache.carbondata.core.util.CarbonProperties;
@@ -107,19 +106,26 @@ public class HdfsFileLock extends AbstractCarbonLock {
   try {
 dataOutputStream.close();
   } catch (IOException e) {
+try {
+  if (!FileFactory.isFileExist(location, 
FileFactory.getFileType(location))) {
+return true;
+  }
+} catch (IOException e1) {
+  LOGGER.error("Exception in isFileExist of the lock file " + 
e1.getMessage());
+}
+LOGGER.error("Exception in unlocking of the lock file " + 
e.getMessage());
 return false;
   } finally {
-CarbonFile carbonFile =
-FileFactory.getCarbonFile(location, 
FileFactory.getFileType(location));
-if (carbonFile.exists()) {
-  if (carbonFile.delete()) {
-LOGGER.info("Deleted the lock file " + location);
-  } else {
-LOGGER.error("Not able to delete the lock file " + location);
+try {
+  if (FileFactory.isFileExist(location, 
FileFactory.getFileType(location))) {
+if (FileFactory.getCarbonFile(location, 
FileFactory.getFileType(location)).delete()) {
+  LOGGER.info("Deleted the lock file " + location);
+} else {
+  LOGGER.error("Not able to delete the lock file " + location);
+}
   }
-} else {
-  LOGGER.error("Not able to delete the lock file because "
-  + "it is not existed in location " + location);
+} catch (IOException e) {
+  LOGGER.error("Exception in isFileExist of the lock file " + 
e.getMessage());
 }
   }
 }

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/df3dea90/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetadataUtil.scala
--
diff --git 
a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetadataUtil.scala
 
b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetadataUtil.scala
index fc5730b..657d7de 100644
--- 
a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetadataUtil.scala
+++ 
b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetadataUtil.scala
@@ -41,7 +41,8 @@ object CarbonHiveMetadataUtil {
   tableName: String,
   sparkSession: SparkSession): Unit = {
 try {
-  sparkSession.sql(s"DROP TABLE IF EXISTS $databaseName.$tableName")
+  
sparkSession.sharedState.externalCatalog.asInstanceOf[HiveExternalCatalog].client.
+runSqlHive(s"DROP TABLE IF EXISTS $databaseName.$tableName")
 } catch {
   case e: Exception =>
 LOGGER.audit(

[38/49] incubator-carbondata git commit: Removed USE_KETTLE options reference from documentation

[jackylk]

Removed USE_KETTLE options reference from documentation


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/97d27895
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/97d27895
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/97d27895

Branch: refs/heads/12-dev
Commit: 97d27895da323313d299d326fb8788e02b1ed6a7
Parents: 1624c5f
Author: Divya 
Authored: Wed Apr 5 13:58:45 2017 +0530
Committer: chenliang613 
Committed: Thu Apr 6 16:51:18 2017 +0530

--
 docs/dml-operation-on-carbondata.md | 9 -
 1 file changed, 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/97d27895/docs/dml-operation-on-carbondata.md
--
diff --git a/docs/dml-operation-on-carbondata.md 
b/docs/dml-operation-on-carbondata.md
index 74fa0b0..6f67543 100644
--- a/docs/dml-operation-on-carbondata.md
+++ b/docs/dml-operation-on-carbondata.md
@@ -133,14 +133,6 @@ You can use the following options to load data:
 
 NOTE: Date formats are specified by date pattern strings. The date pattern 
letters in CarbonData are same as in JAVA. Refer to 
[SimpleDateFormat](http://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html).
 
-- **USE_KETTLE:** This option is used to specify whether to use kettle for 
loading data or not. By default kettle is not used for data loading.
-
-```
-OPTIONS('USE_KETTLE'='FALSE')
-```
-
-   Note :  It is recommended to set the value for this option as false.
-
 - **SINGLE_PASS:** Single Pass Loading enables single job to finish data 
loading with dictionary generation on the fly. It enhances performance in the 
scenarios where the subsequent data loading after initial load involves fewer 
incremental updates on the dictionary.
 
This option specifies whether to use single pass for loading data or not. 
By default this option is set to FALSE.
@@ -165,7 +157,6 @@ options('DELIMITER'=',', 'QUOTECHAR'='"','COMMENTCHAR'='#',
 'MULTILINE'='true','ESCAPECHAR'='\','COMPLEX_DELIMITER_LEVEL_1'='$',
 'COMPLEX_DELIMITER_LEVEL_2'=':',
 'ALL_DICTIONARY_PATH'='/opt/alldictionary/data.dictionary',
-'USE_KETTLE'='FALSE',
 'SINGLE_PASS'='TRUE'
 )
 ```

[37/49] incubator-carbondata git commit: [CARBONDATA-847] Fixed NullPointerException for DictionaryBasedVectorResultCollector during alter table. This closes #724

[jackylk]

[CARBONDATA-847] Fixed NullPointerException for 
DictionaryBasedVectorResultCollector during alter table. This closes #724


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/1624c5f7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/1624c5f7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/1624c5f7

Branch: refs/heads/12-dev
Commit: 1624c5f7a8064eb7c149f8b38710ae7bdd90a69e
Parents: 413cb93 22be345
Author: ravipesala 
Authored: Thu Apr 6 16:45:11 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 16:45:11 2017 +0530

--
 .../RestructureBasedRawResultCollector.java | 13 +++--
 .../RestructureBasedVectorResultCollector.java  |  7 ++-
 .../src/test/resources/restructure/data1.csv|  2 +-
 .../vectorreader/AddColumnTestCases.scala   | 61 
 4 files changed, 75 insertions(+), 8 deletions(-)
--

[33/49] incubator-carbondata git commit: changes in troubleshooting.md file This closes #746

[jackylk]

changes in troubleshooting.md file This closes #746


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/631be314
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/631be314
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/631be314

Branch: refs/heads/12-dev
Commit: 631be3147dc8bc01d7a38992a37c1b7b795afefc
Parents: 1b20c2d 951c507
Author: chenliang613 
Authored: Thu Apr 6 16:37:01 2017 +0530
Committer: chenliang613 
Committed: Thu Apr 6 16:37:01 2017 +0530

--
 docs/troubleshooting.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[18/49] incubator-carbondata git commit: [CARBONDATA-814] bad record log file writing is not correct. This closes #694

[jackylk]

[CARBONDATA-814] bad record log file writing is not correct. This closes #694


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/95549714
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/95549714
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/95549714

Branch: refs/heads/12-dev
Commit: 95549714b4f9ea0a244a936b0b5a8ec73ad61b41
Parents: 05b790a 3af2d65
Author: ravipesala 
Authored: Thu Apr 6 14:57:21 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 14:57:21 2017 +0530

--
 .../core/scan/expression/RangeExpressionEvaluator.java   | 7 +++
 .../newflow/steps/DataConverterProcessorStepImpl.java| 8 
 .../processing/store/CarbonFactDataHandlerColumnar.java  | 5 -
 3 files changed, 15 insertions(+), 5 deletions(-)
--

[34/49] incubator-carbondata git commit: changes in release-guide.md

[jackylk]

changes in release-guide.md


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/60ac23d3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/60ac23d3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/60ac23d3

Branch: refs/heads/12-dev
Commit: 60ac23d32d015a5e8ba01bf3e6a4ad7427ae16e6
Parents: 631be31
Author: swati 
Authored: Thu Apr 6 13:26:23 2017 +0530
Committer: chenliang613 
Committed: Thu Apr 6 16:38:39 2017 +0530

--
 docs/release-guide.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/60ac23d3/docs/release-guide.md
--
diff --git a/docs/release-guide.md b/docs/release-guide.md
index 50a0e8a..6d55192 100644
--- a/docs/release-guide.md
+++ b/docs/release-guide.md
@@ -67,7 +67,7 @@ Determine your Apache GPG key and key ID, as follows:
 gpg --list-keys
 ```
 
-This will list your GPG keys. One of these should reflect your Apache account, 
for exemple:
+This will list your GPG keys. One of these should reflect your Apache account, 
for example:
 
 ```
 pub   2048R/845E6689 2016-02-23
@@ -401,7 +401,7 @@ _Checklist to proceed to the final step:_
 
 Any issue identified during the community review and vote should be fixed in 
this step.
 
-To fully cacel a vote:
+To fully cancel a vote:
 
 * Cancel the current release and verify the version is back to the correct 
SNAPSHOT:
 

[46/49] incubator-carbondata git commit: Formatted Document

[jackylk]

Formatted Document


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/7962027d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/7962027d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/7962027d

Branch: refs/heads/12-dev
Commit: 7962027d8ea4d9fb5c20cb3723b65b7ed1e476ec
Parents: 24d9fd2
Author: ksimar 
Authored: Thu Apr 6 13:16:34 2017 +0530
Committer: chenliang613 
Committed: Fri Apr 7 14:51:55 2017 +0530

--
 docs/quick-start-guide.md | 17 +++--
 1 file changed, 7 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/7962027d/docs/quick-start-guide.md
--
diff --git a/docs/quick-start-guide.md b/docs/quick-start-guide.md
index c29a8d3..5eec075 100644
--- a/docs/quick-start-guide.md
+++ b/docs/quick-start-guide.md
@@ -34,9 +34,7 @@ This tutorial provides a quick introduction to using 
CarbonData.
   EOF
   ```
 
-## Interactive Analysis with Spark Shell
-
-## Version 2.1
+## Interactive Analysis with Spark Shell Version 2.1
 
 Apache Spark Shell provides a simple way to learn the API, as well as a 
powerful tool to analyze data interactively. Please visit [Apache Spark 
Documentation](http://spark.apache.org/docs/latest/) for more details on Spark 
shell.
 
@@ -68,13 +66,13 @@ val carbon = 
SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(
 
  Executing Queries
 
-# Creating a Table
+## Creating a Table
 
 ```
 scala>carbon.sql("CREATE TABLE IF NOT EXISTS test_table(id string, name 
string, city string, age Int) STORED BY 'carbondata'")
 ```
 
-# Loading Data to a Table
+## Loading Data to a Table
 
 ```
 scala>carbon.sql("LOAD DATA INPATH 'sample.csv file path' INTO TABLE 
test_table")
@@ -89,8 +87,7 @@ scala>carbon.sql("SELECT * FROM test_table").show()
 scala>carbon.sql("SELECT city, avg(age), sum(age) FROM test_table GROUP BY 
city").show()
 ```
 
-## Interactive Analysis with Spark Shell
-## Version 1.6
+## Interactive Analysis with Spark Shell Version 1.6
 
  Basics
 
@@ -117,7 +114,7 @@ val cc = new CarbonContext(sc, "")
 
  Executing Queries
 
-# Creating a Table
+## Creating a Table
 
 ```
 scala>cc.sql("CREATE TABLE IF NOT EXISTS test_table (id string, name string, 
city string, age Int) STORED BY 'carbondata'")
@@ -128,14 +125,14 @@ To see the table created :
 scala>cc.sql("SHOW TABLES").show()
 ```
 
-# Loading Data to a Table
+## Loading Data to a Table
 
 ```
 scala>cc.sql("LOAD DATA INPATH 'sample.csv file path' INTO TABLE test_table")
 ```
 **NOTE**: Please provide the real file path of `sample.csv` for the above 
script.
 
-# Query Data from a Table
+## Query Data from a Table
 
 ```
 scala>cc.sql("SELECT * FROM test_table").show()

[01/49] incubator-carbondata git commit: Insert Select Into Same Table [Forced Update!]

[jackylk]

Repository: incubator-carbondata
Updated Branches:
  refs/heads/12-dev dccd2ad0d -> 914d61e32 (forced update)


Insert Select Into Same Table


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/f527d3d2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/f527d3d2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/f527d3d2

Branch: refs/heads/12-dev
Commit: f527d3d2e460076705834393182199c589ed318e
Parents: 950a6d0
Author: sounakr 
Authored: Mon Apr 3 12:48:20 2017 +0530
Committer: sounakr 
Committed: Wed Apr 5 23:52:20 2017 +0530

--
 .../InsertIntoCarbonTableTestCase.scala | 28 +
 .../spark/sql/hive/CarbonAnalysisRules.scala| 43 +++---
 .../spark/sql/CarbonCatalystOperators.scala | 19 +++
 .../sql/CarbonDatasourceHadoopRelation.scala| 13 +
 .../sql/execution/command/DDLStrategy.scala |  5 +-
 .../execution/command/carbonTableSchema.scala   |  3 +-
 .../spark/sql/hive/CarbonAnalysisRules.scala| 60 
 .../spark/sql/hive/CarbonSessionState.scala | 30 +-
 8 files changed, 163 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/f527d3d2/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/allqueries/InsertIntoCarbonTableTestCase.scala
--
diff --git 
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/allqueries/InsertIntoCarbonTableTestCase.scala
 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/allqueries/InsertIntoCarbonTableTestCase.scala
index e84e62a..0b491bf 100644
--- 
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/allqueries/InsertIntoCarbonTableTestCase.scala
+++ 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/allqueries/InsertIntoCarbonTableTestCase.scala
@@ -210,6 +210,32 @@ class InsertIntoCarbonTableTestCase extends QueryTest with 
BeforeAndAfterAll {
 )
   }
 
+  test("insert select from same table") {
+val timeStampPropOrig = 
CarbonProperties.getInstance().getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT)
+CarbonProperties.getInstance()
+  
.addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+
+sql("drop table if exists CarbonDest")
+sql("create table CarbonDest (imei string,deviceInformationId int,MAC 
string,deviceColor string,device_backColor string,modelId string,marketName 
string,AMSize string,ROMSize string,CUPAudit string,CPIClocked string,series 
string,productionDate timestamp,bomCode string,internalModels string, 
deliveryTime string, channelsId string, channelsName string , deliveryAreaId 
string, deliveryCountry string, deliveryProvince string, deliveryCity 
string,deliveryDistrict string, deliveryStreet string, oxSingleNumber string, 
ActiveCheckTime string, ActiveAreaId string, ActiveCountry string, 
ActiveProvince string, Activecity string, ActiveDistrict string, ActiveStreet 
string, ActiveOperatorId string, Active_releaseId string, Active_EMUIVersion 
string, Active_operaSysVersion string, Active_BacVerNumber string, 
Active_BacFlashVer string, Active_webUIVersion string, Active_webUITypeCarrVer 
string,Active_webTypeDataVerNumber string, Active_operatorsVersion string, 
Active_phonePADPartitionedVersio
 ns string, Latest_YEAR int, Latest_MONTH int, Latest_DAY Decimal(30,10), 
Latest_HOUR string, Latest_areaId string, Latest_country string, 
Latest_province string, Latest_city string, Latest_district string, 
Latest_street string, Latest_releaseId string, Latest_EMUIVersion string, 
Latest_operaSysVersion string, Latest_BacVerNumber string, Latest_BacFlashVer 
string, Latest_webUIVersion string, Latest_webUITypeCarrVer string, 
Latest_webTypeDataVerNumber string, Latest_operatorsVersion string, 
Latest_phonePADPartitionedVersions string, Latest_operatorId string, 
gamePointDescription string,gamePointId double,contractNumber BigInt) STORED BY 
'org.apache.carbondata.format'")
+
+sql("drop table if exists HiveDest")
+sql("create table HiveDest (imei string,deviceInformationId int,MAC 
string,deviceColor string,device_backColor string,modelId string,marketName 
string,AMSize string,ROMSize string,CUPAudit string,CPIClocked string,series 
string,productionDate timestamp,bomCode string,internalModels string, 
deliveryTime string, channelsId string, channelsName string , deliveryAreaId 
string, deliveryCountry string, deliveryProvince string, deliveryCity 
string,deliveryDistrict string, deliveryStreet 

[22/49] incubator-carbondata git commit: [CARBONDATA-400] Error message for dataload with a column having more than 100000 characters. This closes #398

[jackylk]

[CARBONDATA-400] Error message for dataload with a column having more than 
10 characters. This closes #398


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/58cd9339
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/58cd9339
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/58cd9339

Branch: refs/heads/12-dev
Commit: 58cd9339e992a2e5b5d67d714210c0ef59bd099c
Parents: fa7421a c99cf06
Author: ravipesala 
Authored: Thu Apr 6 15:34:24 2017 +0530
Committer: ravipesala 
Committed: Thu Apr 6 15:34:24 2017 +0530

--
 .../spark/rdd/CarbonGlobalDictionaryRDD.scala   |  3 +++
 .../spark/util/GlobalDictionaryUtil.scala   | 25 +---
 .../processing/csvload/CSVInputFormat.java  |  1 +
 3 files changed, 26 insertions(+), 3 deletions(-)
--

[2/2] incubator-carbondata git commit: [CARBONDATA-885] Fixed inconsistent usage in queries in ddl operations This closes #758

[chenliang613]

[CARBONDATA-885] Fixed inconsistent usage in queries in ddl operations This 
closes #758


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/33544d32
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/33544d32
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/33544d32

Branch: refs/heads/master
Commit: 33544d32e2e9e35f739b59b819b895681bf548cb
Parents: 914d61e 98e6bcb
Author: chenliang613 
Authored: Fri Apr 7 18:41:02 2017 +0530
Committer: chenliang613 
Committed: Fri Apr 7 18:41:02 2017 +0530

--
 docs/ddl-operation-on-carbondata.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[2/4] incubator-carbondata git commit: clean issue in java/scala doc

[jackylk]

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/086b06d1/processing/src/test/resources/CSV.ktr
--
diff --git a/processing/src/test/resources/CSV.ktr 
b/processing/src/test/resources/CSV.ktr
deleted file mode 100644
index dd20cc3..000
--- a/processing/src/test/resources/CSV.ktr
+++ /dev/null
@@ -1 +0,0 @@
-6e3e8095ef11293bca8c287ca6ee5e0ca7b7d968cba787b42a519a5053bf2e1f6b60d8c9eb8d99d348800c633d156303d9d51d7651e391ca8fab29cd538d4394ce9ea491f50db686ba20b11374dc8763736da191ad0aa7d711d8ebcdb04296c290b9fc5dd86ecd9efbc8a6742d33a1f5734943183b79b6268e0c3b0672befd65ca43816a891a22e53dce91ac7a33207137374218a40bebcda6be2b2cac2e8a3fb67878965a7d5c19f8db8382080b0dfd9f993bb2da04da32dd0fd66557b3fcce01c3d545aa535f3b4bf271738971a516ef094594d35a3abb232f509f5538bec1b274807256266ec373788fa33a35ad568cb92cdc384cb1d1ae29f74096b15c8a0b4120ddd676d1206e9664dedee46a3ab9748b47970e4ca72265f00da397851807ba6dc275e939af1a8115b64100e7e5bff27914121971520bced1f2eb7c5af462d88af8de7ead569cf9520f687b28f2956b9d2cf48002798241c5dcfa14c8f43cc246c84b8039895875f1e16147f9ea750b4c21526951c70dfa7f8d83a50bdfb9185dfe7aebc15180a78034147524220a4441b8d671b29d75d65691a275a23ea371cb5258f555dee464e24eb5aabeeb5527204c0eee7468c9a02f0977e2cfca66872f79e9acb9138b462a6e69b512e79b302b1b756df87b23e3761b3853632e90ecfd9292a478adc9ac7e327e0190a887278
 
a3a87bab2326b585a0a79f3212626571f49ab9c42ed47b2cde2c8780222796fc62a8cd7a18bafe5b6416c70e034daf1c9c984c9bdef080dd36ea4078286950d4ee3228e6ef139b6ad40b00e7f02165b4ef2869b56fe36131b1c791398d40d7a8f09d4a7236b322d346f7e56564a91c33821141d5c9cc373ad48685e88c1a3eb8a5c862b6c729d687d9a106ee7abe7db6fcad10bab7f2c07d33173944c0cf73f382c212d0b6e25562e24ff982371796fc62a8cd7a18bafe5b6416c70e034075924b7fd5d5b28a8b78736ae728ac3457a6962d73772b1c6430984645d91479d86b411f1d78affe8b7aa48c2cde0ec6f4020a08ff1706bb99dd92f5ffa9fdef789424754ee2af574ba03b36dbcace50951d2a1f45ee3dc3b7c8688a6c6811428f170dad241f4825d4cd30b56e5ac051a6e2733ec4905046a4a7983594675a4891d5c55bdf4c4ab50155c899d56579bacd59ce20e499fc266a4e6526a1797f1712ebe5edbd222b0c706205dd6d3319a7081c37ad2e929cbe240a62a8faaae7842eb5d44e11daeba2d0579400717ff0be7db6fcad10bab7f2c07d33173944c0cabbd5e80d9b108dc73852cb8a8b4567901cecd11052975d322180d183a6073ab26109332e378026402e4855c14b5a7be01cb4d239f38b0422ed4e3254a5387762e894976411b75b5a6dd45fa8c9f13d466b0530c08
 
f26b8e90b880f1a39b7cd089afa53e0b8463523d08b4fa408d4117ebb3a18b0ba857c142dffbf37139ddad946bfc22c94411958e3e71b881efca31ef645064003f8dd510d069769ec6e7f29624610b153f9571d3728ebc1926e28bfd2d0fb9c09e1bb18dc2748a98764f4e6a84896b58cf1f2d99a9fba5227b7f610d35da47398a5c98070d93fb014237959b4ffcbb7ad125dffa26b206574bf4b0362d7bb92988ba0cb11dd3b31f94f7009535d73113172ecde1df065d55211c73f7e522b0008d557bea7474a38df5288f0ea6e67753882eaaa2335c20ef69af32712ebe5edbd222b0c706205dd6d3319aded4c93ca11cfc14a44ce94f845ccb35457a6962d73772b1c6430984645d91475dff6fec1ca88e410c0f89642bceb5c484f0a99bd1e329a8ee604f3f13d18e8bb84e187b6913173060259d7ebe8067a0e8b558feecb6c373e478365437ab8fb1a371cb5258f555dee464e24eb5aabeeb1e9c35620033d496f0fb86f8985238b1b9249c61f52f5951edd9c25168370eb2953c898bc9353bf279448dd40485bc6f22472757d567c0713abac9da073c6357bb3fc857c5a7fc76caaa290de3d57030f8c7501452ade0cfa81dbd9d01f92c7991c33821141d5c9cc373ad48685e88c147adb8e6d01e31efba1617f78d291055a371cb5258f555dee464e24eb5aabeeba05aa6d4701efc2
 
2ddda2cd1b584977494e2b4669676a8cf08f462454723b2145ac75cf474b9baa51aaa7327f17fa3d3950d4ee3228e6ef139b6ad40b00e7f028c7786d239d22e46fafd31a7ad40457d443226781cb881b0983d40dcc4470c864aceaa532bea841fc1436a5dc5f7a2cdf593392d92863d32cbc4ed57a44590569535d73113172ecde1df065d55211c735f53ae3b1f2f2e59a9c0683a46a6c10dd867bbb46bd5a09fa4f4f9e49ec255f0f474fdd6f26bf256f4cca9ca0f1c18739c797a35d90f0f78a5eb168c91521cd70ce8cd649b8157c72348cf8a6dc7ab7e2cec771b705f5f2fbc74e4ea3b99792ccb40a80d16fd776f5ffa02afc51bc16ddd52137ced516f91bb7329a19b79049ea5593e9d30cd3c26c55a6068d8b6c24fa99c1abead06b20131d7266dcb80bc6df0dcb0bf553fd1db9a628a33ff90eaa5278dc020b60db1c9447f8dbddd7de1f5c443206c0b1dda67b45e34579eb2934b45e0d85b8e2e62398a7e9ab4b6230ab7060f390a4a86a567491fdd6705ee940f0ce8cd649b8157c72348cf8a6dc7ab7ea3f573f3bddfe427003a79df2026d4e3796fc62a8cd7a18bafe5b6416c70e034f5e63c668e57da6451ce028fed5c0b519624610b153f9571d3728ebc1926e28ba1b763bf0581dce4ad52a4b4d2ff3e51e08ced936704a13a946bb8b35f8bbee6b0865b05eec7fee1fbd7
 

[1/4] incubator-carbondata git commit: clean issue in java/scala doc

[jackylk]

Repository: incubator-carbondata
Updated Branches:
  refs/heads/master 1e5db53a4 -> 75a1db4ba


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/086b06d1/processing/src/test/resources/DATA_FACT_SMALL1.ktr
--
diff --git a/processing/src/test/resources/DATA_FACT_SMALL1.ktr 
b/processing/src/test/resources/DATA_FACT_SMALL1.ktr
deleted file mode 100644
index df9155f..000
--- a/processing/src/test/resources/DATA_FACT_SMALL1.ktr
+++ /dev/null
@@ -1 +0,0 @@
-c912c1ec30ff4709b49423a3e0c8cfd992d3d2ae8361a315d5bf0f6193ec53cfa3ea47591d8a6749f63e1d7c810214e60c9a0a4e7706a7f865c3e66a110b1b0b324ab1d138c847ce42a76b7365dd781d3026c09802a63417e6b3dd23dbd0339ab4a111912970c834610f4070c47ad267b67878965a7d5c19f8db8382080b0dfddc92b7490ca63da6b5d4be92931dfc259c0360a497a7bbdfa62b2d5032ae8e2ec639cdbe779436396956e0adf119ec3673ab1b9dac1c6be702cbd5ea39fff5776ed0c45eb95e9a8079241e699076784f85bbd396fe74f76f620b9ff4ba5690d4f3152ceda226c9c44d69de383a171a1d295c781bd82bf48491b931f13ec61290441c1a49a1e02037caedb359125fb4dd60fdc19c619953fdad76010c49d0febd035e183c2b57977f781adc1cec9ada9a729a40661bc206a0d0096592c8ebaae39732698c599725e9f925613eea843185e083e4b4ae970eb07d61731b0e1e619dc2af3123c998e4a06d634325fc9542eb590e63e49192bc78cfd8b3381b55952191db70f4dc5880d40030032779d11bdeebee0d651d9a75a72439eaaf3fb9ce55ee9123eea047eb888dabe2a71b3d80e08595540256a4b5a0918ca6153870468bdadc24177ff4cecc991eb7db7b151a289535d73113172ecde1df065d55211c73fc4647dd673948f8f2e620ad02da2a96d3a69
 
fa299c171bf7ec7947da1054f83f6aaa3786db4482da297f3e86b89bee6b09036b8d65869257407d7391e206db10ce8cd649b8157c72348cf8a6dc7ab7ecd7d8e04562b8148232c0a2773872eee3a60d6aab3dff03d55a074f2135d475972b3db61ce850d8e2105e0e41b98e2f839cdd11dc50eefa81a4ea69f08ae715652955dd851c0522f12a7fe73f1d1f428c710db75cde42b3b4ee8a4731982cbddb71695d97b02a2500813dbe3d28c27f76e3fe09193693f3d8f5aaa74e614681a7dca599668108f2fa17e8fb4fdfb086d2229085da1d55b3e1220fe1f847edc1cb6969b0fa3cbf2f18c9244d63f80ea6d06ba08d816a53b9e7c15b50850d8c0eb1bae9c5669b8de8cd0015430fdf1e046457a6962d73772b1c6430984645d91479d86b411f1d78affe8b7aa48c2cde0ecb6d7b4e25a0287dac07b8a43177e62c4ebb3a18b0ba857c142dffbf37139ddad946bfc22c94411958e3e71b881efca31ab9549313b48f124dc21b3207eadef4239cdd11dc50eefa81a4ea69f08ae7156e2a84f19b79625aa2fe99f28abf9f435e4f15fd81da2c0c555cbe3eefa71a243f789424754ee2af574ba03b36dbcace50951d2a1f45ee3dc3b7c8688a6c68114285ddd5584afeafd5bb96c9e05736564a371cb5258f555dee464e24eb5aabeebd71533a9d2110668d9d00edf6bcf98677a0fd670ab
 
fb1e94051b633af56f45bb1352033153c068509d8ab9ced98f01d9660d63f28f87999d404464f5d89e41de2e894976411b75b5a6dd45fa8c9f13d466b0530c08f26b8e90b880f1a39b7cd0e7d9ca804a9e096a1b43f66f0916965959b7e2ca2833381cbb94da3ec18c3b9e91c33821141d5c9cc373ad48685e88c1030d6ebfe3d5680ea038549a9ecba80f950d4ee3228e6ef139b6ad40b00e7f02ea727a159e7592e76a37ac2fd8962ff64c50bfd309c54937d448fe297d5bd5cdebb3a18b0ba857c142dffbf37139ddad946bfc22c94411958e3e71b881efca3101ef2d007f1b5942f0371d5ff8f6d8af7dca599668108f2fa17e8fb4fdfb086d44345d85e932a40dd74abf456904de88e3f4a5e425598347cb60fdaaaf7f711300803240593ba88497a8e1a465027a4a91db70f4dc5880d40030032779d11bde5ef25a3c4e49f4d96c7953db88c6e0ee17ea1da6d170fa564416d835a5dac095946bfc22c94411958e3e71b881efca31d2b81682f7866e7bfcb097bee2693a2be7db6fcad10bab7f2c07d33173944c0c6a591ac003ff3dea6d3cc3ef493e83893a60d6aab3dff03d55a074f2135d475953e534a02da4c56ea97759933666ab9c91db70f4dc5880d40030032779d11bde36b576328c2d9e70d20676edfd20932bb6b2468f13a4317265c9cf94f679c651bc483304b5efe72
 
a83c8f069b6083feb6fb798f46b80754850f6b3f7aae1f2e50ce8cd649b8157c72348cf8a6dc7ab7e1cb300741dc150cc5e8c39f514b1299ce3f4a5e425598347cb60fdaaaf7f7113754f4c8105127e6210ad66f6c2430d01950d4ee3228e6ef139b6ad40b00e7f02dcfd2b034d124795dc4cd62bd0be0b033031cd8e03815979043b48abf109536d0c6dc757075e5f3ac652bf50f6eaf0759dc54f67997461b7c2614856adb842e800d617d8187a2e99a5a01446c757423104c72141fdf374b4132d5d6c4bd6dd4e19f9be249296ad40b33ce0ad5852edf2198deb70d45062696b19ea5d4069ee6806eea0954059472ec6af83f910a2fd60a1bc3506cb5723b0074e0494128b6db45507f77f85434255a12142e9706fee3f4306ba9dd0db89e628a3974ce6dec58d9eb3885dca73f6d798a454b34e947789ad953b38f54b48f73189642e47051de04a8a4fa4a214cc193ff792d300e7c0c400d617d8187a2e99a5a01446c75742315c511a497b52f82d5c60fec96af08c5d833abc309298134230453f80a0d81094263e4138c61861b81a3c7ff1d939d9b528ac9a9baf3afab700a90e8b3d2d8e4c66b0530c08f26b8e90b880f1a39b7cd0692b5bf55624e1a8acf7a93e7b7390e191c33821141d5c9cc373ad48685e88c147adb8e6d01e31efba1617f78d291055a371cb5258f555dee464
 

[1/2] incubator-carbondata git commit: clean code

[jackylk]

Repository: incubator-carbondata
Updated Branches:
  refs/heads/master 33544d32e -> 1e5db53a4


clean code


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/6c73d885
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/6c73d885
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/6c73d885

Branch: refs/heads/master
Commit: 6c73d88578b0c5675b26502768974f958594a820
Parents: 33544d3
Author: QiangCai 
Authored: Fri Apr 7 20:49:14 2017 +0530
Committer: QiangCai 
Committed: Fri Apr 7 20:49:14 2017 +0530

--
 assembly/src/main/assembly/assembly.xml   | 2 +-
 common/src/test/java/log4j.properties | 7 ++-
 .../carbondata/common/logging/ft/LoggingServiceTest_FT.java   | 4 ++--
 .../core/datastore/impl/btree/AbstractBTreeBuilder.java   | 2 +-
 .../core/fileoperations/AtomicFileOperationsImpl.java | 3 ---
 5 files changed, 6 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/6c73d885/assembly/src/main/assembly/assembly.xml
--
diff --git a/assembly/src/main/assembly/assembly.xml 
b/assembly/src/main/assembly/assembly.xml
index 14d679f..1b27074 100644
--- a/assembly/src/main/assembly/assembly.xml
+++ b/assembly/src/main/assembly/assembly.xml
@@ -24,7 +24,7 @@
 
 
 
-org.huawei.datasight:*:jar
+org.apache.carbondata:*:jar
 
 
 

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/6c73d885/common/src/test/java/log4j.properties
--
diff --git a/common/src/test/java/log4j.properties 
b/common/src/test/java/log4j.properties
index 3d35725..db0fbe2 100644
--- a/common/src/test/java/log4j.properties
+++ b/common/src/test/java/log4j.properties
@@ -15,12 +15,9 @@
 #  See the License for the specific language governing permissions and
 # limitations under the License.
 #
-log4j.logger.com.huawei=INFO,R5
+log4j.logger.org.apache.carbondata=INFO,R5
 
log4j.appender.R5=org.apache.carbondata.common.logging.impl.AuditExtendedRollingFileAppender
-log4j.appender.R5.File=./unibiaudit.log
+log4j.appender.R5.File=./carbondataaudit.log
 
log4j.appender.R5.threshold=AUDIT#org.apache.carbondata.common.logging.AuditLevel
 log4j.appender.R5.layout=org.apache.log4j.PatternLayout
 log4j.appender.R5.layout.ConversionPattern=%d [%t] %p [%c] %X{CLIENT_IP} 
%X{USER_NAME} %X{MODULE} %X{OPERATRION}- %m%n
-
-
-

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/6c73d885/common/src/test/java/org/apache/carbondata/common/logging/ft/LoggingServiceTest_FT.java
--
diff --git 
a/common/src/test/java/org/apache/carbondata/common/logging/ft/LoggingServiceTest_FT.java
 
b/common/src/test/java/org/apache/carbondata/common/logging/ft/LoggingServiceTest_FT.java
index c72dde1..1f000ce 100644
--- 
a/common/src/test/java/org/apache/carbondata/common/logging/ft/LoggingServiceTest_FT.java
+++ 
b/common/src/test/java/org/apache/carbondata/common/logging/ft/LoggingServiceTest_FT.java
@@ -54,14 +54,14 @@ public class LoggingServiceTest_FT extends TestCase {
   @Test public void testAudit() {
 
 String expectedAuditLine =
-"[main] AUDIT 
[com.huawei.iweb.platform.logging.ft.LoggingServiceTest_FT] 127.0.0.1 "
+"[main] AUDIT 
[org.apache.carbondata.common.logging.ft.LoggingServiceTest_FT] 127.0.0.1 "
 + "testuser Function Test log- audit message created";
 logger.audit("audit message created");
 
 LogManager.shutdown();
 
 try {
-  FileInputStream fstream = new FileInputStream("./unibiaudit.log");
+  FileInputStream fstream = new FileInputStream("./carbondataaudit.log");
   BufferedReader br = new BufferedReader(new InputStreamReader(fstream));
   String actualAuditLine = null;
   String strLine = null;

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/6c73d885/core/src/main/java/org/apache/carbondata/core/datastore/impl/btree/AbstractBTreeBuilder.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/datastore/impl/btree/AbstractBTreeBuilder.java
 
b/core/src/main/java/org/apache/carbondata/core/datastore/impl/btree/AbstractBTreeBuilder.java
index e71f1eb..4e9f4b1 100644
--- 
a/core/src/main/java/org/apache/carbondata/core/datastore/impl/btree/AbstractBTreeBuilder.java
+++ 
b/core/src/main/java/org/apache/carbondata/core/datastore/impl/btree/AbstractBTreeBuilder.java
@@ -52,7 +52,7 @@ public 

[3/4] incubator-carbondata git commit: clean issue in java/scala doc

[jackylk]

clean issue in java/scala doc


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/086b06d1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/086b06d1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/086b06d1

Branch: refs/heads/master
Commit: 086b06d119d4fb855fa944ecf79f3f637855cb56
Parents: 1e5db53
Author: QiangCai 
Authored: Fri Apr 7 21:53:00 2017 +0530
Committer: jackylk 
Committed: Sat Apr 8 00:56:08 2017 +0800

--
 .../impl/ColumnGroupDimensionDataChunk.java |   7 +-
 .../impl/VariableLengthDimensionDataChunk.java  |   7 +-
 .../chunk/reader/MeasureColumnChunkReader.java  |   8 +-
 ...mpressedDimensionChunkFileBasedReaderV3.java |   2 +-
 ...CompressedMeasureChunkFileBasedReaderV3.java |   8 +-
 .../chunk/store/DimensionDataChunkStore.java|   4 +-
 .../SafeAbsractDimensionDataChunkStore.java |   2 -
 .../SafeFixedLengthDimensionDataChunkStore.java |   4 +-
 ...nsafeFixedLengthDimensionDataChunkStore.java |   4 +-
 ...afeVariableLengthDimesionDataChunkStore.java |   2 +-
 .../TimeStampDirectDictionaryGenerator.java |   2 +-
 .../core/scan/executor/util/QueryUtil.java  |   2 +-
 .../visitor/RangeDirectDictionaryVisitor.java   |   1 -
 .../core/scan/model/CarbonQueryPlan.java|   3 -
 .../core/scan/processor/BlockletIterator.java   |   4 +-
 .../core/scan/scanner/impl/FilterScanner.java   |   9 +-
 .../core/util/CarbonMetadataUtil.java   |   9 +-
 .../apache/carbondata/core/util/CarbonUtil.java |   2 +-
 .../carbondata/core/util/DataTypeUtil.java  |   2 +-
 .../dictionary/ForwardDictionaryCacheTest.java  |   2 +-
 .../impl/ColumnGroupDimensionDataChunkTest.java |   2 +-
 .../impl/btree/BTreeBlockFinderTest.java|   3 +-
 .../hadoop/util/CarbonInputSplitTaskInfo.java   |   2 +-
 .../spark/load/DeleteLoadFolders.java   |  12 +-
 .../carbondata/spark/splits/TableSplit.java |   3 -
 .../carbondata/spark/util/LoadMetadataUtil.java |   6 +-
 .../processing/merger/CarbonDataMergerUtil.java |   2 +-
 .../processing/model/CarbonLoadModel.java   |   3 -
 .../newflow/sort/unsafe/sort/TimSort.java   |   2 +-
 .../store/colgroup/ColGroupDataHolder.java  |   5 +-
 .../processing/util/NonDictionaryUtil.java  |   3 -
 processing/src/test/resources/CSV.ktr   |   1 -
 .../src/test/resources/DATA_FACT_SMALL1.ktr |   1 -
 .../src/test/resources/Vishal5SecondsTest1.xml  | Bin 12280 -> 0 bytes
 .../Vishal5SecondsTest1_en_Invalid.xml  | 150 ---
 35 files changed, 52 insertions(+), 227 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/086b06d1/core/src/main/java/org/apache/carbondata/core/datastore/chunk/impl/ColumnGroupDimensionDataChunk.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/datastore/chunk/impl/ColumnGroupDimensionDataChunk.java
 
b/core/src/main/java/org/apache/carbondata/core/datastore/chunk/impl/ColumnGroupDimensionDataChunk.java
index dc5f93f..8234f67 100644
--- 
a/core/src/main/java/org/apache/carbondata/core/datastore/chunk/impl/ColumnGroupDimensionDataChunk.java
+++ 
b/core/src/main/java/org/apache/carbondata/core/datastore/chunk/impl/ColumnGroupDimensionDataChunk.java
@@ -30,7 +30,8 @@ public class ColumnGroupDimensionDataChunk extends 
AbstractDimensionDataChunk {
* Constructor for this class
*
* @param dataChunk   data chunk
-   * @param chunkAttributes chunk attributes
+   * @param columnValueSize chunk attributes
+   * @param numberOfRows
*/
   public ColumnGroupDimensionDataChunk(byte[] dataChunk, int columnValueSize, 
int numberOfRows) {
 this.dataChunkStore = DimensionChunkStoreFactory.INSTANCE
@@ -62,7 +63,8 @@ public class ColumnGroupDimensionDataChunk extends 
AbstractDimensionDataChunk {
* @param rowId
* @param columnIndex
* @param row
-   * @param restructuringInfo @return
+   * @param info  KeyStructureInfo
+   * @return
*/
   @Override public int fillConvertedChunkData(int rowId, int columnIndex, 
int[] row,
   KeyStructureInfo info) {
@@ -79,7 +81,6 @@ public class ColumnGroupDimensionDataChunk extends 
AbstractDimensionDataChunk {
* Below method will be used to get the masked key
*
* @param data   data
-   * @param offset offset of
* @param info
* @return
*/

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/086b06d1/core/src/main/java/org/apache/carbondata/core/datastore/chunk/impl/VariableLengthDimensionDataChunk.java
--
diff --git 

[4/4] incubator-carbondata git commit: clean java/scala doc issue This closes #762

[jackylk]

clean java/scala doc issue This closes #762


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/75a1db4b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/75a1db4b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/75a1db4b

Branch: refs/heads/master
Commit: 75a1db4ba869829a483b1d0d714c97adf21a917e
Parents: 1e5db53 086b06d
Author: jackylk 
Authored: Sat Apr 8 00:56:42 2017 +0800
Committer: jackylk 
Committed: Sat Apr 8 00:56:42 2017 +0800

--
 .../impl/ColumnGroupDimensionDataChunk.java |   7 +-
 .../impl/VariableLengthDimensionDataChunk.java  |   7 +-
 .../chunk/reader/MeasureColumnChunkReader.java  |   8 +-
 ...mpressedDimensionChunkFileBasedReaderV3.java |   2 +-
 ...CompressedMeasureChunkFileBasedReaderV3.java |   8 +-
 .../chunk/store/DimensionDataChunkStore.java|   4 +-
 .../SafeAbsractDimensionDataChunkStore.java |   2 -
 .../SafeFixedLengthDimensionDataChunkStore.java |   4 +-
 ...nsafeFixedLengthDimensionDataChunkStore.java |   4 +-
 ...afeVariableLengthDimesionDataChunkStore.java |   2 +-
 .../TimeStampDirectDictionaryGenerator.java |   2 +-
 .../core/scan/executor/util/QueryUtil.java  |   2 +-
 .../visitor/RangeDirectDictionaryVisitor.java   |   1 -
 .../core/scan/model/CarbonQueryPlan.java|   3 -
 .../core/scan/processor/BlockletIterator.java   |   4 +-
 .../core/scan/scanner/impl/FilterScanner.java   |   9 +-
 .../core/util/CarbonMetadataUtil.java   |   9 +-
 .../apache/carbondata/core/util/CarbonUtil.java |   2 +-
 .../carbondata/core/util/DataTypeUtil.java  |   2 +-
 .../dictionary/ForwardDictionaryCacheTest.java  |   2 +-
 .../impl/ColumnGroupDimensionDataChunkTest.java |   2 +-
 .../impl/btree/BTreeBlockFinderTest.java|   3 +-
 .../hadoop/util/CarbonInputSplitTaskInfo.java   |   2 +-
 .../spark/load/DeleteLoadFolders.java   |  12 +-
 .../carbondata/spark/splits/TableSplit.java |   3 -
 .../carbondata/spark/util/LoadMetadataUtil.java |   6 +-
 .../processing/merger/CarbonDataMergerUtil.java |   2 +-
 .../processing/model/CarbonLoadModel.java   |   3 -
 .../newflow/sort/unsafe/sort/TimSort.java   |   2 +-
 .../store/colgroup/ColGroupDataHolder.java  |   5 +-
 .../processing/util/NonDictionaryUtil.java  |   3 -
 processing/src/test/resources/CSV.ktr   |   1 -
 .../src/test/resources/DATA_FACT_SMALL1.ktr |   1 -
 .../src/test/resources/Vishal5SecondsTest1.xml  | Bin 12280 -> 0 bytes
 .../Vishal5SecondsTest1_en_Invalid.xml  | 150 ---
 35 files changed, 52 insertions(+), 227 deletions(-)
--

[1/6] incubator-carbondata git commit: Refactored query in example

[jackylk]

Repository: incubator-carbondata
Updated Branches:
  refs/heads/12-dev f60f6b62c -> 5783718a9


Refactored query in example

Update ddl-operation-on-carbondata.md


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/98e6bcbd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/98e6bcbd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/98e6bcbd

Branch: refs/heads/12-dev
Commit: 98e6bcbd3263fd2f2137abbe7d1dc080fdafc566
Parents: 914d61e
Author: raman 
Authored: Fri Apr 7 15:59:07 2017 +0530
Committer: chenliang613 
Committed: Fri Apr 7 18:39:15 2017 +0530

--
 docs/ddl-operation-on-carbondata.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/98e6bcbd/docs/ddl-operation-on-carbondata.md
--
diff --git a/docs/ddl-operation-on-carbondata.md 
b/docs/ddl-operation-on-carbondata.md
index 392bb39..2744691 100644
--- a/docs/ddl-operation-on-carbondata.md
+++ b/docs/ddl-operation-on-carbondata.md
@@ -189,8 +189,8 @@ of columns is used.
CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
 [(col_name data_type, ...)]
STORED BY 'carbondata'
-   TBLPROPERTIES(“BUCKETNUMBER”=”noOfBuckets”,
-   “BUCKETCOLUMNS”=’’columnname”)
+   TBLPROPERTIES('BUCKETNUMBER'='noOfBuckets',
+   'BUCKETCOLUMNS'='columnname')
 ```
   
 ## Parameter Description

[3/6] incubator-carbondata git commit: sort columns

[jackylk]

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/a76b418f/processing/src/main/java/org/apache/carbondata/processing/newflow/steps/NoSortProcessorStepImpl.java
--
diff --git 
a/processing/src/main/java/org/apache/carbondata/processing/newflow/steps/NoSortProcessorStepImpl.java
 
b/processing/src/main/java/org/apache/carbondata/processing/newflow/steps/NoSortProcessorStepImpl.java
new file mode 100644
index 000..bde89ed
--- /dev/null
+++ 
b/processing/src/main/java/org/apache/carbondata/processing/newflow/steps/NoSortProcessorStepImpl.java
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.processing.newflow.steps;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.util.DataTypeUtil;
+import org.apache.carbondata.processing.newflow.AbstractDataLoadProcessorStep;
+import org.apache.carbondata.processing.newflow.CarbonDataLoadConfiguration;
+import org.apache.carbondata.processing.newflow.DataField;
+import org.apache.carbondata.processing.newflow.row.CarbonRow;
+import org.apache.carbondata.processing.util.CarbonDataProcessorUtil;
+import org.apache.carbondata.processing.util.NonDictionaryUtil;
+
+/**
+ * if the table doesn't have sort_columns, just convert row format.
+ */
+public class NoSortProcessorStepImpl extends AbstractDataLoadProcessorStep {
+
+  private int dimensionCount;
+
+  private int dimensionWithComplexCount;
+
+  private int noDictCount;
+
+  private int measureCount;
+
+  private boolean[] isNoDictionaryDimensionColumn;
+
+  private char[] aggType;
+
+  public NoSortProcessorStepImpl(CarbonDataLoadConfiguration configuration,
+  AbstractDataLoadProcessorStep child) {
+super(configuration, child);
+this.dimensionWithComplexCount = configuration.getDimensionCount();
+this.noDictCount =
+configuration.getNoDictionaryCount() + 
configuration.getComplexDimensionCount();
+this.dimensionCount = configuration.getDimensionCount() - this.noDictCount;
+this.measureCount = configuration.getMeasureCount();
+this.isNoDictionaryDimensionColumn =
+
CarbonDataProcessorUtil.getNoDictionaryMapping(configuration.getDataFields());
+this.aggType = CarbonDataProcessorUtil
+.getAggType(configuration.getMeasureCount(), 
configuration.getMeasureFields());
+  }
+
+  @Override public DataField[] getOutput() {
+return child.getOutput();
+  }
+
+  @Override public void initialize() throws IOException {
+child.initialize();
+  }
+
+  /**
+   * convert input CarbonRow to output CarbonRow
+   * e.g. There is a table as following,
+   * the number of dictionary dimensions is a,
+   * the number of no-dictionary dimensions is b,
+   * the number of complex dimensions is c,
+   * the number of measures is d.
+   * input CarbonRow format:  the length of Object[] data is a+b+c+d, the 
number of all columns.
+   * 

+   * | Part | Object item| describe
 |
+   * 

+   * | Object[0 ~ a+b-1]| Integer, byte[], Integer, ...  | dict + no 
dict dimensions|
+   * 

+   * | Object[a+b ~ a+b+c-1]| byte[], byte[], ...| complex 
dimensions   |
+   * 

+   * | Object[a+b+c ~ a+b+c+d-1]| int, byte[], ...   | measures
 |
+   * 

+   * output CarbonRow format: the length of object[] data is 3.
+   * 

+   * | Part | Object item| describe
 |
+   * 

[2/6] incubator-carbondata git commit: [CARBONDATA-885] Fixed inconsistent usage in queries in ddl operations This closes #758

[jackylk]

[CARBONDATA-885] Fixed inconsistent usage in queries in ddl operations This 
closes #758


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/33544d32
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/33544d32
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/33544d32

Branch: refs/heads/12-dev
Commit: 33544d32e2e9e35f739b59b819b895681bf548cb
Parents: 914d61e 98e6bcb
Author: chenliang613 
Authored: Fri Apr 7 18:41:02 2017 +0530
Committer: chenliang613 
Committed: Fri Apr 7 18:41:02 2017 +0530

--
 docs/ddl-operation-on-carbondata.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[4/6] incubator-carbondata git commit: sort columns

[jackylk]

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/a76b418f/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortcolumns/TestSortColumns.scala
--
diff --git 
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortcolumns/TestSortColumns.scala
 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortcolumns/TestSortColumns.scala
new file mode 100644
index 000..88b0421
--- /dev/null
+++ 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/sortcolumns/TestSortColumns.scala
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.spark.testsuite.sortcolumns
+
+import org.apache.spark.sql.common.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+class TestSortColumns extends QueryTest with BeforeAndAfterAll {
+
+  override def beforeAll {
+dropTable
+
+sql("CREATE TABLE origintable1 (empno int, empname String, designation 
String, doj Timestamp, workgroupcategory int, workgroupcategoryname String, 
deptno int, deptname String, projectcode int, projectjoindate Timestamp, 
projectenddate Timestamp,attendance int,utilization int,salary int) STORED BY 
'org.apache.carbondata.format'")
+sql(s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE 
origintable1 OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '\"')""")
+  }
+
+  test("create table with no dictionary sort_columns") {
+sql("CREATE TABLE sorttable1 (empno int, empname String, designation 
String, doj Timestamp, workgroupcategory int, workgroupcategoryname String, 
deptno int, deptname String, projectcode int, projectjoindate Timestamp, 
projectenddate Timestamp,attendance int,utilization int,salary int) STORED BY 
'org.apache.carbondata.format' tblproperties('sort_columns'='empno')")
+sql(s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE 
sorttable1 OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '\"')""")
+checkAnswer(sql("select empno from sorttable1"), sql("select empno from 
sorttable1 order by empno"))
+  }
+
+  test("create table with dictionary sort_columns") {
+sql("CREATE TABLE sorttable2 (empno int, empname String, designation 
String, doj Timestamp, workgroupcategory int, workgroupcategoryname String, 
deptno int, deptname String, projectcode int, projectjoindate Timestamp, 
projectenddate Timestamp,attendance int,utilization int,salary int) STORED BY 
'org.apache.carbondata.format' tblproperties('sort_columns'='empname')")
+sql(s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE 
sorttable2 OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '\"')""")
+checkAnswer(sql("select empname from sorttable2"),sql("select empname from 
origintable1"))
+  }
+
+  test("create table with direct-dictioanry sort_columns") {
+sql("CREATE TABLE sorttable3 (empno int, empname String, designation 
String, doj Timestamp, workgroupcategory int, workgroupcategoryname String, 
deptno int, deptname String, projectcode int, projectjoindate Timestamp, 
projectenddate Timestamp,attendance int,utilization int,salary int) STORED BY 
'org.apache.carbondata.format' tblproperties('sort_columns'='doj')")
+sql(s"""LOAD DATA local inpath '$resourcesPath/data.csv' INTO TABLE 
sorttable3 OPTIONS('DELIMITER'= ',', 'QUOTECHAR'= '\"')""")
+checkAnswer(sql("select doj from sorttable3"), sql("select doj from 
sorttable3 order by doj"))
+  }
+
+  test("create table with multi-sort_columns and data loading with offheap 
safe") {
+try {
+  setLoadingProperties("true", "false", "false")
+  sql("CREATE TABLE sorttable4_offheap_safe (empno int, empname String, 
designation String, doj Timestamp, workgroupcategory int, workgroupcategoryname 
String, deptno int, deptname String, projectcode int, projectjoindate 
Timestamp, projectenddate Timestamp,attendance int,utilization int,salary int) 
STORED BY 'org.apache.carbondata.format' 
tblproperties('sort_columns'='workgroupcategory, 

[5/6] incubator-carbondata git commit: sort columns

[jackylk]

sort columns


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/a76b418f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/a76b418f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/a76b418f

Branch: refs/heads/12-dev
Commit: a76b418fe0a6ea76e62004ca19673fd7998c63d8
Parents: 33544d3
Author: QiangCai 
Authored: Thu Mar 2 17:48:54 2017 +0800
Committer: jackylk 
Committed: Fri Apr 7 22:15:46 2017 +0800

--
 .../core/constants/CarbonCommonConstants.java   |   1 +
 .../core/datastore/block/SegmentProperties.java |  42 +++
 .../impl/VariableLengthDimensionDataChunk.java  |  58 +++-
 .../impl/btree/BTreeDataRefNodeFinder.java  |  24 +-
 .../core/keygenerator/mdkey/Bits.java   |   6 +-
 .../ThriftWrapperSchemaConverterImpl.java   |  18 +
 .../core/metadata/schema/table/CarbonTable.java |  24 ++
 .../schema/table/column/ColumnSchema.java   |  10 +
 .../impl/DictionaryBasedResultCollector.java|  12 +-
 ...structureBasedDictionaryResultCollector.java |   4 +-
 .../scan/filter/FilterExpressionProcessor.java  |   4 +-
 .../carbondata/core/scan/filter/FilterUtil.java |  98 --
 ...velRangeLessThanEqualFilterExecuterImpl.java |   2 +-
 .../RowLevelRangeLessThanFiterExecuterImpl.java |   2 +-
 .../visitor/NoDictionaryTypeVisitor.java|   3 +-
 .../visitor/RangeNoDictionaryTypeVisitor.java   |   3 +-
 .../AbstractDetailQueryResultIterator.java  |   4 +-
 .../scan/result/vector/CarbonColumnVector.java  |   7 +
 .../util/AbstractDataFileFooterConverter.java   |   7 +
 .../apache/carbondata/core/util/ByteUtil.java   | 336 ++-
 .../apache/carbondata/core/util/CarbonUtil.java |  18 +
 .../carbondata/core/util/DataTypeUtil.java  |  70 
 .../impl/btree/BTreeBlockFinderTest.java|  10 +-
 .../core/scan/filter/FilterUtilTest.java|   7 +-
 .../carbondata/hadoop/CarbonInputFormat.java|   4 +-
 .../internal/index/impl/InMemoryBTreeIndex.java |   4 +-
 .../impl/DictionaryDecodeReadSupport.java   |   3 +
 .../hadoop/util/BlockLevelTraverser.java|   4 +-
 .../hadoop/test/util/StoreCreator.java  |   5 +
 .../testsuite/sortcolumns/TestSortColumns.scala | 267 +++
 .../spark/sql/catalyst/CarbonDDLSqlParser.scala | 125 +++
 .../execution/command/carbonTableSchema.scala   |  66 ++--
 .../apache/spark/sql/TestCarbonSqlParser.scala  |  56 +---
 .../readsupport/SparkRowReadSupportImpl.java|  13 +-
 .../vectorreader/ColumnarVectorWrapper.java |  13 +
 .../org/apache/spark/sql/TableCreator.scala | 145 
 .../merger/CompactionResultSortProcessor.java   |  10 +-
 .../merger/RowResultMergerProcessor.java|  14 +-
 .../newflow/CarbonDataLoadConfiguration.java|  24 ++
 .../newflow/DataLoadProcessBuilder.java |  24 +-
 .../impl/NonDictionaryFieldConverterImpl.java   |  21 +-
 .../sort/impl/ParallelReadMergeSorterImpl.java  |   3 +-
 ...arallelReadMergeSorterWithBucketingImpl.java |   3 +-
 .../sort/unsafe/UnsafeCarbonRowPage.java|  11 +-
 .../newflow/sort/unsafe/UnsafeSortDataRows.java |  11 +-
 .../unsafe/comparator/UnsafeRowComparator.java  |  10 +-
 .../UnsafeRowComparatorForNormalDIms.java   |   8 +-
 .../holder/UnsafeFinalMergePageHolder.java  |   4 +-
 .../unsafe/holder/UnsafeInmemoryHolder.java |   5 +-
 .../holder/UnsafeSortTempFileChunkHolder.java   |   2 +-
 .../UnsafeSingleThreadFinalSortFilesMerger.java |   4 +-
 .../newflow/steps/NoSortProcessorStepImpl.java  | 153 +
 .../sortdata/IntermediateFileMerger.java|   3 +-
 .../sortdata/NewRowComparator.java  |  12 +-
 .../sortdata/NewRowComparatorForNormalDims.java |  10 +-
 .../sortandgroupby/sortdata/RowComparator.java  |  10 +-
 .../sortdata/RowComparatorForNormalDims.java|  10 +-
 .../sortandgroupby/sortdata/SortDataRows.java   |  13 +-
 .../sortandgroupby/sortdata/SortParameters.java |  43 +++
 .../sortdata/SortTempFileChunkHolder.java   |  11 +-
 .../store/CarbonFactDataHandlerColumnar.java|  45 ++-
 .../store/CarbonFactDataHandlerModel.java   |   7 +
 .../store/SingleThreadFinalSortFilesMerger.java |   8 +-
 63 files changed, 1601 insertions(+), 353 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/a76b418f/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
 
b/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
index b82d53c..6cee7f6 100644
--- 

[6/6] incubator-carbondata git commit: [CARBONDATA-782]support SORT_COLUMNS This closes #757

[jackylk]

[CARBONDATA-782]support SORT_COLUMNS This closes #757


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/5783718a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/5783718a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/5783718a

Branch: refs/heads/12-dev
Commit: 5783718a96e46806a085d82b3cf2763e2c7da2db
Parents: f60f6b6 a76b418
Author: jackylk 
Authored: Fri Apr 7 22:18:38 2017 +0800
Committer: jackylk 
Committed: Fri Apr 7 22:18:38 2017 +0800

--
 docs/ddl-operation-on-carbondata.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[2/2] incubator-carbondata git commit: [CARBONDATA-842]implement CarbonRowDataWriterProcessorStepImpl for no SORT_COLUMN This closes #741

[jackylk]

[CARBONDATA-842]implement CarbonRowDataWriterProcessorStepImpl for no 
SORT_COLUMN This closes #741


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/12bf455a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/12bf455a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/12bf455a

Branch: refs/heads/12-dev
Commit: 12bf455a819fc7864aee44e724b19b52237fb1fa
Parents: 5783718 9c9af81
Author: jackylk 
Authored: Fri Apr 7 22:28:02 2017 +0800
Committer: jackylk 
Committed: Fri Apr 7 22:28:02 2017 +0800

--
 .../processing/merger/CarbonDataMergerUtil.java |   2 +-
 .../processing/model/CarbonLoadModel.java   |   6 +-
 .../newflow/DataLoadProcessBuilder.java |  38 ++-
 .../CarbonRowDataWriterProcessorStepImpl.java   | 321 +++
 .../newflow/steps/NoSortProcessorStepImpl.java  | 153 -
 .../store/CarbonDataFileAttributes.java |  10 +-
 .../store/CarbonFactDataHandlerModel.java   |   2 +-
 .../store/writer/AbstractFactDataWriter.java|   4 +-
 8 files changed, 361 insertions(+), 175 deletions(-)
--