carbondata git commit: [CARBONDATA-2098]Add Documentation for Pre-Aggregate tables

2018-02-02 Thread chenliang613 
Repository: carbondata
Updated Branches:
  refs/heads/master da129d527 -> 71f8828be


[CARBONDATA-2098]Add Documentation for Pre-Aggregate tables

Add Documentation for Pre-Aggregate tables

This closes #1886


Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/71f8828b
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/71f8828b
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/71f8828b

Branch: refs/heads/master
Commit: 71f8828be56ae9f3927a5fc4a5047794a740c6d1
Parents: da129d5
Author: Raghunandan S 
Authored: Mon Jan 29 08:54:49 2018 +0530
Committer: chenliang613 
Committed: Sat Feb 3 15:45:30 2018 +0800

--
 docs/data-management-on-carbondata.md   | 245 +++
 .../examples/PreAggregateTableExample.scala | 145 +++
 .../TimeSeriesPreAggregateTableExample.scala| 103 
 3 files changed, 493 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/carbondata/blob/71f8828b/docs/data-management-on-carbondata.md
--
diff --git a/docs/data-management-on-carbondata.md 
b/docs/data-management-on-carbondata.md
index 3119935..0b35ed9 100644
--- a/docs/data-management-on-carbondata.md
+++ b/docs/data-management-on-carbondata.md
@@ -25,6 +25,7 @@ This tutorial is going to introduce all commands and data 
operations on CarbonDa
 * [UPDATE AND DELETE](#update-and-delete)
 * [COMPACTION](#compaction)
 * [PARTITION](#partition)
+* [PRE-AGGREGATE TABLES](#agg-tables)
 * [BUCKETING](#bucketing)
 * [SEGMENT MANAGEMENT](#segment-management)
 
@@ -748,6 +749,250 @@ This tutorial is going to introduce all commands and data 
operations on CarbonDa
   * The partitioned column can be excluded from SORT_COLUMNS, this will let 
other columns to do the efficient sorting.
   * When writing SQL on a partition table, try to use filters on the partition 
column.
 
+## PRE-AGGREGATE TABLES
+  Carbondata supports pre aggregating of data so that OLAP kind of queries can 
fetch data 
+  much faster.Aggregate tables are created as datamaps so that the handling is 
as efficient as 
+  other indexing support.Users can create as many aggregate tables they 
require as datamaps to 
+  improve their query performance,provided the storage requirements and 
loading speeds are 
+  acceptable.
+  
+  For main table called **sales** which is defined as 
+  
+  ```
+  CREATE TABLE sales (
+  order_time timestamp,
+  user_id string,
+  sex string,
+  country string,
+  quantity int,
+  price bigint)
+  STORED BY 'carbondata'
+  ```
+  
+  user can create pre-aggregate tables using the DDL
+  
+  ```
+  CREATE DATAMAP agg_sales
+  ON TABLE sales
+  USING "preaggregate"
+  AS
+  SELECT country, sex, sum(quantity), avg(price)
+  FROM sales
+  GROUP BY country, sex
+  ```
+  
+Functions supported in pre-aggregate tables
+
+| Function | Rollup supported |
+|---||
+| SUM | Yes |
+| AVG | Yes |
+| MAX | Yes |
+| MIN | Yes |
+| COUNT | Yes |
+
+
+# How pre-aggregate tables are selected
+For the main table **sales** and pre-aggregate table **agg_sales** created 
above, queries of the 
+kind
+```
+SELECT country, sex, sum(quantity), avg(price) from sales GROUP BY country, sex
+
+SELECT sex, sum(quantity) from sales GROUP BY sex
+
+SELECT sum(price), country from sales GROUP BY country
+``` 
+
+will be transformed by Query Planner to fetch data from pre-aggregate table 
**agg_sales**
+
+But queries of kind
+```
+SELECT user_id, country, sex, sum(quantity), avg(price) from sales GROUP BY 
country, sex
+
+SELECT sex, avg(quantity) from sales GROUP BY sex
+
+SELECT max(price), country from sales GROUP BY country
+```
+
+will fetch the data from the main table **sales**
+
+# Loading data to pre-aggregate tables
+For existing table with loaded data, data load to pre-aggregate table will be 
triggered by the 
+CREATE DATAMAP statement when user creates the pre-aggregate table.
+For incremental loads after aggregates tables are created, loading data to 
main table triggers 
+the load to pre-aggregate tables once main table loading is complete.These 
loads are automic 
+meaning that data on main table and aggregate tables are only visible to the 
user after all tables 
+are loaded
+
+# Querying data from pre-aggregate tables
+Pre-aggregate tables cannot be queries directly.Queries are to be made on main 
table.Internally 
+carbondata will check associated pre-aggregate tables with the main table and 
if the 
+pre-aggregate tables satisfy the query condition, the plan is transformed 
automatically to use 
+pre-aggregate table to fetch the data
+
+# Compacting pre-aggregate tables
+Compaction is an optional operation for pre-aggregate

carbondata git commit: [CARBONDATA-2110]deprecate 'tempCSV' option of dataframe load

2018-02-02 Thread jackylk 
Repository: carbondata
Updated Branches:
  refs/heads/master 6c097cbf3 -> da129d527


[CARBONDATA-2110]deprecate 'tempCSV' option of dataframe load

deprecate 'tempCSV' option of dataframe load, it won't generate temp file on 
hdfs, no matter the value of tempCSV

This closes #1916


Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/da129d52
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/da129d52
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/da129d52

Branch: refs/heads/master
Commit: da129d5277babe498fa5686fe53d01433d112bab
Parents: 6c097cb
Author: qiuchenjian <807169...@qq.com>
Authored: Sat Feb 3 00:14:07 2018 +0800
Committer: Jacky Li 
Committed: Sat Feb 3 15:29:08 2018 +0800

--
 .../testsuite/dataload/TestLoadDataFrame.scala  | 19 
 .../spark/sql/CarbonDataFrameWriter.scala   | 98 +---
 2 files changed, 20 insertions(+), 97 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/carbondata/blob/da129d52/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestLoadDataFrame.scala
--
diff --git 
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestLoadDataFrame.scala
 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestLoadDataFrame.scala
index 6f03493..693c145 100644
--- 
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestLoadDataFrame.scala
+++ 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestLoadDataFrame.scala
@@ -29,6 +29,7 @@ class TestLoadDataFrame extends QueryTest with 
BeforeAndAfterAll {
   var df: DataFrame = _
   var dataFrame: DataFrame = _
   var df2: DataFrame = _
+  var df3: DataFrame = _
   var booldf:DataFrame = _
 
 
@@ -52,6 +53,10 @@ class TestLoadDataFrame extends QueryTest with 
BeforeAndAfterAll {
   .map(x => ("key_" + x, "str_" + x, x, x * 2, x * 3))
   .toDF("c1", "c2", "c3", "c4", "c5")
 
+df3 = sqlContext.sparkContext.parallelize(1 to 3)
+  .map(x => (x.toString + "te,s\nt", x))
+  .toDF("c1", "c2")
+
 val boolrdd = sqlContext.sparkContext.parallelize(
   Row("anubhav",true) ::
 Row("prince",false) :: Nil)
@@ -74,6 +79,7 @@ class TestLoadDataFrame extends QueryTest with 
BeforeAndAfterAll {
 sql("DROP TABLE IF EXISTS carbon9")
 sql("DROP TABLE IF EXISTS carbon10")
 sql("DROP TABLE IF EXISTS carbon11")
+sql("DROP TABLE IF EXISTS carbon12")
 sql("DROP TABLE IF EXISTS df_write_sort_column_not_specified")
 sql("DROP TABLE IF EXISTS df_write_specify_sort_column")
 sql("DROP TABLE IF EXISTS df_write_empty_sort_column")
@@ -261,6 +267,19 @@ test("test the boolean data type"){
 val isStreaming: String = 
descResult.collect().find(row=>row(0).asInstanceOf[String].trim.equalsIgnoreCase("streaming")).get.get(1).asInstanceOf[String]
 assert(isStreaming.contains("true"))
   }
+
+  test("test datasource table with specified char") {
+
+df3.write
+  .format("carbondata")
+  .option("tableName", "carbon12")
+  .option("tempCSV", "true")
+  .mode(SaveMode.Overwrite)
+  .save()
+checkAnswer(
+  sql("select count(*) from carbon12"), Row(3)
+)
+  }
   private def getSortColumnValue(tableName: String): Array[String] = {
 val desc = sql(s"desc formatted $tableName")
 val sortColumnRow = desc.collect.find(r =>

http://git-wip-us.apache.org/repos/asf/carbondata/blob/da129d52/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala
--
diff --git 
a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala
 
b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala
index 2b06375..2be89b1 100644
--- 
a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala
+++ 
b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala
@@ -17,16 +17,12 @@
 
 package org.apache.spark.sql
 
-import org.apache.hadoop.fs.Path
-import org.apache.hadoop.io.compress.GzipCodec
 import org.apache.spark.sql.execution.command.management.CarbonLoadDataCommand
 import org.apache.spark.sql.types._
 import org.apache.spark.sql.util.CarbonException
 
 import org.apache.carbondata.common.logging.LogServiceFactory
-import org.apache.carbondata.core.constants.CarbonCommonConstants
 import org.apache.carbondata.core.metadata.datatype.{DataTypes => CarbonType}
-import org.apache.carbondata.core.util.CarbonProperties
 import

Jenkins build became unstable: carbondata-master-spark-2.2 #41

2018-02-02 Thread Apache Jenkins Server 
See

Jenkins build became unstable: carbondata-master-spark-2.2 » Apache CarbonData :: Spark Common Test #41

2018-02-02 Thread Apache Jenkins Server 
See

Jenkins build is back to stable : carbondata-master-spark-2.1 » Apache CarbonData :: Hadoop #2004

2018-02-02 Thread Apache Jenkins Server 
See

Jenkins build is back to stable : carbondata-master-spark-2.1 #2004

2018-02-02 Thread Apache Jenkins Server 
See

Jenkins build is back to stable : carbondata-master-spark-2.1 » Apache CarbonData :: Processing #2004

2018-02-02 Thread Apache Jenkins Server 
See

carbondata git commit: [CARBONDATA-2120]Fixed is null filter issue

2018-02-02 Thread ravipesala 
Repository: carbondata
Updated Branches:
  refs/heads/master 27ec6515a -> 6c097cbf3


[CARBONDATA-2120]Fixed is null filter issue

Problem: Is null filter is failing for numeric data type(No dictionary column).

Root cause: Min max calculation is wrong when no dictionary column is not the 
first column.

As it is not the first column null value can come in between and min max for 
null value is getting updated only when first row is null

Solution: Update the min max in all the case when value is null or not null for 
all type

This closes #1912


Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/6c097cbf
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/6c097cbf
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/6c097cbf

Branch: refs/heads/master
Commit: 6c097cbf310e8d2199e57cde4fcc417122a8a1ca
Parents: 27ec651
Author: kumarvishal 
Authored: Fri Feb 2 17:53:57 2018 +0530
Committer: ravipesala 
Committed: Sat Feb 3 00:23:39 2018 +0530

--
 .../page/statistics/LVStringStatsCollector.java | 28 -
 .../core/util/path/CarbonTablePath.java |  2 +-
 .../src/test/resources/newsample.csv|  7 +
 .../testsuite/filterexpr/TestIsNullFilter.scala | 32 
 4 files changed, 53 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/carbondata/blob/6c097cbf/core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/LVStringStatsCollector.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/LVStringStatsCollector.java
 
b/core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/LVStringStatsCollector.java
index 61acec9..23795c5 100644
--- 
a/core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/LVStringStatsCollector.java
+++ 
b/core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/LVStringStatsCollector.java
@@ -73,28 +73,26 @@ public class LVStringStatsCollector implements 
ColumnPageStatsCollector {
   @Override
   public void update(byte[] value) {
 // input value is LV encoded
+byte[] newValue = null;
 assert (value.length >= 2);
 if (value.length == 2) {
   assert (value[0] == 0 && value[1] == 0);
-  if (min == null && max == null) {
-min = new byte[0];
-max = new byte[0];
-  }
-  return;
+  newValue = new byte[0];
+} else {
+  int length = (value[0] << 8) + (value[1] & 0xff);
+  assert (length > 0);
+  newValue = new byte[value.length - 2];
+  System.arraycopy(value, 2, newValue, 0, newValue.length);
 }
-int length = (value[0] << 8) + (value[1] & 0xff);
-assert (length > 0);
-byte[] v = new byte[value.length - 2];
-System.arraycopy(value, 2, v, 0, v.length);
 if (min == null && max == null) {
-  min = v;
-  max = v;
+  min = newValue;
+  max = newValue;
 } else {
-  if (ByteUtil.UnsafeComparer.INSTANCE.compareTo(min, v) > 0) {
-min = v;
+  if (ByteUtil.UnsafeComparer.INSTANCE.compareTo(min, newValue) > 0) {
+min = newValue;
   }
-  if (ByteUtil.UnsafeComparer.INSTANCE.compareTo(max, v) < 0) {
-max = v;
+  if (ByteUtil.UnsafeComparer.INSTANCE.compareTo(max, newValue) < 0) {
+max = newValue;
   }
 }
   }

http://git-wip-us.apache.org/repos/asf/carbondata/blob/6c097cbf/core/src/main/java/org/apache/carbondata/core/util/path/CarbonTablePath.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/util/path/CarbonTablePath.java 
b/core/src/main/java/org/apache/carbondata/core/util/path/CarbonTablePath.java
index d8c64c4..5a63d2f 100644
--- 
a/core/src/main/java/org/apache/carbondata/core/util/path/CarbonTablePath.java
+++ 
b/core/src/main/java/org/apache/carbondata/core/util/path/CarbonTablePath.java
@@ -75,7 +75,7 @@ public class CarbonTablePath extends Path {
* @param carbonFilePath
*/
   public static String getFolderContainingFile(String carbonFilePath) {
-return carbonFilePath.substring(0, 
carbonFilePath.lastIndexOf(File.separator));
+return carbonFilePath.substring(0, carbonFilePath.lastIndexOf('/'));
   }
 
   /**

http://git-wip-us.apache.org/repos/asf/carbondata/blob/6c097cbf/integration/spark-common-test/src/test/resources/newsample.csv
--
diff --git a/integration/spark-common-test/src/test/resources/newsample.csv 
b/integration/spark-common-test/src/test/resources/newsample.csv
new file mode 100644
index 000..38cd3dd
--- /dev/null
+++

carbondata git commit: [CARBONDATA-2108]Updated unsafe sort memory configuration

2018-02-02 Thread ravipesala 
Repository: carbondata
Updated Branches:
  refs/heads/master 2610a6091 -> 27ec6515a


[CARBONDATA-2108]Updated unsafe sort memory configuration

Deprecated old property: sort.inmemory.size.inmb
Added new property: carbon.sort.storage.inmemory.size.inmb,
If user has configured old property then internally it will be converted to new 
property
for ex: If user has configured sort.inmemory.size.inmb then 20% memory will be 
used as working memory and rest for storage memory

This closes #1896


Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/27ec6515
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/27ec6515
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/27ec6515

Branch: refs/heads/master
Commit: 27ec6515a143dc3b697ac914bfcd4cfe10a49e17
Parents: 2610a60
Author: kumarvishal 
Authored: Wed Jan 31 18:43:02 2018 +0530
Committer: ravipesala 
Committed: Fri Feb 2 23:18:21 2018 +0530

--
 .../core/constants/CarbonCommonConstants.java   |  5 +
 .../core/memory/UnsafeMemoryManager.java|  2 +-
 .../core/memory/UnsafeSortMemoryManager.java|  6 +-
 .../carbondata/core/util/CarbonProperties.java  | 99 
 4 files changed, 108 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/carbondata/blob/27ec6515/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
 
b/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
index 87eec8a..8480758 100644
--- 
a/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
+++ 
b/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
@@ -1585,6 +1585,11 @@ public final class CarbonCommonConstants {
 
   public static final String 
CARBON_ENABLE_PAGE_LEVEL_READER_IN_COMPACTION_DEFAULT = "true";
 
+  @CarbonProperty
+  public static final String IN_MEMORY_STORAGE_FOR_SORTED_DATA_IN_MB =
+  "carbon.sort.storage.inmemory.size.inmb";
+  public static final String IN_MEMORY_STORAGE_FOR_SORTED_DATA_IN_MB_DEFAULT = 
"512";
+
   private CarbonCommonConstants() {
   }
 }

http://git-wip-us.apache.org/repos/asf/carbondata/blob/27ec6515/core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java 
b/core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java
index 4222e14..d3b9b48 100644
--- 
a/core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java
+++ 
b/core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java
@@ -47,7 +47,7 @@ public class UnsafeMemoryManager {
   .getProperty(CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB,
   CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT));
 } catch (Exception e) {
-  size = 
Long.parseLong(CarbonCommonConstants.IN_MEMORY_FOR_SORT_DATA_IN_MB_DEFAULT);
+  size = 
Long.parseLong(CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT);
   LOGGER.info("Wrong memory size given, "
   + "so setting default value to " + size);
 }

http://git-wip-us.apache.org/repos/asf/carbondata/blob/27ec6515/core/src/main/java/org/apache/carbondata/core/memory/UnsafeSortMemoryManager.java
--
diff --git 
a/core/src/main/java/org/apache/carbondata/core/memory/UnsafeSortMemoryManager.java
 
b/core/src/main/java/org/apache/carbondata/core/memory/UnsafeSortMemoryManager.java
index c63b320..67bb6cc 100644
--- 
a/core/src/main/java/org/apache/carbondata/core/memory/UnsafeSortMemoryManager.java
+++ 
b/core/src/main/java/org/apache/carbondata/core/memory/UnsafeSortMemoryManager.java
@@ -75,10 +75,10 @@ public class UnsafeSortMemoryManager {
 long size;
 try {
   size = Long.parseLong(CarbonProperties.getInstance()
-  .getProperty(CarbonCommonConstants.IN_MEMORY_FOR_SORT_DATA_IN_MB,
-  CarbonCommonConstants.IN_MEMORY_FOR_SORT_DATA_IN_MB_DEFAULT));
+  
.getProperty(CarbonCommonConstants.IN_MEMORY_STORAGE_FOR_SORTED_DATA_IN_MB,
+  
CarbonCommonConstants.IN_MEMORY_STORAGE_FOR_SORTED_DATA_IN_MB_DEFAULT));
 } catch (Exception e) {
-  size = 
Long.parseLong(CarbonCommonConstants.IN_MEMORY_FOR_SORT_DATA_IN_MB_DEFAULT);
+  size = 
Long.parseLong(CarbonCommonConstants.IN_MEMORY_STORAGE_FOR_SORTED_DATA_IN_MB_DEFAULT);
   LOGGER.info("Wrong memory size

Jenkins build is back to stable : carbondata-master-spark-2.2 » Apache CarbonData :: Spark2 #39

2018-02-02 Thread Apache Jenkins Server 
See

Jenkins build became unstable: carbondata-master-spark-2.1 » Apache CarbonData :: Processing #2003

2018-02-02 Thread Apache Jenkins Server 
See

Jenkins build became unstable: carbondata-master-spark-2.1 » Apache CarbonData :: Hadoop #2003

2018-02-02 Thread Apache Jenkins Server 
See

Jenkins build became unstable: carbondata-master-spark-2.1 #2003

2018-02-02 Thread Apache Jenkins Server 
See

carbondata git commit: [CARBONDATA-1918] Incorrect data is displayed when String is updated using Sentences

2018-02-02 Thread kumarvishal09 
Repository: carbondata
Updated Branches:
  refs/heads/master a9a0201b4 -> 2610a6091


[CARBONDATA-1918] Incorrect data is displayed when String is updated using 
Sentences

Incorrect data is displayed when updating a String column using Sentences UDF. 
Sentences UDF will give us a Array, When updating string with array, wrong data 
is getting updated. Therefore, we have to check for the supported type before 
updating.

This closes  #1704


Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/2610a609
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/2610a609
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/2610a609

Branch: refs/heads/master
Commit: 2610a6091623c271552b7a69d402dded79ba3517
Parents: a9a0201
Author: dhatchayani 
Authored: Wed Dec 20 18:16:10 2017 +0530
Committer: kumarvishal 
Committed: Fri Feb 2 21:22:30 2018 +0530

--
 .../sdv/generated/DataLoadingIUDTestCase.scala  |  8 
 .../testsuite/iud/UpdateCarbonTableTestCase.scala   | 13 +
 .../mutation/CarbonProjectForUpdateCommand.scala| 16 
 3 files changed, 33 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/carbondata/blob/2610a609/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/DataLoadingIUDTestCase.scala
--
diff --git 
a/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/DataLoadingIUDTestCase.scala
 
b/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/DataLoadingIUDTestCase.scala
index b4459ab..4c232be 100644
--- 
a/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/DataLoadingIUDTestCase.scala
+++ 
b/integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/DataLoadingIUDTestCase.scala
@@ -1858,13 +1858,13 @@ ignore("IUD-01-01-01_040-23", Include) {

 
 //Check for updating carbon table set column value to a value returned by 
split function
+//Split will give us array value
 test("IUD-01-01-01_040-25", Include) {
sql(s"""create table if not exists default.t_carbn01 (Active_status 
String,Item_type_cd INT,Qty_day_avg INT,Qty_total INT,Sell_price 
BIGINT,Sell_pricep DOUBLE,Discount_price DOUBLE,Profit DECIMAL(3,2),Item_code 
String,Item_name String,Outlet_name String,Update_time TIMESTAMP,Create_date 
String)STORED BY 'org.apache.carbondata.format'""").collect
  sql(s"""insert into default.t_carbn01  select * from 
default.t_carbn01b""").collect
- sql(s"""update default.t_carbn01  set (active_status)= (split('t','a')) 
""").collect
-  checkAnswer(s""" select active_status from default.t_carbn01  group by 
active_status """,
-Seq(Row("t\\")), "DataLoadingIUDTestCase_IUD-01-01-01_040-25")
-   sql(s"""drop table default.t_carbn01  """).collect
+ intercept[Exception] {
+   sql(s"""update default.t_carbn01  set (active_status)= (split('t','a')) 
""").collect
+ }
 }

 

http://git-wip-us.apache.org/repos/asf/carbondata/blob/2610a609/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
--
diff --git 
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
index cf4fc07..98c9a16 100644
--- 
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
+++ 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
@@ -691,6 +691,19 @@ class UpdateCarbonTableTestCase extends QueryTest with 
BeforeAndAfterAll {
  CarbonCommonConstants.FILE_SEPARATOR + "Part0")
 assert(f.list().length == 2)
   }
+  test("test sentences func in update statement") {
+sql("drop table if exists senten")
+sql("create table senten(name string, comment string) stored by 
'carbondata'")
+sql("insert into senten select 'aaa','comment for aaa'")
+sql("insert into senten select 'bbb','comment for bbb'")
+sql("select * from senten").show()
+val errorMessage = intercept[Exception] {
+  sql("update senten set(comment)=(sentences('Hello there! How are 
you?'))").show()
+}.getMessage
+errorMessage
+  .contains("Unsupported data type: Array")
+sql("drop table if exists senten")
+  }
 
   override def afterAll {

Jenkins build became unstable: carbondata-master-spark-2.2 #38

2018-02-02 Thread Apache Jenkins Server 
See

Jenkins build became unstable: carbondata-master-spark-2.2 » Apache CarbonData :: Spark2 #38

2018-02-02 Thread Apache Jenkins Server 
See

carbondata git commit: [CARBONDATA-2082] Timeseries pre-aggregate table should support the blank space

2018-02-02 Thread kumarvishal09 
Repository: carbondata
Updated Branches:
  refs/heads/master d3b228fb8 -> a9a0201b4


[CARBONDATA-2082] Timeseries pre-aggregate table should support the blank space

Timeseries pre-aggregate table should support the blank space, 
including:event_time,different franularity

This closes  #1902


Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/a9a0201b
Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/a9a0201b
Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/a9a0201b

Branch: refs/heads/master
Commit: a9a0201b468505c79d1881607fb0673ee588d85a
Parents: d3b228f
Author: xubo245 <601450...@qq.com>
Authored: Thu Feb 1 15:32:36 2018 +0800
Committer: kumarvishal 
Committed: Fri Feb 2 18:38:44 2018 +0530

--
 .../timeseries/TestTimeSeriesCreateTable.scala  | 76 
 .../datamap/CarbonCreateDataMapCommand.scala| 17 +++--
 .../command/timeseries/TimeSeriesUtil.scala | 11 ++-
 3 files changed, 92 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/carbondata/blob/a9a0201b/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala
--
diff --git 
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala
 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala
index b63fd53..f3bbcaf 100644
--- 
a/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala
+++ 
b/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeSeriesCreateTable.scala
@@ -368,6 +368,82 @@ class TestTimeSeriesCreateTable extends QueryTest with 
BeforeAndAfterAll {
 assert(e.getMessage.contains("identifier matching regex"))
   }
 
+  test("test timeseries create table 33: support event_time and granularity 
key with space") {
+sql("DROP DATAMAP IF EXISTS agg1_month ON TABLE maintable")
+sql(
+  s"""CREATE DATAMAP agg1_month ON TABLE mainTable
+ |USING '$timeSeries'
+ |DMPROPERTIES (
+ |   ' event_time '='dataTime',
+ |   ' MONTH_GRANULARITY '='1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+""".stripMargin)
+checkExistence(sql("SHOW DATAMAP ON TABLE maintable"), true, 
"maintable_agg1_month")
+sql("DROP DATAMAP IF EXISTS agg1_month ON TABLE maintable")
+  }
+
+
+  test("test timeseries create table 34: support event_time value with space") 
{
+sql("DROP DATAMAP IF EXISTS agg1_month ON TABLE maintable")
+sql(
+  s"""CREATE DATAMAP agg1_month ON TABLE mainTable
+ |USING '$timeSeries'
+ |DMPROPERTIES (
+ |   'event_time '=' dataTime',
+ |   'MONTH_GRANULARITY '='1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+""".stripMargin)
+checkExistence(sql("SHOW DATAMAP ON TABLE maintable"), true, 
"maintable_agg1_month")
+sql("DROP DATAMAP IF EXISTS agg1_month ON TABLE maintable")
+  }
+
+  test("test timeseries create table 35: support granularity value with 
space") {
+sql("DROP DATAMAP IF EXISTS agg1_month ON TABLE maintable")
+sql(
+  s"""CREATE DATAMAP agg1_month ON TABLE mainTable
+ |USING '$timeSeries'
+ |DMPROPERTIES (
+ |   'event_time '='dataTime',
+ |   'MONTH_GRANULARITY '=' 1')
+ |AS SELECT dataTime, SUM(age) FROM mainTable
+ |GROUP BY dataTime
+""".stripMargin)
+checkExistence(sql("SHOW DATAMAP ON TABLE maintable"), true, 
"maintable_agg1_month")
+sql("DROP DATAMAP IF EXISTS agg1_month ON TABLE maintable")
+  }
+
+  test("test timeseries create table 36: support event_time and granularity 
value with space") {
+sql("DROP DATAMAP IF EXISTS agg1_month ON TABLE maintable")
+sql(
+  s"""
+ | CREATE DATAMAP agg1_month ON TABLE mainTable
+ | USING '$timeSeries'
+ | DMPROPERTIES (
+ |   'EVENT_TIME'='dataTime   ',
+ |   'MONTH_GRANULARITY'=' 1  ')
+ | AS SELECT dataTime, SUM(age) FROM mainTable
+ | GROUP BY dataTime
+""".stripMargin)
+checkExistence(sql("SHOW DATAMAP ON TABLE maintable"), true, 
"maintable_agg1_month")
+  }
+
+  test("test timeseries create table 37:  unsupport event_time error value") {
+sql("DROP DATAMAP IF EXISTS agg1_month ON TABLE maintable")
+intercept[NullPointerException] {
+  sql(
+s"""CREATE DATAMAP

Jenkins build is back to stable : carbondata-master-spark-2.2 » Apache CarbonData :: Spark2 #37

2018-02-02 Thread Apache Jenkins Server 
See

Jenkins build is back to stable : carbondata-master-spark-2.2 #37

2018-02-02 Thread Apache Jenkins Server 
See

Jenkins build is back to stable : carbondata-master-spark-2.2 » Apache CarbonData :: Spark Common Test #37

2018-02-02 Thread Apache Jenkins Server 
See