[GitHub] carbondata issue #2083: [CARBONDATA-2269]Support Query On PreAggregate table...

[kumarvishal09]

Github user kumarvishal09 commented on the issue:

https://github.com/apache/carbondata/pull/2083
  
retest this please 


---

[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1853
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4679/



---

[GitHub] carbondata issue #2106: [CARBONDATA-2291] Added datamap status and refresh c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2106
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4187/



---

[GitHub] carbondata issue #2114: [CARBONDATA-2295] Added property for UnsafeMemory co...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2114
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4680/



---

[GitHub] carbondata issue #2114: [CARBONDATA-2295] Added property for UnsafeMemory co...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2114
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3453/



---

[GitHub] carbondata issue #2083: [CARBONDATA-2269]Support Query On PreAggregate table...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2083
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4186/



---

[GitHub] carbondata pull request #71: [CARBONDATA-155] Code refactor to avoid the Typ...

[mohammadshahidkhan]

Github user mohammadshahidkhan closed the pull request at:

https://github.com/apache/carbondata/pull/71


---

[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1853
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3452/



---

[GitHub] carbondata pull request #2114: [CARBONDATA-2295] Added property for UnsafeMe...

[bhavya411]

Github user bhavya411 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2114#discussion_r178230596
  
--- Diff: 
integration/presto/src/main/java/org/apache/carbondata/presto/impl/CarbonTableReader.java
 ---
@@ -366,7 +368,10 @@ public TBase create() {
   public List getInputSplits2(CarbonTableCacheModel 
tableCacheModel,
  Expression filters)  {
 List result = new ArrayList<>();
-
+if(config.getUnsafeMemoryInMb() != null) {
+  
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB,
--- End diff --

Done


---

[GitHub] carbondata issue #2106: [CARBONDATA-2291] Added datamap status and refresh c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2106
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4678/



---

[GitHub] carbondata issue #2083: [CARBONDATA-2269]Support Query On PreAggregate table...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2083
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4677/



---

[GitHub] carbondata issue #2106: [CARBONDATA-2291] Added datamap status and refresh c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2106
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3451/



---

[GitHub] carbondata pull request #2114: [CARBONDATA-2295] Added property for UnsafeMe...

[chenliang613]

Github user chenliang613 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2114#discussion_r178229105
  
--- Diff: 
integration/presto/src/main/java/org/apache/carbondata/presto/impl/CarbonTableReader.java
 ---
@@ -366,7 +368,10 @@ public TBase create() {
   public List getInputSplits2(CarbonTableCacheModel 
tableCacheModel,
  Expression filters)  {
 List result = new ArrayList<>();
-
+if(config.getUnsafeMemoryInMb() != null) {
+  
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB,
--- End diff --

please add this configuration description to presto/README.md


---

[GitHub] carbondata issue #2111: [CARBONDATA-2292] Fix the problem of different modul...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2111
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4185/



---

[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/1853
  
retest this please


---

[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/1853
  
retest sdv please


---

[GitHub] carbondata issue #2083: [CARBONDATA-2269]Support Query On PreAggregate table...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2083
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3450/



---

[GitHub] carbondata pull request #2092: [CARBONDATA-2270] Write segment file in loadi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2092


---

[GitHub] carbondata pull request #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/1857


---

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
LGTM. SDV failure is not due to this PR, will fix it later


---

[GitHub] carbondata issue #2111: [CARBONDATA-2292] Fix the problem of different modul...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2111
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3448/



---

[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1853
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4184/



---

[GitHub] carbondata issue #2111: [CARBONDATA-2292] Fix the problem of different modul...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2111
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4675/



---

[GitHub] carbondata pull request #2109: [CARBONDATA-2294] Partition preaggregate supp...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2109#discussion_r178223172
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateTableHelper.scala
 ---
@@ -59,8 +60,22 @@ case class PreAggregateTableHelper(
 val df = sparkSession.sql(updatedQuery)
 val fieldRelationMap = 
PreAggregateUtil.validateActualSelectPlanAndGetAttributes(
   df.logicalPlan, queryString)
+val partitionInfo = parentTable.getPartitionInfo
 val fields = fieldRelationMap.keySet.toSeq
 val tableProperties = mutable.Map[String, String]()
+val childPartitionColumns = if (partitionInfo != null &&
+  partitionInfo.getPartitionType == 
PartitionType.NATIVE_HIVE) {
+  val parentPartitionColumns = 
partitionInfo.getColumnSchemaList.asScala.map(_.getColumnName)
+  PreAggregateUtil
+.extractPartitionInfoForAggregateTable(df.logicalPlan, 
parentPartitionColumns)
+} else {Seq()}
+val partitionerFields = fieldRelationMap.collect {
+  case (field, dataMapField) if childPartitionColumns
+.exists(childCol =>
+  
childCol.equals(dataMapField.columnTableRelationList.get.head.parentColumnName) 
&&
+  dataMapField.aggregateFunction.isEmpty) =>
+  PartitionerField(field.name.get, field.dataType, field.columnComment)
+}.toSeq
--- End diff --

PLease add some tests to make sure the order of partition columns are same 
as parent partition columns


---

[GitHub] carbondata pull request #2109: [CARBONDATA-2294] Partition preaggregate supp...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2109#discussion_r178222846
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateTableHelper.scala
 ---
@@ -59,8 +60,22 @@ case class PreAggregateTableHelper(
 val df = sparkSession.sql(updatedQuery)
 val fieldRelationMap = 
PreAggregateUtil.validateActualSelectPlanAndGetAttributes(
   df.logicalPlan, queryString)
+val partitionInfo = parentTable.getPartitionInfo
 val fields = fieldRelationMap.keySet.toSeq
 val tableProperties = mutable.Map[String, String]()
+val childPartitionColumns = if (partitionInfo != null &&
+  partitionInfo.getPartitionType == 
PartitionType.NATIVE_HIVE) {
+  val parentPartitionColumns = 
partitionInfo.getColumnSchemaList.asScala.map(_.getColumnName)
+  PreAggregateUtil
+.extractPartitionInfoForAggregateTable(df.logicalPlan, 
parentPartitionColumns)
+} else {Seq()}
+val partitionerFields = fieldRelationMap.collect {
+  case (field, dataMapField) if childPartitionColumns
+.exists(childCol =>
+  
childCol.equals(dataMapField.columnTableRelationList.get.head.parentColumnName) 
&&
+  dataMapField.aggregateFunction.isEmpty) =>
+  PartitionerField(field.name.get, field.dataType, field.columnComment)
+}.toSeq
--- End diff --

The order of partition columns is important. Please make sure that parent 
partition column order and child partition column order is same.


---

[GitHub] carbondata pull request #2109: [CARBONDATA-2294] Partition preaggregate supp...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2109#discussion_r178222584
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateTableHelper.scala
 ---
@@ -59,8 +60,22 @@ case class PreAggregateTableHelper(
 val df = sparkSession.sql(updatedQuery)
 val fieldRelationMap = 
PreAggregateUtil.validateActualSelectPlanAndGetAttributes(
   df.logicalPlan, queryString)
+val partitionInfo = parentTable.getPartitionInfo
 val fields = fieldRelationMap.keySet.toSeq
 val tableProperties = mutable.Map[String, String]()
+val childPartitionColumns = if (partitionInfo != null &&
+  partitionInfo.getPartitionType == 
PartitionType.NATIVE_HIVE) {
+  val parentPartitionColumns = 
partitionInfo.getColumnSchemaList.asScala.map(_.getColumnName)
+  PreAggregateUtil
+.extractPartitionInfoForAggregateTable(df.logicalPlan, 
parentPartitionColumns)
+} else {Seq()}
--- End diff --

format it properly


---

[GitHub] carbondata pull request #2109: [CARBONDATA-2294] Partition preaggregate supp...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2109#discussion_r178222428
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateTableHelper.scala
 ---
@@ -59,8 +60,22 @@ case class PreAggregateTableHelper(
 val df = sparkSession.sql(updatedQuery)
 val fieldRelationMap = 
PreAggregateUtil.validateActualSelectPlanAndGetAttributes(
   df.logicalPlan, queryString)
+val partitionInfo = parentTable.getPartitionInfo
 val fields = fieldRelationMap.keySet.toSeq
 val tableProperties = mutable.Map[String, String]()
+val childPartitionColumns = if (partitionInfo != null &&
+  partitionInfo.getPartitionType == 
PartitionType.NATIVE_HIVE) {
+  val parentPartitionColumns = 
partitionInfo.getColumnSchemaList.asScala.map(_.getColumnName)
+  PreAggregateUtil
+.extractPartitionInfoForAggregateTable(df.logicalPlan, 
parentPartitionColumns)
--- End diff --

Why again require to extract the columns? Can't you get from 
fieldRelationMap


---

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4183/



---

[GitHub] carbondata issue #2083: [CARBONDATA-2269]Support Query On PreAggregate table...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2083
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3449/



---

[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1853
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3447/



---

[GitHub] carbondata issue #2083: [CARBONDATA-2269]Support Query On PreAggregate table...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2083
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4676/



---

[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1853
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4674/



---

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3446/



---

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4673/



---

[GitHub] carbondata pull request #2109: [CARBONDATA-2294] Partition preaggregate supp...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2109#discussion_r178219637
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateTableHelper.scala
 ---
@@ -59,8 +60,22 @@ case class PreAggregateTableHelper(
 val df = sparkSession.sql(updatedQuery)
 val fieldRelationMap = 
PreAggregateUtil.validateActualSelectPlanAndGetAttributes(
   df.logicalPlan, queryString)
+val partitionInfo = parentTable.getPartitionInfo
 val fields = fieldRelationMap.keySet.toSeq
 val tableProperties = mutable.Map[String, String]()
+val childPartitionColumns = if (partitionInfo != null &&
--- End diff --

just use parentTable.isHivePartitionTable


---

[GitHub] carbondata pull request #2109: [CARBONDATA-2294] Partition preaggregate supp...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2109#discussion_r178219012
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala
 ---
@@ -18,6 +18,7 @@
 package org.apache.spark.sql.execution.command.partition
 
 import java.util
+import java.util.UUID
--- End diff --

Do you allow dropping of partitions directly on aggregate table? I mean not 
through parent table.


---

[GitHub] carbondata pull request #2111: [CARBONDATA-2292] Fix the problem of differen...

[xubo245]

Github user xubo245 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2111#discussion_r178218492
  
--- Diff: integration/presto/pom.xml ---
@@ -436,13 +436,13 @@
   org.apache.spark
   spark-network-common_2.11
--- End diff --

ok,done


---

[GitHub] carbondata pull request #2111: [CARBONDATA-2292] Fix the problem of differen...

[xubo245]

Github user xubo245 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2111#discussion_r178218505
  
--- Diff: integration/presto/pom.xml ---
@@ -436,13 +436,13 @@
   org.apache.spark
   spark-network-common_2.11
   test
-  2.1.0
+  ${spark.version}
 
 
   org.apache.spark
   spark-core_2.11
--- End diff --

ok,done


---

[GitHub] carbondata pull request #2109: [CARBONDATA-2294] Partition preaggregate supp...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2109#discussion_r178218263
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
 ---
@@ -149,7 +143,16 @@ case class CarbonLoadDataCommand(
 val LOGGER: LogService = 
LogServiceFactory.getLogService(this.getClass.getCanonicalName)
 val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
 carbonProperty.addProperty("zookeeper.enable.lock", "false")
-
+currPartitions = if (table.isHivePartitionTable) {
--- End diff --

This is a metastore call, not supposed keep under processData


---

[GitHub] carbondata pull request #2109: [CARBONDATA-2294] Partition preaggregate supp...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2109#discussion_r178217935
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/AggregateDataMapCompactor.scala
 ---
@@ -47,7 +47,15 @@ class AggregateDataMapCompactor(carbonLoadModel: 
CarbonLoadModel,
   override def executeCompaction(): Unit = {
 val carbonTable = 
carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable
 val loadMetaDataDetails = identifySegmentsToBeMerged()
-val segments = loadMetaDataDetails.asScala.map(_.getLoadName)
+// If segmentFile name is specified in load details then segment is 
for partition table
+// therefore the segment file name should be 
loadName#segmentFileName.segment
+val segments = loadMetaDataDetails.asScala.map {
+  loadDetail => if (loadDetail.getSegmentFile != null) {
+loadDetail.getLoadName + "#" + loadDetail.getSegmentFile
+  } else {
+loadDetail.getLoadName
+  }
--- End diff --

Just use new Segment(loadDetail.getLoadName, 
loadDetail.getSegmentFile).toString instead of doing this


---

[GitHub] carbondata issue #2106: [CARBONDATA-2291] Added datamap status and refresh c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2106
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4182/



---

[GitHub] carbondata pull request #2109: [CARBONDATA-2294] Partition preaggregate supp...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2109#discussion_r178217187
  
--- Diff: 
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/standardpartition/StandardPartitionWithPreaggregateTestCase.scala
 ---
@@ -0,0 +1,440 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain id copy of the License at
+*
+*http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.carbondata.spark.testsuite.standardpartition
+
+import java.io.{File, FileWriter}
+
+import org.apache.commons.io.FileUtils
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.hive.CarbonRelation
+import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, CarbonEnv, 
Row}
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class StandardPartitionWithPreaggregateTestCase extends QueryTest with 
BeforeAndAfterAll {
--- End diff --

Please handle and add overwrite test cases for partition with pre-aggregate 
scenario


---

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
retest this please


---

[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...

[xubo245]

Github user xubo245 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1856#discussion_r178216676
  
--- Diff: 
integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/timeseries/TestTimeseriesDataLoad.scala
 ---
@@ -167,59 +170,60 @@ class TestTimeseriesDataLoad extends QueryTest with 
BeforeAndAfterAll {
""".stripMargin)
 
   }
-  test("test Year level timeseries data validation1 ") {
-checkAnswer( sql("select count(*) from table_03_ag1_year"),
+
+  test("test timeseries table selection 1: year level timeseries data 
validation1 ") {
+checkAnswer(sql("SELECT COUNT(*) FROM table_03_ag1_year"),
   Seq(Row(4)))
   }
 
-  test("test month level timeseries data validation1 ") {
-checkAnswer( sql("select count(*) from table_03_ag1_month"),
+  test("test timeseries table selection 2: month level timeseries data 
validation1 ") {
+checkAnswer(sql("SELECT COUNT(*) FROM table_03_ag1_month"),
   Seq(Row(4)))
   }
 
-  test("test day level timeseries data validation1 ") {
-checkAnswer( sql("select count(*) from table_03_ag1_day"),
+  test("test timeseries table selection 3: day level timeseries data 
validation1 ") {
+checkAnswer(sql("SELECT COUNT(*) FROM table_03_ag1_day"),
--- End diff --

I confirmed with jacky and raghu


---

[GitHub] carbondata issue #2106: [CARBONDATA-2291] Added datamap status and refresh c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2106
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4672/



---

[GitHub] carbondata issue #2106: [CARBONDATA-2291] Added datamap status and refresh c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2106
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3445/



---

[GitHub] carbondata issue #2112: [CARBONDATA-2270] Fixed tests on PR 2092

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2112
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3444/



---

[GitHub] carbondata issue #2112: [CARBONDATA-2270] Fixed tests on PR 2092

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2112
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4670/



---

[GitHub] carbondata pull request #2106: [CARBONDATA-2291] Added datamap status and re...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2106#discussion_r178210972
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala
 ---
@@ -185,6 +185,16 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser 
{
 CarbonDataMapShowCommand(tableIdent)
 }
 
+  /**
+   * The syntax of show datamap is used to show datamaps on the table
+   * REFRESH DATAMAP datamapname ON TABLE tableName
--- End diff --

ok


---

[GitHub] carbondata pull request #2106: [CARBONDATA-2291] Added datamap status and re...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2106#discussion_r178210981
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datamap/status/DiskBasedDataMapStatusProvider.java
 ---
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datamap.status;
+
+import java.io.*;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.fileoperations.AtomicFileOperations;
+import org.apache.carbondata.core.fileoperations.AtomicFileOperationsImpl;
+import org.apache.carbondata.core.fileoperations.FileWriteOperation;
+import org.apache.carbondata.core.locks.CarbonLockFactory;
+import org.apache.carbondata.core.locks.CarbonLockUtil;
+import org.apache.carbondata.core.locks.ICarbonLock;
+import org.apache.carbondata.core.locks.LockUsage;
+import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
+import org.apache.carbondata.core.util.CarbonProperties;
+import org.apache.carbondata.core.util.CarbonUtil;
+
+import com.google.gson.Gson;
+
+public class DiskBasedDataMapStatusProvider implements 
DataMapStatusStorageProvider {
+
+  private static final LogService LOG =
+  
LogServiceFactory.getLogService(DiskBasedDataMapStatusProvider.class.getName());
+
+  private static final String DATAMAP_STATUS_FILE = "datamapstatus";
+
+  @Override public DataMapStatusDetail[] getDataMapStatusDetails() throws 
IOException {
+String statusPath = 
CarbonProperties.getInstance().getSystemFolderLocation()
++ CarbonCommonConstants.FILE_SEPARATOR + DATAMAP_STATUS_FILE;
+Gson gsonObjectToRead = new Gson();
+DataInputStream dataInputStream = null;
+BufferedReader buffReader = null;
+InputStreamReader inStream = null;
+DataMapStatusDetail[] dataMapStatusDetails;
+try {
+  if (!FileFactory.isFileExist(statusPath)) {
+return new DataMapStatusDetail[0];
+  }
+  dataInputStream =
+  FileFactory.getDataInputStream(statusPath, 
FileFactory.getFileType(statusPath));
+  inStream = new InputStreamReader(dataInputStream,
+  Charset.forName(CarbonCommonConstants.DEFAULT_CHARSET));
+  buffReader = new BufferedReader(inStream);
+  dataMapStatusDetails = gsonObjectToRead.fromJson(buffReader, 
DataMapStatusDetail[].class);
+} catch (IOException e) {
+  LOG.error(e, "Failed to read datamap status");
+  throw e;
+} finally {
+  CarbonUtil.closeStreams(buffReader, inStream, dataInputStream);
+}
+
+// if dataMapStatusDetails is null, return empty array
+if (null == dataMapStatusDetails) {
+  return new DataMapStatusDetail[0];
+}
+
+return dataMapStatusDetails;
+  }
+
+  @Override
+  public void updateDataMapStatus(List dataMapSchemas, 
DataMapStatus dataMapStatus)
+  throws IOException {
+ICarbonLock carbonTableStatusLock = getDataMapStatusLock();
--- End diff --

ok


---

[GitHub] carbondata pull request #2106: [CARBONDATA-2291] Added datamap status and re...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2106#discussion_r178210924
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datamap/status/DiskBasedDataMapStatusProvider.java
 ---
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datamap.status;
+
+import java.io.*;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.fileoperations.AtomicFileOperations;
+import org.apache.carbondata.core.fileoperations.AtomicFileOperationsImpl;
+import org.apache.carbondata.core.fileoperations.FileWriteOperation;
+import org.apache.carbondata.core.locks.CarbonLockFactory;
+import org.apache.carbondata.core.locks.CarbonLockUtil;
+import org.apache.carbondata.core.locks.ICarbonLock;
+import org.apache.carbondata.core.locks.LockUsage;
+import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
+import org.apache.carbondata.core.util.CarbonProperties;
+import org.apache.carbondata.core.util.CarbonUtil;
+
+import com.google.gson.Gson;
+
+public class DiskBasedDataMapStatusProvider implements 
DataMapStatusStorageProvider {
--- End diff --

ok


---

[GitHub] carbondata pull request #2106: [CARBONDATA-2291] Added datamap status and re...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2106#discussion_r178210938
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datamap/status/DiskBasedDataMapStatusProvider.java
 ---
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datamap.status;
+
+import java.io.*;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.fileoperations.AtomicFileOperations;
+import org.apache.carbondata.core.fileoperations.AtomicFileOperationsImpl;
+import org.apache.carbondata.core.fileoperations.FileWriteOperation;
+import org.apache.carbondata.core.locks.CarbonLockFactory;
+import org.apache.carbondata.core.locks.CarbonLockUtil;
+import org.apache.carbondata.core.locks.ICarbonLock;
+import org.apache.carbondata.core.locks.LockUsage;
+import org.apache.carbondata.core.metadata.schema.table.DataMapSchema;
+import org.apache.carbondata.core.util.CarbonProperties;
+import org.apache.carbondata.core.util.CarbonUtil;
+
+import com.google.gson.Gson;
+
+public class DiskBasedDataMapStatusProvider implements 
DataMapStatusStorageProvider {
+
+  private static final LogService LOG =
+  
LogServiceFactory.getLogService(DiskBasedDataMapStatusProvider.class.getName());
+
+  private static final String DATAMAP_STATUS_FILE = "datamapstatus";
+
+  @Override public DataMapStatusDetail[] getDataMapStatusDetails() throws 
IOException {
--- End diff --

ok


---

[GitHub] carbondata issue #2114: [CARBONDATA-2295] Added property for UnsafeMemory co...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2114
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4180/



---

[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2113
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4179/



---

[GitHub] carbondata issue #2112: [CARBONDATA-2270] Fixed tests on PR 2092

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2112
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4178/



---

[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2109
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4177/



---

[GitHub] carbondata issue #2104: [CARBONDATA-2285] Spark integration code refactor

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2104
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4176/



---

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4175/



---

[GitHub] carbondata issue #2097: [CARBONDATA-2275]Query Failed for 0 byte deletedelta...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2097
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4669/



---

[GitHub] carbondata issue #2097: [CARBONDATA-2275]Query Failed for 0 byte deletedelta...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2097
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3443/



---

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4174/



---

[GitHub] carbondata issue #2097: [CARBONDATA-2275]Query Failed for 0 byte deletedelta...

[BJangir]

Github user BJangir commented on the issue:

https://github.com/apache/carbondata/pull/2097
  
retest this please


---

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4173/



---

[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1853
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4172/



---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4171/



---

[GitHub] carbondata issue #2114: [CARBONDATA-2295] Added property for UnsafeMemory co...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2114
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4668/



---

[GitHub] carbondata issue #2114: [CARBONDATA-2295] Added property for UnsafeMemory co...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2114
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3442/



---

[GitHub] carbondata issue #2112: [CARBONDATA-2270] Fixed tests on PR 2092

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2112
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4666/



---

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4170/



---

[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2113
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4667/



---

[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2113
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3441/



---

[GitHub] carbondata issue #2112: [CARBONDATA-2270] Fixed tests on PR 2092

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2112
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3440/



---

[GitHub] carbondata pull request #2114: [CARBONDATA-2295] Added property for UnsafeMe...

[bhavya411]

GitHub user bhavya411 opened a pull request:

https://github.com/apache/carbondata/pull/2114

[CARBONDATA-2295] Added property for UnsafeMemory configuration

Added property for setting up the unsafe memory limit in MB



 - [ X] Testing done
   Added a new property in the test cases to make sure they all run fine
Tested manually on the presto client as well




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bhavya411/incubator-carbondata CARBONDATA-2295

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2114.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2114


commit 9e44537b97340e71829f733d878e1dfab8affbdc
Author: Bhavya 
Date:   2018-03-29T15:03:34Z

Added property for UnsafeMemory configuration




---

[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...

[akashrn5]

Github user akashrn5 commented on the issue:

https://github.com/apache/carbondata/pull/2113
  
When the loading is triggered after creation of table and datamap , it 
failed with lock exception.
Analysis:
The index write location for the lucene is same, and to IndexWriter will 
take a lock file called write.lock in write location while writing the index 
files. In carbon loading the writer tasks are launched parallel and those many 
writers are opened,
Since the write.lock file is acquired by one writer, all other tasks will 
fail and dataloading will fail.

Solution
so in this PR, the write path can be configured on the fly, with task id 
and timestamp mapping.


Handling in reader part:
In query side, there was a single indexSearch object which does search on 
index file present in a single location.
Now with thsi dataload fix, there will be multiple directories which will 
have index files, and cannot give a single path to search object, as serch 
object is one.

Probable Solutions:
1. create those many number of search objects, as number of index file 
directories created in data load. and finally return all the final Blocklets.
2. you can have one searcher in one executor to search the local folder, 
this part im working how to do



---

[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2109
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4169/



---

[GitHub] carbondata pull request #2113: [WIP][LUCENE_DATAMAP]load issue in lucene dat...

[akashrn5]

GitHub user akashrn5 opened a pull request:

https://github.com/apache/carbondata/pull/2113

[WIP][LUCENE_DATAMAP]load issue in lucene datamap, make multiple directory 
based on taskId

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/akashrn5/incubator-carbondata lucene

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2113.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2113


commit 4453e9218d2997923fa1ab3d60e4cac74cc14dba
Author: akashrn5 
Date:   2018-03-29T14:29:36Z

load issue in lucene datamap, make multiple directory based on taskId




---

[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2109
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4665/



---

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4663/



---

[GitHub] carbondata pull request #2111: [CARBONDATA-2292] Fix the problem of differen...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2111#discussion_r178073457
  
--- Diff: integration/presto/pom.xml ---
@@ -436,13 +436,13 @@
   org.apache.spark
   spark-network-common_2.11
   test
-  2.1.0
+  ${spark.version}
 
 
   org.apache.spark
   spark-core_2.11
--- End diff --

use `${scala.binary.version}`


---

[GitHub] carbondata pull request #2111: [CARBONDATA-2292] Fix the problem of differen...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2111#discussion_r178073421
  
--- Diff: integration/presto/pom.xml ---
@@ -436,13 +436,13 @@
   org.apache.spark
   spark-network-common_2.11
--- End diff --

use `${scala.binary.version}`


---

[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2109
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3439/



---

[jira] [Resolved] (CARBONDATA-2285) spark integration code refactor

[Jacky Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacky Li resolved CARBONDATA-2285.
--
   Resolution: Fixed
Fix Version/s: 1.4.0

> spark integration code refactor
> ---
>
> Key: CARBONDATA-2285
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2285
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Mohammad Shahid Khan
>Assignee: Mohammad Shahid Khan
>Priority: Minor
> Fix For: 1.4.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Spark integration code refactor to improve extensible and re-usability



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #2104: [CARBONDATA-2285] Spark integration code refa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2104


---

[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2109
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4168/



---

[GitHub] carbondata issue #2104: [CARBONDATA-2285] Spark integration code refactor

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2104
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3438/



---

[GitHub] carbondata issue #2104: [CARBONDATA-2285] Spark integration code refactor

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2104
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4664/



---

[GitHub] carbondata issue #2111: [CARBONDATA-2292] Fix the problem of different modul...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2111
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4167/



---

[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1853
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3435/



---

[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1853
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4662/



---

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3437/



---

[GitHub] carbondata pull request #2104: [CARBONDATA-2285] Spark integration code refa...

[mohammadshahidkhan]

Github user mohammadshahidkhan commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2104#discussion_r178051841
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/hive/ICarbonSessionCatalog.scala
 ---
@@ -0,0 +1,67 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements.  See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License.  You may obtain a copy of the License at
+*
+*http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.spark.sql.hive
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.sql.{CarbonEnv, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, 
CatalogTablePartition}
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+import org.apache.carbondata.common.annotations.{InterfaceAudience, 
InterfaceStability}
+
+/**
+ * This interface defines those common api used by carbon for spark-2.1 
and spark-2.2 integration,
+ * but are not defined in SessionCatalog or HiveSessionCatalog to give 
contract to the
+ * Concrete implementation classes.
+ * For example CarbonSessionCatalog defined in 2.1 and 2.2.
+ *
+ */
+@InterfaceAudience.Internal
+@InterfaceStability.Stable
+trait ICarbonSessionCatalog {
--- End diff --

Fixed renames ICarbonSessionCatalog  to CarbonSessionCatalog 
and CarbonSessionCatalog  to ICarbonHiveSessionCatalog 


---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4661/



---

[jira] [Created] (CARBONDATA-2295) Add UNSAFE_WORKING_MEMORY_IN_MB as a configuration parameter in presto integration

[anubhav tarar (JIRA)]

anubhav tarar created CARBONDATA-2295:
-

 Summary: Add UNSAFE_WORKING_MEMORY_IN_MB  as a configuration 
parameter in presto integration
 Key: CARBONDATA-2295
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2295
 Project: CarbonData
  Issue Type: Improvement
  Components: presto-integration
Affects Versions: 1.4.0
Reporter: anubhav tarar


Add UNSAFE_WORKING_MEMORY_IN_MB as a configuration parameter for presto 
integration so we can tackle the unsafe memory related issues when querying 
carbon data from presto



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[kunal642]

Github user kunal642 commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
LGTM


---

[GitHub] carbondata pull request #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...

[xubo245]

Github user xubo245 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1857#discussion_r178048887
  
--- Diff: 
integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/preaggregate/TestPreAggregateIUDSuite.scala
 ---
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.integration.spark.testsuite.preaggregate
+
+import org.apache.spark.sql.Row
+import org.apache.spark.util.SparkUtil4Test
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.spark.util.SparkQueryTest
+
+class TestPreAggregateIUDSuite extends SparkQueryTest with 
BeforeAndAfterAll with BeforeAndAfterEach {
--- End diff --

OK, I find. and remove the test case: TestPreAggregateIUDSuite.scala


---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3434/



---

[jira] [Resolved] (CARBONDATA-1516) Support pre-aggregate tables and timeseries in carbondata

[Jacky Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacky Li resolved CARBONDATA-1516.
--
   Resolution: Fixed
Fix Version/s: 1.4.0

> Support pre-aggregate tables and timeseries in carbondata
> -
>
> Key: CARBONDATA-1516
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1516
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Ravindra Pesala
>Priority: Major
> Fix For: 1.4.0
>
> Attachments: CarbonData Pre-aggregation Table.pdf, CarbonData 
> Pre-aggregation Table_v1.1.pdf, CarbonData Pre-aggregation Table_v1.2.pdf, 
> CarbonData Pre-aggregation Table_v1.3.pdf
>
>
> Currently Carbondata has standard SQL capability on distributed data 
> sets.Carbondata should support pre-aggregating tables for timeseries and 
> improve query performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (CARBONDATA-2073) Add test cases for pre aggregate table

[Jacky Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacky Li resolved CARBONDATA-2073.
--
Resolution: Fixed

> Add test cases for pre aggregate table
> --
>
> Key: CARBONDATA-2073
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2073
> Project: CarbonData
>  Issue Type: Test
>  Components: test
>Affects Versions: 1.3.0
>Reporter: xubo245
>Assignee: xubo245
>Priority: Major
> Fix For: 1.4.0
>
>  Time Spent: 41h 50m
>  Remaining Estimate: 0h
>
> Add test cases for pre aggregate table, include timeseries and so on.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
LGTM


---

[GitHub] carbondata pull request #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/1856


---