date:20180123

[GitHub] carbondata pull request #1841: [CARBONDATA-2062] Configure the temp director...

2018-01-23 Thread sraghunandan

Github user sraghunandan commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1841#discussion_r163167471
  
--- Diff: 
integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
 ---
@@ -941,4 +943,39 @@ object CommonUtil {
 }
   }
 
+  def setTempStoreLocation(
+  index: Int,
+  carbonLoadModel: CarbonLoadModel,
+  isCompactionFlow: Boolean,
+  isAltPartitionFlow: Boolean) : Unit = {
+var storeLocation: String = null
+
+// this property is used to determine whether temp location for carbon 
is inside
+// container temp dir or is yarn application directory.
+val carbonUseLocalDir = CarbonProperties.getInstance()
+  .getProperty("carbon.use.local.dir", "false")
+
+if (carbonUseLocalDir.equalsIgnoreCase("true")) {
+
+  val storeLocations = Util.getConfiguredLocalDirs(SparkEnv.get.conf)
+  if (null != storeLocations && storeLocations.nonEmpty) {
+storeLocation = 
storeLocations(Random.nextInt(storeLocations.length))
+  }
+  if (storeLocation == null) {
+storeLocation = System.getProperty("java.io.tmpdir")
+  }
+} else {
+  storeLocation = System.getProperty("java.io.tmpdir")
+}
+storeLocation = storeLocation + '/' + "carbon" + System.nanoTime() + 
'_' + index
--- End diff --

handled


---

[GitHub] carbondata issue #1845: [CARBONDATA-2066] Add Local keyword when loading dat...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1845
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1817/



---

[GitHub] carbondata pull request #1846: [CARBONDATA-2067] Fix NPE exception in Stream...

2018-01-23 Thread QiangCai

GitHub user QiangCai opened a pull request:

https://github.com/apache/carbondata/pull/1846

[CARBONDATA-2067] Fix NPE exception in StreamHandoffRDD

To avoid NPE of carbon table in StreamHandoffRDD, add carbon table to cache 
in internalCompute method

 - [x] Any interfaces changed?
 no
 - [x] Any backward compatibility impacted?
 no
 - [x] Document update required?
 no
 - [x] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
   local mode doesn't have this issue
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [x] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
  small changes


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QiangCai/carbondata handoff_nullpoint

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1846.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1846


commit ca75a647434ebd190c502d8bc966257f7f86516d
Author: QiangCai 
Date:   2018-01-23T08:20:19Z

fix NPE exception in StreamHandoffRDD




---

[GitHub] carbondata issue #1841: [CARBONDATA-2062] Configure the temp directory to be...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1841
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3040/



---

[GitHub] carbondata issue #1845: [CARBONDATA-2066] Add Local keyword when loading dat...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1845
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3048/



---

[GitHub] carbondata issue #1845: [CARBONDATA-2066] Add Local keyword when loading dat...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1845
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3039/



---

[jira] [Assigned] (CARBONDATA-2067) Streaming hand off operation throw NullPointerException

2018-01-23 Thread QiangCai (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

QiangCai reassigned CARBONDATA-2067:


Assignee: QiangCai

> Streaming hand off operation throw NullPointerException
> ---
>
> Key: CARBONDATA-2067
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2067
> Project: CarbonData
>  Issue Type: Bug
>Reporter: QiangCai
>Assignee: QiangCai
>Priority: Major
>
> 18/01/23 16:01:10 ERROR CompactionResultSortProcessor: Executor task launch 
> worker for task 0 Compaction failed: null java.lang.NullPointerException at 
> org.apache.carbondata.processing.util.CarbonDataProcessorUtil.getLocalDataFolderLocation(CarbonDataProcessorUtil.java:152)
>  at 
> org.apache.carbondata.processing.merger.CompactionResultSortProcessor.initTempStoreLocation(CompactionResultSortProcessor.java:424)
>  at 
> org.apache.carbondata.processing.merger.CompactionResultSortProcessor.execute(CompactionResultSortProcessor.java:156)
>  at 
> org.apache.carbondata.streaming.StreamHandoffRDD.internalCompute(StreamHandoffRDD.scala:113)
>  at org.apache.carbondata.spark.rdd.CarbonRDD.compute(CarbonRDD.scala:60) at 
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at 
> org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at 
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at 
> org.apache.spark.scheduler.Task.run(Task.scala:108) at 
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (CARBONDATA-2067) Streaming hand off operation throw NullPointerException

2018-01-23 Thread QiangCai (JIRA)

QiangCai created CARBONDATA-2067:


 Summary: Streaming hand off operation throw NullPointerException
 Key: CARBONDATA-2067
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2067
 Project: CarbonData
  Issue Type: Bug
Reporter: QiangCai


18/01/23 16:01:10 ERROR CompactionResultSortProcessor: Executor task launch 
worker for task 0 Compaction failed: null java.lang.NullPointerException at 
org.apache.carbondata.processing.util.CarbonDataProcessorUtil.getLocalDataFolderLocation(CarbonDataProcessorUtil.java:152)
 at 
org.apache.carbondata.processing.merger.CompactionResultSortProcessor.initTempStoreLocation(CompactionResultSortProcessor.java:424)
 at 
org.apache.carbondata.processing.merger.CompactionResultSortProcessor.execute(CompactionResultSortProcessor.java:156)
 at 
org.apache.carbondata.streaming.StreamHandoffRDD.internalCompute(StreamHandoffRDD.scala:113)
 at org.apache.carbondata.spark.rdd.CarbonRDD.compute(CarbonRDD.scala:60) at 
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at 
org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at 
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at 
org.apache.spark.scheduler.Task.run(Task.scala:108) at 
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1840: [CARBONDATA-2054]Add an example: how to use CarbonDa...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1840
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1820/



---

[GitHub] carbondata pull request #1847: [CARBONDATA-2064] Add compaction listener

2018-01-23 Thread dhatchayani

GitHub user dhatchayani opened a pull request:

https://github.com/apache/carbondata/pull/1847

[CARBONDATA-2064] Add compaction listener

Added compaction listeners

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [x] Testing done
Manual Testing
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhatchayani/incubator-carbondata 
compactionListeners

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1847.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1847


commit 4b679aae54ee8fcd87bd666d0a8641ad1b50a0e3
Author: dhatchayani 
Date:   2018-01-23T09:56:26Z

[CARBONDATA-2064] Add compaction listener




---

[GitHub] carbondata issue #1840: [CARBONDATA-2054]Add an example: how to use CarbonDa...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1840
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3042/



---

[GitHub] carbondata issue #1847: [WIP][CARBONDATA-2064] Add compaction listener

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1847
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3044/



---

[GitHub] carbondata issue #1846: [CARBONDATA-2067] Fix NPE exception in StreamHandoff...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1846
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1819/



---

[GitHub] carbondata issue #1846: [CARBONDATA-2067] Fix NPE exception in StreamHandoff...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1846
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3041/



---

[GitHub] carbondata issue #1840: [CARBONDATA-2054]Add an example: how to use CarbonDa...

2018-01-23 Thread zzcclp

Github user zzcclp commented on the issue:

https://github.com/apache/carbondata/pull/1840
  
retest this please.


---

[GitHub] carbondata issue #1840: [CARBONDATA-2054]Add an example: how to use CarbonDa...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1840
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1821/



---

[GitHub] carbondata issue #1840: [CARBONDATA-2054]Add an example: how to use CarbonDa...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1840
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3051/



---

[GitHub] carbondata issue #1822: [CARBONDATA-2043] Configurable wait time for request...

2018-01-23 Thread kushalsaha

Github user kushalsaha commented on the issue:

https://github.com/apache/carbondata/pull/1822
  
retest SDV this please


---

[GitHub] carbondata issue #1831: [CARBONDATA-1993] Carbon properties default values f...

2018-01-23 Thread kushalsaha

Github user kushalsaha commented on the issue:

https://github.com/apache/carbondata/pull/1831
  
retest SDV please


---

[GitHub] carbondata issue #1831: [CARBONDATA-1993] Carbon properties default values f...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1831
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3043/



---

[GitHub] carbondata issue #1847: [WIP][CARBONDATA-2064] Add compaction listener

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1847
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3053/



---

[GitHub] carbondata issue #1840: [CARBONDATA-2054]Add an example: how to use CarbonDa...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1840
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3052/



---

[GitHub] carbondata issue #1841: [CARBONDATA-2062] Configure the temp directory to be...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1841
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3049/



---

[GitHub] carbondata issue #1841: [CARBONDATA-2062] Configure the temp directory to be...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1841
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1818/



---

[GitHub] carbondata issue #1846: [CARBONDATA-2067] Fix NPE exception in StreamHandoff...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1846
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3050/



---

[GitHub] carbondata issue #1847: [WIP][CARBONDATA-2064] Add compaction listener

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1847
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1822/



---

[GitHub] carbondata issue #1847: [WIP][CARBONDATA-2064] Add compaction listener

2018-01-23 Thread dhatchayani

Github user dhatchayani commented on the issue:

https://github.com/apache/carbondata/pull/1847
  
retest this please


---

[GitHub] carbondata issue #1838: [CARBONDATA-2060]fix insert overwrite on partition t...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1838
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3057/



---

[GitHub] carbondata issue #1847: [WIP][CARBONDATA-2064] Add compaction listener

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1847
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3046/



---

[GitHub] carbondata issue #1838: [CARBONDATA-2060]fix insert overwrite on partition t...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1838
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1825/



---

[GitHub] carbondata issue #1831: [CARBONDATA-1993] Carbon properties default values f...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1831
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3047/



---

[GitHub] carbondata issue #1848: [WIP][CARBONDATA-2068][CARBONDATA-1516][Tests] Drop ...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1848
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1826/



---

[GitHub] carbondata issue #1847: [WIP][CARBONDATA-2064] Add compaction listener

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1847
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3056/



---

[GitHub] carbondata issue #1831: [CARBONDATA-1993] Carbon properties default values f...

2018-01-23 Thread kushalsaha

Github user kushalsaha commented on the issue:

https://github.com/apache/carbondata/pull/1831
  
retest SDV please


---

[GitHub] carbondata issue #1822: [CARBONDATA-2043] Configurable wait time for request...

2018-01-23 Thread kushalsaha

Github user kushalsaha commented on the issue:

https://github.com/apache/carbondata/pull/1822
  
retest SDV please


---

[jira] [Created] (CARBONDATA-2068) Drop datamap should work for timeseries

2018-01-23 Thread xubo245 (JIRA)

xubo245 created CARBONDATA-2068:
---

 Summary: Drop datamap should  work for timeseries 
 Key: CARBONDATA-2068
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2068
 Project: CarbonData
  Issue Type: Bug
  Components: core, spark-integration
Affects Versions: 1.3.0
Reporter: xubo245
 Fix For: 1.3.0


Drop datamap is not  work after creating timeseries datamap for preaggregate 
table,
but it should  work.

refer：

https://issues.apache.org/jira/browse/CARBONDATA-1516



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1840: [CARBONDATA-2054]Add an example: how to use CarbonDa...

2018-01-23 Thread zzcclp

Github user zzcclp commented on the issue:

https://github.com/apache/carbondata/pull/1840
  
retest this please


---

[GitHub] carbondata issue #1840: [CARBONDATA-2054]Add an example: how to use CarbonDa...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1840
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3054/



---

[GitHub] carbondata issue #1840: [CARBONDATA-2054]Add an example: how to use CarbonDa...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1840
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1823/



---

[GitHub] carbondata issue #1847: [WIP][CARBONDATA-2064] Add compaction listener

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1847
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3045/



---

[GitHub] carbondata pull request #1848: [CARBONDATA-2068][CARBONDATA-1516] Drop datam...

2018-01-23 Thread xubo245

GitHub user xubo245 opened a pull request:

https://github.com/apache/carbondata/pull/1848

[CARBONDATA-2068][CARBONDATA-1516] Drop datamap should work for timeseries 
preaggregate table



 - [ ] Any interfaces changed?
  No
 - [ ] Any backward compatibility impacted?
No
 - [ ] Document update required?
No
 - [ ] Testing done
add test case for drop timeseries pre aggregate table
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xubo245/carbondata dropTimeseriesDataMap

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1848.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1848


commit f0b0d2ec406aa1b8a2bccd1e1b21435f5924fcbf
Author: xubo245 <601450868@...>
Date:   2018-01-23T12:05:06Z

[CARBONDATA-2068] Drop datamap should work for timeseries preaggregate table




---

[GitHub] carbondata issue #1847: [WIP][CARBONDATA-2064] Add compaction listener

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1847
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1824/



---

[GitHub] carbondata issue #1848: [WIP][CARBONDATA-2068][CARBONDATA-1516][Tests] Drop ...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1848
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3058/



---

[jira] [Commented] (CARBONDATA-2068) Drop datamap should work for timeseries

2018-01-23 Thread kumar vishal (JIRA)


[ 
https://issues.apache.org/jira/browse/CARBONDATA-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335758#comment-16335758
 ] 

kumar vishal commented on CARBONDATA-2068:
--

[~xubo245] can u please add your testcase for the above scenario 

> Drop datamap should  work for timeseries 
> -
>
> Key: CARBONDATA-2068
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2068
> Project: CarbonData
>  Issue Type: Bug
>  Components: core, spark-integration
>Affects Versions: 1.3.0
>Reporter: xubo245
>Priority: Major
> Fix For: 1.3.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Drop datamap is not  work after creating timeseries datamap for preaggregate 
> table,
> but it should  work.
> refer：
> https://issues.apache.org/jira/browse/CARBONDATA-1516



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1849: [HOTFIX] process VM error for the streaming w...

2018-01-23 Thread QiangCai

GitHub user QiangCai opened a pull request:

https://github.com/apache/carbondata/pull/1849

[HOTFIX] process VM error for the streaming writer

process VM error for the streaming writer by using Throwable instead of 
Exception

 - [x] Any interfaces changed?
 
 - [x] Any backward compatibility impacted?
 
 - [x] Document update required?

 - [x] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [x] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QiangCai/carbondata catch_jvm_error

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1849.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1849


commit f09703a5cc91c6a3e9327dd5f5f7cf0a28611c6a
Author: QiangCai 
Date:   2018-01-23T13:12:32Z

process vw error for streaming writer




---

[GitHub] carbondata issue #1847: [WIP][CARBONDATA-2064] Add compaction listener

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1847
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3059/



---

[GitHub] carbondata issue #1847: [WIP][CARBONDATA-2064] Add compaction listener

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1847
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1827/



---

[GitHub] carbondata pull request #1693: [CARBONDATA-1909] Load is failing during inse...

2018-01-23 Thread gvramana

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1693#discussion_r163241946
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
 ---
@@ -864,6 +864,18 @@
   public static final String CARBON_MERGE_SORT_PREFETCH = 
"carbon.merge.sort.prefetch";
   public static final String CARBON_MERGE_SORT_PREFETCH_DEFAULT = "true";
 
+  /**
+   * to determine to use the rdd persist or not for insert query.
--- End diff --

Please write clear comment, when this should be used and what is the impact 
if not used, why default value is false?


---

[GitHub] carbondata issue #1822: [CARBONDATA-2043] Configurable wait time for request...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1822
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3048/



---

[jira] [Created] (CARBONDATA-2070) when hive metastore is enabled, create preaggregate table on decimal column of main table is failing

2018-01-23 Thread Akash R Nilugal (JIRA)

Akash R Nilugal created CARBONDATA-2070:
---

 Summary: when hive metastore is enabled, create preaggregate table 
on decimal column of main table is failing
 Key: CARBONDATA-2070
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2070
 Project: CarbonData
  Issue Type: Bug
Reporter: Akash R Nilugal
Assignee: Akash R Nilugal


{color:#33}steps:{color}

{color:#33}Enable hive metastore and run the following queries{color}

{color:#33}1){color}

{color:#33}CREATE TABLE uniqdata(CUST_ID int,CUST_NAME 
String,ACTIVE_EMUI_VERSION string,DOB timestamp,DOJ timestamp, BIGINT_COLUMN1 
bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10),DECIMAL_COLUMN2 
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 
int) STORED BY 'org.apache.carbondata.format'{color}

 

{color:#33}2){color}

{color:#33}insert into uniqdata select 
9000,'CUST_NAME_0','ACTIVE_EMUI_VERSION_0','1970-01-01 
01:00:03','1970-01-01 
02:00:03',123372036854,-223372036854,12345678901.123400,22345678901.123400,11234567489.797600,-11234567489.797600,1{color}

 

{color:#33}3){color}

{color:#33}create datamap uniqdata_agg on table uniqdata using 
'preaggregate' as select min(DECIMAL_COLUMN1) from uniqdata group by 
DECIMAL_COLUMN1{color}

 

{color:#33}java.lang.ClassCastException: 
org.apache.carbondata.core.metadata.datatype.DataType cannot be cast to 
org.apache.carbondata.core.metadata.datatype.DecimalType
    at 
org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema.write(ColumnSchema.java:478)
    at 
org.apache.carbondata.core.metadata.schema.table.TableSchema.write(TableSchema.java:215)
    at 
org.apache.carbondata.core.metadata.schema.table.DataMapSchema.write(DataMapSchema.java:99)
    at 
org.apache.carbondata.core.metadata.schema.table.TableInfo.write(TableInfo.java:245)
    at 
org.apache.carbondata.core.metadata.schema.table.TableInfo.serialize(TableInfo.java:304)
    at 
org.apache.spark.sql.CarbonDatasourceHadoopRelation.buildScan(CarbonDatasourceHadoopRelation.scala:83)
    at 
org.apache.spark.sql.execution.strategy.CarbonLateDecodeStrategy$$anonfun$1.apply(CarbonLateDecodeStrategy.scala:63)
    at 
org.apache.spark.sql.execution.strategy.CarbonLateDecodeStrategy$$anonfun$1.apply(CarbonLateDecodeStrategy.scala:63)
    at 
org.apache.spark.sql.execution.strategy.CarbonLateDecodeStrategy$$anonfun$pruneFilterProject$1.apply(CarbonLateDecodeStrategy.scala:178)
    at 
org.apache.spark.sql.execution.strategy.CarbonLateDecodeStrategy$$anonfun$pruneFilterProject$1.apply(CarbonLateDecodeStrategy.scala:177)
    at 
org.apache.spark.sql.execution.strategy.CarbonLateDecodeStrategy.getDataSourceScan(CarbonLateDecodeStrategy.scala:366)
    at 
org.apache.spark.sql.execution.strategy.CarbonLateDecodeStrategy.pruneFilterProjectRaw(CarbonLateDecodeStrategy.scala:299)
    at 
org.apache.spark.sql.execution.strategy.CarbonLateDecodeStrategy.pruneFilterProject(CarbonLateDecodeStrategy.scala:172)
    at 
org.apache.spark.sql.execution.strategy.CarbonLateDecodeStrategy.apply(CarbonLateDecodeStrategy.scala:59)
    at 
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:62)
    at 
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:62)
    at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)
    at 
org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:92)
    at 
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:77)
    at 
org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$2$$anonfun$apply$2.apply(QueryPlanner.scala:74)
    at 
scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
    at 
scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
    at scala.collection.Iterator$class.foreach(Iterator.scala:893)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
    at 
scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157){color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (CARBONDATA-2069) Data is not loaded into preaggregate table when table is created when data load is in progress for main table

2018-01-23 Thread Kunal Kapoor (JIRA)

Kunal Kapoor created CARBONDATA-2069:


 Summary: Data is not loaded into preaggregate table when table is 
created when data load is in progress for main table
 Key: CARBONDATA-2069
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2069
 Project: CarbonData
  Issue Type: Bug
Reporter: Kunal Kapoor
Assignee: Kunal Kapoor


Problem:
 # Load data into maintable
 # create datamap parallelly

preaggregate table will not have any data while data load is successful for 
main table. This will make the pre-aggregate table inconsistent



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1850: [CARBONDATA-2069] Restrict create datamap whe...

2018-01-23 Thread kunal642

GitHub user kunal642 opened a pull request:

https://github.com/apache/carbondata/pull/1850

[CARBONDATA-2069] Restrict create datamap when load is in progress

**Problem:**
1. Load data into maintable
2. create datamap parallelly
preaggregate table will not have any data while data load is successful for 
main table. This will make the pre-aggregate table inconsistent

**Solution:** Restrict creation of pre-aggregate table when load is in 
progress on main table

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [X] Any interfaces changed?
 
 - [X] Any backward compatibility impacted?
 
 - [X] Document update required?

 - [X] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [X] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kunal642/carbondata restrict_create

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1850.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1850


commit 0e47c0e8841c1e0f1ec0291c18d7c7caea2f58aa
Author: kunal642 
Date:   2018-01-23T13:22:48Z

restrict create datamap when load is in progress




---

[jira] [Closed] (CARBONDATA-2068) Drop datamap should work for timeseries

2018-01-23 Thread xubo245 (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xubo245 closed CARBONDATA-2068.
---
Resolution: Won't Fix

sql(s"drop datamap if exists agg1 on table mainTable") not support,
Carbon support sql(s"drop datamap if exists agg1_year on table mainTable").

> Drop datamap should  work for timeseries 
> -
>
> Key: CARBONDATA-2068
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2068
> Project: CarbonData
>  Issue Type: Bug
>  Components: core, spark-integration
>Affects Versions: 1.3.0
>Reporter: xubo245
>Priority: Major
> Fix For: 1.3.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Drop datamap is not  work after creating timeseries datamap for preaggregate 
> table,
> but it should  work.
> refer：
> https://issues.apache.org/jira/browse/CARBONDATA-1516



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1853
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1841/



---

[GitHub] carbondata issue #1850: [CARBONDATA-2069] Restrict create datamap when load ...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1850
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3065/



---

[GitHub] carbondata issue #1850: [CARBONDATA-2069] Restrict create datamap when load ...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1850
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1845/



---

[GitHub] carbondata issue #1781: [CARBONDATA-2012] Add support to load pre-aggregate ...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1781
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3066/



---

[GitHub] carbondata issue #1844: [CARBONDATA-2061] Check for only valid IN_PROGRESS s...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1844
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3079/



---

[GitHub] carbondata issue #1844: [CARBONDATA-2061] Check for only valid IN_PROGRESS s...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1844
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1848/



---

[GitHub] carbondata issue #1851: [CARBONDATA-2071] Added block size to BblockletDataM...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1851
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3064/



---

[GitHub] carbondata issue #1851: [CARBONDATA-2071] Added block size to BblockletDataM...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1851
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3074/



---

[GitHub] carbondata issue #1851: [CARBONDATA-2071] Added block size to BblockletDataM...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1851
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1843/



---

[GitHub] carbondata issue #1852: [CARBONDATA-2070]fix create preaggregate on decimal ...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1852
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1844/



---

[GitHub] carbondata issue #1781: [CARBONDATA-2012] Add support to load pre-aggregate ...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1781
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1846/



---

[GitHub] carbondata issue #1852: [CARBONDATA-2070]fix create preaggregate on decimal ...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1852
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3075/



---

[GitHub] carbondata issue #1851: [CARBONDATA-2071] Added block size to BblockletDataM...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1851
  
retest this please


---

[GitHub] carbondata issue #1850: [CARBONDATA-2069] Restrict create datamap when load ...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1850
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3076/



---

[GitHub] carbondata issue #1851: [CARBONDATA-2071] Added block size to BblockletDataM...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1851
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1847/



---

[GitHub] carbondata issue #1844: [CARBONDATA-2061] Check for only valid IN_PROGRESS s...

2018-01-23 Thread dhatchayani

Github user dhatchayani commented on the issue:

https://github.com/apache/carbondata/pull/1844
  
retest this please


---

[GitHub] carbondata issue #1852: [CARBONDATA-2070]fix create preaggregate on decimal ...

2018-01-23 Thread akashrn5

Github user akashrn5 commented on the issue:

https://github.com/apache/carbondata/pull/1852
  
retest this please


---

[GitHub] carbondata issue #1781: [CARBONDATA-2012] Add support to load pre-aggregate ...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1781
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3077/



---

[GitHub] carbondata issue #1851: [CARBONDATA-2071] Added block size to BblockletDataM...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1851
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3078/



---

[jira] [Created] (CARBONDATA-2072) Add dropTables method for optimizing drop table operation in test cases

2018-01-23 Thread xubo245 (JIRA)

xubo245 created CARBONDATA-2072:
---

 Summary: Add dropTables method for optimizing drop table operation 
in test cases
 Key: CARBONDATA-2072
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2072
 Project: CarbonData
  Issue Type: Test
  Components: test
Affects Versions: 1.3.0
Reporter: xubo245
Assignee: xubo245
 Fix For: 1.3.0


There are many drop table in beforeAll or afterAll of test cases,like this:


{code:java}
 override def afterAll {
sql("drop table if exists load")
sql("drop table if exists inser")
sql("DROP TABLE IF EXISTS THive")
sql("DROP TABLE IF EXISTS TCarbon")
sql("drop table if exists TCarbonLocal")
sql("drop table if exists TCarbonSource")
sql("drop table if exists loadtable")
sql("drop table if exists insertTable")
sql("drop table if exists CarbonDest")
sql("drop table if exists HiveDest")
sql("drop table if exists CarbonOverwrite")
sql("drop table if exists HiveOverwrite")
sql("drop table if exists tcarbonsourceoverwrite")
sql("drop table if exists carbon_table1")
sql("drop table if exists carbon_table")
sql("DROP TABLE IF EXISTS student")
sql("DROP TABLE IF EXISTS uniqdata")
sql("DROP TABLE IF EXISTS show_insert")
sql("drop table if exists OverwriteTable_t1")
sql("drop table if exists OverwriteTable_t2")
}
{code}

in 
org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase

It can be optimized by a public method in QueryTest




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1854: [WIP] Test insert fail issue

2018-01-23 Thread ravipesala

GitHub user ravipesala opened a pull request:

https://github.com/apache/carbondata/pull/1854

[WIP] Test insert fail issue

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ravipesala/incubator-carbondata 
fix-insert-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1854.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1854


commit a43caa3755435a08fd4ced23dd9e203b10417348
Author: ravipesala 
Date:   2018-01-24T03:47:07Z

Test insert fail issue




---

[GitHub] carbondata issue #1854: [WIP] Test insert fail issue

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1854
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1842/



---

[GitHub] carbondata pull request #1851: [CARBONDATA-2071] Added block size to Bblockl...

2018-01-23 Thread ravipesala

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1851#discussion_r163450767
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/indexstore/BlockletDataMapIndexStore.java
 ---
@@ -244,4 +247,22 @@ public void clearAccessCount(
   cacheable.clear();
 }
   }
+
+  public static class BlockMetaInfo {
+private String[] locationInfo;
+private long size;
--- End diff --

ok


---

[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1853
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3072/



---

[GitHub] carbondata issue #1853: [CARBONDATA-2072][TEST] Add dropTables method for op...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1853
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3062/



---

[GitHub] carbondata pull request #1851: [CARBONDATA-2071] Added block size to Bblockl...

2018-01-23 Thread jackylk

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1851#discussion_r163449480
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/indexstore/BlockletDataMapIndexStore.java
 ---
@@ -78,17 +78,19 @@ public BlockletDataMap 
get(TableBlockIndexUniqueIdentifier identifier)
 String segmentPath = CarbonTablePath.getSegmentPath(
 identifier.getAbsoluteTableIdentifier().getTablePath(),
 identifier.getSegmentId());
-Map locationMap = new HashMap<>();
+Map blockMetaInfoMap = new HashMap<>();
 CarbonFile carbonFile = FileFactory.getCarbonFile(segmentPath);
 CarbonFile[] carbonFiles = carbonFile.locationAwareListFiles();
 SegmentIndexFileStore indexFileStore = new SegmentIndexFileStore();
 indexFileStore.readAllIIndexOfSegment(carbonFiles);
 PartitionMapFileStore partitionFileStore = new 
PartitionMapFileStore();
 partitionFileStore.readAllPartitionsOfSegment(carbonFiles, 
segmentPath);
 for (CarbonFile file : carbonFiles) {
-  locationMap.put(file.getAbsolutePath(), file.getLocations());
+  blockMetaInfoMap
+  .put(file.getAbsolutePath(), new 
BlockMetaInfo(file.getLocations(), file.getSize()));
--- End diff --

move `.put` to previous line


---

[GitHub] carbondata pull request #1851: [CARBONDATA-2071] Added block size to Bblockl...

2018-01-23 Thread jackylk

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1851#discussion_r163449593
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/indexstore/BlockletDataMapIndexStore.java
 ---
@@ -244,4 +247,22 @@ public void clearAccessCount(
   cacheable.clear();
 }
   }
+
+  public static class BlockMetaInfo {
+private String[] locationInfo;
+private long size;
--- End diff --

please add comment for these two variable


---

[GitHub] carbondata pull request #1851: [CARBONDATA-2071] Added block size to Bblockl...

2018-01-23 Thread jackylk

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1851#discussion_r163449568
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/indexstore/BlockletDataMapIndexStore.java
 ---
@@ -244,4 +247,22 @@ public void clearAccessCount(
   cacheable.clear();
 }
   }
+
+  public static class BlockMetaInfo {
--- End diff --

Is it only used in this class? I think it is better to move to outside if 
not just used in this class


---

[jira] [Commented] (CARBONDATA-2068) Drop datamap should work for timeseries

2018-01-23 Thread xubo245 (JIRA)


[ 
https://issues.apache.org/jira/browse/CARBONDATA-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336735#comment-16336735
 ] 

xubo245 commented on CARBONDATA-2068:
-

sql(s"drop datamap if exists agg1 on table mainTable") not support,
Carbon support sql(s"drop datamap if exists agg1_year on table mainTable").

> Drop datamap should  work for timeseries 
> -
>
> Key: CARBONDATA-2068
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2068
> Project: CarbonData
>  Issue Type: Bug
>  Components: core, spark-integration
>Affects Versions: 1.3.0
>Reporter: xubo245
>Priority: Major
> Fix For: 1.3.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Drop datamap is not  work after creating timeseries datamap for preaggregate 
> table,
> but it should  work.
> refer：
> https://issues.apache.org/jira/browse/CARBONDATA-1516



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1853: [CARBONDATA-2072][TEST] Add dropTables method...

2018-01-23 Thread xubo245

GitHub user xubo245 opened a pull request:

https://github.com/apache/carbondata/pull/1853

[CARBONDATA-2072][TEST] Add dropTables method for optimizing drop table 
method in test cases

Add dropTables method for optimizing drop table method in test cases
Please see jira

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 No
 - [ ] Any backward compatibility impacted?
 No
 - [ ] Document update required?
No
 - [ ] Testing done
  only change test cases, no new test cases
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
No


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xubo245/carbondata dropTables

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1853.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1853


commit 48760ee0554d7cf3a645da0db697bdd5bee7bc47
Author: xubo245 <601450868@...>
Date:   2018-01-24T02:00:44Z

[CARBONDATA-2072][TEST] Add dropTables method for optimizing drop table 
method in test cases




---

[GitHub] carbondata issue #1851: [CARBONDATA-2071] Added block size to BblockletDataM...

2018-01-23 Thread QiangCai

Github user QiangCai commented on the issue:

https://github.com/apache/carbondata/pull/1851
  
verified on local standalone mode

LGTM


---

[GitHub] carbondata issue #1851: [CARBONDATA-2071] Added block size to BblockletDataM...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1851
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3054/



---

[GitHub] carbondata issue #1693: [CARBONDATA-1909] Load is failing during insert into...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1693
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3057/



---

[GitHub] carbondata issue #1852: [CARBONDATA-2070]fix create preaggregate on decimal ...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1852
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1840/



---

[GitHub] carbondata issue #1838: [CARBONDATA-2060]fix insert overwrite on partition t...

2018-01-23 Thread manishgupta88

Github user manishgupta88 commented on the issue:

https://github.com/apache/carbondata/pull/1838
  
LGTM


---

[jira] [Resolved] (CARBONDATA-2060) Fix InsertOverwrite on partition table

2018-01-23 Thread Manish Gupta (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manish Gupta resolved CARBONDATA-2060.
--
   Resolution: Fixed
Fix Version/s: 1.3.0

> Fix InsertOverwrite on partition table
> --
>
> Key: CARBONDATA-2060
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2060
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Akash R Nilugal
>Assignee: Akash R Nilugal
>Priority: Minor
> Fix For: 1.3.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> {color:#33}when partition table overwrite with empty table , it is not 
> overwriting the partition table , and when insert overwrite is done on 
> dynamic partition table , overwrite was not happening.{color}
>  
> {color:#33}sql("create table partitionLoadTable(name string, age int) 
> PARTITIONED BY(address string) stored by 'carbondata'"){color}
> {color:#33}sql("insert into partitionLoadTable select 
> 'abc',4,'def'"){color}
> {color:#33}sql("insert into partitionLoadTable select 
> 'abd',5,'xyz'"){color}
> {color:#33}sql("create table noLoadTable (name string, age int, address 
> string) stored by 'carbondata'"){color}
> {color:#33}sql("insert overwrite table partitionLoadTable select * from 
> noLoadTable"){color}
> {color:#33}when we do select * after insert overwrite operation, ideally 
> it should give empty data, but it is giving all data.{color}
>  
> {color:#33}sql("CREATE TABLE uniqdata_hive_static (CUST_ID int,CUST_NAME 
> String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, 
> BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), 
> DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double, 
> INTEGER_COLUMN1 int)ROW FORMAT DELIMITED FIELDS TERMINATED BY ','"){color}
> {color:#33}sql("CREATE TABLE uniqdata_string_static(CUST_ID int,CUST_NAME 
> String,DOB timestamp,DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 
> bigint,DECIMAL_COLUMN1 decimal(30,10),DECIMAL_COLUMN2 
> decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 
> int) PARTITIONED BY(ACTIVE_EMUI_VERSION string) STORED BY 
> 'org.apache.carbondata.format' TBLPROPERTIES ('TABLE_BLOCKSIZE'= '256 
> MB')"){color}
> {color:#33}sql(s"LOAD DATA INPATH '$resourcesPath/partData.csv' into 
> table uniqdata_string_static OPTIONS('FILEHEADER'='CUST_ID,CUST_NAME 
> ,ACTIVE_EMUI_VERSION,DOB,DOJ, 
> BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1, 
> Double_COLUMN2,INTEGER_COLUMN1','BAD_RECORDS_ACTION'='FORCE')"){color}
> {color:#33}sql(s"LOAD DATA INPATH '$resourcesPath/partData.csv' into 
> table uniqdata_string_static OPTIONS('FILEHEADER'='CUST_ID,CUST_NAME 
> ,ACTIVE_EMUI_VERSION,DOB,DOJ, 
> BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1, 
> Double_COLUMN2,INTEGER_COLUMN1','BAD_RECORDS_ACTION'='FORCE')"){color}
> {color:#33}sql("insert overwrite table uniqdata_string_static select 
> CUST_ID, CUST_NAME,DOB,doj, bigint_column1, bigint_column2, decimal_column1, 
> decimal_column2,double_column1, 
> double_column2,integer_column1,active_emui_version from uniqdata_hive_static 
> limit 10"){color}
>  
> {color:#33}after this, select * was giving result, ideally it should give 
> empty result.{color}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1693: [CARBONDATA-1909] Load is failing during insert into...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1693
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3056/



---

[GitHub] carbondata issue #1852: [CARBONDATA-2070]fix create preaggregate on decimal ...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1852
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3071/



---

[GitHub] carbondata issue #1854: [WIP] Test insert fail issue

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1854
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1849/



---

[GitHub] carbondata issue #1844: [CARBONDATA-2061] Check for only valid IN_PROGRESS s...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1844
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3081/



---

[GitHub] carbondata issue #1841: [CARBONDATA-2062] Configure the temp directory to be...

2018-01-23 Thread QiangCai

Github user QiangCai commented on the issue:

https://github.com/apache/carbondata/pull/1841
  
retest this please


---

[GitHub] carbondata issue #1844: [CARBONDATA-2061] Check for only valid IN_PROGRESS s...

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1844
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3067/



---

[GitHub] carbondata issue #1854: [WIP] Test insert fail issue

2018-01-23 Thread ravipesala

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1854
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3068/



---

[jira] [Created] (CARBONDATA-2071) Add block size to BblockletDataMap while initialising

2018-01-23 Thread Ravindra Pesala (JIRA)

Ravindra Pesala created CARBONDATA-2071:
---

 Summary: Add block size to BblockletDataMap while initialising
 Key: CARBONDATA-2071
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2071
 Project: CarbonData
  Issue Type: Improvement
Reporter: Ravindra Pesala


Add block size to blocklet datamap so that features like small file merge can 
use them



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1852: [CARBONDATA-2070]fix create preaggregate on d...

2018-01-23 Thread akashrn5

GitHub user akashrn5 opened a pull request:

https://github.com/apache/carbondata/pull/1852

[CARBONDATA-2070]fix create preaggregate on decimal column in hive metastore

Problem: when hive metastore is enabled and aggregate table is tried to 
create on the decimal column of main table, cast exception is thrown for 
Decimal datatype

solution:During creation of TableInfo from hivemetastore the DataMapSchemas 
and the columns
   dataTypes are not converted to the appropriate child classes.
   convert to actual datatype

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [X] Any interfaces changed?
 NA
 - [X] Any backward compatibility impacted?
 NA
 - [X] Document update required?
NA
 - [X] Testing done
Added UT
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/akashrn5/incubator-carbondata hive_pre

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1852.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1852


commit d8c38d5d11fbb1f4fcfb590168b8785497ab868a
Author: akashrn5 
Date:   2018-01-23T13:29:51Z

fix create preaggregate on decimal column in hive metastore




---

[GitHub] carbondata pull request #1693: [CARBONDATA-1909] Load is failing during inse...

2018-01-23 Thread ManoharVanam

Github user ManoharVanam commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1693#discussion_r163250438
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
 ---
@@ -864,6 +864,18 @@
   public static final String CARBON_MERGE_SORT_PREFETCH = 
"carbon.merge.sort.prefetch";
   public static final String CARBON_MERGE_SORT_PREFETCH_DEFAULT = "true";
 
+  /**
+   * to determine to use the rdd persist or not for insert query.
--- End diff --

done


---

[GitHub] carbondata issue #1848: [WIP][CARBONDATA-2068][CARBONDATA-1516][Tests] Drop ...

2018-01-23 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1848
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1829/



---

1 2 >

1 - 100 of 139 matches

Mail list logo