[GitHub] carbondata issue #2056: [CARBONDATA-2238][DataLoad] Merge and spill in-memor...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2056
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3873/



---

[GitHub] carbondata pull request #2058: [CARBONDATA-2249] Fixed bug for querying data...

[geetikagupta16]

GitHub user geetikagupta16 opened a pull request:

https://github.com/apache/carbondata/pull/2058

[CARBONDATA-2249] Fixed bug for querying data through presto

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/geetikagupta16/incubator-carbondata 
CARBONDATA-2249

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2058.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2058


commit 897172913db783a3a7d3770bdf08772b8008e9d0
Author: Geetika Gupta 
Date:   2018-03-13T05:21:06Z

Added dependency for lz4




---

[GitHub] carbondata issue #2034: [CARBONDATA-2288] [Test] Exception is Masked Inside ...

[anubhav100]

Github user anubhav100 commented on the issue:

https://github.com/apache/carbondata/pull/2034
  
retest sdv please


---

[GitHub] carbondata issue #2057: [CARBONDATA-2248]Fixed Memory leak in parser/CarbonS...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2057
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4235/



---

[GitHub] carbondata issue #2057: [CARBONDATA-2248]Fixed Memory leak in parser/CarbonS...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2057
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2991/



---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2990/



---

[GitHub] carbondata issue #2052: [CARBONDATA-2246][DataLoad] Fix exhausted memory pro...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2052
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3872/



---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4234/



---

[jira] [Created] (CARBONDATA-2248) Removing parsers thread local objects after parsing of carbon query

[kumar vishal (JIRA)]

kumar vishal created CARBONDATA-2248:


 Summary: Removing parsers thread local objects after parsing of 
carbon query
 Key: CARBONDATA-2248
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2248
 Project: CarbonData
  Issue Type: Bug
Reporter: kumar vishal
Assignee: kumar vishal


In some scenarios where more sessions are created, there are many parser 
failure objects are accumulated in memory inside thread locals.

Solution: Remove the parser object from thread local after parsing of the query



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #2057: [CARBONDATA-2248]Fixed Memory leak in parser/...

[kumarvishal09]

GitHub user kumarvishal09 opened a pull request:

https://github.com/apache/carbondata/pull/2057

[CARBONDATA-2248]Fixed Memory leak in parser/CarbonSparkSqlParser.scala

In some scenarios where more sessions are created, there are many parser 
failure objects are accumulated in memory inside thread locals and causing 
memory leak in long run.

Solution: Remove the parser object from thread local after parsing of the 
query

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kumarvishal09/incubator-carbondata 12-03-208

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2057.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2057


commit d4df8ceeaa29a6281e731271c5f2f05861c73d9e
Author: kumarvishal 
Date:   2018-03-12T15:30:22Z

Fixed Memory leak in parser/CarbonSparkSqlParser.scala




---

[GitHub] carbondata issue #2053: [CARBONDATA-2247][SDK] Support write Index file in C...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2053
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2989/



---

[GitHub] carbondata issue #2053: [CARBONDATA-2247][SDK] Support write Index file in C...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2053
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4233/



---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2987/



---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4231/



---

[GitHub] carbondata issue #2056: [CARBONDATA-2238][DataLoad] Merge and spill in-memor...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2056
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2988/



---

[GitHub] carbondata issue #2052: [CARBONDATA-2246][DataLoad] Fix exhausted memory pro...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2052
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2985/



---

[GitHub] carbondata issue #2056: [CARBONDATA-2238][DataLoad] Merge and spill in-memor...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2056
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4232/



---

[GitHub] carbondata issue #2052: [CARBONDATA-2246][DataLoad] Fix exhausted memory pro...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2052
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4229/



---

[GitHub] carbondata pull request #2053: [CARBONDATA-2247][SDK] Support write Index fi...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2053#discussion_r174009976
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java
 ---
@@ -42,6 +45,16 @@
 
   private List otherColumns = new LinkedList<>();
 
+  private int blockSize;
+
+  public TableSchemaBuilder blockSize(int blockSize) {
+if (blockSize == 0) {
--- End diff --

ok


---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3871/



---

[GitHub] carbondata issue #2056: [CARBONDATA-2238][DataLoad] Merge and spill in-memor...

[xuchuanyin]

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/2056
  
retest this please


---

[GitHub] carbondata issue #2056: [CARBONDATA-2238][DataLoad] Merge and spill in-memor...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2056
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4230/



---

[GitHub] carbondata issue #2056: [CARBONDATA-2238][DataLoad] Merge and spill in-memor...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2056
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2986/



---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4228/



---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2983/



---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2984/



---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4227/



---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
retest this please


---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3870/



---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
@kumarvishal09 Please review it


---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
retest sdv please


---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2982/



---

[GitHub] carbondata issue #1929: [CARBONDATA-2129][CARBONDATA-2094][CARBONDATA-1516] ...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1929
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3869/



---

[GitHub] carbondata issue #1920: [CARBONDATA-2110] Remove tempCsv option in test case...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1920
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4226/



---

[GitHub] carbondata issue #1920: [CARBONDATA-2110] Remove tempCsv option in test case...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1920
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2981/



---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
Any one can help to review it? Please. I rebased this PR more than 8 
times... @ravipesala  @jackylk @QiangCai @chenliang613 @sraghunandan  
@kumarvishal09 


---

[GitHub] carbondata issue #2052: [CARBONDATA-2246][DataLoad] Fix exhausted memory pro...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2052
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2980/



---

[GitHub] carbondata issue #2052: [CARBONDATA-2246][DataLoad] Fix exhausted memory pro...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2052
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4225/



---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4224/



---

[GitHub] carbondata issue #2055: [CARBONDATA-2224][File Level Reader Support] Externa...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2055
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3868/



---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2979/



---

[GitHub] carbondata issue #1920: [CARBONDATA-2110] Remove tempCsv option in test case...

[xuchuanyin]

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/1920
  
retest this please


---

[GitHub] carbondata issue #2052: [CARBONDATA-2246][DataLoad] Fix exhausted memory pro...

[xuchuanyin]

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/2052
  
retest this please


---

[GitHub] carbondata issue #1990: [CARBONDATA-2195] Add new test case for partition fe...

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/1990
  
Please review it. @ravipesala @jackylk @QiangCai @chenliang613 
@sraghunandan @kumarvishal09 


---

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
Any one can help to review it? Please  @ravipesala @jackylk @QiangCai 
@chenliang613 @sraghunandan @kumarvishal09


---

[GitHub] carbondata pull request #2053: [CARBONDATA-2247][SDK] Support write Index fi...

[xuchuanyin]

Github user xuchuanyin commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2053#discussion_r173991546
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java
 ---
@@ -42,6 +45,16 @@
 
   private List otherColumns = new LinkedList<>();
 
+  private int blockSize;
+
+  public TableSchemaBuilder blockSize(int blockSize) {
+if (blockSize == 0) {
--- End diff --

Logic and prompt are inconsistent, should be '<='


---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[xubo245]

Github user xubo245 commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
retest this please


---

[GitHub] carbondata issue #2054: [CARBONDATA-2224][File Level Reader Support] Externa...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2054
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3867/



---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[zzcclp]

Github user zzcclp commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
@jackylk @ravipesala please help to review, thanks


---

[GitHub] carbondata issue #2031: [CARBONDATA-2223] Remove unused listeners

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2031
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3866/



---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2978/



---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4223/



---

[GitHub] carbondata issue #2031: [CARBONDATA-2223] Remove unused listeners

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2031
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3865/



---

[GitHub] carbondata issue #2050: [CARBONDATA-2244]fix creating pre-aggregate table bu...

[zzcclp]

Github user zzcclp commented on the issue:

https://github.com/apache/carbondata/pull/2050
  
@jackylk @ravipesala please help to review, thanks


---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[zzcclp]

Github user zzcclp commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
retest this please


---

[GitHub] carbondata issue #2031: [CARBONDATA-2223] Remove unused listeners

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2031
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3864/



---

[GitHub] carbondata issue #2031: [CARBONDATA-2223] Remove unused listeners

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2031
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2977/



---

[GitHub] carbondata issue #2050: [CARBONDATA-2244]fix creating pre-aggregate table bu...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2050
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3863/



---

[GitHub] carbondata issue #2031: [CARBONDATA-2223] Remove unused listeners

[dhatchayani]

Github user dhatchayani commented on the issue:

https://github.com/apache/carbondata/pull/2031
  
retest this please


---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3862/



---

[GitHub] carbondata issue #2050: [CARBONDATA-2244]fix creating pre-aggregate table bu...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2050
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3861/



---

[GitHub] carbondata issue #2055: [CARBONDATA-2224][File Level Reader Support] Externa...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2055
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2976/



---

[GitHub] carbondata issue #2055: [CARBONDATA-2224][File Level Reader Support] Externa...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2055
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4221/



---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3860/



---

[GitHub] carbondata issue #2055: [CARBONDATA-2224][File Level Reader Support] Externa...

[sounakr]

Github user sounakr commented on the issue:

https://github.com/apache/carbondata/pull/2055
  
Retest this please


---

[GitHub] carbondata issue #2055: [CARBONDATA-2224][File Level Reader Support] Externa...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2055
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2975/



---

[GitHub] carbondata issue #2055: [CARBONDATA-2224][File Level Reader Support] Externa...

[sounakr]

Github user sounakr commented on the issue:

https://github.com/apache/carbondata/pull/2055
  
Retest this please


---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3859/



---

[GitHub] carbondata issue #2054: [CARBONDATA-2224][File Level Reader Support] Externa...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2054
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2973/



---

[GitHub] carbondata issue #2054: [CARBONDATA-2224][File Level Reader Support] Externa...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2054
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4218/



---

[GitHub] carbondata issue #2031: [CARBONDATA-2223] Remove unused listeners

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2031
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2972/



---

[GitHub] carbondata issue #2031: [CARBONDATA-2223] Remove unused listeners

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2031
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4217/



---

[GitHub] carbondata issue #2054: [CARBONDATA-2224][File Level Reader Support] Externa...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2054
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3858/



---

[GitHub] carbondata issue #2050: [CARBONDATA-2244]fix creating pre-aggregate table bu...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2050
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2971/



---

[GitHub] carbondata issue #2050: [CARBONDATA-2244]fix creating pre-aggregate table bu...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2050
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4216/



---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2970/



---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4215/



---

[GitHub] carbondata issue #2031: [CARBONDATA-2223] Remove unused listeners

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2031
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3857/



---

[GitHub] carbondata issue #2050: [CARBONDATA-2244]fix creating pre-aggregate table bu...

[zzcclp]

Github user zzcclp commented on the issue:

https://github.com/apache/carbondata/pull/2050
  
retest this please


---

[GitHub] carbondata issue #2050: [CARBONDATA-2244]fix creating pre-aggregate table bu...

[zzcclp]

Github user zzcclp commented on the issue:

https://github.com/apache/carbondata/pull/2050
  
retest sdv please


---

[GitHub] carbondata pull request #2045: [CARBONDATA-2230]Add a path into table path t...

[zzcclp]

Github user zzcclp commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2045#discussion_r173887885
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/locks/CarbonLockUtil.java ---
@@ -107,4 +115,37 @@ public static int getLockProperty(String property, int 
defaultValue) {
 }
   }
 
+  /**
+   * Currently the segment lock files are not deleted immediately when 
unlock,
+   * so it needs to delete expired lock files before delete loads.
+   */
+  public static void deleteExpiredSegmentLockFiles(CarbonTable 
carbonTable) {
+LoadMetadataDetails[] details =
+
SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath());
+if (details != null && details.length > 0) {
+  AbsoluteTableIdentifier absoluteTableIdentifier = 
carbonTable.getAbsoluteTableIdentifier();
+  long segmentLockFilesPreservTime =
+  
CarbonProperties.getInstance().getSegmentLockFilesPreserveHours();
+  long currTime = System.currentTimeMillis();
+  for (LoadMetadataDetails oneRow : details) {
+if (oneRow.getVisibility().equalsIgnoreCase("false") ||
+SegmentStatus.SUCCESS == oneRow.getSegmentStatus() ||
+SegmentStatus.LOAD_FAILURE == oneRow.getSegmentStatus() ||
+SegmentStatus.LOAD_PARTIAL_SUCCESS == 
oneRow.getSegmentStatus() ||
+SegmentStatus.COMPACTED == oneRow.getSegmentStatus()) {
+  String location = CarbonTablePath
+  .getLockFilesDirPath(absoluteTableIdentifier.getTablePath()) 
+
+  CarbonCommonConstants.FILE_SEPARATOR +
+  CarbonTablePath.addSegmentPrefix(oneRow.getLoadName()) + 
LockUsage.LOCK;
+  CarbonFile carbonFile =
+  FileFactory.getCarbonFile(location, 
FileFactory.getFileType(location));
+  if (carbonFile.exists()) {
--- End diff --

Added static function 'isFileExist(String filePath)' in FileFactory.
But I didn't use this function directly, it still needs to use 'CarbonFile' 
to get the last modified time below.


---

[GitHub] carbondata pull request #2045: [CARBONDATA-2230]Add a path into table path t...

[zzcclp]

Github user zzcclp commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2045#discussion_r173886926
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/locks/CarbonLockUtil.java ---
@@ -107,4 +115,37 @@ public static int getLockProperty(String property, int 
defaultValue) {
 }
   }
 
+  /**
+   * Currently the segment lock files are not deleted immediately when 
unlock,
+   * so it needs to delete expired lock files before delete loads.
+   */
+  public static void deleteExpiredSegmentLockFiles(CarbonTable 
carbonTable) {
+LoadMetadataDetails[] details =
+
SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath());
+if (details != null && details.length > 0) {
+  AbsoluteTableIdentifier absoluteTableIdentifier = 
carbonTable.getAbsoluteTableIdentifier();
+  long segmentLockFilesPreservTime =
+  
CarbonProperties.getInstance().getSegmentLockFilesPreserveHours();
+  long currTime = System.currentTimeMillis();
+  for (LoadMetadataDetails oneRow : details) {
+if (oneRow.getVisibility().equalsIgnoreCase("false") ||
+SegmentStatus.SUCCESS == oneRow.getSegmentStatus() ||
+SegmentStatus.LOAD_FAILURE == oneRow.getSegmentStatus() ||
+SegmentStatus.LOAD_PARTIAL_SUCCESS == 
oneRow.getSegmentStatus() ||
+SegmentStatus.COMPACTED == oneRow.getSegmentStatus()) {
+  String location = CarbonTablePath
+  .getLockFilesDirPath(absoluteTableIdentifier.getTablePath()) 
+
+  CarbonCommonConstants.FILE_SEPARATOR +
+  CarbonTablePath.addSegmentPrefix(oneRow.getLoadName()) + 
LockUsage.LOCK;
--- End diff --

Done


---

[GitHub] carbondata issue #2054: [CARBONDATA-2224][File Level Reader Support] Externa...

[ajantha-bhat]

Github user ajantha-bhat commented on the issue:

https://github.com/apache/carbondata/pull/2054
  
@jackylk : Binary files [carbondata and index files (sdk Writer output)] 
are intentionally added for test cases of this requirement. Test cases will 
fail If we remove them.


---

[GitHub] carbondata issue #2050: [CARBONDATA-2244]fix creating pre-aggregate table bu...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2050
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2968/



---

[GitHub] carbondata pull request #2054: [CARBONDATA-2224][File Level Reader Support] ...

[ajantha-bhat]

Github user ajantha-bhat commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2054#discussion_r173885274
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java
 ---
@@ -826,6 +826,12 @@ public boolean isExternalTable() {
 return external != null && external.equalsIgnoreCase("true");
   }
 
+  public boolean isFileLevelExternalTable() {
--- End diff --

**stored by 'carbondatafileformat' is mapped with _filelevelexternal.**
So, In carbonScanRDD, when mapReduce service or hadoop service calls 
carbonScanRDD.
based on _filelevelexternal, new File level reader [CarbonFileInputFormat] 
will be called.

External table can be table level (stored by 'carbondata') or file level 
(stored by 'carbondatafileformat')
**This is used to identify the file level external table.**


---

[GitHub] carbondata issue #2050: [CARBONDATA-2244]fix creating pre-aggregate table bu...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2050
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4213/



---

[GitHub] carbondata issue #1856: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1856
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3856/



---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4214/



---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2969/



---

[GitHub] carbondata issue #2053: [CARBONDATA-2247][SDK] Support write Index file in C...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2053
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2967/



---

[GitHub] carbondata issue #2050: [CARBONDATA-2244]fix creating pre-aggregate table bu...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2050
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4211/



---

[GitHub] carbondata issue #2031: [CARBONDATA-2223] Remove unused listeners

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2031
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2966/



---

[GitHub] carbondata issue #2053: [CARBONDATA-2247][SDK] Support write Index file in C...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2053
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4212/



---

[GitHub] carbondata issue #1857: [CARBONDATA-2073][CARBONDATA-1516][Tests] Add test c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1857
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3855/



---

[GitHub] carbondata issue #2050: [CARBONDATA-2244]fix creating pre-aggregate table bu...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2050
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2965/



---

[GitHub] carbondata issue #2054: [CARBONDATA-2224][File Level Reader Support] Externa...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2054
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4210/



---

[GitHub] carbondata issue #2050: [CARBONDATA-2244]fix creating pre-aggregate table bu...

[zzcclp]

Github user zzcclp commented on the issue:

https://github.com/apache/carbondata/pull/2050
  
@jackylk please help to review, thanks.


---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2963/



---

[GitHub] carbondata issue #2045: [CARBONDATA-2230]Add a path into table path to store...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2045
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4208/



---

[GitHub] carbondata issue #2053: [CARBONDATA-2247][SDK] Support write Index file in C...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2053
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3854/



---