[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2131
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4829/



---

[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2131
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3606/



---

[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...

[ajantha-bhat]

Github user ajantha-bhat commented on the issue:

https://github.com/apache/carbondata/pull/2131
  
retest this please


---

[GitHub] carbondata pull request #2134: [WIP][CARBONDATA-2310] Refactored code to imp...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2134#discussion_r179341666
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java
 ---
@@ -186,23 +193,28 @@ public void init(DataMapModel dataMapModel) throws 
IOException, MemoryException
 }
   }
 }
-if (unsafeMemoryDMStore != null) {
-  unsafeMemoryDMStore.finishWriting();
-}
 if (null != unsafeMemorySummaryDMStore) {
   addTaskSummaryRowToUnsafeMemoryStore(
   summaryRow,
   schemaBinary,
   filePath,
   fileName,
   segmentId);
-  unsafeMemorySummaryDMStore.finishWriting();
 }
 LOGGER.info(
 "Time taken to load blocklet datamap from file : " + 
dataMapModel.getFilePath() + "is " + (
 System.currentTimeMillis() - startTime));
   }
 
+  @Override public void commit() throws MemoryException, IOException {
--- End diff --

This is specific to BlockletDataMap so lets only keep this class not at 
interface level.
And also I am not sure why we should separate a commit method. Is there any 
way to avoid this method?


---

[GitHub] carbondata pull request #2134: [WIP][CARBONDATA-2310] Refactored code to imp...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2134#discussion_r179341377
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/indexstore/UnsafeMemoryDMStore.java
 ---
@@ -52,11 +62,23 @@
 
   private final long taskId = 
ThreadLocalTaskInfo.getCarbonTaskInfo().getTaskId();
 
-  public UnsafeMemoryDMStore(CarbonRowSchema[] schema) throws 
MemoryException {
+  private boolean addRowToUnsafe = true;
--- End diff --

Class name itself `UnsafeMemoryDMStore` so it does not make sense to have 
this variable.
Please create AbstractClass and give two implementations with safe and 
Unsafe.


---

[GitHub] carbondata pull request #2134: [WIP][CARBONDATA-2310] Refactored code to imp...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2134#discussion_r179340305
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datamap/dev/DataMapFactory.java 
---
@@ -62,6 +63,12 @@
*/
   void fireEvent(Event event);
 
+  /**
+   * Add the dataMap to cache
+   * @param dataMap
+   */
+  void addDataMapToCache(DataMap dataMap) throws IOException, 
MemoryException;
--- End diff --

Better add these methods in another interface CacheableDataMap and 
implement it to BlockletDataMapFactory directly.
```
CacheableDataMap {
  void cache(List dataMaps)
  List getAllUncachedDataMaps()
}
```


---

[GitHub] carbondata pull request #2134: [WIP][CARBONDATA-2310] Refactored code to imp...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2134#discussion_r179339971
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datamap/dev/DataMap.java ---
@@ -67,4 +68,14 @@
*/
   void clear();
 
+  /**
+   * This method will be required for dataMaps that require 2 stage of 
construction.
+   * Ideal scenario will be first stage contains all the processing logic 
and second
+   * stage includes updating to database.
+   * Method usage can differ based on scenario and implementation
+   *
+   * @throws MemoryException
+   */
+  void commit() throws MemoryException, IOException;
--- End diff --

I think it does not make sense to have this method in interface level. 
Caching should only be restricted to DataMapFactory not to the level of 
DataMap. 


---

[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2131
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4300/



---

[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2131
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3604/



---

[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2131
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4827/



---

[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2113
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4298/



---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3603/



---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4826/



---

[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2113
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4297/



---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[anubhav100]

Github user anubhav100 commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
retest this please


---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4825/



---

[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2113
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4296/



---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3602/



---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[anubhav100]

Github user anubhav100 commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
@chenliang613 please review this pr


---

[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2131
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4295/



---

[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2109
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3600/



---

[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2109
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4823/



---

[GitHub] carbondata issue #1940: [CARBONDATA-2140 ] Refactoring code to improve perfo...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1940
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3599/



---

[GitHub] carbondata issue #1940: [CARBONDATA-2140 ] Refactoring code to improve perfo...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1940
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4822/



---

[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2131
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4294/



---

[GitHub] carbondata issue #2135: [CARBONDATA-2311][Streaming] Fix bug to avoid to app...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2135
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4293/



---

[GitHub] carbondata pull request #2131: [CARBONDATA-2313] Support unmanaged carbon ta...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2131#discussion_r179154952
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
@@ -2910,10 +2914,16 @@ private static long getMaxOfBlockAndFileSize(long 
blockSize, long fileSize) {
* @return
*/
   public static String getBlockId(AbsoluteTableIdentifier identifier, 
String filePath,
-  String segmentId) {
+  String segmentId, boolean isUnmangedTable) {
 String blockId;
 String blockName = filePath.substring(filePath.lastIndexOf("/") + 1, 
filePath.length());
 String tablePath = identifier.getTablePath();
+
--- End diff --

this block id should be commonly constructed for unmanaged and managed 
case. As unmanged partition is also possible


---

[GitHub] carbondata pull request #2131: [CARBONDATA-2313] Support unmanaged carbon ta...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2131#discussion_r179150325
  
--- Diff: 
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
 ---
@@ -128,12 +134,18 @@ protected CarbonTable 
getOrCreateCarbonTable(Configuration configuration) throws
   @Override
   public List getSplits(JobContext job) throws IOException {
 AbsoluteTableIdentifier identifier = 
getAbsoluteTableIdentifier(job.getConfiguration());
-LoadMetadataDetails[] loadMetadataDetails = SegmentStatusManager
-
.readTableStatusFile(CarbonTablePath.getTableStatusFilePath(identifier.getTablePath()));
+
 CarbonTable carbonTable = 
getOrCreateCarbonTable(job.getConfiguration());
 if (null == carbonTable) {
   throw new IOException("Missing/Corrupt schema file for table.");
 }
+this.readCommittedScope = getReadCommitted(job, identifier);
--- End diff --

When empty data.. select should give empty records


---

[GitHub] carbondata pull request #2131: [CARBONDATA-2313] Support unmanaged carbon ta...

[sounakr]

Github user sounakr commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2131#discussion_r179148873
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/processing/loading/model/CarbonLoadModel.java
 ---
@@ -47,6 +47,13 @@
 
   private String tablePath;
 
+  /*
+ This points if the carbonTable is a Unmanaged Table or not.
+ The path will be pointed by the tablePath. And there will be
+ no Metadata folder present for the unmanaged Table.
+   */
+  private boolean carbonUnmanagedTable;
--- End diff --

Done


---

[GitHub] carbondata issue #2137: [CARBONDATA-2314] Removed block for Streaming with P...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2137
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4292/



---

[GitHub] carbondata issue #2137: [CARBONDATA-2314] Removed block for Streaming with P...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2137
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4821/



---

[GitHub] carbondata issue #2137: [CARBONDATA-2314] Removed block for Streaming with P...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2137
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3598/



---

[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support

[manishgupta88]

Github user manishgupta88 commented on the issue:

https://github.com/apache/carbondata/pull/2109
  
retest sdv please


---

[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2109
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4291/



---

[jira] [Created] (CARBONDATA-2314) Data mismatch in Pre-Aggregate table after Streaming load due to threadset issue

[Praveen M P (JIRA)]

Praveen M P created CARBONDATA-2314:
---

 Summary: Data mismatch in Pre-Aggregate table after Streaming load 
due to threadset issue
 Key: CARBONDATA-2314
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2314
 Project: CarbonData
  Issue Type: Bug
Reporter: Praveen M P
Assignee: Praveen M P


Wrong data insertion into Pre-Aggregate table on Streaming table due to 
ThreadSet issue



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2131
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3597/



---

[GitHub] carbondata issue #2131: [CARBONDATA-2313] Support unmanaged carbon table rea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2131
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4820/



---

[GitHub] carbondata pull request #2131: [CARBONDATA-2313] Support unmanaged carbon ta...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2131#discussion_r179085175
  
--- Diff: 
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java 
---
@@ -152,15 +167,23 @@ private CarbonTable buildCarbonTable() {
   new StructField(field.getFieldName(), field.getDataType()),
   sortColumnsList.contains(field.getFieldName()));
 }
-String tableName = "_tempTable";
-String dbName = "_tempDB";
+String tableName;
--- End diff --

Not required to add tablename here, in this class, keep it same as previous


---

[GitHub] carbondata pull request #2131: [CARBONDATA-2313] Support unmanaged carbon ta...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2131#discussion_r179088904
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/processing/loading/model/CarbonLoadModel.java
 ---
@@ -47,6 +47,13 @@
 
   private String tablePath;
 
+  /*
+ This points if the carbonTable is a Unmanaged Table or not.
+ The path will be pointed by the tablePath. And there will be
+ no Metadata folder present for the unmanaged Table.
+   */
+  private boolean carbonUnmanagedTable;
--- End diff --

already class name has carbon word, so not required to repeat, this again. 
better use "unmanagedTable"


---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4819/



---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3596/



---

[jira] [Created] (CARBONDATA-2313) Support Reading unmanaged carbon table

[Ajantha Bhat (JIRA)]

Ajantha Bhat created CARBONDATA-2313:


 Summary: Support Reading unmanaged carbon table
 Key: CARBONDATA-2313
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2313
 Project: CarbonData
  Issue Type: Bug
Reporter: Ajantha Bhat






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #2131: [WIP] Support unmanaged carbon table read and write

[sounakr]

Github user sounakr commented on the issue:

https://github.com/apache/carbondata/pull/2131
  
Retest this please


---

[GitHub] carbondata pull request #2131: [WIP] Support unmanaged carbon table read and...

[sounakr]

Github user sounakr commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2131#discussion_r179098290
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonCountStar.scala ---
@@ -76,6 +76,11 @@ case class CarbonCountStar(
 SparkHadoopUtil.get.addCredentials(jobConf)
 val job = new Job(jobConf)
 FileInputFormat.addInputPath(job, new 
Path(absoluteTableIdentifier.getTablePath))
+if (carbonTable.getTableInfo.isUnManagedTable) {
+  CarbonInputFormat.setCarbonUnmanagedTable(job.getConfiguration, 
"true")
--- End diff --

Done


---

[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2134
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3591/



---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3595/



---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[anubhav100]

Github user anubhav100 commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
retest this please


---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4818/



---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[anubhav100]

Github user anubhav100 commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
retest this please


---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4816/



---

[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2134
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3588/



---

[GitHub] carbondata issue #2135: [CARBONDATA-2311][Streaming] Fix bug to avoid to app...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2135
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3587/



---

[GitHub] carbondata issue #2135: [CARBONDATA-2311][Streaming] Fix bug to avoid to app...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2135
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4811/



---

[GitHub] carbondata pull request #2139: [CARBONDATA-2267] [Presto] Support Reading Ca...

[anubhav100]

GitHub user anubhav100 opened a pull request:

https://github.com/apache/carbondata/pull/2139

[CARBONDATA-2267] [Presto] Support Reading CarbonData Partition From Presto 
Integration

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anubhav100/incubator-carbondata 
prestopartition

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2139.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2139


commit 9a8cf766e97882a3671cba8d566489f5918cc948
Author: anubhav100 
Date:   2018-04-04T08:18:15Z

added logic for fetching the partition columns in presto

commit 3faf8e0e32e175e89b43edaceef23da0a03927b6
Author: Geetika Gupta 
Date:   2018-04-04T08:37:36Z

Refactored CarbonTableReader to add partition spec to configuration object




---

[GitHub] carbondata pull request #2136: [CARBONDATA-2307] Fix OOM issue when using Da...

[manishgupta88]

Github user manishgupta88 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2136#discussion_r179080266
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datastore/chunk/impl/MeasureRawColumnChunk.java
 ---
@@ -106,9 +106,13 @@ public ColumnPage convertToColumnPageWithOutCache(int 
index) {
   for (int i = 0; i < columnPages.length; i++) {
 if (columnPages[i] != null) {
   columnPages[i].freeMemory();
+  columnPages[i] = null;
 }
   }
 }
+if (rawData != null) {
+  rawData = null;
+}
--- End diff --

Same comment as above


---

[GitHub] carbondata pull request #2136: [CARBONDATA-2307] Fix OOM issue when using Da...

[manishgupta88]

Github user manishgupta88 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2136#discussion_r179080396
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datastore/chunk/impl/DimensionRawColumnChunk.java
 ---
@@ -106,9 +106,13 @@ public DimensionColumnPage 
convertToDimColDataChunkWithOutCache(int index) {
   for (int i = 0; i < dataChunks.length; i++) {
 if (dataChunks[i] != null) {
   dataChunks[i].freeMemory();
+  dataChunks[i] = null;
 }
   }
 }
+if (rawData != null) {
+  rawData = null;
+}
--- End diff --

No need for null check as rawData is being set only to null. Better to 
implement freeMemory method in super class AbstractRawColumnChunk and set 
rawData to null in that and call super.freeMemory() from child classes.


---

[GitHub] carbondata pull request #2136: [CARBONDATA-2307] Fix OOM issue when using Da...

[manishgupta88]

Github user manishgupta88 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2136#discussion_r179074922
  
--- Diff: 
integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/CarbonScanRDD.scala
 ---
@@ -438,7 +438,9 @@ class CarbonScanRDD(
   // for usage of unsafe method for processing of one blocklet and if 
there is any exception
   // while doing that the unsafe memory occupied for that task will 
not get cleared
   context.addTaskCompletionListener { _ =>
-reader.close()
+if (reader != null) {
+  reader.close()
+}
--- End diff --

Remove null check and reader.close() call and call closeReader() method here


---

[GitHub] carbondata pull request #2136: [CARBONDATA-2307] Fix OOM issue when using Da...

[manishgupta88]

Github user manishgupta88 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2136#discussion_r179074140
  
--- Diff: 
integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/CarbonScanRDD.scala
 ---
@@ -402,7 +402,7 @@ class CarbonScanRDD(
   // one query id per table
   model.setQueryId(queryId)
   // get RecordReader by FileFormat
-  val reader: RecordReader[Void, Object] = inputSplit.getFileFormat 
match {
+  var reader: RecordReader[Void, Object] = inputSplit.getFileFormat 
match {
--- End diff --

reader can be val...is there any specific purpose for making it var?


---

[GitHub] carbondata issue #2134: [WIP][CARBONDATA-2310] Refactored code to improve Di...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2134
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4812/



---

[GitHub] carbondata issue #2131: [WIP] Support unmanaged carbon table read and write

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2131
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3585/



---

[GitHub] carbondata issue #2131: [WIP] Support unmanaged carbon table read and write

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2131
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4809/



---

[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2109
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4808/



---

[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2109
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3584/



---

[GitHub] carbondata issue #2135: [CARBONDATA-2311][Streaming] Fix bug to avoid to app...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2135
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4807/



---

[GitHub] carbondata issue #2135: [CARBONDATA-2311][Streaming] Fix bug to avoid to app...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2135
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3583/



---

[GitHub] carbondata issue #2136: [CARBONDATA-2307] Fix OOM issue when using DataFrame...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2136
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4804/



---

[GitHub] carbondata issue #2138: [CARBONDATA-2230][Documentation]add documentation fo...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2138
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3582/



---

[GitHub] carbondata issue #2138: [CARBONDATA-2230][Documentation]add documentation fo...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2138
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4806/



---

[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2109
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3580/



---

[GitHub] carbondata issue #2109: [CARBONDATA-2294] Partition preaggregate support

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2109
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4803/



---

[GitHub] carbondata issue #2103: [CARBONDATA-2312]Support In Memory Catalog

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2103
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4802/



---

[GitHub] carbondata issue #2138: [CARBONDATA-2230][Documentation]add documentation fo...

[sgururajshetty]

Github user sgururajshetty commented on the issue:

https://github.com/apache/carbondata/pull/2138
  
LGTM


---

[GitHub] carbondata issue #2103: [CARBONDATA-2312]Support In Memory Catalog

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2103
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3579/



---

[GitHub] carbondata pull request #2138: [CARBONDATA-2230][Documentation]add documenta...

[sgururajshetty]

Github user sgururajshetty commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2138#discussion_r179039767
  
--- Diff: docs/configuration-parameters.md ---
@@ -39,6 +39,7 @@ This section provides the details of all the 
configurations required for the Car
 | carbon.streaming.auto.handoff.enabled | true | If this parameter value 
is set to true, auto trigger handoff function will be enabled.|
 | carbon.streaming.segment.max.size | 102400 | This parameter defines 
the maximum size of the streaming segment. Setting this parameter to 
appropriate value will avoid impacting the streaming ingestion. The value is in 
bytes.|
 | carbon.query.show.datamaps | true | If this parameter value is set to 
true, show tables command will list all the tables including datatmaps(eg: 
Preaggregate table), else datamaps will be excluded from the table list. |
+| carbon.segment.lock.files.preserve.hours | 48 | This property value 
indicates the number of hours the segment lock files will be preserved after 
dataload. These lock fils will be deleted with clean files command after the 
configured amount of hours. |
--- End diff --

Spelling error "fils"

These lock files will be deleted with the clean command after the 
configured number of hours. 


---

[GitHub] carbondata pull request #2138: [CARBONDATA-2230][Documentation]add documenta...

[akashrn5]

GitHub user akashrn5 opened a pull request:

https://github.com/apache/carbondata/pull/2138

[CARBONDATA-2230][Documentation]add documentation for segment lock files 
clean up configuration

added documentation for segment lock files clean up configuration

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/akashrn5/incubator-carbondata patch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2138.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2138


commit 97424c066d1d8f719566aff186ad2373475d748b
Author: akashrn5 
Date:   2018-04-04T06:36:21Z

Update configuration-parameters.md




---

[GitHub] carbondata issue #2136: [CARBONDATA-2307] Fix OOM issue when using DataFrame...

[Xaprice]

Github user Xaprice commented on the issue:

https://github.com/apache/carbondata/pull/2136
  
retest this please


---