[GitHub] carbondata issue #1104: [CARBONDATA-1239] Add validation for set command par...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1104
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1700/



---

[GitHub] carbondata issue #1823: [CARBONDATA-2045][PreAggregate]Fixed Pre Aggregate f...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1823
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/2952/



---

[GitHub] carbondata issue #1821: [HOTFIX] Listeners not getting registered to the bus...

[kunal642]

Github user kunal642 commented on the issue:

https://github.com/apache/carbondata/pull/1821
  
@jackylk handled all the review comments.


---

[GitHub] carbondata pull request #1829: [CARBONDATA-2050] Add example of query data w...

[chenliang613]

GitHub user chenliang613 opened a pull request:

https://github.com/apache/carbondata/pull/1829

[CARBONDATA-2050] Add example of query data with specified segments

Add example of query data with specified segments and also consider 
compaction sceanrios.

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [X] Any interfaces changed?
 NA
 - [X] Any backward compatibility impacted?
 NA
 - [X] Document update required?
YES
 - [X] Testing done
YES
   
 - [X] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
NA


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chenliang613/carbondata querysegment

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1829.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1829


commit f778c024802559dd06ea233a5244ca01ed1c5530
Author: chenliang613 
Date:   2018-01-18T07:50:23Z

[CARBONDATA-2050] Add example of query data with specified segments




---

[GitHub] carbondata issue #1815: [CARBONDATA-2047] Clean up temp folder after task co...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1815
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2932/



---

[GitHub] carbondata issue #1824: [CARBONDATA-2046]agg Query failed when non supported...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1824
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/2951/



---

[GitHub] carbondata issue #1815: [CARBONDATA-2047] Clean up temp folder after task co...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1815
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1699/



---

[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1825
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/2950/



---

[GitHub] carbondata issue #1826: [CARBONDATA-2048] Data delete should be rejected whe...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1826
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/2949/



---

[GitHub] carbondata issue #1827: [HOTFIX] Ignore one testcase to improve CI stability

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1827
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/2948/



---

[GitHub] carbondata issue #1815: [CARBONDATA-2047] Clean up temp folder after task co...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1815
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/2947/



---

[GitHub] carbondata pull request #1821: [HOTFIX] Listeners not getting registered to ...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1821#discussion_r162264895
  
--- Diff: 
core/src/main/java/org/apache/carbondata/events/OperationEventListener.java ---
@@ -19,13 +19,29 @@
 /**
  * Event listener interface which describes the possible events
  */
-public interface OperationEventListener {
+public abstract class OperationEventListener {
 
   /**
* Called on a specified event occurrence
*
* @param event
* @param operationContext
*/
-  void onEvent(Event event, OperationContext operationContext) throws 
Exception;
+  protected abstract void onEvent(Event event, OperationContext 
operationContext) throws Exception;
+
+  @Override
+  public boolean equals(Object obj) {
+if (obj == null || !(obj instanceof OperationEventListener)) {
+  return false;
+}
+return getComparisonName().equals(((OperationEventListener) 
obj).getComparisonName());
+  }
+
+  private String getComparisonName() {
+return getClass().getName();
+  }
+
+  @Override public int hashCode() {
--- End diff --

move @Override to previous line


---

[GitHub] carbondata issue #1822: [WIP][CARBONDATA-2043] Configurable wait time for re...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1822
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2931/



---

[GitHub] carbondata pull request #1821: [HOTFIX] Listeners not getting registered to ...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1821#discussion_r162264804
  
--- Diff: integration/spark2/src/main/spark2.1/CarbonSessionState.scala ---
@@ -75,6 +75,8 @@ class CarbonSessionCatalog(
 env
   }
 
+  CarbonEnv.initListeners()
--- End diff --

There is a CarbonEnv.init function, can you put this initialization inside 
that


---

[GitHub] carbondata pull request #1828: [CARBONDATA-2049] CarbonCleanFilesCommand tab...

[mohammadshahidkhan]

GitHub user mohammadshahidkhan opened a pull request:

https://github.com/apache/carbondata/pull/1828

[CARBONDATA-2049] CarbonCleanFilesCommand table path problem

**Problem:**
In CarbonCleanFilesCommand datbaseLocation is being passed instead of the 
tablePath in case of forceclean.
And in case of cleanGarbageData, storeLocation is being passed instead of 
the tablePath.


 - [X] Any interfaces changed?
 None
 - [X] Any backward compatibility impacted?
 None
 - [X] Document update required?
None
 - [X] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
Corrected test case
 - [X] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
NA


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mohammadshahidkhan/incubator-carbondata 
cleancommand_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1828.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1828


commit 8bfeb75b9af0de7d02f94a593c283d6d99df6f92
Author: mohammadshahidkhan 
Date:   2018-01-16T06:19:54Z

[CARBONDATA-2049] CarbonCleanFilesCommand table path problem




---

[GitHub] carbondata issue #1822: [WIP][CARBONDATA-2043] Configurable wait time for re...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1822
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1698/



---

[GitHub] carbondata issue #1792: [CARBONDATA-2018][DataLoad] Optimization in reading/...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1792
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2930/



---

[GitHub] carbondata pull request #1827: [HOTFIX] Ignore one testcase to improve CI st...

[jackylk]

GitHub user jackylk opened a pull request:

https://github.com/apache/carbondata/pull/1827

[HOTFIX] Ignore one testcase to improve CI stability

Ignore one testcase to improve CI stability

 - [X] Any interfaces changed?
 No
 - [X] Any backward compatibility impacted?
 No
 - [X] Document update required?
No
 - [X] Testing done
One testcase removed
   
 - [X] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
NA

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jackylk/incubator-carbondata remove_listener

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1827.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1827


commit 9f99761cd07dc210b4228529d6d992032db6f96a
Author: Jacky Li 
Date:   2018-01-18T07:01:30Z

ignore one testcase




---

[GitHub] carbondata issue #1792: [CARBONDATA-2018][DataLoad] Optimization in reading/...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1792
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1697/



---

[jira] [Created] (CARBONDATA-2050) Add example of query data with specified segments

[Liang Chen (JIRA)]

Liang Chen created CARBONDATA-2050:
--

 Summary: Add example of query data with specified segments
 Key: CARBONDATA-2050
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2050
 Project: CarbonData
  Issue Type: Task
  Components: examples
Reporter: Liang Chen
Assignee: Liang Chen


Add example of query data with specified segments and also consider compaction 
sceanrios

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (CARBONDATA-2049) CarbonCleanFilesCommand table path problem

[Mohammad Shahid Khan (JIRA)]

Mohammad Shahid Khan created CARBONDATA-2049:


 Summary: CarbonCleanFilesCommand table path problem 
 Key: CARBONDATA-2049
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2049
 Project: CarbonData
  Issue Type: Bug
Reporter: Mohammad Shahid Khan
Assignee: Mohammad Shahid Khan


In CarbonCleanFilesCommand datbaseLocation is being passed instead of the 
tablePath in case of forceclean.

And in case of cleanGarbageData, storeLocation is being passed instead of the 
tablePath.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1808: [CARBONDATA-2023][DataLoad] Add size base block allo...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1808
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2929/



---

[GitHub] carbondata issue #1808: [CARBONDATA-2023][DataLoad] Add size base block allo...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1808
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1696/



---

[GitHub] carbondata issue #1815: [CARBONDATA-2047] Clean up temp folder after task co...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1815
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2928/



---

[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...

[xuchuanyin]

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/1825
  
@ravipesala thanks, I got your point...


---

[GitHub] carbondata issue #1815: [CARBONDATA-2047] Clean up temp folder after task co...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1815
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1695/



---

[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...

[geetikagupta16]

Github user geetikagupta16 commented on the issue:

https://github.com/apache/carbondata/pull/1819
  
@mohammadshahidkhan I have made some changes. Please check


---

[GitHub] carbondata issue #1774: [CARBONDATA-2001] Unable to Save DataFrame As Carbon...

[anubhav100]

Github user anubhav100 commented on the issue:

https://github.com/apache/carbondata/pull/1774
  
retest this please


---

[GitHub] carbondata issue #1792: [CARBONDATA-2018][DataLoad] Optimization in reading/...

[xuchuanyin]

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/1792
  
retest this please


---

[GitHub] carbondata issue #1808: [CARBONDATA-2023][DataLoad] Add size base block allo...

[xuchuanyin]

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/1808
  
retest this please


---

[GitHub] carbondata issue #1816: [CARBONDATA-2038][Tests] use junit assertion in java...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1816
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2927/



---

[GitHub] carbondata issue #1816: [CARBONDATA-2038][Tests] use junit assertion in java...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1816
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1694/



---

[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1825
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2926/



---

[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1825
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1693/



---

[GitHub] carbondata pull request #1810: [CARBONDATA-2037]Store carbondata locations i...

[manishgupta88]

Github user manishgupta88 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1810#discussion_r162253805
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java
 ---
@@ -598,16 +627,16 @@ public boolean isScanRequired(FilterResolverIntf 
filterExp) {
 if (filterExp == null) {
   int rowCount = unsafeMemoryDMStore.getRowCount();
   for (int i = 0; i < rowCount; i++) {
-DataMapRow unsafeRow = unsafeMemoryDMStore.getUnsafeRow(i);
-blocklets.add(createBlocklet(unsafeRow, i));
+DataMapRow safeRow = 
unsafeMemoryDMStore.getUnsafeRow(i).convertToSafeRow();
+blocklets.add(createBlocklet(safeRow, i));
   }
 } else {
   int startIndex = findStartIndex(convertToRow(searchStartKey), 
comparator);
   int endIndex = findEndIndex(convertToRow(searchEndKey), comparator);
   FilterExecuter filterExecuter =
   FilterUtil.getFilterExecuterTree(filterExp, segmentProperties, 
null);
   while (startIndex <= endIndex) {
-DataMapRow unsafeRow = 
unsafeMemoryDMStore.getUnsafeRow(startIndex);
+DataMapRow unsafeRow = 
unsafeMemoryDMStore.getUnsafeRow(startIndex).convertToSafeRow();
--- End diff --

Please rename this variable also to safeRow


---

[GitHub] carbondata issue #1806: modify default config: change the default of tempCSV...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1806
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2924/



---

[GitHub] carbondata issue #1826: [CARBONDATA-2048] Data delete should be rejected whe...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1826
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1692/



---

[GitHub] carbondata issue #1826: [CARBONDATA-2048] Data delete should be rejected whe...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1826
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2925/



---

[GitHub] carbondata issue #1806: modify default config: change the default of tempCSV...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1806
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1691/



---

[GitHub] carbondata issue #1815: [CARBONDATA-2047] Clean up temp folder after task co...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1815
  
retest this please


---

[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1825
  
@xuchuanyin There is a reason why we do copy instead of directly writing to 
HDFS.
1. We make sure that one complete carbondata file goes to one HDFS block 
only, while copying it to HDFS from local disk we specify the block size. Other 
wise it impacts query performance a lot.
2. Remove the overhead of writing to HDFS directly (it internally writes to 
replication as well) , so start copying in a different thread to avoid blocking 
of main loading flow.


---

[GitHub] carbondata issue #1822: [CARBONDATA-2043] Configurable wait time for request...

[mohammadshahidkhan]

Github user mohammadshahidkhan commented on the issue:

https://github.com/apache/carbondata/pull/1822
  
retest this please


---

[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...

[mohammadshahidkhan]

Github user mohammadshahidkhan commented on the issue:

https://github.com/apache/carbondata/pull/1819
  
@geetikagupta16  
org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala:507
if (loadStatus == SegmentStatus.LOAD_PARTIAL_SUCCESS &&
  status(0)._2._2.failureCauses == FailureCauses.BAD_RECORDS &&
  **carbonLoadModel.getBadRecordsAction.split(",")(1) == 
LoggerAction.FAIL.name**) 
The problem is checking equally without ignoring the case.
The same problem will arise even with load options option 
'bad_records_action'='fail'.
Please correct there as well and add test case to cover this scenario.


---

[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1812
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2923/



---

[GitHub] carbondata issue #1817: [CARBONDATA-2040] Add standard partiton example and ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1817
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1689/



---

[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1812
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1690/



---

[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1812
  
@Xaprice I think we should have validation for the order of segments to be 
merged. For suppose we have segments of 1 to 8, and the user gives the 
compaction on 1, 5, 8 then this should not be valid as it will impact the order 
of data it is inserted. 


---

[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...

[chenliang613]

Github user chenliang613 commented on the issue:

https://github.com/apache/carbondata/pull/1812
  
Two questions:
1. Why only consider major compaction with specified segments, no need to 
consider minor compaction?
2. Whether can keep consistent syntax as "query with specified segments", 
or not ?
a. First set segment id : "SET carbon.input.segments.dbname.tablename=1,3"
b.Do compaction : ALTER TABLE tablename compact 'MAJOR' 



---

[GitHub] carbondata pull request #1812: [CARBONDATA-2033]support user specified segme...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1812#discussion_r162245791
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonDataMergerUtil.java
 ---
@@ -441,6 +452,30 @@ public int compare(LoadMetadataDetails seg1, 
LoadMetadataDetails seg2) {
 });
   }
 
+  /**
+   * This method will return the list of loads which are specified by user 
in SQL.
+   *
+   * @param listOfSegmentsLoadedInSameDateInterval
+   * @param segmentIds
+   * @return
+   */
+  private static List 
identitySegmentsToBeMergedBasedOnSpecifiedSegments(
+  List listOfSegmentsLoadedInSameDateInterval,
+  Set segmentIds) {
+List listOfSegmentsSpecified =
+new ArrayList<>(CarbonCommonConstants.DEFAULT_COLLECTION_SIZE);
+if (segmentIds != null && segmentIds.size() != 0) {
+  for (LoadMetadataDetails detail : 
listOfSegmentsLoadedInSameDateInterval) {
+if (isSegmentValid(detail) && 
segmentIds.contains(detail.getLoadName())) {
--- End diff --

If the specified segment is not valid, better throw exception about invalid 
segments instead of ignoring it


---

[GitHub] carbondata pull request #1826: [CARBONDATA-2048] Data delete should be rejec...

[jackylk]

GitHub user jackylk opened a pull request:

https://github.com/apache/carbondata/pull/1826

[CARBONDATA-2048] Data delete should be rejected when insert overwrite is 
in progress

Add testcase and fix bug: data delete should be rejected when insert 
overwrite is in progress

 - [X] Any interfaces changed?
 No
 - [X] Any backward compatibility impacted?
 No
 - [X] Document update required?
No
 - [X] Testing done
Testcase added
 - [X] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
NA

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jackylk/incubator-carbondata add_testcase

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1826.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1826


commit 212074e5fbd1f2cd01ddf2b507bf0c65d4296a34
Author: Jacky Li 
Date:   2018-01-18T04:03:56Z

add test

commit b2acee20523e8f58bfea387bf2b692636c41d91e
Author: Jacky Li 
Date:   2018-01-18T04:09:20Z

update




---

[jira] [Created] (CARBONDATA-2048) Data delete should be rejected when insert overwrite is in progress

[Jacky Li (JIRA)]

Jacky Li created CARBONDATA-2048:


 Summary: Data delete should be rejected when insert overwrite is 
in progress
 Key: CARBONDATA-2048
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2048
 Project: CarbonData
  Issue Type: Bug
Reporter: Jacky Li
 Fix For: 1.3.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1816: [CARBONDATA-2038][Tests] use junit assertion in java...

[xuchuanyin]

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/1816
  
retest this please


---

[jira] [Closed] (CARBONDATA-2035) Incorrect assert in code leads to tests failed

[xuchuanyin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xuchuanyin closed CARBONDATA-2035.
--
Resolution: Duplicate

> Incorrect assert in code leads to tests failed
> --
>
> Key: CARBONDATA-2035
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2035
> Project: CarbonData
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.3.0
>Reporter: xuchuanyin
>Assignee: xuchuanyin
>Priority: Major
> Fix For: 1.3.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Today I came across tests failure when I ran tests in Intellij Idea. The code 
> was on the master branch, and it had not been modified after I fetched the 
> latest code.
>  
> The failed test lies in `TestLoadDataGeneral.test("test load / insert / 
> update with data more than 32000 bytes - dictionary_exclude")`.  It complains 
> that `"DataLoad failure: there is an unexpected error: null" did not contain 
> "Dataload failure: DataLoad failed, String size cannot exceed 32000 bytes"`.
>  
> After I study the code, I find out that in 
> `NonDictionaryFieldConverterImpl#line74 and #line77`, carbondata use an 
> assert to raise the exception. But the exception is not thrown when I debug 
> this tests. After some research, I found that the tests will success only if 
> I enable `-ea` option. (run -> Edit configurations. Configuration -> VM 
> parameters)
> By default, the "VM parameters" is empty and `-ea` is disabled, so the assert 
> sentence in code will not come into effect, thus causing this problem.
> In conclusion, carbondata should not detect exception in this way, we 
> need to correct it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1814: [CARBONDATA-2035][DataLoad][Tests] Fix bugs i...

[xuchuanyin]

Github user xuchuanyin closed the pull request at:

https://github.com/apache/carbondata/pull/1814


---

[GitHub] carbondata issue #1814: [CARBONDATA-2035][DataLoad][Tests] Fix bugs in raisi...

[xuchuanyin]

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/1814
  
This issue has been fixed by another PR, so close it here


---

[GitHub] carbondata pull request #1825: [CARBONDATA-2032][DataLoad] directly write ca...

[xuchuanyin]

GitHub user xuchuanyin opened a pull request:

https://github.com/apache/carbondata/pull/1825

[CARBONDATA-2032][DataLoad] directly write carbon data files to HDFS

Currently in data loading, carbondata write the final data files to local 
disk and then copy it to HDFS.
For saving disk IO, carbondata can skip this procedure and directly write 
these files to HDFS.

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [x] Any interfaces changed?
 `Only internal interfaces has been changed`
 - [x] Any backward compatibility impacted?
 `No`
 - [x] Document update required?
`No`
 - [x] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
`No`
- How it is tested? Please attach test report.
`Tested in local node and a 3-nodes cluster`
- Is it a performance related change? Please attach the performance 
test report.
`Yes. The disk IO has decreased`
- Any additional information to help reviewers in testing this 
change.
`No`
 - [x] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
`Not related`


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xuchuanyin/carbondata 
0118_opt_write_data_files_directly_to_hdfs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1825.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1825


commit fdced241cd5508d2fc7da457ed6e2e57dcaee4f1
Author: xuchuanyin 
Date:   2018-01-18T03:24:34Z

directly write carbon data files to HDFS

directly write carbon data files to hdfs to reduce disk I/O




---

[jira] [Created] (CARBONDATA-2047) Clean up temp folder after task completion in case of partitioning

[Ravindra Pesala (JIRA)]

Ravindra Pesala created CARBONDATA-2047:
---

 Summary: Clean up temp folder after task completion in case of 
partitioning
 Key: CARBONDATA-2047
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2047
 Project: CarbonData
  Issue Type: Improvement
Reporter: Ravindra Pesala


h1. Clean up temp folder after task completion in case of partitioning



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...

[chenliang613]

Github user chenliang613 commented on the issue:

https://github.com/apache/carbondata/pull/1812
  
please change the title to : [CARBONDATA-2033] Support user specified 
segments in major compaction


---

[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...

[chenliang613]

Github user chenliang613 commented on the issue:

https://github.com/apache/carbondata/pull/1812
  
retest this please


---

[GitHub] carbondata issue #1817: [CARBONDATA-2040] Add standard partiton example and ...

[chenliang613]

Github user chenliang613 commented on the issue:

https://github.com/apache/carbondata/pull/1817
  
retest this please


---

[GitHub] carbondata pull request #1800: [HOTFIX] Fix concurrent testcase random failu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/1800


---

[GitHub] carbondata issue #1800: [HOTFIX] Fix concurrent testcase random failure

[chenliang613]

Github user chenliang613 commented on the issue:

https://github.com/apache/carbondata/pull/1800
  
verified, LGTM


---

[GitHub] carbondata issue #1815: [WIP]Clean up temp folder after task completion in c...

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/1815
  
It seems there are two testcase failure. And please change PR title


---

[GitHub] carbondata issue #1800: [HOTFIX] Fix concurrent testcase random failure

[chenliang613]

Github user chenliang613 commented on the issue:

https://github.com/apache/carbondata/pull/1800
  
retest this please


---

[GitHub] carbondata issue #1805: [CARBONDATA-1827] S3 Carbon Implementation

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/1805
  
please rebase and drop the first two commits which are merged to 
carbonstore branch already


---

[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1795
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2921/



---

[GitHub] carbondata issue #1815: [WIP]Clean up temp folder after task completion in c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1815
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2920/



---

[GitHub] carbondata issue #1815: [WIP]Clean up temp folder after task completion in c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1815
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1687/



---

[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1795
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1688/



---

[GitHub] carbondata issue #1821: [HOTFIX] Listeners not getting registered to the bus...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1821
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2919/



---

[GitHub] carbondata issue #1821: [HOTFIX] Listeners not getting registered to the bus...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1821
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1686/



---

[GitHub] carbondata issue #1749: [CARBONDATA-1968] Add external table support

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1749
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1685/



---

[GitHub] carbondata issue #1772: [CARBONDATA-1995] Unify all writer steps and make te...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1772
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2916/



---

[GitHub] carbondata issue #1768: [CARBONDATA-2025] Unify all path construction throug...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1768
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2917/



---

[GitHub] carbondata issue #1768: [CARBONDATA-2025] Unify all path construction throug...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1768
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1684/



---

[GitHub] carbondata issue #1772: [CARBONDATA-1995] Unify all writer steps and make te...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1772
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1683/



---

[GitHub] carbondata issue #1792: [CARBONDATA-2018][DataLoad] Optimization in reading/...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1792
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2915/



---

[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1795
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2914/



---

[GitHub] carbondata issue #1792: [CARBONDATA-2018][DataLoad] Optimization in reading/...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1792
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1682/



---

[GitHub] carbondata issue #1805: [CARBONDATA-1827] S3 Carbon Implementation

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1805
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2912/



---

[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1795
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1681/



---

[GitHub] carbondata issue #1798: [CARBONDATA-1995][CARBONDATA-1996] Support file leve...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1798
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2913/



---

[GitHub] carbondata issue #1798: [CARBONDATA-1995][CARBONDATA-1996] Support file leve...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1798
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1680/



---

[GitHub] carbondata issue #1807: [CARBONDATA-2030]avg with Aggregate table for double...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1807
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2911/



---

[GitHub] carbondata issue #1808: [CARBONDATA-2023][DataLoad] Add size base block allo...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1808
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2910/



---

[GitHub] carbondata issue #1805: [CARBONDATA-1827] S3 Carbon Implementation

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1805
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1679/



---

[GitHub] carbondata issue #1807: [CARBONDATA-2030]avg with Aggregate table for double...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1807
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1678/



---

[GitHub] carbondata issue #1824: [CARBONDATA-2046]agg Query failed when non supported...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1824
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2908/



---

[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1795
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2909/



---

[GitHub] carbondata issue #1808: [CARBONDATA-2023][DataLoad] Add size base block allo...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1808
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1677/



---

[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1795
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1676/



---

[GitHub] carbondata pull request #1824: [CARBONDATA-2046]agg Query failed when non su...

[kumarvishal09]

Github user kumarvishal09 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1824#discussion_r162137213
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonPreAggregateRules.scala
 ---
@@ -347,9 +347,9 @@ case class CarbonPreAggregateQueryRules(sparkSession: 
SparkSession) extends Rule
 }
   // case of handling aggregation query with filter
   case agg@Aggregate(
-grExp,
-aggExp,
-Filter(expression, child@CarbonSubqueryAlias(_, l: 
LogicalRelation)))
+  grExp,
--- End diff --

Please add 2 space for all 


---

[jira] [Resolved] (CARBONDATA-2030) avg with Aggregate table for double data type is failed.

[kumar vishal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kumar vishal resolved CARBONDATA-2030.
--
   Resolution: Fixed
 Assignee: Babulal
Fix Version/s: 1.3.0

> avg with Aggregate table for double data type is failed. 
> -
>
> Key: CARBONDATA-2030
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2030
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Babulal
>Assignee: Babulal
>Priority: Major
> Fix For: 1.3.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> spark.sql("drop table if exists y ")
> spark.sql("create table y(year int,month int,name string,salary double) 
> stored by 'carbondata' 
> tblproperties('sort_scope'='Global_sort','table_blocksize'='23','sort_columns'='month,year,name')")
> spark.sql("insert into y select 10,11,'babu',12.89")
> spark.sql("insert into y select 10,11,'babu',12.89")
> spark.sql("create datamap y1_sum1 on table y using 'preaggregate' as select 
> name,avg(salary) from y group by name")
> spark.sql("select name,avg(salary) from y group by name").show(false)
>  
>  
> Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot 
> resolve '(sum(y_y1_sum1.`y_salary_sum`) / sum(y_y1_sum1.`y_salary_count`))' 
> due to data type mismatch: differing types in '(sum(y_y1_sum1.`y_salary_sum`) 
> / sum(y_y1_sum1.`y_salary_count`))' (double and bigint).;;
> 'Aggregate [y_name#25], [y_name#25 AS name#41, (sum(y_salary_sum#26) / 
> sum(y_salary_count#27L)) AS avg(salary)#46]
> +- Relation[y_name#25,y_salary_sum#26,y_salary_count#27L] 
> CarbonDatasourceHadoopRelation [ Database name :default, Table name 
> :y_y1_sum1, Schema :Some(StructType(StructField(y_name,StringType,true), 
> StructField(y_salary_sum,DoubleType,true), 
> StructField(y_salary_count,LongType,true))) ]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1807: [CARBONDATA-2030]avg with Aggregate table for...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/1807


---

[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...

[anubhav100]

Github user anubhav100 commented on the issue:

https://github.com/apache/carbondata/pull/1795
  
retest this please


---

[GitHub] carbondata issue #1800: [HOTFIX] Fix concurrent testcase random failure

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1800
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2906/



---

[GitHub] carbondata issue #1824: [CARBONDATA-2046]agg Query failed when non supported...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1824
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1675/



---

[GitHub] carbondata issue #1800: [HOTFIX] Fix concurrent testcase random failure

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1800
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1674/



---

[GitHub] carbondata issue #1795: [CARBONDATA-2016] Exception displays while executing...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1795
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2904/



---

[GitHub] carbondata pull request #1815: [WIP]Clean up temp folder after task completi...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1815#discussion_r162129482
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
 ---
@@ -317,8 +313,7 @@ public void initializeWriter() throws 
CarbonDataWriterException {
 .getCarbonDataFileName(fileCount, 
model.getCarbonDataFileAttributes().getTaskId(),
 model.getBucketId(), model.getTaskExtension(),
 "" + model.getCarbonDataFileAttributes().getFactTimeStamp());
-this.carbonDataFileTempPath = chosenTempLocation + File.separator
-+ carbonDataFileName + 
CarbonCommonConstants.FILE_INPROGRESS_STATUS;
+this.carbonDataFileTempPath = chosenTempLocation + File.separator + 
carbonDataFileName;
--- End diff --

ok


---