[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2619
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6558/



---

[GitHub] carbondata issue #2619: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2619
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7833/



---

[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/2616
  
LGTM


---

[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...

[praveenmeenakshi56]

Github user praveenmeenakshi56 commented on the issue:

https://github.com/apache/carbondata/pull/2617
  
retest SDV please


---

[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2617
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6205/



---

[GitHub] carbondata pull request #2619: [CARBONDATA-2819] Fixed cannot drop preagg da...

[Sssan520]

GitHub user Sssan520 opened a pull request:

https://github.com/apache/carbondata/pull/2619

[CARBONDATA-2819] Fixed cannot drop preagg datamap on table if the tab…

1.Use "SET carbon.datamap.visible.{dbName}.{mainTable}.{datamapName} = 
{true or false}",if datamap provider is 'preagg',block this configuration.
2.When create preagg datamap, now also create its datamap schema file in 
system folder.
3.Use command "show datamap on table {tableName}" will get its all datamaps 
from datamap schema files

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?No
 
 - [ ] Any backward compatibility impacted? No
 
 - [ ] Document update required? No

 - [ ] Testing done 
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Sssan520/carbondata droppreagg

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2619.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2619


commit 7884ba86f13b89fb23895400bbac12eaa4e62546
Author: Sssan520 
Date:   2018-08-08T06:27:41Z

[CARBONDATA-2819]Fixed cannot drop preagg datamap on table if the table has 
other index datamaps




---

[GitHub] carbondata pull request #2618: [CARBONDATA-2819] Fixed cannot drop preagg da...

[lianganping]

GitHub user lianganping opened a pull request:

https://github.com/apache/carbondata/pull/2618

[CARBONDATA-2819] Fixed cannot drop preagg datamap on table if the tab…

1.Use "SET carbon.datamap.visible.{dbName}.{mainTable}.{datamapName} = 
{true or false}",if datamap provider is 'preagg',block this configuration.
2.When create preagg datamap, now also create its datamap schema file in 
system folder.
3.Use command "show datamap on table {tableName}" will get its all datamaps 
from datamap schema files

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed? No
 
 - [ ] Any backward compatibility impacted? No
 
 - [ ] Document update required? No

 - [ ] Testing done NA
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Sssan520/carbondata droppreagg

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2618.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2618


commit 7884ba86f13b89fb23895400bbac12eaa4e62546
Author: Sssan520 
Date:   2018-08-08T06:27:41Z

[CARBONDATA-2819]Fixed cannot drop preagg datamap on table if the table has 
other index datamaps




---

[GitHub] carbondata pull request #2618: [CARBONDATA-2819] Fixed cannot drop preagg da...

[lianganping]

Github user lianganping closed the pull request at:

https://github.com/apache/carbondata/pull/2618


---

[GitHub] carbondata issue #2618: [CARBONDATA-2819] Fixed cannot drop preagg datamap o...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2618
  
Can one of the admins verify this patch?


---

[jira] [Resolved] (CARBONDATA-2659) Support partitioned carbon table by DataFrame.write

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-2659.
-
   Resolution: Fixed
Fix Version/s: (was: 1.5.0)

> Support partitioned carbon table by DataFrame.write
> ---
>
> Key: CARBONDATA-2659
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2659
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Jacky Li
>Priority: Major
> Fix For: 1.4.1
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Currently only partition table is only supported by SQL, it should be 
> supported by Spark DataFrame API also.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #2415: [CARBONDATA-2659] Support partition table by ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2415


---

[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2415
  
LGTM


---

[GitHub] carbondata pull request #2613: [HOTFIX] Modified code to fix the degrade in ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2613


---

[GitHub] carbondata issue #2617: [CARBONDATA-2838] Added SDV test cases for Local Dic...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2617
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6204/



---

[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2613
  
LGTM


---

[jira] [Resolved] (CARBONDATA-2834) Refactor code to remove nested for loop to extract invalidTimestampRange.

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-2834.
-
   Resolution: Fixed
Fix Version/s: 1.4.1

> Refactor code to remove nested for loop to extract invalidTimestampRange.
> -
>
> Key: CARBONDATA-2834
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2834
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Kunal Kapoor
>Assignee: Kunal Kapoor
>Priority: Major
> Fix For: 1.4.1
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Reactor getInvalidTimestampRange method in SegmentUpdateStatusManager because 
> it has an unnecessary nested loop to get timestamp from invalid segments.
> This will cause query performance degradation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #2612: [CARBONDATA-2834] Remove unnecessary nested l...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2612


---

[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2612
  
LGTM


---

[GitHub] carbondata pull request #2617: [CARBONDATA-2838] Added SDV test cases for Lo...

[praveenmeenakshi56]

GitHub user praveenmeenakshi56 opened a pull request:

https://github.com/apache/carbondata/pull/2617

[CARBONDATA-2838] Added SDV test cases for Local Dictionary Support

Added SDV test cases for Local Dictionary Support

 - [ ] Any interfaces changed?
NA
 - [ ] Any backward compatibility impacted?
NA
 - [ ] Document update required?
NA
 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
Test cases are added
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
NA


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/praveenmeenakshi56/carbondata local_dict_sdv

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2617.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2617


commit d05ba074b239e3d52672e1638eece39367fddd40
Author: praveenmeenakshi56 
Date:   2018-08-08T06:15:49Z

Added SDV test cases for Local Dictionary Support




---

[jira] [Created] (CARBONDATA-2838) Add SDV test cases for Local Dictionary Support

[Praveen M P (JIRA)]

Praveen M P created CARBONDATA-2838:
---

 Summary: Add SDV test cases for Local Dictionary Support
 Key: CARBONDATA-2838
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2838
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Praveen M P
Assignee: Praveen M P






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #2606: [CARBONDATA-2817]Thread Leak in Update and in...

[kumarvishal09]

Github user kumarvishal09 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2606#discussion_r208460810
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/processing/loading/steps/CarbonRowDataWriterProcessorStepImpl.java
 ---
@@ -169,38 +171,59 @@ private void doExecute(Iterator 
iterator, int iteratorIndex) thr
   if (rowsNotExist) {
 rowsNotExist = false;
 dataHandler = 
CarbonFactHandlerFactory.createCarbonFactHandler(model);
+this.carbonFactHandlers.add(dataHandler);
 dataHandler.initialise();
   }
   processBatch(iterator.next(), dataHandler, iteratorIndex);
 }
-if (!rowsNotExist) {
-  finish(dataHandler, iteratorIndex);
+try {
+  if (!rowsNotExist) {
+finish(dataHandler, iteratorIndex);
+  }
+} catch (CarbonDataWriterException e) {
--- End diff --

remove this catch block


---

[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2606
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6555/



---

[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2606
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7830/



---

[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...

[brijoobopanna]

Github user brijoobopanna commented on the issue:

https://github.com/apache/carbondata/pull/2606
  
retest this please



---

[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2616
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6554/



---

[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2613
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6553/



---

[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2415
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6551/



---

[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2616
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7829/



---

[GitHub] carbondata issue #2615: [HOTFIX] [presto] presto code cleanup

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2615
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6552/



---

[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2564
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6550/



---

[GitHub] carbondata issue #2615: [HOTFIX] [presto] presto code cleanup

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2615
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7828/



---

[GitHub] carbondata issue #2616: [HOTFIX]Fixed int overflow and comparison gone wrong...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2616
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6203/



---

[GitHub] carbondata pull request #2616: [HOTFIX]Fixed int overflow and comparison gon...

[ravipesala]

GitHub user ravipesala opened a pull request:

https://github.com/apache/carbondata/pull/2616

[HOTFIX]Fixed int overflow and comparison gone wrong during blocklet min/max

Problem: During calculating min/max for blocklet, it needs to calculate 
from all the pages. During that comparison, it is typecasting to int and 
overflows, so there is a chance the negative becomes positive and positive 
become negative. That's why min max of long comes wrong for bigger values. 
Solution: Don't typecast directly, instead check first the negative or 
positive and then return.

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ravipesala/incubator-carbondata 
blocklet-min-max-issue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2616.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2616


commit a99d0516ee06a285f8be3880d7005bf161f1c940
Author: ravipesala 
Date:   2018-08-07T15:49:36Z

Fixed int overflow and comparasion gone wrong during blocklet min/max




---

[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2613
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7827/



---

[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2415
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7826/



---

[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2612
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6547/



---

[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2588
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6548/



---

[GitHub] carbondata issue #2615: [HOTFIX] [presto] presto code cleanup

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2615
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6202/



---

[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2612
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7824/



---

[GitHub] carbondata pull request #2615: [HOTFIX] [presto] presto code cleanup

[ajantha-bhat]

GitHub user ajantha-bhat opened a pull request:

https://github.com/apache/carbondata/pull/2615

[HOTFIX] [presto] presto code cleanup

Changes in the PR:
a. Removed spark jars from release jars dependency
b. enable check style for presto module
c. fixed all the checkstyle for presto module

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed? NA
 
 - [ ] Any backward compatibility impacted? NA
 
 - [ ] Document update required? NA

 - [ ] Testing done. NA
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA.  NA



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ajantha-bhat/carbondata master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2615.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2615


commit 399ca66addcd6767bc43ac7fa15514eb705546d4
Author: ajantha-bhat 
Date:   2018-08-06T12:21:03Z

[HOTFIX] presto code cleanup

a. Removed spark jars from release jars dependency
b. enable check style for presto module
c. fixed all the checkstyle for presto module




---

[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2564
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7825/



---

[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2613
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7822/



---

[jira] [Resolved] (CARBONDATA-2807) Fixed data load performance issue with more number of records

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-2807.
-
   Resolution: Fixed
Fix Version/s: 1.4.1

> Fixed data load performance issue with more number of records
> -
>
> Key: CARBONDATA-2807
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2807
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: kumar vishal
>Assignee: kumar vishal
>Priority: Major
> Fix For: 1.4.1
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> **Problem:**Data Loading is taking more time when number of records are high.
> **Root cause:** As number of records are high intermediate merger is taking 
> more time.
> **Solution:** Checking the number of files present in file list is done is 
> synchronized block because of this 
> each intermediate request is taking sometime and when number of records are 
> high it impacting overall data loading performance



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2588


---

[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2588
  
LGTM


---

[GitHub] carbondata issue #2602: [CARBONDATA-2539]Fix mv classcast exception issue.

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2602
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6544/



---

[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2613
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6546/



---

[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2588
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6545/



---

[jira] [Resolved] (CARBONDATA-2831) Support Merge index files read from non transactional table.

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-2831.
-
   Resolution: Fixed
Fix Version/s: 1.4.1

> Support Merge index files read from non transactional table.
> 
>
> Key: CARBONDATA-2831
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2831
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Ajantha Bhat
>Assignee: Ajantha Bhat
>Priority: Major
> Fix For: 1.4.1
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> problem : Currently  SDK read/ nontransactional table read from external 
> table gives null output when carbonMergeindex file is present instead of 
> carobnindex files. 
> cause : In LatestFileReadCommitted, while taking snapshot, merge index files 
> were not considered.
> solution: consider the merge index files while taking snapshot
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #2610: [CARBONDATA-2831] Added Support Merge index f...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2610


---

[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2415
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6200/



---

[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2588
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7823/



---

[GitHub] carbondata issue #2602: [CARBONDATA-2539]Fix mv classcast exception issue.

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2602
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7820/



---

[GitHub] carbondata issue #2537: [CARBONDATA-2768][CarbonStore] Fix error in tests fo...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2537
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6199/



---

[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2610
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7819/



---

[GitHub] carbondata issue #2564: [CARBONDATA-2785][ExternalFormat] Optimize table pru...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2564
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6198/



---

[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2610
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6543/



---

[GitHub] carbondata pull request #2537: [CARBONDATA-2768][CarbonStore] Fix error in t...

[xuchuanyin]

Github user xuchuanyin closed the pull request at:

https://github.com/apache/carbondata/pull/2537


---

[GitHub] carbondata pull request #2415: [CARBONDATA-2659] Support partition table by ...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2415#discussion_r208224446
  
--- Diff: 
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala
 ---
@@ -92,11 +89,38 @@ class CarbonDataFrameWriter(sqlContext: SQLContext, val 
dataFrame: DataFrame) {
 ).filter(_._2.isDefined)
   .map(property => s"'${property._1}' = 
'${property._2.get}'").mkString(",")
 
+val partition: Seq[String] = if (options.partitionColumns.isDefined) {
+  if (options.partitionColumns.get.toSet.size != 
options.partitionColumns.get.length) {
+throw new MalformedCarbonCommandException(s"repeated partition 
column")
+  }
+  options.partitionColumns.get.map { column =>
+val field = schema.fields.find(_.name.equalsIgnoreCase(column))
+if (field.isEmpty) {
+  throw new MalformedCarbonCommandException(s"invalid partition 
column: $column")
+}
+s"$column ${field.get.dataType.typeName}"
+  }
+} else {
+  Seq()
+}
+
+val schemaWithoutPartition = if (options.partitionColumns.isDefined) {
+  val fields = schema.filterNot(field => 
options.partitionColumns.get.contains(field.name))
--- End diff --

fixed


---

[GitHub] carbondata issue #2614: [CARBONDATA-2837] Added MVExample in example module

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2614
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6542/



---

[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2610
  
LGTM


---

[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2612
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6197/



---

[GitHub] carbondata issue #2614: [CARBONDATA-2837] Added MVExample in example module

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2614
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7818/



---

[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2606
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6541/



---

[GitHub] carbondata pull request #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStor...

[jackylk]

Github user jackylk closed the pull request at:

https://github.com/apache/carbondata/pull/2589


---

[GitHub] carbondata issue #2391: [CARBONDATA-2625] Optimize the performance of Carbon...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2391
  
@xubo245 Please close it as it is handled in 
https://github.com/apache/carbondata/pull/2441


---

[GitHub] carbondata issue #2537: [CARBONDATA-2768][CarbonStore] Fix error in tests fo...

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/2537
  
LGTM
merged into external-format


---

[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2415
  
retest this please


---

[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2613
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6196/



---

[GitHub] carbondata issue #2423: [CARBONDATA-2530][MV] Fix wrong data displayed when ...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2423
  
@xubo245 Please close it, it is handled in 
https://github.com/apache/carbondata/pull/2474


---

[GitHub] carbondata pull request #2602: [CARBONDATA-2539]Fix mv classcast exception i...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2602


---

[GitHub] carbondata pull request #2605: [CARBONDATA-2585] Fix local dictionary for bo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2605


---

[GitHub] carbondata issue #2602: [CARBONDATA-2539]Fix mv classcast exception issue.

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/2602
  
LGTM


---

[GitHub] carbondata pull request #2612: [CARBONDATA-2834] Remove unnecessary nested l...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2612#discussion_r208211297
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
 ---
@@ -779,6 +779,32 @@ public UpdateVO getInvalidTimestampRange(String 
segmentId) {
 }
 return range;
   }
+
+  /**
+   * Returns the invalid timestamp range of a segment.
+   * @return
+   */
+  public List getInvalidTimestampRange() {
+List ranges = new ArrayList();
+for (LoadMetadataDetails segment : segmentDetails) {
+  if (!(segment.getSegmentStatus().equals(SegmentStatus.SUCCESS) ||
+  
segment.getSegmentStatus().equals(SegmentStatus.LOAD_PARTIAL_SUCCESS))) {
--- End diff --

Please refer SegmentStatusManager for checking the invalid segments


---

[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2606
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7817/



---

[GitHub] carbondata issue #2612: [CARBONDATA-2834] Remove unnecessary nested looping ...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2612
  
@kunal642 Please fix checkstyle


---

[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...

[akashrn5]

Github user akashrn5 commented on the issue:

https://github.com/apache/carbondata/pull/2605
  
2.1 build has random failure, which is not related to this change


---

[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2605
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7816/



---

[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2605
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6540/



---

[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/2605
  
LGTM


---

[GitHub] carbondata pull request #2614: [CARBONDATA-2837] Added MVExample in example ...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2614#discussion_r208206503
  
--- Diff: examples/spark2/pom.xml ---
@@ -49,6 +49,11 @@
   carbondata-store-sdk
   ${project.version}
 
+
+  org.apache.carbondata
+  carbondata-mv-core
--- End diff --

This is adding always or in case of `mv` profile is selected?


---

[GitHub] carbondata issue #2602: [CARBONDATA-2539]Fix mv classcast exception issue.

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2602
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6195/



---

[GitHub] carbondata issue #2588: [CARBONDATA-2807] Fixed data load performance issue ...

[brijoobopanna]

Github user brijoobopanna commented on the issue:

https://github.com/apache/carbondata/pull/2588
  
retest this please



---

[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2613
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6539/



---

[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2613
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7815/



---

[GitHub] carbondata issue #2614: [CARBONDATA-2837] Added MVExample in example module

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2614
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6194/



---

[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...

[mohammadshahidkhan]

Github user mohammadshahidkhan commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2588#discussion_r208199760
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeIntermediateMerger.java
 ---
@@ -111,18 +108,15 @@ public void addFileToMerge(File sortTempFile) {
   }
 
   public void startFileMergingIfPossible() {
-File[] fileList = null;
-synchronized (lockObject) {
-  if (procFiles.size() >= 
parameters.getNumberOfIntermediateFileToBeMerged()) {
+File[] fileList;
+if (procFiles.size() >= 
parameters.getNumberOfIntermediateFileToBeMerged()) {
--- End diff --

ok


---

[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...

[mohammadshahidkhan]

Github user mohammadshahidkhan commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2588#discussion_r208199601
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeIntermediateMerger.java
 ---
@@ -111,18 +108,15 @@ public void addFileToMerge(File sortTempFile) {
   }
 
   public void startFileMergingIfPossible() {
-File[] fileList = null;
-synchronized (lockObject) {
-  if (procFiles.size() >= 
parameters.getNumberOfIntermediateFileToBeMerged()) {
+File[] fileList;
+if (procFiles.size() >= 
parameters.getNumberOfIntermediateFileToBeMerged()) {
+  synchronized (lockObject) {
 fileList = procFiles.toArray(new File[procFiles.size()]);
 this.procFiles = new ArrayList();
-if (LOGGER.isDebugEnabled()) {
-  LOGGER
-  .debug("Submitting request for intermediate merging no of 
files: " + fileList.length);
-}
   }
-}
-if (null != fileList) {
+  if (LOGGER.isDebugEnabled()) {
+LOGGER.debug("Sumitting request for intermediate merging no of 
files: " + fileList.length);
+  }
   startIntermediateMerging(fileList);
--- End diff --

ok


---

[GitHub] carbondata pull request #2588: [CARBONDATA-2807] Fixed data load performance...

[kumarvishal09]

Github user kumarvishal09 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2588#discussion_r208198390
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/merger/UnsafeIntermediateMerger.java
 ---
@@ -111,18 +108,15 @@ public void addFileToMerge(File sortTempFile) {
   }
 
   public void startFileMergingIfPossible() {
-File[] fileList = null;
-synchronized (lockObject) {
-  if (procFiles.size() >= 
parameters.getNumberOfIntermediateFileToBeMerged()) {
+File[] fileList;
+if (procFiles.size() >= 
parameters.getNumberOfIntermediateFileToBeMerged()) {
--- End diff --

@mohammadshahidkhan Yes you are right but in this case UnsafeSortDataRow 
processing will be slower as it will read/ sort and write so chances of above 
condition is negligible, because of this double check is not added here 


---

[GitHub] carbondata issue #2606: [CARBONDATA-2817]Thread Leak in Update and in No sor...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2606
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6193/



---

[jira] [Resolved] (CARBONDATA-2836) Fixed data loading performance issue

[Ravindra Pesala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravindra Pesala resolved CARBONDATA-2836.
-
   Resolution: Fixed
Fix Version/s: 1.4.1

> Fixed data loading performance issue
> 
>
> Key: CARBONDATA-2836
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2836
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: kumar vishal
>Assignee: kumar vishal
>Priority: Major
> Fix For: 1.4.1
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Problem: Data Loading is taking more time when number of records are high(3.5 
> billion) records
> Root Cause: In case of Final merge sort temp row conversion is done in main 
> thread because of this final step processing became slower.
> Solution: Mode conversion logic to pre-fetch thread for parallel processing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #2611: [CARBONDATA-2836]Fixed data loading performan...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2611


---

[GitHub] carbondata issue #2611: [CARBONDATA-2836]Fixed data loading performance issu...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2611
  
LGTM


---

[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...

[ajantha-bhat]

Github user ajantha-bhat commented on the issue:

https://github.com/apache/carbondata/pull/2610
  
retest this please


---

[GitHub] carbondata pull request #2614: [CARBONDATA-2837] Added MVExample in example ...

[ravipesala]

GitHub user ravipesala opened a pull request:

https://github.com/apache/carbondata/pull/2614

[CARBONDATA-2837] Added MVExample in example module

Added MVExample in example module

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ravipesala/incubator-carbondata mvexample

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2614.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2614


commit f50e72201dd1d0da720bcfaf68c5e4df14a51624
Author: ravipesala 
Date:   2018-08-07T11:07:17Z

Added MVExample




---

[jira] [Created] (CARBONDATA-2837) Add MV Example in examples module

[Ravindra Pesala (JIRA)]

Ravindra Pesala created CARBONDATA-2837:
---

 Summary: Add MV Example in examples module
 Key: CARBONDATA-2837
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2837
 Project: CarbonData
  Issue Type: Improvement
Reporter: Ravindra Pesala






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #2605: [CARBONDATA-2585] Fix local dictionary for both tabl...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2605
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6192/



---

[GitHub] carbondata issue #2613: [HOTFIX] Modified code to fix the degrade in compact...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2613
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/6191/



---

[GitHub] carbondata pull request #2605: [CARBONDATA-2585] Fix local dictionary for bo...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2605#discussion_r208186221
  
--- Diff: 
integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
 ---
@@ -641,6 +641,78 @@ object CarbonScalaUtil {
 }
   }
 
+  /**
+   * This method validates all the child columns of complex column 
recursively to check whether
+   * any of the child column is of string dataType or not
+   *
+   * @param field
+   */
+  def validateChildColumnsRecursively(field: Field): Boolean = {
+if (field.children.isDefined && null != field.children.get) {
+  field.children.get.exists { childColumn =>
+if (childColumn.children.isDefined && null != 
childColumn.children.get) {
+  validateChildColumnsRecursively(childColumn)
+} else {
+  childColumn.dataType.get.equalsIgnoreCase("string")
+}
+  }
+} else {
+  false
+}
+  }
+
+  /**
+   * This method validates the local dictionary configured columns
+   *
+   * @param fields
--- End diff --

please add parameter description for all newly added parameter


---

[GitHub] carbondata issue #2610: [CARBONDATA-2831] Added Support Merge index files re...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2610
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6536/



---