[GitHub] carbondata pull request #2606: [CARBONDATA-2817]Thread Leak in Update and in...

2018-08-08 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2606


---

[GitHub] carbondata pull request #2606: [CARBONDATA-2817]Thread Leak in Update and in...

2018-08-08 Thread BJangir 
Github user BJangir commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2606#discussion_r208515484
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/processing/loading/steps/CarbonRowDataWriterProcessorStepImpl.java
 ---
@@ -169,24 +171,36 @@ private void doExecute(Iterator 
iterator, int iteratorIndex) thr
   if (rowsNotExist) {
 rowsNotExist = false;
 dataHandler = 
CarbonFactHandlerFactory.createCarbonFactHandler(model);
+this.carbonFactHandlers.add(dataHandler);
 dataHandler.initialise();
   }
   processBatch(iterator.next(), dataHandler, iteratorIndex);
 }
-if (!rowsNotExist) {
-  finish(dataHandler, iteratorIndex);
+try {
+  if (!rowsNotExist) {
+finish(dataHandler, iteratorIndex);
+  }
+} finally {
+  carbonFactHandlers.remove(dataHandler);
 }
+
+
   }
 
   @Override protected String getStepName() {
 return "Data Writer";
   }
 
   private void finish(CarbonFactHandler dataHandler, int iteratorIndex) {
+CarbonDataWriterException exception = null;
--- End diff --

Fixed


---

[GitHub] carbondata pull request #2606: [CARBONDATA-2817]Thread Leak in Update and in...

2018-08-07 Thread kumarvishal09 
Github user kumarvishal09 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2606#discussion_r208460810
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/processing/loading/steps/CarbonRowDataWriterProcessorStepImpl.java
 ---
@@ -169,38 +171,59 @@ private void doExecute(Iterator 
iterator, int iteratorIndex) thr
   if (rowsNotExist) {
 rowsNotExist = false;
 dataHandler = 
CarbonFactHandlerFactory.createCarbonFactHandler(model);
+this.carbonFactHandlers.add(dataHandler);
 dataHandler.initialise();
   }
   processBatch(iterator.next(), dataHandler, iteratorIndex);
 }
-if (!rowsNotExist) {
-  finish(dataHandler, iteratorIndex);
+try {
+  if (!rowsNotExist) {
+finish(dataHandler, iteratorIndex);
+  }
+} catch (CarbonDataWriterException e) {
--- End diff --

remove this catch block


---

[GitHub] carbondata pull request #2606: [CARBONDATA-2817]Thread Leak in Update and in...

2018-08-06 Thread kumarvishal09 
Github user kumarvishal09 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2606#discussion_r207961430
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/processing/loading/steps/CarbonRowDataWriterProcessorStepImpl.java
 ---
@@ -169,24 +171,36 @@ private void doExecute(Iterator 
iterator, int iteratorIndex) thr
   if (rowsNotExist) {
 rowsNotExist = false;
 dataHandler = 
CarbonFactHandlerFactory.createCarbonFactHandler(model);
+this.carbonFactHandlers.add(dataHandler);
 dataHandler.initialise();
   }
   processBatch(iterator.next(), dataHandler, iteratorIndex);
 }
-if (!rowsNotExist) {
-  finish(dataHandler, iteratorIndex);
+try {
+  if (!rowsNotExist) {
+finish(dataHandler, iteratorIndex);
+  }
+} finally {
+  carbonFactHandlers.remove(dataHandler);
 }
+
+
   }
 
   @Override protected String getStepName() {
 return "Data Writer";
   }
 
   private void finish(CarbonFactHandler dataHandler, int iteratorIndex) {
+CarbonDataWriterException exception = null;
--- End diff --

Please handle for closeHandler method as it can also throw exception 


---

[GitHub] carbondata pull request #2606: [CARBONDATA-2817]Thread Leak in Update and in...

2018-08-02 Thread kumarvishal09 
Github user kumarvishal09 commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2606#discussion_r207321150
  
--- Diff: 
processing/src/main/java/org/apache/carbondata/processing/loading/steps/CarbonRowDataWriterProcessorStepImpl.java
 ---
@@ -80,11 +82,16 @@
 
   private Map 
localDictionaryGeneratorMap;
 
+  private List carbonFactHandlers;
+
+  ExecutorService executorService = null;
--- End diff --

make it private


---

[GitHub] carbondata pull request #2606: [CARBONDATA-2817]Thread Leak in Update and in...

2018-08-02 Thread BJangir 
GitHub user BJangir opened a pull request:

https://github.com/apache/carbondata/pull/2606

[CARBONDATA-2817]Thread Leak in Update and in No sort flow

Issue :-  After Update Command is finished , Loading threads are not 
getting stopped. 

Root Cause :- 
1. In Update flow  DataLoadExecutor 's close method is not called so all 
Executors services are not closed.
2. In Exceptions are not handled property in AFDW class's 
closeExecutorService() which is cuasing Thread leak if Job is killed from 
SparkUI..  

Solution :- 
1.  Add Task Completion Listener and  call close method of DataLoadExecutor 
 to it .
2. Handle Exception in closeExecutor Service so that all Writer steps 
Threads can be closed.



Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 NO
 - [ ] Any backward compatibility impacted?
 NO
 - [ ] Document update required?
NO
 - [ ] Testing done
  Verified in cluster manually .  
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
  NA


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/BJangir/incubator-carbondata update_threadLeak

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2606.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2606


commit c6c8b484284842169d9239ca403af7d46c67a346
Author: BJangir 
Date:   2018-08-02T16:21:07Z

[CARBONDATA-2817]Thread Leak in Update and in No sort flow




---