kevinjmh commented on a change in pull request #3847:
URL: https://github.com/apache/carbondata/pull/3847#discussion_r460750438
##########
File path:
processing/src/main/java/org/apache/carbondata/processing/loading/sort/impl/UnsafeParallelReadMergeSorterWithColumnRangeImpl.java
##########
@@ -99,6 +101,8 @@ public void initialize(SortParameters sortParameters) {
UnsafeSortDataRows[] sortDataRows = new
UnsafeSortDataRows[columnRangeInfo.getNumOfRanges()];
intermediateFileMergers = new
UnsafeIntermediateMerger[columnRangeInfo.getNumOfRanges()];
SortParameters[] sortParameterArray = new
SortParameters[columnRangeInfo.getNumOfRanges()];
+ this.writeService =
Executors.newFixedThreadPool(originSortParameters.getNumberOfCores(),
Review comment:
@ajantha-bhat Good point. So the only difference is adding threads
horizontally or vertically. If each thread takes same time to process the data
and writes at same time, performance may degrade caused by IO preemption. But
the different may not big when number of input split is large enough. @shunlean
could you please do some test to confirm ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
