[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi…
ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700]
Optimize pruning performance when prunning with multi…
URL: https://github.com/apache/carbondata/pull/3620#discussion_r379988354
##
File path:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/blockprune/BlockPruneQueryTestCase.scala
##
@@ -18,16 +18,30 @@ package org.apache.carbondata.spark.testsuite.blockprune
import java.io.DataOutputStream
+import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.spark.sql.Row
import org.scalatest.BeforeAndAfterAll
import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.util.CarbonProperties
import org.apache.spark.sql.test.util.QueryTest
/**
* This class contains test cases for block prune query
*/
class BlockPruneQueryTestCase extends QueryTest with BeforeAndAfterAll {
val outputPath = s"$resourcesPath/block_prune_test.csv"
+ val MULTI_THREAD_ENABLE_FILES_COUNT = "1";
Review comment:
Still it won't prune multi-thread as other conditions may not satisfy
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi…
ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700]
Optimize pruning performance when prunning with multi…
URL: https://github.com/apache/carbondata/pull/3620#discussion_r379988354
##
File path:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/blockprune/BlockPruneQueryTestCase.scala
##
@@ -18,16 +18,30 @@ package org.apache.carbondata.spark.testsuite.blockprune
import java.io.DataOutputStream
+import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.spark.sql.Row
import org.scalatest.BeforeAndAfterAll
import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.util.CarbonProperties
import org.apache.spark.sql.test.util.QueryTest
/**
* This class contains test cases for block prune query
*/
class BlockPruneQueryTestCase extends QueryTest with BeforeAndAfterAll {
val outputPath = s"$resourcesPath/block_prune_test.csv"
+ val MULTI_THREAD_ENABLE_FILES_COUNT = "1";
Review comment:
Still it won't prune multi-thread as other conditions may not satisfy
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi…
ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700]
Optimize pruning performance when prunning with multi…
URL: https://github.com/apache/carbondata/pull/3620#discussion_r379988045
##
File path:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/blockprune/BlockPruneQueryTestCase.scala
##
@@ -18,16 +18,30 @@ package org.apache.carbondata.spark.testsuite.blockprune
import java.io.DataOutputStream
+import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.spark.sql.Row
import org.scalatest.BeforeAndAfterAll
import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.util.CarbonProperties
import org.apache.spark.sql.test.util.QueryTest
/**
* This class contains test cases for block prune query
*/
class BlockPruneQueryTestCase extends QueryTest with BeforeAndAfterAll {
val outputPath = s"$resourcesPath/block_prune_test.csv"
+ val MULTI_THREAD_ENABLE_FILES_COUNT = "1";
Review comment:
use camel case for variable names
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi…
ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700]
Optimize pruning performance when prunning with multi…
URL: https://github.com/apache/carbondata/pull/3620#discussion_r379988045
##
File path:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/blockprune/BlockPruneQueryTestCase.scala
##
@@ -18,16 +18,30 @@ package org.apache.carbondata.spark.testsuite.blockprune
import java.io.DataOutputStream
+import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.spark.sql.Row
import org.scalatest.BeforeAndAfterAll
import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.util.CarbonProperties
import org.apache.spark.sql.test.util.QueryTest
/**
* This class contains test cases for block prune query
*/
class BlockPruneQueryTestCase extends QueryTest with BeforeAndAfterAll {
val outputPath = s"$resourcesPath/block_prune_test.csv"
+ val MULTI_THREAD_ENABLE_FILES_COUNT = "1";
Review comment:
use small case for variable names
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi…
ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700]
Optimize pruning performance when prunning with multi…
URL: https://github.com/apache/carbondata/pull/3620#discussion_r379987838
##
File path:
core/src/main/java/org/apache/carbondata/core/datamap/TableDataMap.java
##
@@ -138,8 +138,12 @@ public CarbonTable getTable() {
}
}
int numOfThreadsForPruning = CarbonProperties.getNumOfThreadsForPruning();
+int carbonDriverPruningMultiThreadEnableFilesCount =
Review comment:
Need to add validation for carbon property, if someone configures negative
value. Need to use the default value
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700] Optimize pruning performance when prunning with multi…
ajantha-bhat commented on a change in pull request #3620: [CARBONDATA-3700]
Optimize pruning performance when prunning with multi…
URL: https://github.com/apache/carbondata/pull/3620#discussion_r379987630
##
File path:
core/src/main/java/org/apache/carbondata/core/datamap/TableDataMap.java
##
@@ -138,8 +138,12 @@ public CarbonTable getTable() {
}
}
int numOfThreadsForPruning = CarbonProperties.getNumOfThreadsForPruning();
+int carbonDriverPruningMultiThreadEnableFilesCount =
+Integer.parseInt(CarbonProperties.getInstance().getProperty(
+
CarbonCommonConstants.CARBON_DRIVER_PRUNING_MULTI_THREAD_ENABLE_FILES_COUNT,
Review comment:
Need update the document for the new property added
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
