[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version
CarbonDataQA1 commented on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649583529 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3221/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version
CarbonDataQA1 commented on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649581253 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1494/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version
ajantha-bhat commented on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649537393 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3772: [CARBONDATA-3832]Added block and blocket pruning for the polygon expression processing
CarbonDataQA1 commented on pull request #3772: URL: https://github.com/apache/carbondata/pull/3772#issuecomment-649533543 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1493/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3772: [CARBONDATA-3832]Added block and blocket pruning for the polygon expression processing
CarbonDataQA1 commented on pull request #3772: URL: https://github.com/apache/carbondata/pull/3772#issuecomment-649532771 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3220/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3772: [CARBONDATA-3832]Added block and blocket pruning for the polygon expression processing
VenuReddy2103 commented on a change in pull request #3772:
URL: https://github.com/apache/carbondata/pull/3772#discussion_r445481119
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java
##
@@ -188,6 +189,14 @@ private static FilterExecuter createFilterExecuterTree(
return new FalseFilterExecutor();
case ROWLEVEL:
default:
+ if (filterExpressionResolverTree.getFilterExpression() instanceof
UnknownExpression) {
+FilterExecuter filterExecuter =
+((UnknownExpression)
filterExpressionResolverTree.getFilterExpression())
+.getFilterExecuter(filterExpressionResolverTree,
segmentProperties);
+if (filterExecuter != null) {
Review comment:
I understand what you meant. I think, it is better to create the
RowLevelFilterExecuterImpl only at this place. If UnknownExpression do not have
any specific implemenation, it will be like a fallback to
RowLevelFilterExecuterImpl.
Also, if we have return RowLevelFilterExecuterImpl from
UnknownExpression.getFilterExecuter, we need to pass complexDimensionInfoMap
and all overiding methods will have this argument. It is an unused argument as
current UnKnown expressions do not support complex dimensions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3772: [CARBONDATA-3832]Added block and blocket pruning for the polygon expression processing
VenuReddy2103 commented on a change in pull request #3772:
URL: https://github.com/apache/carbondata/pull/3772#discussion_r445475371
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/geo/GeoTest.scala
##
@@ -134,8 +133,31 @@ class GeoTest extends QueryTest with BeforeAndAfterAll
with BeforeAndAfterEach {
Row(116285807, 40084087)))
}
- test("test polygon query on table partitioned by timevalue column")
- {
+ test("test block pruning for polygon query") {
+createTable()
+sql(s"insert into $table1 select 157542840,116285807,40084087")
+sql(s"insert into $table1 select 157542840,116372142,40129503")
+sql(s"insert into $table1 select 157542840,116187332,39979316")
+sql(s"insert into $table1 select 157542840,116337069,39951887")
+sql(s"insert into $table1 select 157542840,116359102,40154684")
+sql(s"insert into $table1 select 157542840,116736367,39970323")
+sql(s"insert into $table1 select 157542840,116362699,39942444")
+sql(s"insert into $table1 select 157542840,116325378,39963129")
+sql(s"insert into $table1 select 157542840,116302895,39930753")
+sql(s"insert into $table1 select 157542840,116288955,3101")
+val df = sql(s"select longitude, latitude from $table1 where
IN_POLYGON('116.321011 " +
+ s"40.123503, 116.137676 39.947911, 116.560993 39.935276,
116.321011 40.123503')")
+assert(df.rdd.getNumPartitions == 6)
+checkAnswer(df,
Review comment:
Ok. Instead of making the Seq of rows with same rows in all testcases,
have defined it once and used same in all the testcases.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version
CarbonDataQA1 commented on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649439729 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3219/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version
CarbonDataQA1 commented on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649437666 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1492/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3771: [CARBONDATA-3849] pushdown array_contains filter to carbon for array of primitive types
CarbonDataQA1 commented on pull request #3771: URL: https://github.com/apache/carbondata/pull/3771#issuecomment-649431612 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1490/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3771: [CARBONDATA-3849] pushdown array_contains filter to carbon for array of primitive types
CarbonDataQA1 commented on pull request #3771: URL: https://github.com/apache/carbondata/pull/3771#issuecomment-649430423 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3217/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat opened a new pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark 2.4 profile
ajantha-bhat opened a new pull request #3807: URL: https://github.com/apache/carbondata/pull/3807 ### Why is this PR needed? After #3700 , spark binary version addition. a) Module name always taking deafult profile spark binary version. b) IDE test cases are running with spark 2.3 even for spark2.3.4 because of above problem ### What changes were proposed in this PR? a) keep a simple module name without binary version b) Use the package option with jar file name to have binary version in the final jar. ### Does this PR introduce any user interface change? - No ### Is any new testcase added? - No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark 2.4 profile
ajantha-bhat commented on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649391566 @kunal642 , @akashrn5 : check it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat edited a comment on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark 2.4 profile
ajantha-bhat edited a comment on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649391566 @kunal642 , @akashrn5 : please check it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3801: adding comments with issue numbers.
CarbonDataQA1 commented on pull request #3801: URL: https://github.com/apache/carbondata/pull/3801#issuecomment-649388349 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1491/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3801: adding comments with issue numbers.
CarbonDataQA1 commented on pull request #3801: URL: https://github.com/apache/carbondata/pull/3801#issuecomment-649384032 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3218/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] chenliang613 commented on pull request #3801: adding comments with issue numbers.
chenliang613 commented on pull request #3801: URL: https://github.com/apache/carbondata/pull/3801#issuecomment-649381896 this pr is too big. be careful to merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] chenliang613 commented on pull request #3801: adding comments with issue numbers.
chenliang613 commented on pull request #3801: URL: https://github.com/apache/carbondata/pull/3801#issuecomment-649381380 add to whitelist This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3771: [CARBONDATA-3849] pushdown array_contains filter to carbon for array of primitive types
ajantha-bhat commented on a change in pull request #3771:
URL: https://github.com/apache/carbondata/pull/3771#discussion_r445361657
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java
##
@@ -222,49 +228,103 @@ public BitSetGroup applyFilter(RawBlockletColumnChunks
rawBlockletColumnChunks,
}
}
BitSetGroup bitSetGroup = new BitSetGroup(pageNumbers);
-for (int i = 0; i < pageNumbers; i++) {
- BitSet set = new BitSet(numberOfRows[i]);
- RowIntf row = new RowImpl();
- BitSet prvBitset = null;
- // if bitset pipe line is enabled then use rowid from previous bitset
- // otherwise use older flow
- if (!useBitsetPipeLine ||
- null == rawBlockletColumnChunks.getBitSetGroup() ||
- null == bitSetGroup.getBitSet(i) ||
- rawBlockletColumnChunks.getBitSetGroup().getBitSet(i).isEmpty()) {
+if (isDimensionPresentInCurrentBlock.length == 1 &&
isDimensionPresentInCurrentBlock[0]
Review comment:
I think using equalTo expression I can reuse most of the code. what do
you think ? @QiangCai
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
