[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version

2020-06-25 Thread GitBox


CarbonDataQA1 commented on pull request #3807:
URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649583529


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3221/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version

2020-06-25 Thread GitBox


CarbonDataQA1 commented on pull request #3807:
URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649581253


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1494/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3771: [CARBONDATA-3849] pushdown array_contains filter to carbon for array of primitive types

2020-06-25 Thread GitBox


ajantha-bhat commented on a change in pull request #3771:
URL: https://github.com/apache/carbondata/pull/3771#discussion_r445361657



##
File path: 
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java
##
@@ -222,49 +228,103 @@ public BitSetGroup applyFilter(RawBlockletColumnChunks 
rawBlockletColumnChunks,
   }
 }
 BitSetGroup bitSetGroup = new BitSetGroup(pageNumbers);
-for (int i = 0; i < pageNumbers; i++) {
-  BitSet set = new BitSet(numberOfRows[i]);
-  RowIntf row = new RowImpl();
-  BitSet prvBitset = null;
-  // if bitset pipe line is enabled then use rowid from previous bitset
-  // otherwise use older flow
-  if (!useBitsetPipeLine ||
-  null == rawBlockletColumnChunks.getBitSetGroup() ||
-  null == bitSetGroup.getBitSet(i) ||
-  rawBlockletColumnChunks.getBitSetGroup().getBitSet(i).isEmpty()) {
+if (isDimensionPresentInCurrentBlock.length == 1 && 
isDimensionPresentInCurrentBlock[0]

Review comment:
   I think using equalTo expression I can reuse most of the code. what do 
you think ? @QiangCai 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] chenliang613 commented on pull request #3801: adding comments with issue numbers.

2020-06-25 Thread GitBox


chenliang613 commented on pull request #3801:
URL: https://github.com/apache/carbondata/pull/3801#issuecomment-649381380


   add to whitelist



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] chenliang613 commented on pull request #3801: adding comments with issue numbers.

2020-06-25 Thread GitBox


chenliang613 commented on pull request #3801:
URL: https://github.com/apache/carbondata/pull/3801#issuecomment-649381896


   this pr is too big.  be careful to merge.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3801: adding comments with issue numbers.

2020-06-25 Thread GitBox


CarbonDataQA1 commented on pull request #3801:
URL: https://github.com/apache/carbondata/pull/3801#issuecomment-649388349


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1491/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3771: [CARBONDATA-3849] pushdown array_contains filter to carbon for array of primitive types

2020-06-25 Thread GitBox


CarbonDataQA1 commented on pull request #3771:
URL: https://github.com/apache/carbondata/pull/3771#issuecomment-649431612


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1490/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version

2020-06-25 Thread GitBox


CarbonDataQA1 commented on pull request #3807:
URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649439729


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3219/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3801: adding comments with issue numbers.

2020-06-25 Thread GitBox


CarbonDataQA1 commented on pull request #3801:
URL: https://github.com/apache/carbondata/pull/3801#issuecomment-649384032


   Build Failed  with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3218/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark 2.4 profile

2020-06-25 Thread GitBox


ajantha-bhat commented on pull request #3807:
URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649391566


   @kunal642 , @akashrn5 : check it



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat edited a comment on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark 2.4 profile

2020-06-25 Thread GitBox


ajantha-bhat edited a comment on pull request #3807:
URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649391566


   @kunal642 , @akashrn5 : please check it



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat opened a new pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark 2.4 profile

2020-06-25 Thread GitBox


ajantha-bhat opened a new pull request #3807:
URL: https://github.com/apache/carbondata/pull/3807


### Why is this PR needed?
After #3700 , spark binary version addition. 
   a) Module name always taking deafult profile spark binary version.
   b)   IDE test cases are running with spark 2.3 even for spark2.3.4 because 
of above problem

### What changes were proposed in this PR?
   a)  keep a simple module name without binary version
   b) Use the package option with jar file name to have binary version in the 
final jar. 
   
### Does this PR introduce any user interface change?
- No
   
### Is any new testcase added?
- No
   
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version

2020-06-25 Thread GitBox


CarbonDataQA1 commented on pull request #3807:
URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649437666


   Build Failed  with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1492/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3771: [CARBONDATA-3849] pushdown array_contains filter to carbon for array of primitive types

2020-06-25 Thread GitBox


CarbonDataQA1 commented on pull request #3771:
URL: https://github.com/apache/carbondata/pull/3771#issuecomment-649430423


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3217/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3772: [CARBONDATA-3832]Added block and blocket pruning for the polygon expression processing

2020-06-25 Thread GitBox


CarbonDataQA1 commented on pull request #3772:
URL: https://github.com/apache/carbondata/pull/3772#issuecomment-649532771


   Build Success with Spark 2.3.4, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3220/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3772: [CARBONDATA-3832]Added block and blocket pruning for the polygon expression processing

2020-06-25 Thread GitBox


CarbonDataQA1 commented on pull request #3772:
URL: https://github.com/apache/carbondata/pull/3772#issuecomment-649533543


   Build Success with Spark 2.4.5, Please check CI 
http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1493/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3772: [CARBONDATA-3832]Added block and blocket pruning for the polygon expression processing

2020-06-25 Thread GitBox


VenuReddy2103 commented on a change in pull request #3772:
URL: https://github.com/apache/carbondata/pull/3772#discussion_r445481119



##
File path: 
core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java
##
@@ -188,6 +189,14 @@ private static FilterExecuter createFilterExecuterTree(
   return new FalseFilterExecutor();
 case ROWLEVEL:
 default:
+  if (filterExpressionResolverTree.getFilterExpression() instanceof 
UnknownExpression) {
+FilterExecuter filterExecuter =
+((UnknownExpression) 
filterExpressionResolverTree.getFilterExpression())
+.getFilterExecuter(filterExpressionResolverTree, 
segmentProperties);
+if (filterExecuter != null) {

Review comment:
   I understand what you meant. I think, it is better to create the 
RowLevelFilterExecuterImpl only at this place. If UnknownExpression do not have 
any specific implemenation, it will be like a fallback to 
RowLevelFilterExecuterImpl.
Also, if we have return RowLevelFilterExecuterImpl from 
UnknownExpression.getFilterExecuter, we need to pass complexDimensionInfoMap 
and all overiding methods will have this argument. It is an unused argument as 
current UnKnown expressions do not support complex dimensions.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3772: [CARBONDATA-3832]Added block and blocket pruning for the polygon expression processing

2020-06-25 Thread GitBox


VenuReddy2103 commented on a change in pull request #3772:
URL: https://github.com/apache/carbondata/pull/3772#discussion_r445475371



##
File path: 
integration/spark/src/test/scala/org/apache/carbondata/geo/GeoTest.scala
##
@@ -134,8 +133,31 @@ class GeoTest extends QueryTest with BeforeAndAfterAll 
with BeforeAndAfterEach {
 Row(116285807, 40084087)))
   }
 
-  test("test polygon query on table partitioned by timevalue column")
-  {
+  test("test block pruning for polygon query") {
+createTable()
+sql(s"insert into $table1 select 157542840,116285807,40084087")
+sql(s"insert into $table1 select 157542840,116372142,40129503")
+sql(s"insert into $table1 select 157542840,116187332,39979316")
+sql(s"insert into $table1 select 157542840,116337069,39951887")
+sql(s"insert into $table1 select 157542840,116359102,40154684")
+sql(s"insert into $table1 select 157542840,116736367,39970323")
+sql(s"insert into $table1 select 157542840,116362699,39942444")
+sql(s"insert into $table1 select 157542840,116325378,39963129")
+sql(s"insert into $table1 select 157542840,116302895,39930753")
+sql(s"insert into $table1 select 157542840,116288955,3101")
+val df = sql(s"select longitude, latitude from $table1 where 
IN_POLYGON('116.321011 " +
+ s"40.123503, 116.137676 39.947911, 116.560993 39.935276, 
116.321011 40.123503')")
+assert(df.rdd.getNumPartitions == 6)
+checkAnswer(df,

Review comment:
   Ok. Instead of making the Seq of rows with same rows in all testcases, 
have defined it once and used same in all the testcases.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [carbondata] ajantha-bhat commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version

2020-06-25 Thread GitBox


ajantha-bhat commented on pull request #3807:
URL: https://github.com/apache/carbondata/pull/3807#issuecomment-649537393


   retest this please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org