[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3771: [WIP] pushdown array_contains filter to carbon
CarbonDataQA1 commented on pull request #3771: URL: https://github.com/apache/carbondata/pull/3771#issuecomment-633064918 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3056/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3771: [WIP] pushdown array_contains filter to carbon
CarbonDataQA1 commented on pull request #3771: URL: https://github.com/apache/carbondata/pull/3771#issuecomment-633063889 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1336/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on pull request #3771: [WIP] pushdown array_contains filter to carbon
QiangCai commented on pull request #3771: URL: https://github.com/apache/carbondata/pull/3771#issuecomment-633044918 if the query has only one simple filter(without and/or), maybe we can try to push down "limit" to filter. So the filter will not require to read all values of all rows. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #3771: [WIP] pushdown array_contains filter to carbon
QiangCai commented on a change in pull request #3771:
URL: https://github.com/apache/carbondata/pull/3771#discussion_r429538025
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/complextypes/ComplexQueryType.java
##
@@ -67,4 +67,18 @@ private DimensionColumnPage
getDecodedDimensionPage(DimensionColumnPage[][] dime
}
return dimensionColumnPages[columnIndex][pageNumber];
}
+
+ /**
+ * Method will copy the block chunk holder data and return the cloned value.
+ * This method is also used by child.
+ */
+ protected byte[] copyBlockDataChunkWithoutClone(DimensionRawColumnChunk[]
rawColumnChunks,
+ DimensionColumnPage[][] dimensionColumnPages, int rowNumber, int
pageNumber) {
+byte[] data =
+getDecodedDimensionPage(dimensionColumnPages,
rawColumnChunks[columnIndex], pageNumber)
Review comment:
how about to cache the page, it will not require to decode for each row
again.
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java
##
@@ -222,49 +224,90 @@ public BitSetGroup applyFilter(RawBlockletColumnChunks
rawBlockletColumnChunks,
}
}
BitSetGroup bitSetGroup = new BitSetGroup(pageNumbers);
-for (int i = 0; i < pageNumbers; i++) {
- BitSet set = new BitSet(numberOfRows[i]);
- RowIntf row = new RowImpl();
- BitSet prvBitset = null;
- // if bitset pipe line is enabled then use rowid from previous bitset
- // otherwise use older flow
- if (!useBitsetPipeLine ||
- null == rawBlockletColumnChunks.getBitSetGroup() ||
- null == bitSetGroup.getBitSet(i) ||
- rawBlockletColumnChunks.getBitSetGroup().getBitSet(i).isEmpty()) {
-for (int index = 0; index < numberOfRows[i]; index++) {
- createRow(rawBlockletColumnChunks, row, i, index);
- Boolean rslt = false;
- try {
-rslt = exp.evaluate(row).getBoolean();
- }
- // Any invalid member while evaluation shall be ignored, system will
log the
- // error only once since all rows the evaluation happens so inorder
to avoid
- // too much log inforation only once the log will be printed.
- catch (FilterIllegalMemberException e) {
-FilterUtil.logError(e, false);
- }
- if (null != rslt && rslt) {
-set.set(index);
+
+if (isDimensionPresentInCurrentBlock.length == 1 &&
isDimensionPresentInCurrentBlock[0]) {
+ // fill default value here
+ DimColumnResolvedFilterInfo dimColumnEvaluatorInfo =
dimColEvaluatorInfoList.get(0);
+ // if filter dimension is not present in the current add its default
value
+ if (dimColumnEvaluatorInfo.getDimension().getDataType().isComplexType())
{
+for (int i = 0; i < pageNumbers; i++) {
+ BitSet set = new BitSet(numberOfRows[i]);
+ RowIntf row = new RowImpl();
+ for (int index = 0; index < numberOfRows[i]; index++) {
+ArrayQueryType complexType =
+(ArrayQueryType)
complexDimensionInfoMap.get(dimensionChunkIndex[i]);
+int[] numberOfChild = complexType
+
.getNumberOfChild(rawBlockletColumnChunks.getDimensionRawColumnChunks(), null,
Review comment:
how about to get all numbers of the child once
##
File path:
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java
##
@@ -222,49 +224,90 @@ public BitSetGroup applyFilter(RawBlockletColumnChunks
rawBlockletColumnChunks,
}
}
BitSetGroup bitSetGroup = new BitSetGroup(pageNumbers);
-for (int i = 0; i < pageNumbers; i++) {
- BitSet set = new BitSet(numberOfRows[i]);
- RowIntf row = new RowImpl();
- BitSet prvBitset = null;
- // if bitset pipe line is enabled then use rowid from previous bitset
- // otherwise use older flow
- if (!useBitsetPipeLine ||
- null == rawBlockletColumnChunks.getBitSetGroup() ||
- null == bitSetGroup.getBitSet(i) ||
- rawBlockletColumnChunks.getBitSetGroup().getBitSet(i).isEmpty()) {
-for (int index = 0; index < numberOfRows[i]; index++) {
- createRow(rawBlockletColumnChunks, row, i, index);
- Boolean rslt = false;
- try {
-rslt = exp.evaluate(row).getBoolean();
- }
- // Any invalid member while evaluation shall be ignored, system will
log the
- // error only once since all rows the evaluation happens so inorder
to avoid
- // too much log inforation only once the log will be printed.
- catch (FilterIllegalMemberException e) {
-FilterUtil.logError(e, false);
- }
- if (null != rslt && rslt) {
-set.set(index);
+
+if (isDimensionPresentInCurrentBlock.length == 1 &&
isDimensionPresentInCurrentBlock[0]) {
+ // fill default value
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3771: [WIP] pushdown array_contains filter to carbon
ajantha-bhat commented on a change in pull request #3771:
URL: https://github.com/apache/carbondata/pull/3771#discussion_r429519687
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestCompactionComplexType.scala
##
@@ -47,6 +47,33 @@ class TestCompactionComplexType extends QueryTest with
BeforeAndAfterAll {
sql("DROP TABLE IF EXISTS compactComplex")
}
+ test("complex issue") {
+sql("drop table if exists complex1")
+sql("create table complex1 (arr array) stored as carbondata")
+sql("insert into complex1 select array('as') union all " +
+"select array('sd','df','gh') union all " +
+"select array('rt','ew','rtyu','jk','sder') union all " +
+"select array('ghsf','dbv','fg','ty') union all " +
+"select array('hjsd','fggb','nhj','sd','asd')")
+
Review comment:
This is WIP temp, cannot merge this poc code. Why review?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3771: [WIP] pushdown array_contains filter to carbon
Indhumathi27 commented on a change in pull request #3771:
URL: https://github.com/apache/carbondata/pull/3771#discussion_r429519286
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestCompactionComplexType.scala
##
@@ -47,6 +47,33 @@ class TestCompactionComplexType extends QueryTest with
BeforeAndAfterAll {
sql("DROP TABLE IF EXISTS compactComplex")
}
+ test("complex issue") {
+sql("drop table if exists complex1")
+sql("create table complex1 (arr array) stored as carbondata")
+sql("insert into complex1 select array('as') union all " +
+"select array('sd','df','gh') union all " +
+"select array('rt','ew','rtyu','jk','sder') union all " +
+"select array('ghsf','dbv','fg','ty') union all " +
+"select array('hjsd','fggb','nhj','sd','asd')")
+
Review comment:
Please add test scenario with data as null. array(null)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3771: [WIP] pushdown array_contains filter to carbon
ajantha-bhat commented on a change in pull request #3771:
URL: https://github.com/apache/carbondata/pull/3771#discussion_r429519687
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestCompactionComplexType.scala
##
@@ -47,6 +47,33 @@ class TestCompactionComplexType extends QueryTest with
BeforeAndAfterAll {
sql("DROP TABLE IF EXISTS compactComplex")
}
+ test("complex issue") {
+sql("drop table if exists complex1")
+sql("create table complex1 (arr array) stored as carbondata")
+sql("insert into complex1 select array('as') union all " +
+"select array('sd','df','gh') union all " +
+"select array('rt','ew','rtyu','jk','sder') union all " +
+"select array('ghsf','dbv','fg','ty') union all " +
+"select array('hjsd','fggb','nhj','sd','asd')")
+
Review comment:
This is WIP temp, cannot merge this poc code. Why review?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3771: [WIP] pushdown array_contains filter to carbon
Indhumathi27 commented on a change in pull request #3771:
URL: https://github.com/apache/carbondata/pull/3771#discussion_r429519286
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestCompactionComplexType.scala
##
@@ -47,6 +47,33 @@ class TestCompactionComplexType extends QueryTest with
BeforeAndAfterAll {
sql("DROP TABLE IF EXISTS compactComplex")
}
+ test("complex issue") {
+sql("drop table if exists complex1")
+sql("create table complex1 (arr array) stored as carbondata")
+sql("insert into complex1 select array('as') union all " +
+"select array('sd','df','gh') union all " +
+"select array('rt','ew','rtyu','jk','sder') union all " +
+"select array('ghsf','dbv','fg','ty') union all " +
+"select array('hjsd','fggb','nhj','sd','asd')")
+
Review comment:
Please add test scenario with data as null. array(null)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
