zhxiaoping commented on a change in pull request #3209: [CARBONDATA-3373]
Optimize scenes with in numbers in SQL
URL: https://github.com/apache/carbondata/pull/3209#discussion_r285365328
##########
File path:
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/IncludeFilterExecuterImpl.java
##########
@@ -272,29 +273,21 @@ private BitSet getFilteredIndexesForMeasures(ColumnPage
columnPage,
// Get the measure values from the chunk. compare sequentially with the
// the filter values. The one that matches sets it Bitset.
BitSet bitSet = new BitSet(rowsInPage);
- Object[] filterValues = msrColumnExecutorInfo.getFilterKeys();
-
- SerializableComparator comparator =
Comparator.getComparatorByDataTypeForMeasure(msrType);
+ Set filterValuesSet = msrColumnExecutorInfo.getFilterKeysSet();
BitSet nullBitSet = columnPage.getNullBits();
- for (int i = 0; i < filterValues.length; i++) {
- if (filterValues[i] == null) {
- for (int j = nullBitSet.nextSetBit(0); j >= 0; j =
nullBitSet.nextSetBit(j + 1)) {
- bitSet.set(j);
- }
- continue;
- }
- for (int startIndex = 0; startIndex < rowsInPage; startIndex++) {
- if (!nullBitSet.get(startIndex)) {
- // Check if filterValue[i] matches with measure Values.
- Object msrValue = DataTypeUtil
- .getMeasureObjectBasedOnDataType(columnPage, startIndex,
- msrType, msrColumnEvaluatorInfo.getMeasure());
-
- if (comparator.compare(msrValue, filterValues[i]) == 0) {
- // This is a match.
- bitSet.set(startIndex);
- }
+ for (int startIndex = 0; startIndex < rowsInPage; startIndex++) {
+ if (!nullBitSet.get(startIndex)) {
+ // Check if filterValue[i] matches with measure Values.
+ Object msrValue = DataTypeUtil
+ .getMeasureObjectBasedOnDataType(columnPage, startIndex,
+ msrType, msrColumnEvaluatorInfo.getMeasure());
+
+ if (filterValuesSet.contains(msrValue)) {
Review comment:
if you guys review this code without context, it exactly is what you said
above, but the filter value and column value already were converted to be that
type is the same and precision is the same before my modified code. so the
situation you worry about will never happen. I add a test case to the project,
and it all passed. the last point : if I use treemap, it will increase the time
complexity ,please review again, thank you,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
