Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2574#discussion_r205996478
--- Diff:
core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockDataMap.java
---
@@ -542,48 +551,48 @@ protected void
createSummaryDMStore(BlockletDataMapModel blockletDataMapModel)
List<Blocklet> blocklets = new ArrayList<>();
CarbonRowSchema[] schema = getFileFooterEntrySchema();
String filePath = getFilePath();
- int numBlocklets = 0;
- if (filterExp == null) {
- numBlocklets = memoryDMStore.getRowCount();
- for (int i = 0; i < numBlocklets; i++) {
- DataMapRow safeRow = memoryDMStore.getDataMapRow(schema,
i).convertToSafeRow();
- blocklets.add(createBlocklet(safeRow,
getFileNameWithFilePath(safeRow, filePath),
- getBlockletId(safeRow), false));
- }
- } else {
- // Remove B-tree jump logic as start and end key prepared is not
- // correct for old store scenarios
- int startIndex = 0;
- numBlocklets = memoryDMStore.getRowCount();
- FilterExecuter filterExecuter = FilterUtil
- .getFilterExecuterTree(filterExp, getSegmentProperties(), null,
getMinMaxCacheColumns());
- // flag to be used for deciding whether use min/max in executor
pruning for BlockletDataMap
- boolean useMinMaxForPruning = useMinMaxForExecutorPruning(filterExp);
- // min and max for executor pruning
- while (startIndex < numBlocklets) {
- DataMapRow safeRow = memoryDMStore.getDataMapRow(schema,
startIndex).convertToSafeRow();
- String fileName = getFileNameWithFilePath(safeRow, filePath);
- short blockletId = getBlockletId(safeRow);
- boolean isValid =
- addBlockBasedOnMinMaxValue(filterExecuter,
getMinMaxValue(safeRow, MAX_VALUES_INDEX),
- getMinMaxValue(safeRow, MIN_VALUES_INDEX), fileName,
blockletId);
- if (isValid) {
- blocklets.add(createBlocklet(safeRow, fileName, blockletId,
useMinMaxForPruning));
+ ByteBuffer byteBuffer =
ByteBuffer.wrap(getBlockletRowCountForEachBlock());
--- End diff --
for legacy store, the pruned result of other index datamap will be
processed by âcreateBlockletFromRelativeBlockletIdâ which will also try to
get blocklet count for each block --- will this also be a problem?
---
