xuchuanyin commented on a change in pull request #3102: [CARBONDATA-3272]fix
ArrayIndexOutOfBoundsException of horizontal compaction during update, when
cardinality changes within a segment
URL: https://github.com/apache/carbondata/pull/3102#discussion_r251230402
##########
File path:
processing/src/main/java/org/apache/carbondata/processing/merger/CarbonCompactionExecutor.java
##########
@@ -140,25 +139,72 @@ public CarbonCompactionExecutor(Map<String,
TaskBlockInfo> segmentMapping,
|| !CarbonCompactionUtil.isSorted(listMetadata.get(0));
for (String task : taskBlockListMapping) {
list = taskBlockInfo.getTableBlockInfoList(task);
- Collections.sort(list);
- LOGGER.info(
- "for task -" + task + "- in segment id -" + segmentId + "- block
size is -" + list
- .size());
- queryModel.setTableBlockInfos(list);
- if (sortingRequired) {
- resultList.get(CarbonCompactionUtil.UNSORTED_IDX).add(
- new RawResultIterator(executeBlockList(list, segmentId, task,
configuration),
- sourceSegProperties, destinationSegProperties, false));
- } else {
- resultList.get(CarbonCompactionUtil.SORTED_IDX).add(
- new RawResultIterator(executeBlockList(list, segmentId, task,
configuration),
- sourceSegProperties, destinationSegProperties, false));
+ // during update there may be a chance that the cardinality may change
within the segment
Review comment:
@akashrn5 Can you explain in which scenario that the cardinality in one
segment will change? I thought there may be other problems besides compaction:
for example for bloomfilter datamap's rebuilding procedure we only use one
keyGenerator for each segment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
