Github user kevinjmh commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2713#discussion_r242454233
--- Diff:
datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMapFactory.java
---
@@ -218,56 +218,46 @@ public DataMapBuilder createBuilder(Segment segment,
String shardName,
this.bloomFilterSize, this.bloomFilterFpp, bloomCompress);
}
- /**
- * returns all shard directories of bloom index files for query
- * if bloom index files are merged we should get only one shard path
- */
- private Set<String> getAllShardPaths(String tablePath, String segmentId)
{
- String dataMapStorePath = CarbonTablePath.getDataMapStorePath(
- tablePath, segmentId, dataMapName);
- CarbonFile[] carbonFiles =
FileFactory.getCarbonFile(dataMapStorePath).listFiles();
- Set<String> shardPaths = new HashSet<>();
+
+ private boolean isAllShardsMerged(String dmSegmentPath) {
+ boolean mergeShardExist = false;
boolean mergeShardInprogress = false;
- CarbonFile mergeShardFile = null;
+ CarbonFile[] carbonFiles =
FileFactory.getCarbonFile(dmSegmentPath).listFiles();
for (CarbonFile carbonFile : carbonFiles) {
- if
(carbonFile.getName().equals(BloomIndexFileStore.MERGE_BLOOM_INDEX_SHARD_NAME))
{
- mergeShardFile = carbonFile;
- } else if
(carbonFile.getName().equals(BloomIndexFileStore.MERGE_INPROGRESS_FILE)) {
+ String fileName = carbonFile.getName();
+ if
(fileName.equals(BloomIndexFileStore.MERGE_BLOOM_INDEX_SHARD_NAME)) {
+ mergeShardExist = true;
+ } else if
(fileName.equals(BloomIndexFileStore.MERGE_INPROGRESS_FILE)) {
mergeShardInprogress = true;
--- End diff --
Yes, you are right. We need to fix this. If we allow to use bloom filter
when the index files are merging, maybe any IO Exception will occur in
following steps when the merging is done.
Some simple ideas for this:
1. datamap do not choose bloom when merging is under action
2. change the pruning logic to segment independent, any datamap excepts
default datamap can reject or fail the segment pruning ( by return null or ?),
and no more result blocklet intersection for this datamap, such that this does
not affect final result
---
