[ 
https://issues.apache.org/jira/browse/CARBONDATA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15499422#comment-15499422
 ] 

ASF GitHub Bot commented on CARBONDATA-241:
-------------------------------------------

Github user gvramana commented on a diff in the pull request:

    https://github.com/apache/incubator-carbondata/pull/158#discussion_r79288466
  
    --- Diff: 
core/src/main/java/org/apache/carbondata/core/carbon/datastore/BlockIndexStore.java
 ---
    @@ -260,11 +295,29 @@ public void removeTableBlocks(List<TableBlockInfo> 
removeTableBlocksInfos,
         }
         Map<TableBlockInfo, AbstractIndex> map = 
tableBlocksMap.get(absoluteTableIdentifier);
         // if there is no loaded blocks then return
    -    if (null == map) {
    +    if (null == map || map.isEmpty()) {
    +      return;
    +    }
    +    Map<String, List<TableBlockInfo>> segmentIdToBlockInfoMap =
    +        segmentIdToBlockListMap.get(absoluteTableIdentifier);
    +    if (null == segmentIdToBlockInfoMap || 
segmentIdToBlockInfoMap.isEmpty()) {
           return;
         }
    -    for (TableBlockInfo blockInfos : removeTableBlocksInfos) {
    -      map.remove(blockInfos);
    +    synchronized (lockObject) {
    +      for (String segmentId : segmentsToBeRemoved) {
    +        List<TableBlockInfo> tableBlockInfoList = 
segmentIdToBlockInfoMap.get(segmentId);
    +        if (null == tableBlockInfoList) {
    +          continue;
    +        }
    +        Iterator<TableBlockInfo> tableBlockInfoIterator = 
tableBlockInfoList.iterator();
    +        while (tableBlockInfoIterator.hasNext()) {
    +          TableBlockInfo info = tableBlockInfoIterator.next();
    +          AbstractIndex remove = map.remove(info);
    +          if (null != remove) {
    --- End diff --
    
    tableBlockInfoIterator.remove needs to called irrespective of null != remove


> OOM error during query execution in long run
> --------------------------------------------
>
>                 Key: CARBONDATA-241
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-241
>             Project: CarbonData
>          Issue Type: Bug
>            Reporter: kumar vishal
>            Assignee: kumar vishal
>
> **Problem:** During long run query execution is taking more time and it is 
> throwing out of memory issue.
> **Reason**: In compaction we are compacting segments and each segment 
> metadata is loaded in memory. So after compaction compacted segments are 
> invalid but its meta data is not removed from memory because of this 
> duplicate metadata is pile up and it is taking more memory and after few days 
> query exeution is throwing OOM
> **Solution**: Need to remove invalid blocks from memory
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to