[jira] [Commented] (ASTERIXDB-2103) Too many disk components for InvertedIndex with CorrelatedMergePolicy

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ASTERIXDB-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199708#comment-16199708
 ] 

ASF subversion and git services commented on ASTERIXDB-2103:


Commit 1dc8228b7262e47c06487443da0f735321e63afe in asterixdb's branch 
refs/heads/master from [~luochen01]
[ https://git-wip-us.apache.org/repos/asf?p=asterixdb.git;h=1dc8228 ]

Revert "[ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy"

This reverts commit 21ed0f72681a20ccb6a654f9aa4d54b8d0ea9c5c.

Change-Id: I670545acd09c678f21be25313353ab306be86202
Reviewed-on: https://asterix-gerrit.ics.uci.edu/2063
Tested-by: Jenkins 
Contrib: Jenkins 
Reviewed-by: Ian Maxon 
Integration-Tests: Jenkins 


> Too many disk components for InvertedIndex with CorrelatedMergePolicy
> -
>
> Key: ASTERIXDB-2103
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2103
> Project: Apache AsterixDB
>  Issue Type: Bug
>Reporter: Chen Luo
>Assignee: Chen Luo
>
> As reported by [~javierjia], when using CorrelatedPrefixMergePolicy, the 
> secondary InvertedIndex could have too many disk components (the primary 
> index has ~ 200 disk components, while the InvertedIndex has ~500 disk 
> components).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ASTERIXDB-2103) Too many disk components for InvertedIndex with CorrelatedMergePolicy

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/ASTERIXDB-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198853#comment-16198853
 ] 

ASF subversion and git services commented on ASTERIXDB-2103:


Commit 21ed0f72681a20ccb6a654f9aa4d54b8d0ea9c5c in asterixdb's branch 
refs/heads/master from [~luochen01]
[ https://git-wip-us.apache.org/repos/asf?p=asterixdb.git;h=21ed0f7 ]

[ASTERIXDB-2103][STO] Too many disk components for CorrelatedPolicy

- user model changes: no
- storage format changes: no
- interface changes: yes

Details:
Currently CorrelatedMergePolicy uses component Ids to ensure disk
components of primary and secondary indexes are merged together,
but without synchronization. However, this results in too many disk
components for secondary InvertedIndex. The reason is that secondary
index could miss some round of merges, if the merge policy finds out
the corresponding secondary components are not available (either being
merged or being flushed). Even though flow-control on secondary indexes
can guarantee the secondary index would catch up the next time, it is
still possible that the primary component is finialized, which leaves
the secondary components which miss this round of merge are never merged
again.

This patch fixes this bug by:
- Add the mechanism of depending operations to LSM IO operation. An
operation finishes only after all depending operations have finished.
- For correlated merge policy, the flush/merge of the primary index depends
on all flushes/merges of secondary indexes. This ensures when the
correlated policy schedules merge, all related components of all indexes
are available to merge.

Change-Id: Ib6c06ee23f3bfd16b758802388389c00e29780b1
Reviewed-on: https://asterix-gerrit.ics.uci.edu/2018
Sonar-Qube: Jenkins 
Tested-by: Jenkins 
Contrib: Jenkins 
Integration-Tests: Jenkins 
Reviewed-by: Jianfeng Jia 


> Too many disk components for InvertedIndex with CorrelatedMergePolicy
> -
>
> Key: ASTERIXDB-2103
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2103
> Project: Apache AsterixDB
>  Issue Type: Bug
>Reporter: Chen Luo
>Assignee: Chen Luo
>
> As reported by [~javierjia], when using CorrelatedPrefixMergePolicy, the 
> secondary InvertedIndex could have too many disk components (the primary 
> index has ~ 200 disk components, while the InvertedIndex has ~500 disk 
> components).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)