[
https://issues.apache.org/jira/browse/ASTERIXDB-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309361#comment-17309361
]
ASF subversion and git services commented on ASTERIXDB-2813:
------------------------------------------------------------
Commit 8f1cd017b710c6de4bf5f1b480f8daaeb06de388 in asterixdb's branch
refs/heads/master from luochen
[ https://gitbox.apache.org/repos/asf?p=asterixdb.git;h=8f1cd01 ]
[ASTERIXDB-2813] Limit the number of flush/merge threads
- user model changes: no
- storage format changes: no
- interface changes: yes.
Details:
- Limit the number of flush/merge threads by introducing
the following parameters.
- storage.max.running.flushes.per.partition: the maximum
number of running flushes for each partition.
- storage.max.scheduled.merge.per.partition: the maximum
number of scheduled merges for each partition. This is
mainly used by the greedy scheduler.
- storage.max.running.merges.per.partition: the maximum
number of running mergese per partition.
- Basically, we limit the number of flush/merge threads
and put newly created flush/merge opreations into a wait
queue if the limit is reached.
- For the greedy scheduler, the scheduled merges
(i.e., merge threads) are more than the running merges
so that the scheduler can pick the smallest merge
for each LSM-tree.
Change-Id: I85a55423a1438b1d534c2e6a5968e675a99884c8
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/9183
Integration-Tests: Jenkins <jenk...@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <mhub...@apache.org>
Tested-by: Murtadha Hubail <mhub...@apache.org>
> Limit the number of flush/merge threads
> ---------------------------------------
>
> Key: ASTERIXDB-2813
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2813
> Project: Apache AsterixDB
> Issue Type: Improvement
> Components: STO - Storage
> Reporter: Chen Luo
> Assignee: Chen Luo
> Priority: Major
>
> Currently, AsterixDB uses one thread to execute each flush and merge
> operation. This may result in a large number of I/O threads in some cases,
> e.g., when writing to many datasets at the same time.
>
> A better solution is to enforce some limit on the number of flush and merge
> threads. When there is no thread available, a newly created flush or merge
> operation should be delayed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
