[ https://issues.apache.org/jira/browse/HBASE-24436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117881#comment-17117881 ]
Andrew Kyle Purtell commented on HBASE-24436: --------------------------------------------- In another brainstorming we had where I work there is an idea to use a fork join pool for open work. The pool is allocated at the regionserver level. Open work is recursively decomposed to align with the fork-join model. Each open task is a runnable. Each store file open is a runnable. We let the pool decide what is the appropriate level of parallelism and do not give an explicit bound. I can’t prove it without trying it and collecting metrics but this would allow us to be faster about opening than any option that uses fixed size pools when we have uneven distribution of number of files in stores, some quite large (like date tiered ones). In the meantime if you are going to continue to use fixed sized pools they should be allocated per store not per region so they will at least be proportional to the number of stores (as opposed to number of regions, which is not the same and may be substantially lower, imagine a table with 10 families... you are exchanging 10 fixed size pools for 1 fixed size pool with this proposal, seems the wrong direction) > The store file open and close thread pool should be shared at the region level > ------------------------------------------------------------------------------ > > Key: HBASE-24436 > URL: https://issues.apache.org/jira/browse/HBASE-24436 > Project: HBase > Issue Type: Improvement > Reporter: Junhong Xu > Assignee: Junhong Xu > Priority: Minor > > For now, we provide threads per column family evenly in general, but there > are some cases that some column families have much more store files than > others( maybe that's the life, right? ). So in that case, some Stores have > beed done quickly while others are struggling.We should share the thread pool > at the region level in case of data skew. -- This message was sent by Atlassian Jira (v8.3.4#803005)