[
https://issues.apache.org/jira/browse/IMPALA-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16971338#comment-16971338
]
Vihang Karajgaonkar commented on IMPALA-9140:
---------------------------------------------
Thanks [~stigahuang] for the detailed explanation. The reason I see it as
redundant is as follows:
For async load requests (could be backgroundLoad or prioritized, doesn't matter
since they are all adding the table name to the FIFO queue)
1. A thread in {{TableLoadingMgr.startTableLoadingThreads}} pool makes sure
that the table name which is added in the queue is submitted to the loading
pool {{tblLoadingPool_}}
2. The async threads call {{loadNextTable}} which internally just picks up the
element from the queue and calls {{getOrLoadTable}}. {{getOrLoadTable}} calls
{{replaceTableIfUnchanged}} which adds the table to the catalog.
In case of sync load tables calls during the DDL processing, CatalogOpExecutor
directly issues a {{getOrLoadTable}} which internally calls
{{tableLoadingMgr_.loadAsync}} which does a {{tblLoadingPool_.execute()}}
> Get rid of the unnecessary load submitter thread pool in tblLoadingMgr
> ----------------------------------------------------------------------
>
> Key: IMPALA-9140
> URL: https://issues.apache.org/jira/browse/IMPALA-9140
> Project: IMPALA
> Issue Type: Bug
> Reporter: Vihang Karajgaonkar
> Priority: Major
>
> This JIRA is created as a followup on the discussion on
> https://gerrit.cloudera.org/#/c/14611 related to various pools used for
> loading tables.
> It looks like there are 2 pools of threads both of the size
> {{num_metadata_loading_threads}}. One pool is used to submit the load
> requests to another pool {{tblLoadingPool_}} which does the actual loading of
> the tables. I think we can get rid of the pool which submits the tasks since
> it is not very time-consuming operation and can be done synchronously (all it
> needs to do submit the task in the queue in the front or back based on
> whether its a prioritized load or background load). This will simplify the
> loading code and reduce unnecessary number of threads being created by
> {{TblLoadingMgr}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
