[
https://issues.apache.org/jira/browse/LENS-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322148#comment-15322148
]
Amareshwari Sriramadasu commented on LENS-1168:
-----------------------------------------------
Here are the approaches I'm thinking of :
Approach 1 :
- We will separate query submitters for each driver.
- We might have to remove server level launch constraint checks.
- We separate the queued, waiting and launched queries to be separate for each
driver.
- side effect of this - users will see queue number at driver level.
- Status poller need not poll for all drivers. For example : All JDBC
queries can be registered with notification completion listener and polling is
not required as we already have dedicated thread to execute in jdbc driver.
Approach 1A : Approach 1 + the following
- with pool submitters for each driver : Default size of the pool will be 1.
But overridable through configuration.
- Pool of submitters will be locked on constraint checks to be applied, within
the same lock change the state of the query to 'LAUNCHING' - if any query
passes constraints. So that others will apply the constraints correctly.
Approach 1B : Approach 1 + the following
- with single submitter for each driver
- Queries will be still launched one by one.
- Constraint checks will be applied one by one. Still we could be less
efficient as we are not using the concurrency provided by underlying driver to
the fullest.
Approach 2 :
We have pool of submitters shared across all drivers
- We can have big pool of submitters for scaling.
- Submitters are not tied to any driver.
- Will have server level constraints.
- Each submitter should apply constraint checks in a lock. Within the same
lock change the state of the query to 'LAUNCHING', so that others will apply
the constraints correctly.
> Isolate submissions for each driver
> -----------------------------------
>
> Key: LENS-1168
> URL: https://issues.apache.org/jira/browse/LENS-1168
> Project: Apache Lens
> Issue Type: Improvement
> Components: server
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 2.6
>
>
> We need to isolate query submitters for each driver to be different. We have
> seen scenarios where hive submission takes time and other interactive queries
> on jdbc are queued because querysubmitter was busy.
> Also, each driver can have a pool of submitters - which should benefit
> interactive queries with immediate submissions, especially the ones submitted
> on EXECUTE_WITH_TIMEOUT
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
