[
https://issues.apache.org/jira/browse/LIVY-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274174#comment-16274174
]
Saisai Shao commented on LIVY-421:
----------------------------------
bq. Adding concurrency for sql make sense for me as each sql statements should
be independent.
Not exactly. For example, create table A, load data, select from table A. These
3 sql queries should not be run concurrently.
Actually even for one thread per interpreter, there still has minor issues like
shared variables, like one interpreter register a table, then another
interpreter do some transformations on the registered table.
It's really difficult for Livy to identify what kind of statements can be run
concurrently. I think for the resource under-utilize issue, dynamic allocation
is your friend!
> support multi statements run at the same time
> ---------------------------------------------
>
> Key: LIVY-421
> URL: https://issues.apache.org/jira/browse/LIVY-421
> Project: Livy
> Issue Type: Improvement
> Components: REPL
> Affects Versions: 0.4
> Reporter: zhuweibin
> Labels: performance
>
> In the v0.4.0-incubating's’ implementation, the number of threads used to
> execute the statement code is hardcoded to one. When idle resources are
> sufficient to execute multiple statements, a lot of resources are wasted and
> blocking statements run. We are considering improvements in one of two ways:
> # 1. Add a configuration in livy.conf to globally control the number of
> threads in interpreterExecutor for all repl/sessions to execute statements
> # 2 in the session to create a rest api parameter to set the maximum number
> of statements can be run at the same time the number of statements
> hope to discuss this issue, thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
