[
https://issues.apache.org/jira/browse/LIVY-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775778#comment-16775778
]
weiwenda edited comment on LIVY-544 at 2/23/19 3:47 AM:
--------------------------------------------------------
1. while in my solution, i make a new statement kind named SerialSQL,
and user post
{ "code": "show tables" "kind":"serial_sql" }
to run a SQL in serial mode
2. when user post old kind "sql", statement will run in concurrent mode, the
threadpool size control by "livy.rsc.sql.interpreter.threadPool.size", the
destination spark scheduler pool control by
"livy.rsc.sql.default.scheduler.pool"
3. user have to edit livy-fairscheduler.xml to make SparkContext run in multi
scheduler pools. for example
{quote}<allocations>
<pool name="default">
<schedulingMode>FIFO</schedulingMode>
<weight>1</weight>
<minShare>0</minShare>
</pool>
<pool name="fair">
<schedulingMode>FAIR</schedulingMode>
<weight>1</weight>
<minShare>0</minShare>
</pool>
</allocations>
{quote}
and set livy.rsc.sql.default.scheduler.pool=fair
spark.scheduler.mode=FAIR
4. only "sql" kind statement will run into fair scheduler pool
more details look at [https://github.com/apache/incubator-livy/pull/135]
was (Author: [email protected]):
1. while in my solution, i make a new statement kind named SerialSQL,
and user post {
"code": "show datasources"
"kind":"serial_sql"
} to run a SQL in serial mode
2. when user post old kind "sql", statement will run in concurrent mode, the
threadpool size control by "livy.rsc.sql.interpreter.threadPool.size", the
destination spark scheduler pool control by
"livy.rsc.sql.default.scheduler.pool"
3. user have to edit livy-fairscheduler.xml to make SparkContext run in multi
scheduler pools. for example
{quote}<allocations>
<pool name="default">
<schedulingMode>FIFO</schedulingMode>
<weight>1</weight>
<minShare>0</minShare>
</pool>
<pool name="fair">
<schedulingMode>FAIR</schedulingMode>
<weight>1</weight>
<minShare>0</minShare>
</pool>
</allocations>{quote}
and set livy.rsc.sql.default.scheduler.pool=fair
spark.scheduler.mode=FAIR
4. only "sql" kind statement will run into fair scheduler pool
more details look at [https://github.com/apache/incubator-livy/pull/135]
> Statements in a Session must execute one by one which is time-consuming
> -----------------------------------------------------------------------
>
> Key: LIVY-544
> URL: https://issues.apache.org/jira/browse/LIVY-544
> Project: Livy
> Issue Type: Improvement
> Components: REPL
> Affects Versions: 0.5.0
> Reporter: weiwenda
> Priority: Minor
> Labels: performance
>
> * when user's main usage scenario is not sensitive to the order of execution,
> such as sql Session Kind, user may hope livy server can deal with multi
> statement concurrentlly
> * but as interpreterExecutor initialized as singleThreadExecutor, it's no
> possible to satisfy such requirement.
> {quote}private val interpreterExecutor =
> ExecutionContext.fromExecutorService(Executors.newSingleThreadExecutor())
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
