[
https://issues.apache.org/jira/browse/LIVY-535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16695949#comment-16695949
]
Alexey Romanenko commented on LIVY-535:
---------------------------------------
https://github.com/apache/incubator-livy/pull/129
> Config option “livy.server.session.max-creation” doesn't work properly
> ----------------------------------------------------------------------
>
> Key: LIVY-535
> URL: https://issues.apache.org/jira/browse/LIVY-535
> Project: Livy
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Alexey Romanenko
> Assignee: Alexey Romanenko
> Priority: Major
>
> Config option {{livy.server.session.max-creation}} doesn't work properly.
> For example, I set the value of this option to “1” and try to submit 5 batch
> jobs (example SparkPi) to Livy almost in the same time using a bash command
> like this:
> {code:bash}
> for i in {1..5}; do curl -X POST --data '{"file":
> "/tmp/spark-examples-2.jar", "className":
> "org.apache.spark.examples.SparkPi"}' -H "Content-Type: application/json"
> http://localhost:8999/batches; done
> {code}
> In such case, I expect that the only one job (first) will be submitted
> properly and others will be rejected with a response message “Rejected, too
> many sessions are being created!”. This is what I actually have when I submit
> them manually, one by one. Though, in the case given above, almost always all
> 5 jobs will be submitted properly (sometimes only one will be rejected)
> despite of the value of config option.
> Looks like, that code, where Livy checks the number of already created
> sessions and launch new one, is not atomic:
> https://github.com/apache/incubator-livy/blob/4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f/server/src/main/scala/org/apache/livy/server/SessionServlet.scala#L130
> Also, the calculation of the total number of child processes in
> {{tooManySessions()}} is not atomic operation too.
> So, when I submit the jobs almost in the same moment (POST request takes only
> about tens of milliseconds) then I can spawn more spark-submit processes
> despite of maximum limitation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
