+dev@ I’ve created a Jira [1] and PR [2] for that.
[1] https://issues.apache.org/jira/browse/LIVY-535 <https://issues.apache.org/jira/browse/LIVY-535> [2] https://github.com/apache/incubator-livy/pull/129 <https://github.com/apache/incubator-livy/pull/129> > On 19 Nov 2018, at 19:00, Alexey Romanenko <aromanenko....@gmail.com> wrote: > > Hi all, > > I’d like to confirm that the current behaviour of config option > “livy.server.session.max-creation” is correct or not. > > For example, I set the value of this option to “1” and try to submit 5 batch > jobs (example SparkPi) on Livy almost in the same time using a command like > this: > “for i in {1..5}; do curl -X POST --data '{"file": > "/tmp/spark-examples-2.jar", "className": > "org.apache.spark.examples.SparkPi"}' -H "Content-Type: application/json" > http://localhost:8999/batches <http://localhost:8999/batches>; done”. > > In such case, I expect that the only one job (first) will be submitted > properly and others will be rejected with a response message “Rejected, too > many sessions are being created!”. This is what I actually have when I submit > them manually, one by one. Though, in the case given above, almost always all > 5 jobs will be submitted properly (sometimes only one will be rejected) > despite of the value of config option. > > Looks like, that code, when Livy checks the number of already created > sessions and launch new one, is not atomic: > https://github.com/apache/incubator-livy/blob/4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f/server/src/main/scala/org/apache/livy/server/SessionServlet.scala#L130 > > <https://github.com/apache/incubator-livy/blob/4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f/server/src/main/scala/org/apache/livy/server/SessionServlet.scala#L130> > > Also, the calculation of the total number of child processes in > "tooManySessions()" is not atomic operation. > > So, when I submit the jobs almost in the same moment (POST request takes only > about tens of milliseconds) then I can spawn more spark-submit processes > despite of maximum limitation. > > If it’s a bug then I will create a Jira and try to fix it. > > >
