For the dead/killed session, could you check the yarn app logs ? Hugo Herlanin <hugo.herla...@lendico.com.br> 于2019年7月4日周四 下午9:41写道：
> > Hey, user mail is not working out! > > I am having some problems with livy setup. My use case is as follows: I > use a DAG in airflow (1.10) to create a cluster in EMR (5.24.1, one master > is m4.large and two nodes in m5a.xlarge), and when it is ready, this dag > sends 5 to 7 simultaneous requests to Livy. I think I'm not messing with > the Livy settings, I just set livy.spark.deploy-mode = client and > livy.repl.enable-hive-context = true. > > The problem is that from these ~ 5 to 7 sessions, just one or two opens > (goes to 'idle') and all others go straight to 'dead' or 'killed', in logs > Yarn returns that the sessions were killed by 'livy' user. I tried to > tinker with all possible timeout settings, but this is still happening. If > I send more than ~10 simultaneous requests, livy responds with 500, and if > I continue sending requests, the server freezes. This happens even if EMR > has enough resources available. > > I know the cluster is able to handle that many questions because it works > when I open them via a loop with an interval of 15 seconds or more, but it > feels like livy should be able to deal with that many requests > simultaneously. It seems strange that I should need to manage the queue in > such a way for an API of a distributed system. > > Do you have any clue about where I might be doing wrong? Is there any > known limitation that I'm unaware of? > > Best, > > Hugo Herlanin > > -- Best Regards Jeff Zhang
