[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815931#comment-16815931
]
Manish Maheshwari commented on IMPALA-7802:
-------------------------------------------
Thanks [~tarmstrong]. Adding some more background to the issues that I saw with
the session not closed -
* The client app was a custom scala akka app with connection pools been
maintained using C3P0ConnectionPool and something called skinnyORM that I am
not quite aware of. The system runs on a combination of 6 nodes.
* Somewhere during the testing, there were massive connection leaks from
multiple nodes. During testing, they ran the app multiple times and killed in
arbitrarily. With the connection leaks and app itself been killed, the handle
to the connection is lost.
* On the coordinator nodes, we could see sockets open and had to resort to way
to kill the socket as root. I would expect the remote OS to clean up the
sockets when the process died/killed but somehow that wasn't working.
Below is counts from analysis of `netstat -anlp` from all the coordinator nodes.
!https://mail.google.com/mail/u/0?ui=2&ik=65068d25cd&attid=0.1&permmsgid=msg-a:r-6585308661363146914&th=169faaff147cd389&view=fimg&sz=s0-l75-ft&attbid=ANGjdJ9VZRenh5lNIssq3pNJ9FN5rOSKmPEA3DHzexWK9U9Zlvl2_QxrjPPAFzJydvQ9ncD7FoqJH1qNN0Rjd-owE4NmHfg2wULieCGT6YmOVNz4GDphvQAC1TDAcgM&disp=emb&realattid=ii_ju7pf4nj0!
!image-2019-04-12-11-14-42-938.png!
So, my only options were to either restart the coordinator or go the impala
daemon webui and close all the sessions one by one manually.
If we want to keep the current behaviour ASIS, maybe the auto termination can
be put behind a flag that customers are explicitly set to true in cases of
application bugs like these.
> Implement support for closing idle sessions
> -------------------------------------------
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
> Affects Versions: Impala 3.0, Impala 2.12.0
> Reporter: Michael Ho
> Priority: Critical
> Labels: supportability
> Attachments: image-2019-04-12-11-14-42-938.png
>
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
