[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222074#comment-17222074
]
Miklos Szurap commented on IMPALA-7802:
---
In contrast to the above commit message the new setting's name is
{{idle_client_poll_period_s}} with default value of 30 seconds and comments
from the source code:
bq. The poll period, in seconds, after no activity from an Impala client which
an Impala service thread (beeswax and HS2) wakes up to check if the connection
should be closed. If --idle_session_timeout is also set, a client connection
will be closed if all the sessions associated with it have become idle. Set
this to 0 to disable the polling behavior and clients' connection will remain
opened until they are explicitly closed.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Critical
> Labels: supportability
> Fix For: Impala 3.3.0
>
> Attachments: image-2019-04-12-11-14-42-938.png
>
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869177#comment-16869177
]
ASF subversion and git services commented on IMPALA-7802:
-
Commit 67cd55e0445d037d834ecd120d08b0948b064491 in impala's branch
refs/heads/master from Michael Ho
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=67cd55e ]
IMPALA-7802: Close connections of idle client sessions
Previously, if idle session timeout is set either via
startup flag or query options, a client session will expire
after that set period of inactivity. However, the network
connection and the service thread of an expired session will
still be around until the session is closed by the client.
This is highly undesirable as these idle sessions still count
towards the quota bound by --fe_esrvice_threads, so if the
total number of sessions (including the idle ones) reaches
that upper bound, all incoming new session will block until
some of the existing sessions exit. There is no time bound on
when those expired sessions will be closed. In some sense,
leaving many idle sessions opened is a denial-of-service attack
on Impala.
This change implements support for closing expired client sessions.
In particular, a new flag --idle_client_poll_time_s is added to
specify a time interval in seconds of client's inactivity which
will cause an idle service thread of a client connection to wake up
and check if all sessions associated with the connection are idle.
If so, the connection will be closed. This allows the service threads
to be freed up without waiting for client to close the connections.
Testing done:
- core build
- new targeted test which verifies the connections of expired sessions
are closed.
- verified the flags function as expected in a secure cluster with Kerberos +
SSL
Change-Id: I97c4fb8e1b741add273f8a913fb0967303683e38
Reviewed-on: http://gerrit.cloudera.org:8080/13607
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Critical
> Labels: supportability
> Attachments: image-2019-04-12-11-14-42-938.png
>
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819545#comment-16819545
]
Michael Ho commented on IMPALA-7802:
bq. On the coordinator nodes, we could see sockets open and had to resort to
way to kill the socket as root. I would expect the remote OS to clean up the
sockets when the process died/killed but somehow that wasn't working.
I am pretty sure the connections will be closed on the coordinator side and the
session will be closed if the remote client is terminated.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Priority: Critical
> Labels: supportability
> Attachments: image-2019-04-12-11-14-42-938.png
>
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815931#comment-16815931
]
Manish Maheshwari commented on IMPALA-7802:
---
Thanks [~tarmstrong]. Adding some more background to the issues that I saw with
the session not closed -
* The client app was a custom scala akka app with connection pools been
maintained using C3P0ConnectionPool and something called skinnyORM that I am
not quite aware of. The system runs on a combination of 6 nodes.
* Somewhere during the testing, there were massive connection leaks from
multiple nodes. During testing, they ran the app multiple times and killed in
arbitrarily. With the connection leaks and app itself been killed, the handle
to the connection is lost.
* On the coordinator nodes, we could see sockets open and had to resort to way
to kill the socket as root. I would expect the remote OS to clean up the
sockets when the process died/killed but somehow that wasn't working.
Below is counts from analysis of `netstat -anlp` from all the coordinator nodes.
!https://mail.google.com/mail/u/0?ui=2=65068d25cd=0.1=msg-a:r-6585308661363146914=169faaff147cd389=fimg=s0-l75-ft=ANGjdJ9VZRenh5lNIssq3pNJ9FN5rOSKmPEA3DHzexWK9U9Zlvl2_QxrjPPAFzJydvQ9ncD7FoqJH1qNN0Rjd-owE4NmHfg2wULieCGT6YmOVNz4GDphvQAC1TDAcgM=emb=ii_ju7pf4nj0!
!image-2019-04-12-11-14-42-938.png!
So, my only options were to either restart the coordinator or go the impala
daemon webui and close all the sessions one by one manually.
If we want to keep the current behaviour ASIS, maybe the auto termination can
be put behind a flag that customers are explicitly set to true in cases of
application bugs like these.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Priority: Critical
> Labels: supportability
> Attachments: image-2019-04-12-11-14-42-938.png
>
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812058#comment-16812058
]
Tim Armstrong commented on IMPALA-7802:
---
[~manishm@cloudera.com_impala_1546] that's not quite true - see the previous
comments starting with Zoram's first one.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812035#comment-16812035
]
Manish Maheshwari commented on IMPALA-7802:
---
A badly designed impala client did not close the connections and I had 18,000
connections pile up on 4 coordinators causing an impala service outage. There
seems to be no benefit of keeping this session open as the doc clearly says
`The session remains open, but the only operation you can perform is to close
it.`.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797101#comment-16797101
]
Balazs Jeszenszky commented on IMPALA-7802:
---
I agree with Tim that closing sessions is the lesser evil compared to the
current state of affairs. I also think this is arguably a bug, it'd be good to
be in a slightly better state sooner rather than later.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739902#comment-16739902
]
Zoram Thanga commented on IMPALA-7802:
--
Apologies, [~tarmstrong] for not responding sooner. I hadn't noticed your
comment until today.
It would be nice to have something like
[TThreadedSelectorServer|https://github.com/apache/thrift/blob/master/lib/java/src/org/apache/thrift/server/TThreadedSelectorServer.java],
but that would likely require too much work.
The best we can do might be to find a way to forcibly close the socket once all
sessions in the connection have timed out, as you suggest above. The tricky
thing we need to do is to change TAcceptQueueServer::Task::run()'s call to
TTransport->peek() so that it can be 'woken up' from the ImpalaServer layer.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729165#comment-16729165
]
Tim Armstrong commented on IMPALA-7802:
---
If that turns out to be too complex I could maybe be convinced that just
closing the sockets is strictly better than the current state of things.
Generally clients that hold connections open a long time or recycle connections
need to be able to test connections on their side and reopen if they're stale.
I tested impala-shell and Hue and they both seem to handle closed connections
OK by automatically reconnecting.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718239#comment-16718239
]
Zoram Thanga commented on IMPALA-7802:
--
Thanks for the comment, [~tarmstrong]. You brought up a good point regarding
what the client experience has to be, when the session has expired. I think we
will have to maintain the current behavior, but without tying up FE service
threads. That is, expired sessions should not consume any resources on the
Impala server besides the session states.
I am thinking about moving expired sessions to a separate list (a death row)
where they remain until explicitly cancelled and/or closed by the client.
Perhaps we can have a single thread handle cancellation and closing of expired
sessions.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710806#comment-16710806
]
Tim Armstrong commented on IMPALA-7802:
---
[~zoram] The thing is does do is report a meaningful error message when the
user comes back and tries to use the session - i.e. "Client session expired due
to more than...".
E.g. a quick google revealed this forum thread where the error message pointed
the user in the right direction,
https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Query-blah-expired-due-to-client-inactivity-timeout-is-10m/m-p/66842.
Maybe the error reporting isn't worth the other hassles though - or maybe we
just need to set clearer expectations for client behaviour - that they need to
handle sessions being terminated in this way.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7802) Implement support for closing idle sessions
[
https://issues.apache.org/jira/browse/IMPALA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710704#comment-16710704
]
Zoram Thanga commented on IMPALA-7802:
--
The documentation states that:
{quote}
Once a session is expired, you cannot issue any new query requests to it. The
session remains open, but the only operation you can perform is to close it.
{quote}
This basically says that an expired session serves no useful purpose to any one
- not to Impala as it consumes an fe_service_thread, and not to the client
because the only operation allowed on it is to close it.
I would like to change the session expiry code to always force-close expired
sessions from the server side by calling ImpalaServer::CloseSessionInternal()
or a modified version of it.
> Implement support for closing idle sessions
> ---
>
> Key: IMPALA-7802
> URL: https://issues.apache.org/jira/browse/IMPALA-7802
> Project: IMPALA
> Issue Type: Improvement
> Components: Clients
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Assignee: Zoram Thanga
>Priority: Critical
> Labels: supportability
>
> Currently, the query option {{idle_session_timeout}} specifies a timeout in
> seconds after which all running queries of that idle session will be
> cancelled and no new queries can be issued to it. However, the idle session
> will remain open and it needs to be closed explicitly. Please see the
> [documentation|https://www.cloudera.com/documentation/enterprise/latest/topics/impala_idle_session_timeout.html]
> for details.
> This behavior may be undesirable as each session still consumes an Impala
> frontend service thread. The number of frontend service threads is bound by
> the flag {{fe_service_threads}}. So, in a multi-tenant environment, an Impala
> server can have a lot of idle sessions but they still consume against the
> quota of {{fe_service_threads}}. If the number of sessions established
> reaches {{fe_service_threads}}, all new session creations will block until
> some of the existing sessions exit. There may be no time bound on when these
> zombie idle sessions will be closed and it's at the mercy of the client
> implementation to close them. In some sense, leaving many idle sessions open
> is a way to launch a denial of service attack on Impala.
> To fix this situation, we should have an option to forcefully close a session
> when it's considered idle so it won't unnecessarily consume the limited
> number of frontend service threads. cc'ing [~zoram]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
