[
https://issues.apache.org/jira/browse/PHOENIX-5812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17080049#comment-17080049
]
Mingliang Liu commented on PHOENIX-5812:
----------------------------------------
[~elserj] and [~jisaac] Yes the context I have is pretty lightweight and it's
all about PConnection auto-closing. I think leaked PConnection that is not
closed could cause problems in two ways:
# new connection in CQSI is not allowed if the throttle is enabled for and the
max capacity is reached, defined by
{{phoenix.client.connection.max.allowed.connections}}
# the CQSI will have to maintain a very long connection queue with invalid
items, and the operations over it are very slow.
The reason is, PConnection if not closed, for 1) it will not decrease the
connection count, for 2) it will not get removed from the queue by the renew
lease task. For the two problems I think 1) is observable and user is affected
while 2) is more a performance problem and not very critical since the
reference to PConnection is weak in CQSI. [~dbwong] and [~ckulkarni] can
clarify my understanding.
To solve this problem, those idle PConnections can be auto-closed after some
interval since last access. Closing a PConnection is not necessarily closing a
CQSI. So I think this solution is orthogonal to CQSI clean up. Specially, for
CQSI/HConnection auto-closing, I see the current
{{PhoenixDriver::connectionQueryServicesCache}} evicts the CQSI if it is not
accessed for some time (default 1 day), defined by
{{phoenix.client.connection.max.duration}}. On removal, it closes this CQSI and
hence underlying HConnection, see
[here|https://github.com/apache/phoenix/blob/950331a1e5f8ac16e4b49ed857fe78b2ab9dc6fb/phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDriver.java#L163].
This however may still have the same problem as what we proposed here: user
may get connection closed exception when her PConnection is closed off-band
this way.
It's still possible to make PConnection cleaning up significantly less user
facing. I like this idea a lot. One potential solution in my mind is that:
- we *softly* close those PConnection when they are doomed idle and keep them
in separate connection queue in CQSI
- decrease the connection count so we do not consider them when throttling
- auto-open those PConnection when they become active again
- after a hard lifecycle setting (say, 1 day), those idle connections are
*hard* closed and removed.
Is that viable? I hope there is better way because I guess my this idea adds
much complexity to the implementation. Not sure if we can address that later in
another JIRA.
Thanks,
> Automatically Close "Idle" Long Open Connections
> ------------------------------------------------
>
> Key: PHOENIX-5812
> URL: https://issues.apache.org/jira/browse/PHOENIX-5812
> Project: Phoenix
> Issue Type: Improvement
> Reporter: Daniel Wong
> Priority: Major
>
> As Phoenix may keep a maximum default number of connections. Badly
> performing client calls or internal errors (See PHOENIX-5802). Can cause
> total available connections to go to 0. Proposing a client connection
> monitor with a connection reaper like task to reap idle connections.
> Definition of "Idle"
> Simple may be simple time based say if a connection has been open for
> configurable amount of minutes simply close.
> More complicated solution may be keeping track of last interaction time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
