[
https://issues.apache.org/jira/browse/PHOENIX-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinmay Kulkarni updated PHOENIX-5802:
--------------------------------------
Labels: phoenix-hardening (was: )
> Connection leaks in UPSERT SELECT/DELETE paths due to
> MutatingParallelIteratorFactory iterator not being closed
> ---------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-5802
> URL: https://issues.apache.org/jira/browse/PHOENIX-5802
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 5.0.0, 4.15.0
> Reporter: Chinmay Kulkarni
> Assignee: Chinmay Kulkarni
> Priority: Major
> Labels: phoenix-hardening
> Fix For: 5.1.0, 4.16.0
>
> Attachments: PHOENIX-5802-4.x-v1.patch, PHOENIX-5802-4.x-v2.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> During an UPSERT SELECT query or client-side DELETE, we clone the existing
> Phoenix connection and call
> [mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
> which eventually calls either
> [upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
> or
> [deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
> When getting iterators, if we encounter any exception, then on calling
> [close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
> we attempt to cancel any queued work, while [accumulating already-started
> task
> futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
>
> Later we attempt to block on getting the result (iterator) of
> already-started tasks and [close each
> iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
> Here however, if any of the upsert-select or client-side delete tasks had
> thrown an exception _E_, we wouldn't be able to close the underlying iterator
> (which in-turn would've [closed the cloned
> connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]).
> This manifests as [these
> logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
> {code:java}
> "Failed to execute task during cancel", <exception E stack trace>
> {code}
> This is leading to a connection leak. Because we use CQSI to create the
> Phoenix Connection here, it also gets accounted for during open Phoenix
> connection throttling, thus leaving the client application with less
> connections that can be opened.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
