[
https://issues.apache.org/jira/browse/CALCITE-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987023#comment-16987023
]
Ruben Q L edited comment on CALCITE-3535 at 12/3/19 4:08 PM:
-------------------------------------------------------------
With some invaluable help from [~zabetak], it seems that we have finally
discovered the issue.
The root cause of the problem is that the nested loop join implementation in
EnumerableDefaults#nestedLoopJoin *does not close all its resources*.
Specifically, it uses some enumerators that are never closed. This means, when
running the tests, these un-closed enumerators hold some resources, e.g.
Connection objects, which are never closed, i.e. released, and that causes the
deadlock when trying to get a new Connection from the Connection pool in
further tests.
We can see now this issue because, due to the change in EnumerableJoinRule,
nested loop join algorithm is more used than before, e.g. an inner Join with
condition=true was executed previously via a HashJoin, and now with a
NestedLoopJoin. Since more nested loop joins are executed in the tests, more
un-closed resources are generated, until we eventually reach the problem.
To avoid mixing things, I will commit the fix in
EnumerableDefaults#nestedLoopJoin in a separate tickect. By chance, I happen to
have an open ticket which is somehow related to this topic: "CALCITE-3546 -
Improve EnumerableDefaults nested loop join". That ticket provides a new, more
optimized nested loop join algorithm, so the issue will probably be mitigated.
However, the old (and "resource-greedy") algorithm still remains (the new
optimized algorithm does not support right / full join); so I will commit the
fix to close resources in there.
Then, once CALCITE-3546 gets merged into master, I will rebase this PR and,
hopefully, the issue will be solved.
was (Author: rubenql):
With some invaluable help from [~zabetak], it seems that we have finally
discovered the issue.
The root cause of the problem is that the nested loop join implementation in
EnumerableDefaults#nestedLoopJoin *does not close all its resources*.
Specifically, it uses some enumerators that are never closed. This means, when
running the tests, these un-closed enumerators hold some resources, e.g.
Connection objects, which are never closed, i.e. released, and that causes the
deadlock when trying to get a new Connection from the Connection pool in
further tests.
We can see now this issue because, due to the change in EnumerableJoinRule,
nested loop join algorithm is more used than before, e.g. an inner Join with
condition=true was executed previously via a HashJoin, and now with a
NestedLoopJoin. Since more nested loop joins are executed in the tests, more
un-closed resources are generated, until we eventually reach the problem.
To avoid mixing things, I will commit the fix in
EnumerableDefaults#nestedLoopJoin in a separate tickect. By chance, I happen to
have an open ticket which is somehow related to this topic: "CALCITE-3546 -
Improve EnumerableDefaults nested loop join". That ticket provides a new, more
optimized nested loop join algorithm, so the issue will probably be mitigated.
However, the old (and "resource-greedy") algorithm still remains (the new
optimized algorithm does not support right / full join); so I will commit the
fix to close resources in there.
Then, once CALCITE-3546 gets merged into Trunk, I will rebase this PR and,
hopefully, the issue will be solved.
> EnumerableJoinRule: remove unnecessary Filter on top of INNER Join
> ------------------------------------------------------------------
>
> Key: CALCITE-3535
> URL: https://issues.apache.org/jira/browse/CALCITE-3535
> Project: Calcite
> Issue Type: Task
> Components: core
> Affects Versions: 1.21.0
> Reporter: Ruben Q L
> Assignee: Ruben Q L
> Priority: Minor
> Labels: pull-request-available
> Fix For: 1.22.0
>
> Attachments: threadDump.txt
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> With the implementation of CALCITE-2973, now EnumerableHashJoin supports all
> type of conditions (not just equi joins). However, there is still one
> [TODO|https://github.com/apache/calcite/blob/07c1efbb2b42a0e27f9d01968d357d9c1e17cd51/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableJoinRule.java#L89]
> in EnumerableHashJoinRule that, in case of an INNER Join, creates a Filter
> with the non-equi conditions on top of the EnumerableHashJoin (created only
> with the equi-conditions), this filter is not really needed, since now
> EnumerableHashJoinRule can support the full condition: equi and non-equi
> items.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
