Ryu Kobayashi created HIVE-29702:
------------------------------------
Summary: Reduce-side merge join can produce NULL values or fail
with IllegalStateException under Tez container reuse
Key: HIVE-29702
URL: https://issues.apache.org/jira/browse/HIVE-29702
Project: Hive
Issue Type: Bug
Reporter: Ryu Kobayashi
h2. Summary
Reduce-side merge join (a DummyStore + CommonMergeJoinOperator pipeline used
for SMB / merge-join fallback) can misbehave when Tez reuses a container across
multiple tasks of the same reducer vertex. Depending on timing, this manifests
as either:
(a) a task failure:
{code}
java.lang.IllegalStateException: Was expecting dummy store operator but found:
FS[...]
{code}
(b) or, if no exception happens to be thrown, silent NULL values in the columns
produced by the join, with no error reported to the user.
h2. Root cause 1: stale operator-tree wiring on container reuse (crash)
Hive's per-query {{ObjectCache}} (keyed by vertex name) can hand a task the
same already-initialized {{DummyStoreOperator}} / {{CommonMergeJoinOperator}}
instances that a previous task (sharing the same JVM/container) already used.
{{TezDummyStoreOperator#closeOp}} does not remove itself from its child's
{{parentOperators}}, nor clear its own {{childOperators}}, when closing. When
the cached operator instances are reused by a subsequent task, the
DummyStoreOperator still has a stale child reference left over from the
previous task's execution. This causes
{{ReduceRecordProcessor#getJoinParentOp}} to walk past the DummyStoreOperator
into the wrong subtree, eventually reaching a non-DummyStoreOperator leaf and
throwing {{IllegalStateException}}.
Additionally, {{ReduceRecordProcessor#close()}} closes the main {{reducer}}
before the {{mergeWorkList}} (DummyStore chain). Since
{{DummyStoreOperator#closeOp}} is what removes the stale parent reference from
the MergeJoin side, closing the main reducer first means
{{CommonMergeJoinOperator#allInitializedParentsAreClosed()}} cannot detect all
parents as closed in the correct order to flush the final join group.
h2. Root cause 2: reducer count / partitioning mismatch (NULL values)
(to be filled in once the corresponding fix is finalized)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
