Daniel Becker has uploaded a new patch set (#3). (
http://gerrit.cloudera.org:8080/20953 )
Change subject: IMPALA-12695: Crash with UNION with complex types
......................................................................
IMPALA-12695: Crash with UNION with complex types
If we unnest an array coming from a UNION ALL, we read invalid memory
and in ASAN builds we crash.
Example:
with v as (select arr1 from complextypes_arrays
union all select arr1 from complextypes_arrays)
select am.item from v, v.arr1 am;
The problem seems to be that in the item tuple of the collections, the
item slots are present twice. This is because both the inline view
analyzer and the main analyzer add slots with the same path to the
tuple. This is possible because
- the target tuple is determined based on the path via
Path.getRootDesc(), so it will be the same both in the inline view
and in the main scope
AND
- the inline view analyzer and the main one do not share
'slotPathMap_', so the analyzer cannot recognise that a slot for the
path has already been added.
This commit solves the problem by checking the target tuple whether a
slot with the same path already exists in it, and if it does, we reuse
that slot. Note, however, that when Analyzer.registerSlotRef() is called
with 'duplicateIfCollections=true', a separate slot is added for
collections which should not be reused. This commit adds a set,
'duplicateCollectionSlots', in Analyzer.GlobalState to keep track of
such collection slots, and these slots are never reused.
Note that there is another bug, IMPALA-12753, that a predicate on the
collection item in the above query is only enforced on the first child
of the union. Therefore this commit disallows placing a predicate on a
collection item when the unnested collection comes from a union.
Testing:
- added test queries in nested-array-in-select-list.test,
nested-map-in-select-list.test, zipping-unnest-in-from-clause.test
and zipping-unnest-in-select-list.test
Change-Id: I340adc50e6d7cda6f59dacd7a46b6adc31635d46
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M
testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
M
testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test
M
testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-from-clause.test
M
testdata/workloads/functional-query/queries/QueryTest/zipping-unnest-in-select-list.test
7 files changed, 373 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/20953/3
--
To view, visit http://gerrit.cloudera.org:8080/20953
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I340adc50e6d7cda6f59dacd7a46b6adc31635d46
Gerrit-Change-Number: 20953
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Daniel Becker <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
