[
https://issues.apache.org/jira/browse/IMPALA-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17625200#comment-17625200
]
ASF subversion and git services commented on IMPALA-10838:
----------------------------------------------------------
Commit ca9b08372556ac2010348b7c978c350e538b7b2c in impala's branch
refs/heads/master from Daniel Becker
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ca9b08372 ]
IMPALA-11685: Slot memory sharing between struct and field not working
if the field is also a struct
IMPALA-10838 introduced that if a struct and one of its fields are both
present in the select list, no extra slot is generated in the row for
the struct field but the memory of the struct is reused, i.e. the row
size is the same as when only the struct is queried. It works when the
struct field is a primitive type:
explain select id, outer_struct from
functional_orc_def.complextypes_nested_structs;
row-size=64B
explain select id, outer_struct, outer_struct.str from
functional_orc_def.complextypes_nested_structs;
row-size=64B
However, it does not if the child is itself a struct:
explain select id, outer_struct, outer_struct.inner_struct3 from
functional_orc_def.complextypes_nested_structs;
row-size=80B
This is because struct slot descriptors are registered before others so
that it is easier to reuse the slot memory of the struct fields, but
struct slot descriptors among themselves are sorted in the wrong order
(see
https://github.com/apache/impala/blob/c12ac6c27b2df1eae693b44c157d65499f491d21/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java#L340).
Also, because struct children are also inserted into the Analyzer's
slotPathMap_ when the struct is registered, there is no need for the
mechanism that tries to find ancestors of fields to be able to share
their slot memory - they can be retrieved from the slotPathMap_. This
change deletes the code that dealt with that.
Testing:
- PlannerTest#testStructFieldSlotSharedWithStruct has been updated to
include queries where the struct field is also a struct; the elements
in the select list are permutated to make sure the order in which
they are listed does not matter.
Change-Id: I6d4dee3941fb2d285fbd3836ea5712c859db8848
Reviewed-on: http://gerrit.cloudera.org:8080/19167
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Error when struct returned from WITH() and used in an ORDER BY
> --------------------------------------------------------------
>
> Key: IMPALA-10838
> URL: https://issues.apache.org/jira/browse/IMPALA-10838
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Gabor Kaszab
> Assignee: Daniel Becker
> Priority: Major
> Labels: complextype, nested_types
>
> {code:java}
> with sub as (
> select id, small_struct
> from functional_orc_def.complextypes_structs
> where length(small_struct.s) > 5)
> select sub.id, sub.small_struct from sub order by sub.small_struct.i desc;
> {code}
> The above query results an error when trying to run SlotRef.toThrift()
> {code:java}
> ERROR: IllegalStateException: Illegal reference to non-materialized tuple:
> debugname=InlineViewRef sub alias=sub tid=2
> {code}
> If I rewrite the query a bit to return the member of the struct from the
> inline view (WITH()) and use this in the ORDER by then the query succeeds as
> expected:
> {code:java}
> with sub as (
> select id, small_struct, small_struct.i as si
> from functional_orc_def.complextypes_structs where small_struct.i > 19200)
> select sub.id, sub.small_struct from sub order by sub.si desc;
> {code}
> In SortNode.toThrift() I checked what the sort exprs and the resolved tuple
> exprs are and I see a difference that could be the cause.
> In the problematic case:
> {code:java}
> - sort exprs in SortNode:
> SlotRef{label=small_struct.i, type=INT, id=15}
> - resolved exprs in SortNode:
> SlotRef{label=id, path=id, type=INT, id=0}
> SlotRef{label=small_struct, path=small_struct, type=STRUCT<i:INT,s:STRING>,
> id=1}
> *SlotRef{label=sub.small_struct.i, path=sub.small_struct.i, type=INT,
> id=10}*
> {code}
> In the successful case:
> {code:java}
> - sort exprs in SortNode:
> SlotRef{label=si, type=INT, id=14}
> - resolved exprs in SortNode:
> SlotRef{label=id, path=id, type=INT, id=0}
> SlotRef{label=small_struct, path=small_struct, type=STRUCT<i:INT,s:STRING>,
> id=1}
> *SlotRef{label=small_struct.i, path=small_struct.i, type=INT, id=4}*
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
