Csaba Ringhofer created IMPALA-15241:
----------------------------------------
Summary: Use file IDs instead of paths to get INPUT__FILE__NAME
Key: IMPALA-15241
URL: https://issues.apache.org/jira/browse/IMPALA-15241
Project: IMPALA
Issue Type: Improvement
Components: Backend, Frontend
Reporter: Csaba Ringhofer
See discussion in https://gerrit.cloudera.org/#/c/24572/5//COMMIT_MSG@9 for
motivation.
Currently INPUT__FILE__NAME is used in two ways:
1. common: anti join key in IcebergDeleteJoinNode
2. rare: any other use case, e.g. return in select list
INPUT__FILE__NAME is a StringValue where the actual string is deduplicated, but
may get duplicated when going through exchanges and some other operators.
IMPALA-15171
improves this by nulling the string if it is no longer used after the delete
node, but I consider this a bit hacky.
A longer term solution could be to have a global id->path map and store only an
int32 in tuples and use it to look up the name if it is actually needed. A
similar global map already exists in for directed distribution mode of delete
nodes.
The same lookup mechanism could be used for other things too, for example
storing partition columns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
