[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669100#comment-16669100
]
Jason Dere commented on HIVE-20839:
-----------------------------------
Issue seems to be that there are several column expressions that end up being
converted to constant value 'SSS' during constant folding. Some of these
constant columns end up being pruned from the query, and others might be join
keys which are required in the query. The problem was that
ExprNodeDescUtils.resolveJoinKeysAsRSColumns() (which is used to setup the join
key expressions in the case of dynamically partitioned hashjoin) was trying to
resolve the join key expression for the 'SSS' column and picked the wrong
column - it was also constant 'SSS', but it was actually a different column
that ended up getting pruned.
The ExprNodeConstDesc actually has a getFoldedFromCol() field that shows what
column this constant expression has been folded from, I think this can be used
to help resolve to the correct expression.
> "Cannot find field" error during dynamically partitioned hash join
> ------------------------------------------------------------------
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Jason Dere
> Assignee: Jason Dere
> Priority: Major
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
