[
https://issues.apache.org/jira/browse/HIVE-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shohei Okumiya resolved HIVE-23670.
-----------------------------------
Fix Version/s: 4.1.0
Resolution: Won't Fix
We have discontinued Hive on Spark and EoLed Hive 3. HIVE-26134
> hive on spark may encounter error when use map join
> ---------------------------------------------------
>
> Key: HIVE-23670
> URL: https://issues.apache.org/jira/browse/HIVE-23670
> Project: Hive
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: zhoukang
> Priority: Major
> Fix For: 4.1.0
>
>
> when using spark as hive engine, some task may failed with exception below.
> The cause is that when we load table, we may issued with:
> {code:java}
> if (tableContainer == null) {
> tableContainer = useOptimizedContainer ?
> new MapJoinBytesTableContainer(hconf, valueContext, -1, 0) :
> create(name, metaData);
> }
> {code}
> then cast exception will issued with code below:
> {code:java}
> @Override
> public JoinUtil.JoinResult setFromOther(ReusableGetAdaptor other) {
> assert other instanceof GetAdaptor;
> GetAdaptor other2 = (GetAdaptor)other;
> this.key = other2.key;
> this.isFirstKey = other2.isFirstKey;
> this.currentValue = mHash.get(key);
> if (this.currentValue == null) {
> return JoinUtil.JoinResult.NOMATCH;
> }
> else {
> return JoinUtil.JoinResult.MATCH;
> }
> }
> {code}
> and it is because spark HashTableLoader is based on the files on the given
> path.
> Then it will generate HashMapWrapper:
> {code:java}
> // Get an empty container when the small table is empty.
> private static MapJoinTableContainer
> getDefaultEmptyContainer(MapJoinObjectSerDeContext keyCtx,
> MapJoinObjectSerDeContext valCtx) throws SerDeException {
> MapJoinTableContainer container = new HashMapWrapper();
> container.setSerde(keyCtx, valCtx);
> container.seal();
> return container;
> }
> {code}
> {code:java}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected
> exception from MapJoinOperator :
> org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$GetAdaptor
> cannot be cast to
> org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper$GetAdaptor
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:465)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:148)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:547)
> ... 19 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
