Michael Smith created IMPALA-15437:
--------------------------------------
Summary: Dataload can fail on Hive Iceberg
Key: IMPALA-15437
URL: https://issues.apache.org/jira/browse/IMPALA-15437
Project: IMPALA
Issue Type: Task
Components: Test
Affects Versions: Impala 5.0.0
Reporter: Michael Smith
Assignee: Fang-Yu Rao
On Ubuntu 22.04 (x86_64), I started running into an error with dataload on the
master branch that hadn't been there before
{code:java}
ERROR : FAILED: Execution Error, return code 40000 from
org.apache.hadoop.hive.ql.exec.MoveTask. Error committing job:
job_17902705519450_0022 for table: functional_parquet.iceberg_avro_format
INFO : Completed executing
command(queryId=michael_20260924103245_79db4f29-1da2-4fc6-9adf-63bc6565fcf7);
Time taken: 29.937 seconds
INFO : OK
Error: Error while compiling statement: FAILED: Execution Error, return code
40000 from org.apache.hadoop.hive.ql.exec.MoveTask. Error committing job:
job_17902705519450_0022 for table: functional_parquet.iceberg_avro_format;
Query ID: michael_20260924103245_79db4f29-1da2-4fc6-9adf-63bc6565fcf7
(state=08S01,code=40000)
java.sql.SQLException: Error while compiling statement: FAILED: Execution
Error, return code 40000 from org.apache.hadoop.hive.ql.exec.MoveTask. Error
committing job: job_17902705519450_0022 for table:
functional_parquet.iceberg_avro_format; Query ID:
michael_20260924103245_79db4f29-1da2-4fc6-9adf-63bc6565fcf7
at
org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:472)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:334)
at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1008)
at org.apache.hive.beeline.Commands.execute(Commands.java:1218)
at org.apache.hive.beeline.Commands.sql(Commands.java:1147)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1498)
at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1356)
at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:1330)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1128)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1083)
at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:541)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:523)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.hadoop.util.RunJar.run(RunJar.java:333)
at org.apache.hadoop.util.RunJar.main(RunJar.java:254) {code}
I suspect I redownloaded the CDP Hive tarball and the jar ordering on the
filesystem changed. It seems to have exposed mismatched implementations of
{{org.apache.iceberg.BaseFile}} in {{hive-iceberg-handler-\*.jar}} /
{{hive-iceberg-shading-\*.jar}} and {{patched-iceberg-core-*.jar}}.
I was able to work around it with a patch to {{run-hive-server.sh}}, but it
should really be fixed in the CDP Hive build
{code}
# Hive's lib/ jars are added in unsorted 'find' order, and patched-iceberg-core
has an
# incompatible org.apache.iceberg.BaseFile (serialVersionUID) from the one in
# hive-iceberg-handler that Tez tasks use to serialize commit files. Put the
handler
# first via TEZ_CONF_DIR, the only entry bin/hive places ahead of lib/. The
"/./" keeps
# Hadoop's classpath de-duplication from dropping it in favor of the later lib/
entry.
for jar in "${HIVE_HOME}"/lib/hive-iceberg-handler-*.jar; do
if [[ -f "$jar" ]]; then
export
TEZ_CONF_DIR="${TEZ_CONF_DIR:-/etc/tez/conf}:${HIVE_HOME}/lib/./$(basename
"$jar")"
fi
done
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
