Anika Kelhanka created SPARK-60023:
--------------------------------------
Summary: FileFormatWriter throws NPE in setupJob with
ManifestCommitter
Key: SPARK-60023
URL: https://issues.apache.org/jira/browse/SPARK-60023
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Anika Kelhanka
SPARK-56919 (commit 802b0b0d7e8c) moved committer.setupJob(job) ahead of
materializeAdaptiveSparkPlan(plan) in FileFormatWriter.write so that a failure
during AQE materialization triggers committer.abortJob(job) instead of leaving
an un-setup job that deletes the output directory.
However, job.getConfiguration.set("spark.sql.sources.writeJobUUID",
description.uuid) remained inside the try block after
materializeAdaptiveSparkPlan(plan).
Committers that read spark.sql.sources.writeJobUUID during setupJob or
abortJob—such as Hadoop's ManifestCommitter (the default committer for gs://
and abfs:// since Hadoop 3.4)—fall back to jobContext.getJobID().toString()
when spark.sql.sources.writeJobUUID is unset, throwing a NullPointerException
because job.getJobID() is null for a Job that was never submitted:
{code}
java.lang.NullPointerException: Cannot invoke
"org.apache.hadoop.mapreduce.JobID.toString()" because "jobId" is null
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestCommitterSupport.buildJobUUID(ManifestCommitterSupport.java:118)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterConfig.<init>(ManifestCommitterConfig.java:190)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.enterCommitter(ManifestCommitter.java:181)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.setupJob(ManifestCommitter.java:196)
at
org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:198)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:165)
{code}
Setting spark.sql.sources.writeJobUUID on job.getConfiguration before calling
committer.setupJob(job) resolves the issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
