[
https://issues.apache.org/jira/browse/SPARK-60023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60023:
-----------------------------------
Labels: pull-request-available (was: )
> FileFormatWriter throws NPE in setupJob with ManifestCommitter
> -----------------------------------------------------------------
>
> Key: SPARK-60023
> URL: https://issues.apache.org/jira/browse/SPARK-60023
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Anika Kelhanka
> Priority: Major
> Labels: pull-request-available
>
> SPARK-56919 moved committer.setupJob(job) ahead of
> materializeAdaptiveSparkPlan(plan) in FileFormatWriter.write so that a
> failure during AQE materialization triggers committer.abortJob(job) instead
> of leaving an un-setup job that deletes the output directory.
> However, job.getConfiguration.set("spark.sql.sources.writeJobUUID",
> description.uuid) remained inside the try block after
> materializeAdaptiveSparkPlan(plan).
> Committers that read spark.sql.sources.writeJobUUID during setupJob or
> abortJob—such as Hadoop's ManifestCommitter (the default committer for gs://
> and abfs:// since Hadoop 3.4)—fall back to jobContext.getJobID().toString()
> when spark.sql.sources.writeJobUUID is unset, throwing a NullPointerException
> because job.getJobID() is null for a Job that was never submitted:
> {code:java}
> java.lang.NullPointerException: Cannot invoke
> "org.apache.hadoop.mapreduce.JobID.toString()" because "jobId" is null
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestCommitterSupport.buildJobUUID(ManifestCommitterSupport.java:118)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterConfig.<init>(ManifestCommitterConfig.java:190)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.enterCommitter(ManifestCommitter.java:181)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.setupJob(ManifestCommitter.java:196)
> at
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:198)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:165)
> {code}
> *Proposed Fix:* Setting spark.sql.sources.writeJobUUID on
> job.getConfiguration before calling committer.setupJob(job) resolves the
> issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
