[
https://issues.apache.org/jira/browse/SPARK-60023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anika Kelhanka updated SPARK-60023:
-----------------------------------
Description:
SPARK-56919 moved committer.setupJob(job) ahead of
materializeAdaptiveSparkPlan(plan) in FileFormatWriter.write so that a failure
during AQE materialization triggers committer.abortJob(job) instead of leaving
an un-setup job that deletes the output directory.
However, job.getConfiguration.set("spark.sql.sources.writeJobUUID",
description.uuid) remained inside the try block after
materializeAdaptiveSparkPlan(plan).
Committers that read spark.sql.sources.writeJobUUID during setupJob or
abortJob—such as Hadoop's ManifestCommitter (the default committer for gs://
and abfs:// since Hadoop 3.4)—fall back to jobContext.getJobID().toString()
when spark.sql.sources.writeJobUUID is unset, throwing a NullPointerException
because job.getJobID() is null for a Job that was never submitted:
{code:java}
java.lang.NullPointerException: Cannot invoke
"org.apache.hadoop.mapreduce.JobID.toString()" because "jobId" is null
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestCommitterSupport.buildJobUUID(ManifestCommitterSupport.java:118)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterConfig.<init>(ManifestCommitterConfig.java:190)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.enterCommitter(ManifestCommitter.java:181)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.setupJob(ManifestCommitter.java:196)
at
org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:198)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:165)
{code}
*Proposed Fix:* Setting spark.sql.sources.writeJobUUID on job.getConfiguration
before calling committer.setupJob(job) resolves the issue.
was:
SPARK-56919 moved `committer.setupJob(job)` ahead of
`materializeAdaptiveSparkPlan(plan)` in `FileFormatWriter.write` so that a
failure during AQE materialization triggers `committer.abortJob(job)` instead
of leaving an un-setup job that deletes the output directory.
However, `job.getConfiguration.set("spark.sql.sources.writeJobUUID",
description.uuid)` remained inside the try block after
`materializeAdaptiveSparkPlan(plan)`.
Committers that read `spark.sql.sources.writeJobUUID` during `setupJob` or
`abortJob`—such as Hadoop's ManifestCommitter (the default committer for
`gs://` and `abfs://` since Hadoop 3.4)—fall back to
`jobContext.getJobID().toString()` when `spark.sql.sources.writeJobUUID` is
unset, throwing a `NullPointerException` because `job.getJobID()` is null for a
Job that was never submitted:
{code:java}
java.lang.NullPointerException: Cannot invoke
"org.apache.hadoop.mapreduce.JobID.toString()" because "jobId" is null
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestCommitterSupport.buildJobUUID(ManifestCommitterSupport.java:118)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterConfig.<init>(ManifestCommitterConfig.java:190)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.enterCommitter(ManifestCommitter.java:181)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.setupJob(ManifestCommitter.java:196)
at
org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:198)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:165)
{code}
Setting spark.sql.sources.writeJobUUID on job.getConfiguration before calling
committer.setupJob(job) resolves the issue.
> FileFormatWriter throws NPE in setupJob with ManifestCommitter
> -----------------------------------------------------------------
>
> Key: SPARK-60023
> URL: https://issues.apache.org/jira/browse/SPARK-60023
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Anika Kelhanka
> Priority: Major
>
> SPARK-56919 moved committer.setupJob(job) ahead of
> materializeAdaptiveSparkPlan(plan) in FileFormatWriter.write so that a
> failure during AQE materialization triggers committer.abortJob(job) instead
> of leaving an un-setup job that deletes the output directory.
> However, job.getConfiguration.set("spark.sql.sources.writeJobUUID",
> description.uuid) remained inside the try block after
> materializeAdaptiveSparkPlan(plan).
> Committers that read spark.sql.sources.writeJobUUID during setupJob or
> abortJob—such as Hadoop's ManifestCommitter (the default committer for gs://
> and abfs:// since Hadoop 3.4)—fall back to jobContext.getJobID().toString()
> when spark.sql.sources.writeJobUUID is unset, throwing a NullPointerException
> because job.getJobID() is null for a Job that was never submitted:
> {code:java}
> java.lang.NullPointerException: Cannot invoke
> "org.apache.hadoop.mapreduce.JobID.toString()" because "jobId" is null
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestCommitterSupport.buildJobUUID(ManifestCommitterSupport.java:118)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterConfig.<init>(ManifestCommitterConfig.java:190)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.enterCommitter(ManifestCommitter.java:181)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.setupJob(ManifestCommitter.java:196)
> at
> org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:198)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:165)
> {code}
> *Proposed Fix:* Setting spark.sql.sources.writeJobUUID on
> job.getConfiguration before calling committer.setupJob(job) resolves the
> issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
