[ https://issues.apache.org/jira/browse/SPARK-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16171386#comment-16171386 ]
Steve Loughran commented on SPARK-20886: ---------------------------------------- Not, but related. This is handling the situation where the committer is a classic FileOutputCommitter (or subclass, like the parquet one), but when you ask for a working dir it returns null. SPARK-21549 looks like there's hard-coded expectations of a dest dir being set via the (private) config option used by FileOutputCommitter, and NPEing if its not there > HadoopMapReduceCommitProtocol to fail with message if > FileOutputCommitter.getWorkPath==null > ------------------------------------------------------------------------------------------- > > Key: SPARK-20886 > URL: https://issues.apache.org/jira/browse/SPARK-20886 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.3.0 > Reporter: Steve Loughran > Assignee: Steve Loughran > Priority: Trivial > Fix For: 2.3.0 > > > This is minor, and the root cause is my fault *elsewhere*, but its the patch > I used to track down the problem. > If {{HadoopMapReduceCommitProtocol}} has a {{FileOutputCommitter}} for > committing things, and *somehow* that's been configured with a > {{JobAttemptContext}}, not a {{TaskAttemptContext}}, then the committer NPEs. > A {{require()}} statement can validate the working path and so point the > blame at whoever's code is confused. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org