[ https://issues.apache.org/jira/browse/HADOOP-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218067#comment-15218067 ]
Steve Loughran commented on HADOOP-12979: ----------------------------------------- full stack {code} Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1457) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1445) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1444) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1444) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:809) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:809) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:809) ... Cause: java.io.IOException: ${hadoop.tmp.dir}/s3a not configured at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:269) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:349) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:421) at org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:198) at org.apache.hadoop.fs.s3a.S3AOutputStream.<init>(S3AOutputStream.java:91) at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:488) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:921) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:814) at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123) at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90) {code} > IOE in S3a: ${hadoop.tmp.dir}/s3a not configured > ------------------------------------------------- > > Key: HADOOP-12979 > URL: https://issues.apache.org/jira/browse/HADOOP-12979 > Project: Hadoop Common > Issue Type: Bug > Affects Versions: 2.8.0 > Reporter: Steve Loughran > > Running some spark s3a tests trigger an NPE in Hadoop <=2/7l IOE in 2.8 > saying "${hadoop.tmp.dir}/s3a not configured". > That's correct: there is no configuration option on the conf called > "${hadoop.tmp.dir}/s3a ". There may be one called {{hadoop.tmp.dir}}, however. > Essentially s3a is sending the wrong config option down, if it can't find > {{fs.s3a.buffer.dir}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)