[ https://issues.apache.org/jira/browse/SPARK-11128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-11128. ------------------------------- Resolution: Not A Problem Not a problem with Spark, that is. > strange NPE when writing in non-existing S3 bucket > -------------------------------------------------- > > Key: SPARK-11128 > URL: https://issues.apache.org/jira/browse/SPARK-11128 > Project: Spark > Issue Type: Bug > Components: Input/Output > Affects Versions: 1.5.1 > Reporter: mathieu despriee > Priority: Minor > > For the record, as it's relatively minor, and related to s3n (not tested with > s3a). > By mistake, we tried writing a parquet dataframe to a non-existing s3 bucket, > with a simple df.write.parquet(s3path). > We got a NPE (see stack trace below), which is very misleading. > java.lang.NullPointerException > at > org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:433) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:73) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57) > at > org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:69) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:140) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:138) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) > at > org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:138) > at > org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:933) > at > org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:933) > at > org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:197) > at > org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:146) > at > org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:137) > at > org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:304) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org