Hi,
I want to write my RDDs to multiples files based on a key value. So, i
used groupByKey and iterate over partitions. Here is a the code :
rdd.map(f => (f.substring(0,4), f)).groupByKey().foreachPartition(iterator
=>
iterator.map { case (key, values) =>
val fs: FileSystem = FileSystem.get(new Configuration())
val outputFile = fs.create(new Path("/my_path/" + key + ".txt"))
values.foreach { x => outputFile.write(x.getBytes()) }
outputFile.close()
}
)
I don't see any error on the spark-shell LOG. But, no file is written.
Does any body know where i missed something ?
thank's
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/foreachPartition-write-to-multiple-files-tp15925.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
