update: hangs even when not writing to hdfs. I changed the code to avoid
saveAsTextFile() and instead do a forEachParitition and log the results.
This time it hangs at 96/100 tasks, but still hangs.
I changed the saveAsTextFile to:
stringIntegerJavaPairRDD.foreachPartition(p -> {
while (p.hasNext()) {
LOG.info("{}", p.next());
}
});
Thanks, David.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/saveAsTextFile-hangs-with-hdfs-tp12412p12419.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
