Ongen:
I don't know why your process is hanging, sorry. But I do know that the
way saveAsTextFile works is that you give it a path to a directory, not a
file. The "file" is saved in multiple parts, corresponding to the
partitions. (part-00000, part-00001 etc.)
(Presumably it does this because it allows each partition to be saved on
the local disk, to minimize network traffic. It's how Hadoop works, too.)
On Mon, Mar 24, 2014 at 5:00 PM, Ognen Duzlevski
<og...@nengoiksvelzud.com>wrote:
> Is someRDD.saveAsTextFile("hdfs://ip:port/path/final_filename.txt")
> supposed to work? Meaning, can I save files to the HDFS fs this way?
>
> I tried:
>
> val r = sc.parallelize(List(1,2,3,4,5,6,7,8))
> r.saveAsTextFile("hdfs://ip:port/path/file.txt")
>
> and it is just hanging. At the same time on my HDFS it created file.txt
> but as a directory which has subdirectories (the final one is empty).
>
> Thanks!
> Ognen
>
