It is not worked to configure local dirs to HDFS. Local dirs are mainly
used for data spill and shuffle data persistence, it is not suitable to use
hdfs. If you met capacity problem, you could configure multiple dirs
located in different mounted disks.
On Wed, Jul 6, 2016 at 9:05 AM, Sri <kali.tumm...@gmail.com> wrote:
> Hi ,
>
> Space issue we are currently using /tmp and at the moment we don't have
> any mounted location setup yet.
>
> Thanks
> Sri
>
>
> Sent from my iPhone
>
> On 5 Jul 2016, at 17:22, Jeff Zhang <zjf...@gmail.com> wrote:
>
> Any reason why you want to set this on hdfs ?
>
> On Tue, Jul 5, 2016 at 3:47 PM, kali.tumm...@gmail.com <
> kali.tumm...@gmail.com> wrote:
>
>> Hi All,
>>
>> can I set spark.local.dir to HDFS location instead of /tmp folder ?
>>
>> I tried setting up temp folder to HDFS but it didn't worked can
>> spark.local.dir write to HDFS ?
>>
>> .set("spark.local.dir","hdfs://namednode/spark_tmp/")
>>
>>
>> 16/07/05 15:35:47 ERROR DiskBlockManager: Failed to create local dir in
>> hdfs://namenode/spark_tmp/. Ignoring this directory.
>> java.io.IOException: Failed to create a temp directory (under
>> hdfs://namenode/spark_tmp/) after 10 attempts!
>>
>>
>> Thanks
>> Sri
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-local-dir-to-HDFS-tp27291.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com
>> <http://nabble.com>.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
