Yes, I also tried FUSE before, it is not stable and I don’t recommend it
> On Aug 24, 2016, at 22:15, Saisai Shao <sai.sai.s...@gmail.com> wrote:
>
> Also fuse is another candidate (https://wiki.apache.org/hadoop/MountableHDFS
> <https://wiki.apache.org/hadoop/MountableHDFS>), but not so stable as I tried
> before.
>
> On Wed, Aug 24, 2016 at 10:09 PM, Sun Rui <sunrise_...@163.com
> <mailto:sunrise_...@163.com>> wrote:
> For HDFS, maybe you can try mount HDFS as NFS. But not sure about the
> stability, and also there is additional overhead of network I/O and replica
> of HDFS files.
>
>> On Aug 24, 2016, at 21:02, Saisai Shao <sai.sai.s...@gmail.com
>> <mailto:sai.sai.s...@gmail.com>> wrote:
>>
>> Spark Shuffle uses Java File related API to create local dirs and R/W data,
>> so it can only be worked with OS supported FS. It doesn't leverage Hadoop
>> FileSystem API, so writing to Hadoop compatible FS is not worked.
>>
>> Also it is not suitable to write temporary shuffle data into distributed FS,
>> this will bring unnecessary overhead. In you case if you have large memory
>> on each node, you could use ramfs instead to store shuffle data.
>>
>> Thanks
>> Saisai
>>
>> On Wed, Aug 24, 2016 at 8:11 PM, tony....@tendcloud.com
>> <mailto:tony....@tendcloud.com> <tony....@tendcloud.com
>> <mailto:tony....@tendcloud.com>> wrote:
>> Hi, All,
>> When we run Spark on very large data, spark will do shuffle and the shuffle
>> data will write to local disk. Because we have limited capacity at local
>> disk, the shuffled data will occupied all of the local disk and then will be
>> failed. So is there a way we can write the shuffle spill data to HDFS? Or
>> if we introduce alluxio in our system, can the shuffled data write to
>> alluxio?
>>
>> Thanks and Regards,
>>
>> 阎志涛(Tony)
>>
>> 北京腾云天下科技有限公司
>> --------------------------------------------------------------------------------------------------------
>> 邮箱：tony....@tendcloud.com <mailto:tony....@tendcloud.com>
>> 电话：13911815695
>> 微信： zhitao_yan
>> QQ ： 4707059
>> 地址：北京市东城区东直门外大街39号院2号楼航空服务大厦602室
>> 邮编：100027
>> --------------------------------------------------------------------------------------------------------
>> TalkingData.com <http://talkingdata.com/> - 让数据说话
>>
>
>
