I wish to setup a Tez data analysis framework, where the data resides in memory. Currently, I have tez (and also Hive) setup such that it can read from an in-memory filesystem like Tachyon.
However, the intermediate data is still written to disk at the each processing node. I considered writing to tmpfs, however, such a setup does not fall back to disk gracefully. Does Tez have an interface to write intermediate data to HDFS like filesystem ? If yes, what are the settings ? Does setting "yarn.nodemanager.local-dirs" to some HDFS or Tachyon URI suffice ? Thanks, Raajay
