Re: Spark standalone cluster - Output file stored in temporary directory in worker
I think the properties that you have in your hdfs-site.xml should go in the core-site.xml (at least for the namenode.name and datanote.data ones). I might be wrong here, but that's what I have in my setup. you should also add hadoop.tmp.dir in your core-site.xml. That might be the source of your inconsistency. as for hadoop-env.sh, I just use it to export variable such as HADOOP_PREFIX, LOG_DIR, CONF_DIR and JAVA_HOME. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-standalone-cluster-Output-file-stored-in-temporary-directory-in-worker-tp23653p23697.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark standalone cluster - Output file stored in temporary directory in worker
core-site.xml configuration property namefs.default.name/name valuehdfs://localhost:9000/value /property /configuration hdfs_site.xml - configuration property namedfs.replication/name value1/value /property property namedfs.namenode.name.dir/name valuefile:/usr/local/hadoop_store/hdfs/namenode/value /property property namedfs.datanode.data.dir/name valuefile:/usr/local/hadoop_store/hdfs/datanode/value /property /configuration I have not made any changes to the default hadoo-env.sh apart from manually adding the JAVA_HOME entry. What should the properties be configured to ? To the master HDFS where the file is actually present ? Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-standalone-cluster-Output-file-stored-in-temporary-directory-in-worker-tp23653p23683.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark standalone cluster - Output file stored in temporary directory in worker
Can you share your hadoop configuration file please? - etc/hadoop/core-site.xml - etc/hadoop/hdfs-site.xml - etc/hadoop/hadoo-env.sh AFAIK, the following properties should be configured: hadoop.tmp.dir, dfs.namenode.name.dir, dfs.datanode.data.dir and dfs.namenode.checkpoint.dir Otherwise, an HDFS slave will use it's default temporary folder to save blocks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-standalone-cluster-Output-file-stored-in-temporary-directory-in-worker-tp23653p23656.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
