Hi all, Currently, SystemML makes use of a "scratch space" folder for temporary files during execution. This is currently set to a relative "scratch_space" directory that will be placed relative to the execution path (local mode) or in the user's directory on HDFS. This works okay in some cases, although it can cause confusion as to why the folder exists. In other cases, such as on Databricks Cloud, a relative path for HDFS is not allowed, and thus the user must change this "scratch space" folder to an absolute path, or else a strange error message will occur.
Since this "scratch space" folder is just for temporary files during execution, might it be better to simply query HDFS (which falls back to local FS if need) for a temporary folder, and just use that? If so, this would remove the need to adjust this setting, thus making it easier to use SystemML. Thoughts? - Mike -- Michael W. Dusenberry GitHub: github.com/dusenberrymw LinkedIn: linkedin.com/in/mikedusenberry
