Hi,
I need to access a file from a UDF. In standalone, if I add the file
/tmp/somedata, it ends up in /private/tmp/somedata, as I found out keeping
an eye on the logs. That is actually the same file because of a link
between the directories, nothing related to spark. My expectation reading
some code examples was to have access to a copy of the file in the current
working directory of the UDF, but that is clearly not the case here. I was
wondering if there is a general rule about these added files that is
independent from Spark  mode and persistence layer, say if the file comes
from HDFS vs S3. That would be very helpful for me to achieve the necessary
level of generality for the UDF. Thanks


Antonio

Reply via email to