> Markus, I'm guessing at your config which is probably one or more workers running on the same node as > your master, and the same node as your spark-shell. Which is why you're expecting the workers to be > able to read the same relative path?
> If that's the case, the reason it doesn't work out as you expect is because the workers have > different working directories. Yes, you are in right. The problem was caused by the use of relative paths. (I remember starting the worker and the driver program from the same path though.) However, the system could have been a bit more elaborate that it didn't find the files. Now I got the impression that everything went fine until I got unexpected results. Perhaps there was something alarming in the flood of log messages. Best regards, Markus Losoi ([email protected])
