Hi, I have a question regarding code movements. It’s not clear of how exactly my code is being moved onto Worker nodes to be completed.
My assumption was that by submitting jar file through spark-submit, Spark copies this jar file to Worker nodes and adds this jar to their classpath. My experiments with custom kryo registrators gives me another experience. Without adding jar to SparkConf, Worker nodes simply says that they cannot find class of my CustomKryoRegistrator. Which makes me think that the process is the following. Jar file in spark-submit is not copied to Worker nodes and is not added into their classpath, but instead each function (transformation / action) is serialized and moved to Workers along with all required classes to be actually completed. Since my CustomKryoRegistrator is not part of any function, then it never end up being on a Workers nodes, thus class not found. Is my “new” understanding correct? Could you, please, explain in couple of words how code being moved from Driver to Workers? Could you give me a hint of where to find this in sources? Thanks in advance. -- Eugene Morozov fathers...@list.ru
