Hi, The solution we have is to make a generic spark submit python file (driver.py). This is just a main method which takes a single parameter- the module containing the app you want to run. The main method itself just dynamically loads the module and executes some well know method on it (we use run). As such it is basically capable of running any pyspark app so long as the required module is available.
We then package all our job modules and associated library code into a single zip file (jobs.py). You can pass this zip file to spark submit via pyfiles and use the generic launcher as the driver: Spark-submit —py-files jobs.py driver.py job-module > On 25 May 2019, at 10:02, Riccardo Ferrari <[email protected]> wrote: > > Following up to my previous message, it turns out the issue is a mix between > packaging and spark launcher mechanics. Everything can be summarized as follow > My app is packaged as 'fat-jar' alike. I'm using zipapp to prepare the self > executable archive (with all the needed descriptors) > IF the AppResource does not end with 'py', the launcher believes it is a JVM > application. > I am not allowed to use the '.pyz' or '.zip' extension > IF 'py-files' is set to add a '.py' file it expects a py file and not py-zip > archive > I have no direct evidence of this sentence. Fact is if I add the same package > with '.zip' everything starts working > So at the moment I was able to run my app by shipping the same file with two > different extensions, '.py' to make the AppResource happy and 'zip' to make > the the py-files work.It is what it is, an akward workaournd. > > Can someone share an approach to reach the following goals: > submit complex python applications (multiple classes) > without installing them on every node of the standalone cluster > possibly without requiring any external tool (jobserver/livy/...) > Best, > >> On Fri, May 24, 2019 at 10:24 PM Riccardo Ferrari <[email protected]> wrote: >> Hi list, >> >> I am having an issue ditributing a pandas_udf to my workers. >> I'm using Spark 2.4.1 in standalone mode. >> Submit: >> via SparkLauncher as separate process. I do add the py-files with the >> self-executable zip (with .py extension) before launching the application. >> The whole application (few modules) is zippped into a single zip file with >> 'py' extension >> The app defines a GROUPED_MAP pandas_udf that instanciate an class from >> within the module >> Error: >> When I perform my df.gruopBy(<something>).agg(my_pandas_udf) my workers >> errors out with: >> ==== >> 2019-05-24 19:40:18.577: Caused by: >> org.apache.spark.api.python.PythonException: Traceback (most recent call >> last): >> 2019-05-24 19:40:18.577: File >> "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 366, in main >> 2019-05-24 19:40:18.577: func, profiler, deserializer, serializer = >> read_udfs(pickleSer, infile, eval_type) >> 2019-05-24 19:40:18.577: File >> "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 229, in read_udfs >> 2019-05-24 19:40:18.577: arg_offsets, udf = read_single_udf(pickleSer, >> infile, eval_type, runner_conf) >> 2019-05-24 19:40:18.577: File >> "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 168, in >> read_single_udf >> 2019-05-24 19:40:18.577: f, return_type = read_command(pickleSer, infile) >> 2019-05-24 19:40:18.577: File >> "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 69, in >> read_command >> 2019-05-24 19:40:18.577: command = serializer._read_with_length(file) >> 2019-05-24 19:40:18.577: File >> "/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 172, in >> _read_with_length >> 2019-05-24 19:40:18.577: return self.loads(obj) >> 2019-05-24 19:40:18.577: File >> "/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 580, in >> loads >> 2019-05-24 19:40:18.577: return pickle.loads(obj, encoding=encoding) >> 2019-05-24 19:40:18.577: ModuleNotFoundError: No module named '<my-module>' >> 2019-05-24 19:40:18.577: >> 2019-05-24 19:40:18.577: at >> org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452) >> ==== >> where <my-module> is the name of my base package >> My python application: >> The structure is very simple >> my-application.py (actually a zip): >> __mian__.py >> <my-module >> __init__.py >> <sub-package> >> __init__.py >> ... >> <another-sub-package> >> __init__ >> spark_job.py >> udf.py >> udf.py has a dummy class with just a staticmethod definig the UDF >> spark_job.py: at some point defines the pandas_udf by calling: >> pandas_udf(f=UDFHelper.function, returnType=UDFHelper.get_schema(), >> functionType=PandasUDFType.GROUPED_MAP) >> The error makes me think the executor don't have my archive in the >> PYTHONPATH however the 'addPyFile' should take care of it. >> I don't want to manually install my application everywhere (for env >> constraints) >> >> Any hint to troubleshoot or fix are really appreciated! >> Thanks,
