Following up to my previous message, it turns out the issue is a mix
between packaging and spark launcher mechanics. Everything can be
summarized as follow
- My app is packaged as 'fat-jar' alike. I'm using zipapp to prepare the
self executable archive (with all the needed descriptors)
- IF the AppResource does not end with 'py', the launcher believes it is
a JVM application.
- I am not allowed to use the '.pyz' or '.zip' extension
- IF 'py-files' is set to add a '.py' file it expects a py file and not
py-zip archive
- I have no direct evidence of this sentence. Fact is if I add the
same package with '.zip' everything starts working
So at the moment I was able to run my app by shipping the same file with
two different extensions, '.py' to make the AppResource happy and 'zip' to
make the the py-files work.It is what it is, an akward workaournd.
Can someone share an approach to reach the following goals:
- submit complex python applications (multiple classes)
- without installing them on every node of the standalone cluster
- possibly without requiring any external tool (jobserver/livy/...)
Best,
On Fri, May 24, 2019 at 10:24 PM Riccardo Ferrari <[email protected]>
wrote:
> Hi list,
>
> I am having an issue ditributing a pandas_udf to my workers.
> I'm using Spark 2.4.1 in standalone mode.
> *Submit*:
>
> - via SparkLauncher as separate process. I do add the py-files with
> the self-executable zip (with .py extension) before launching the
> application.
> - The whole application (few modules) is zippped into a single zip
> file with 'py' extension
> - The app defines a GROUPED_MAP pandas_udf that instanciate an class
> from within the module
>
> *Error*:
> When I perform my df.gruopBy(<something>).agg(my_pandas_udf) my workers
> errors out with:
> ====
> 2019-05-24 19:40:18.577: Caused by:
> org.apache.spark.api.python.PythonException: Traceback (most recent call
> last):
> 2019-05-24 19:40:18.577: File
> "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 366, in main
> 2019-05-24 19:40:18.577: func, profiler, deserializer, serializer =
> read_udfs(pickleSer, infile, eval_type)
> 2019-05-24 19:40:18.577: File
> "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 229, in
> read_udfs
> 2019-05-24 19:40:18.577: arg_offsets, udf = read_single_udf(pickleSer,
> infile, eval_type, runner_conf)
> 2019-05-24 19:40:18.577: File
> "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 168, in
> read_single_udf
> 2019-05-24 19:40:18.577: f, return_type = read_command(pickleSer,
> infile)
> 2019-05-24 19:40:18.577: File
> "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 69, in
> read_command
> 2019-05-24 19:40:18.577: command = serializer._read_with_length(file)
> 2019-05-24 19:40:18.577: File
> "/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 172, in
> _read_with_length
> 2019-05-24 19:40:18.577: return self.loads(obj)
> 2019-05-24 19:40:18.577: File
> "/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 580, in
> loads
> 2019-05-24 19:40:18.577: return pickle.loads(obj, encoding=encoding)
> 2019-05-24 19:40:18.577: ModuleNotFoundError: No module named '<my-module>'
> 2019-05-24 19:40:18.577:
> 2019-05-24 19:40:18.577: at
> org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)
> ====
> where <my-module> is the name of my base package
> *My python application:*
> The structure is very simple
> my-application.py (actually a zip):
>
> - __mian__.py
> - <my-module
> - __init__.py
> - <sub-package>
> - __init__.py
> - ...
> - <another-sub-package>
> - __init__
> - spark_job.py
> - udf.py
>
> udf.py has a dummy class with just a staticmethod definig the UDF
> spark_job.py: at some point defines the pandas_udf by calling:
>
> pandas_udf(f=UDFHelper.function, returnType=UDFHelper.get_schema(),
> functionType=PandasUDFType.GROUPED_MAP)
>
> The error makes me think the executor don't have my archive in the
> PYTHONPATH however the 'addPyFile' should take care of it.
> I don't want to manually install my application everywhere (for env
> constraints)
>
> Any hint to troubleshoot or fix are really appreciated!
> Thanks,
>