I’m writing this email to reach out to the community to demisty the py-files
parameter when working with spark-submit and python projects.
Currently I have a project, say
When I zip up the src directory and submit it to spark via emr add step , the
namespacing is lost.
From Modules.module1 import SomeClass
My code returns and error that it cannot find this class, now this works if I
goto the instance download my project, and submit it to spark from within the
EMR instance via spark-submit , but not when adding it as a step in emr from
This message is confidential, intended only for the named recipient(s) and may
contain information that is privileged or exempt from disclosure under
applicable law. If you are not the intended recipient(s), you are notified that
the dissemination, distribution, or copying of this message is strictly
prohibited. If you receive this message in error or are not the named
recipient(s), please notify the sender by return email and delete this message.