What is the recommended way to do this? We have some native database client libraries for which we are adding pyspark bindings.
The pyspark invokes spark-submit. Do we add our libraries to the SPARK_SUBMIT_LIBRARY_PATH ? This issue relates back to an error we have been seeing "Py4jError: Trying to call a package" - the suspicion being that the third party libraries may not be available on the jvm side.
