HyukjinKwon commented on a change in pull request #24772: [MINOR][BRANCH-2.4]
Avoid hardcoded py4j-0.10.7-src.zip in Scala
URL: https://github.com/apache/spark/pull/24772#discussion_r289690508
##########
File path: core/src/main/scala/org/apache/spark/api/python/PythonUtils.scala
##########
@@ -27,12 +27,15 @@ import org.apache.spark.SparkContext
import org.apache.spark.api.java.{JavaRDD, JavaSparkContext}
private[spark] object PythonUtils {
+ val PY4J_ZIP_NAME = "py4j-0.10.7-src.zip"
+
/** Get the PYTHONPATH for PySpark, either from SPARK_HOME, if it is set, or
from our JAR */
def sparkPythonPath: String = {
val pythonPath = new ArrayBuffer[String]
for (sparkHome <- sys.env.get("SPARK_HOME")) {
pythonPath += Seq(sparkHome, "python", "lib",
"pyspark.zip").mkString(File.separator)
- pythonPath += Seq(sparkHome, "python", "lib",
"py4j-0.10.7-src.zip").mkString(File.separator)
+ pythonPath +=
+ Seq(sparkHome, "python", "lib", PY4J_ZIP_NAME).mkString(File.separator)
Review comment:
Yup, just in case we backport other changes. I think it's better to keep the
diff small comparing to the master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
