HyukjinKwon commented on a change in pull request #24770: [MINOR] Avoid use
hardcoded py4j Python zip name in Scala
URL: https://github.com/apache/spark/pull/24770#discussion_r289669057
##########
File path: core/src/main/scala/org/apache/spark/api/python/PythonUtils.scala
##########
@@ -27,13 +27,15 @@ import org.apache.spark.SparkContext
import org.apache.spark.api.java.{JavaRDD, JavaSparkContext}
private[spark] object PythonUtils {
+ val PY4J_ZIP_NAME = "py4j-0.10.8.1-src.zip"
+
/** Get the PYTHONPATH for PySpark, either from SPARK_HOME, if it is set, or
from our JAR */
def sparkPythonPath: String = {
val pythonPath = new ArrayBuffer[String]
for (sparkHome <- sys.env.get("SPARK_HOME")) {
pythonPath += Seq(sparkHome, "python", "lib",
"pyspark.zip").mkString(File.separator)
pythonPath +=
- Seq(sparkHome, "python", "lib",
"py4j-0.10.8.1-src.zip").mkString(File.separator)
+ Seq(sparkHome, "python", "lib", PY4J_ZIP_NAME).mkString(File.separator)
Review comment:
BTW, the way of getting Spark home is different in the places. Looks
deduplicating the name only is enough since that's the problematic part
frequently changing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
