viirya commented on code in PR #58978:
URL: https://github.com/apache/spark/pull/58978#discussion_r4162823148
##########
core/src/main/scala/org/apache/spark/internal/config/Python.scala:
##########
@@ -56,6 +57,25 @@ private[spark] object Python {
.bytesConf(ByteUnit.MiB)
.createOptional
+ val IN_PROCESS_SITE_PACKAGES =
ConfigBuilder("spark.inprocess.python.sitePackages")
+ .doc("Comma-separated executor directories containing packages for
in-process Python UDFs. " +
+ "These directories are processed with site.addsitedir after Spark
distribution paths " +
+ "and the process PYTHONPATH. JEP must be directly importable from these
directories. " +
+ "Paths cannot contain quotes, newlines, NUL, surrogate characters or the
platform " +
Review Comment:
078288e fixes the doc and `require` wording (single quotes; surrogate
characters, including supplementary characters). The plugin now reads the
config before the installation checks, so an invalid value surfaces as
`INVALID_CONF_VALUE.REQUIREMENT` without the checklist.
##########
dev/sparktestsupport/modules.py:
##########
@@ -620,7 +620,7 @@ def __hash__(self):
pyspark_sql = Module(
name="pyspark-sql",
dependencies=[pyspark_core, hive, avro, protobuf],
- source_file_regexes=["python/pyspark/sql"],
+ source_file_regexes=["python/pyspark/sql", "python/pyspark/inprocess"],
Review Comment:
078288e adds `inprocess` to the `pyspark-core` lookahead, so a change under
`python/pyspark/inprocess` now selects the same modules as one under
`python/pyspark/sql`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
