nchammas opened a new pull request, #57763: URL: https://github.com/apache/spark/pull/57763
### What changes were proposed in this pull request? Package everything in `sbin/` when building a PySpark Classic distribution. ### Why are the changes needed? Some `sbin` scripts were first added to PySpark in #23715. There was some disagreement then about whether PySpark should include these scripts, mainly because some committers at the time thought PySpark should be a client-only distribution. In the intervening years, the Spark Connect effort has created a true client-only distribution of Spark in contrast to the "heavier" PySpark Classic which includes all of Spark's assembly JARs. PySpark Classic has included some `sbin` scripts since 3.0.0, and #56907 recently [added the Connect server scripts][1]. #56907, however, neglected to add the corresponding directives to `MANIFEST.in`. [1]: https://github.com/apache/spark/pull/56907/changes#diff-dd3c4e1caaf1748872bda7e102f7287d434044236e3996835008600c070ad43a Since #57452 clarifies the intended use of the Python distributions of Spark -- specifically, that starting a full cluster is [not supported][2], regardless of whether it's possible -- I believe it's conceptually simpler to just package all of `sbin`. That would, for example, prevent the kind of gap identified in #56907. [2]: https://github.com/apache/spark/pull/57452#discussion_r3635301417 ### Does this PR introduce _any_ user-facing change? Yes, it packages additional `sbin` scripts in the PySpark classic distribution. ### How was this patch tested? Distributions are not tested thoroughly. #57645 adds a dedicated distribution validation script. I think we should discuss there any testing we would like to add for this. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
