GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/17437
[SPARK-20102] Fix nightly snapshot and RC packaging scripts w/ two minor build fixes ## What changes were proposed in this pull request? The master snapshot publisher builds are currently broken due to two minor build issues: 1. For unknown reasons, the LFTP `mkdir -p` command began throwing errors when the remote FTP directory already exists. This change of behavior might have been caused by configuration changes in the ASF's SFTP server, but I'm not entirely sure of that. To work around this problem, this patch updates the script to ignore errors from the `lftp mkdir -p` commands. 2. The PySpark `setup.py` file references a non-existent `pyspark.ml.stat` module, causing Python packaging to fail by complaining about a missing directory. The fix is to simply drop that line from the setup script. ## How was this patch tested? The LFTP fix was tested by manually running the failing commands on AMPLab Jenkins against the ASF SFTP server. The PySpark fix was tested locally. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JoshRosen/spark spark-20102 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17437.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17437 ---- commit 213fd2f12bf39dde8311e02e01216af9430c3eb0 Author: Josh Rosen <joshro...@databricks.com> Date: 2017-03-23T20:31:11Z Fix LFTP mkdir -p commit 8d29b5aff5ee45fa54e90d96685d680fc18e9313 Author: Josh Rosen <joshro...@databricks.com> Date: 2017-03-24T00:13:48Z Remove -f commit 867ca2ccab8fe75861a246e166da175086c45248 Author: Josh Rosen <joshro...@databricks.com> Date: 2017-03-24T20:14:07Z Remove pyspark.ml.stat (non-existent?) from setup.py file. ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org