GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/17437
[SPARK-20102] Fix nightly snapshot and RC packaging scripts w/ two minor
build fixes
## What changes were proposed in this pull request?
The master snapshot publisher builds are currently broken due to two minor
build issues:
1. For unknown reasons, the LFTP `mkdir -p` command began throwing errors
when the remote FTP directory already exists. This change of behavior might
have been caused by configuration changes in the ASF's SFTP server, but I'm not
entirely sure of that. To work around this problem, this patch updates the
script to ignore errors from the `lftp mkdir -p` commands.
2. The PySpark `setup.py` file references a non-existent `pyspark.ml.stat`
module, causing Python packaging to fail by complaining about a missing
directory. The fix is to simply drop that line from the setup script.
## How was this patch tested?
The LFTP fix was tested by manually running the failing commands on AMPLab
Jenkins against the ASF SFTP server. The PySpark fix was tested locally.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark spark-20102
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17437.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17437
----
commit 213fd2f12bf39dde8311e02e01216af9430c3eb0
Author: Josh Rosen <[email protected]>
Date: 2017-03-23T20:31:11Z
Fix LFTP mkdir -p
commit 8d29b5aff5ee45fa54e90d96685d680fc18e9313
Author: Josh Rosen <[email protected]>
Date: 2017-03-24T00:13:48Z
Remove -f
commit 867ca2ccab8fe75861a246e166da175086c45248
Author: Josh Rosen <[email protected]>
Date: 2017-03-24T20:14:07Z
Remove pyspark.ml.stat (non-existent?) from setup.py file.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
