Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11350#discussion_r54154973
--- Diff: dev/create-release/release-build.sh ---
@@ -178,11 +198,11 @@ if [[ "$1" == "package" ]]; then
# Copy data
dest_dir="$REMOTE_PARENT_DIR/${DEST_DIR_NAME}-bin"
echo "Copying release tarballs to $dest_dir"
- $SSH $USER_HOST mkdir $dest_dir
- rsync -e "$SSH" spark-* $USER_HOST:$dest_dir
+ LFTP mkdir -p $dest_dir
+ LFTP mput -O $dest_dir spark-*
echo "Linking /latest to $dest_dir"
- $SSH $USER_HOST rm -f "$REMOTE_PARENT_DIR/latest"
- $SSH $USER_HOST ln -s $dest_dir "$REMOTE_PARENT_DIR/latest"
+ LFTP rm -f "$REMOTE_PARENT_DIR/latest"
+ LFTP ln -s $dest_dir "$REMOTE_PARENT_DIR/latest"
--- End diff --
The FTP protocol itself does not support symlinks and the older versions of
`lftp` don't have the `ln` command. I'm going to work around this by uploading
the binaries twice, once to the `latest` folder and once to the other one. With
the symlink, at least the change to `/latest` appeared relatively atomic so you
wouldn't see half-uploaded files in there. I'll try to work around this by
uploading to dest_dir, moving, then uploading again. This isn't perfect but
seems like a decent hack.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
