[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743299#comment-16743299
]
shane knapp commented on SPARK-26565:
-
the build successfully packages up pyspark: {{pyspark-3.0.0.dev0.tar.gz}}
however, it's not looking like the SparkR tarball is being created in a way we
can consume... it's being built, but it's getting deleted somewhere and i
haven't found exactly where yet.
i'll poke around the logs/scripts/etc and see if i can pinpoint wtf is going on.
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, add an {{if}} statement to
> skip the following sections when run on jenkins:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> -another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.-
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742845#comment-16742845
]
Felix Cheung commented on SPARK-26565:
--
ok AFAIK
the first `cannot stat` is from this line
{code:java}
# Move R source package to match the Spark release version if the versions are
not the same.
# NOTE(shivaram): `mv` throws an error on Linux if source and destination are
same file
if [ "$R_PACKAGE_VERSION" != "$VERSION" ]; then
mv "$SPARK_HOME/R/SparkR_$R_PACKAGE_VERSION.tar.gz"
"$SPARK_HOME/R/SparkR_$VERSION.tar.gz"
fi
{code}
which should rename the tarball to SparkR_3.0.0-SNAPSHOT.tar.gz
(maybe it wasn't built?)
the second is from
{code:java}
# Remove the python distribution from dist/ if we built it
if [ "$MAKE_PIP" == "true" ]; then
rm -f "$DISTDIR"/python/dist/pyspark-*.tar.gz
fi
{code}
which is fine, it isn't the output we are looking for (not under dist)
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, add an {{if}} statement to
> skip the following sections when run on jenkins:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> -another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.-
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742361#comment-16742361
]
shane knapp commented on SPARK-26565:
-
[~felixcheung] sounds great.
my goal here is to re-create the release process in an atomic build that does
everything except to sign and push the release artifacts.
the big question i have now, especially given how these scripts "interact", is
what exactly is produced when a release is run... for example,
{{make-distribution.sh}} deletes a bunch of stuff that {{release-build.sh}}
expects to be there:
{noformat}
cp: cannot stat
`spark-3.0.0-SNAPSHOT-bin-without-hadoop-scala-2.12/R/SparkR_3.0.0-SNAPSHOT.tar.gz':
No such file or directory
Skipping signing of R source package
cp: cannot stat
`spark-3.0.0-SNAPSHOT-bin-without-hadoop-scala-2.12/python/dist/pyspark-3.0.0.dev0.tar.gz':
No such file or directory
{noformat}
this stuff hasn't been audited in probably ~5 years, and i'm finding a lot of
cruft and bugs. in all honesty, i'm surprised it all still works.
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, add an {{if}} statement to
> skip the following sections when run on jenkins:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> -another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.-
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16741453#comment-16741453
]
Felix Cheung commented on SPARK-26565:
--
Yeah, my point wasn’t to allow access to unsigned release but to help RM to
check out built packages before kicking off the RC process before release.
For example, often times the build completes successfully but there are some
issue with the content.
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, add an {{if}} statement to
> skip the following sections when run on jenkins:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> -another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.-
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739684#comment-16739684
]
shane knapp commented on SPARK-26565:
-
ok, the PR (https://github.com/apache/spark/pull/23492) should be g2g, but one
conversation i had w/[~felixcheung] was if we wanted to provide nightly,
unsigned packages for people to consume (at their own risk).
i'm ok w/this if it's what we want, but:
1) how useful can unsigned packages be?
2) anyone should be able to build their own dists locally
thoughts?
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, add an {{if}} statement to
> skip the following sections when run on jenkins:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> -another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.-
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737520#comment-16737520
]
shane knapp commented on SPARK-26565:
-
test build passed!
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, add an {{if}} statement to
> skip the following sections when run on jenkins:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> -another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.-
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737463#comment-16737463
]
shane knapp commented on SPARK-26565:
-
test build running:
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-package-sknapptest/1/
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, add an {{if}} statement to
> skip the following sections when run on jenkins:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> -another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.-
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737453#comment-16737453
]
shane knapp commented on SPARK-26565:
-
see: https://github.com/apache/spark/pull/23492
i also have a PR open to add the AMPLAB_JENKINS=1 flag to the package job
configs.
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, add an {{if}} statement to
> skip the following sections when run on jenkins:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> -another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.-
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737420#comment-16737420
]
shane knapp commented on SPARK-26565:
-
ok, thanks for all of the clarification on this. i think i understand now how
everything works... and that my PR in it's current state will break the
release process if it's committed. ;)
since we have the publish and docs builds, it makes sense to me to have the
package build run as well.
i'll update the issue description w/what i'm planning on doing to
{{release-build.sh}} instead.
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, remove ALL of the following
> bits:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736580#comment-16736580
]
Wenchen Fan commented on SPARK-26565:
-
The release tools have been evolved. In the beginning, people need to run
*release-build.sh* multiple times for each step, and now it's automated by the
*do-release-docker.sh*.
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, remove ALL of the following
> bits:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736500#comment-16736500
]
shane knapp commented on SPARK-26565:
-
[~vanzin] – good to have confirmed that `do-release-docker.sh` is the
official/only way to do a release. it appears that there's a LOT of cruft in
the repo WRT release process, and that should probably go away ASAP.
there is some discussion on the dev@ list about the packaging builds, and the
general consensus was they could still be useful to test if packaging itself
still works... however, after looking at `do-release-docker.sh` i feel that
maybe the package builds have reached the limit of their usefulness.
that all being said, the `do-release.sh` script is still used for docs and
maven snapshots (but not packaging).
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, remove ALL of the following
> bits:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736493#comment-16736493
]
Marcelo Vanzin commented on SPARK-26565:
Stepping back a little, what is the purpose of that packaging build?
IIRC it was meant to publish an unofficial nightly "distribution" that people
could download / use with a custom maven repo; and for that you need GPG keys.
If we remove the keys, is that build doing anything useful? If it is, can we
just add that to the existing test scripts instead, and get rid of that build?
To answer one of the questions, the current "official" way to create a release
is to run {{do-release-docker.sh}} from the master branch (regardless of which
release you're creating); the web site may need some updates to reflect that.
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, remove ALL of the following
> bits:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736485#comment-16736485
]
Sean Owen commented on SPARK-26565:
---
I've never done a release myself ... [~cloud_fan] [~vanzin] [~irashid] ?
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, remove ALL of the following
> bits:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736472#comment-16736472
]
shane knapp commented on SPARK-26565:
-
[~srowen] – herein lies my confusion... after reading the release process page
([https://spark.apache.org/release-process.html)] i don't see any mention of
the release manager manually running this script.
i checked for anything in the spark repo that calls this script, and the only
one i found was `do-release.sh`... which also doesn't appear to be run
(according to the release docs).
{noformat}
➜ spark git:(remove-package-signing) grep -r "release-build.sh" *
dev/create-release/do-release.sh: "$SELF/release-build.sh" package
dev/create-release/do-release.sh: "$SELF/release-build.sh" docs
dev/create-release/do-release.sh: "$SELF/release-build.sh" publish-release
dev/create-release/release-build.sh:usage: release-build.sh
{noformat}
so, if release-build.sh is actually still in use during the release process
(package), then i will nuke my changes and add in a jenkins-specific flag to
skip the GPG and svn parts, rather than deleting them outright.
a quick image to sum up my feelings about how all of the release tooling works:
!no-idea.jpg!
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png, no-idea.jpg
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, remove ALL of the following
> bits:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> another, and probably much better option, is to nuke the
> spark-\{branch}-packaging builds and create new ones that just build things
> w/o touching this incredible fragile shell scripting nightmare.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing
[
https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736465#comment-16736465
]
Sean Owen commented on SPARK-26565:
---
Sure, if we're not making releases with it, no need to sign them.
> modify dev/create-release/release-build.sh to let jenkins build packages w/o
> publishing
> ---
>
> Key: SPARK-26565
> URL: https://issues.apache.org/jira/browse/SPARK-26565
> Project: Spark
> Issue Type: Bug
> Components: Build
>Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0
>Reporter: shane knapp
>Assignee: shane knapp
>Priority: Major
> Attachments: fine.png
>
>
> about a year+ ago, we stopped publishing releases directly from jenkins...
> this means that the spark-\{branch}-packaging builds are failing due to gpg
> signing failures, and i would like to update these builds to *just* perform
> packaging.
> example:
> [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console]
> i propose to change dev/create-release/release-build.sh...
> when the script is called w/the 'package' option, remove ALL of the following
> bits:
> 1) gpg signing of the source tarball (lines 184-187)
> 2) gpg signing of the sparkR dist (lines 243-248)
> 3) gpg signing of the python dist (lines 256-261)
> 4) gpg signing of the regular binary dist (lines 264-271)
> 5) the svn push of the signed dists (lines 317-332)
>
> PR coming soon.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
