[
https://issues.apache.org/jira/browse/SPARK-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14660374#comment-14660374
]
Ryan Williams commented on SPARK-1517:
--------------------------------------
Hey [~pwendell], thanks for continuing to push on this.
A workflow I'd like to see supported (and maybe it already is; please let me
know if so) is to more easily fetch these artifacts (both [Maven
snapshots|https://repository.apache.org/content/repositories/snapshots/org/apache/spark/]
and [bundled release {{.tgz}}
files|https://people.apache.org/~pwendell/spark-nightly/]) by their git SHAs.
For the Maven snapshots, I'd like to be able to just change the Spark version
in a downstream project's POM to a git SHA and have Maven fetch the Spark JARs
for that SHA (assuming it's one that has been built by the tools here); I'm
fine with the (presumably necessary) step on my end of adding a Maven
repository to make this work, either per-project or globally.
Today, the Maven snapshots at e.g.
https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-core_2.10/1.5.0-SNAPSHOT/
all seem to be uniquely ID'd by timestamps that I don't know how to get useful
information out of, which has precluded my using them.
On the bundled releases front, I see that the git SHA is being added to the
folders at https://people.apache.org/~pwendell/spark-nightly/spark-master-bin/:
!http://cl.ly/image/0o111a1o0U2N/Screen%20Shot%202015-08-06%20at%201.08.18%20PM.png!
but those don't seem to stick around more than a day or so? Additionally, as
that screenshot shows, there are 3 copies of one SHA there right now, and only
2 SHAs total.
I rolled some of my own scripts for cloning, building, and selecting specific
Spark versions locally at
[ryan-williams/spark-helpers|https://github.com/ryan-williams/spark-helpers],
which currently fetches release {{.tgz}} files for released Spark versions, but
for arbitrary Spark SHAs there doesn't seem to be an easy way to download a
pre-built Spark, so I am just cloning them and running {{mvn package}}.
Let me know if you have thoughts about exposing built artifacts for more SHAs,
the workflows I've described here, etc. Thanks again!
> Publish nightly snapshots of documentation, maven artifacts, and binary builds
> ------------------------------------------------------------------------------
>
> Key: SPARK-1517
> URL: https://issues.apache.org/jira/browse/SPARK-1517
> Project: Spark
> Issue Type: Improvement
> Components: Build, Project Infra
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Priority: Critical
>
> Should be pretty easy to do with Jenkins. The only thing I can think of that
> would be tricky is to set up credentials so that jenkins can publish this
> stuff somewhere on apache infra.
> Ideally we don't want to have to put a private key on every jenkins box
> (since they are otherwise pretty stateless). One idea is to encrypt these
> credentials with a passphrase and post them somewhere publicly visible. Then
> the jenkins build can download the credentials provided we set a passphrase
> in an environment variable in jenkins. There may be simpler solutions as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
