> scp -r
jenkins-controller:<JENKINS_HOME>/jobs/ProjectFolder/jobs/MyProject/branches/
*<HOW_DO_I_COMPUTE_THE_BRANCH_PORTION_OF_PATH?>*
/lastSuccessfulBuild/artifact/<GLOB>
Hi, sorry to revive an old thread but,
I'm going down the `scp` route. However, upon copying
`jobs/${JOB_NAME}/builds/${BUILD_ID}/archive/`
to the controller node, refreshing the Jenkins UI
(https://{JENKINS_URL}/job/{JOB_NAME}/job/{BRANCH_NAME}/) does not show the
copied artifact. Was this a problem you came across, if so how did you
overcome it?
Alternatively, would you mind sharing the implementation of your shared
library function?
On Friday, March 5, 2021 at 5:08:05 PM UTC+13 [email protected] wrote:
> To whom it may concern, I ended up finding the code in Jenkins branch-api
> plugin that's creating that branch path segment (the NameMangler
> <https://github.com/jenkinsci/branch-api-plugin/blob/f2bd7ec715057feb047754f5427f209bbf1b3248/src/main/java/jenkins/branch/NameMangler.java#L55>),
>
> however it turned out to be completely unnecessary since I can just get the
> build directory (to construct the path to the artifacts on the controller)
> from the project/job object, obtained by name from the jenkins instance in
> groovy. So my shared library function is even simpler now, works for any
> project type and safer because it doesn't need to use any core or plug in
> classes.
>
> On Monday, March 1, 2021 at 1:43:02 PM UTC-8 Tim Black wrote:
>
>> I'm trying to do same, but in both directions (archiving AND copying
>> artifacts from upstream). I wonder how the scp approach to copying
>> artifacts would work in multibranch pipelines? Can one deterministically
>> construct the path to a branch job's artifact folder on the controller's
>> disk?
>>
>> As I commented here
>> <https://stackoverflow.com/questions/21268327/rsync-alternative-to-jenkins-copy-artifacts-plugin#comment117378767_25530456>,
>>
>> I'm also seeking massive performance gains by replacing copyArtifact with a
>> shell call in my pipelines. In lieu of installing a proper artifact
>> management system and replacing all archive/copyArtifact with calls to its
>> REST API (which I'll be doing later this year), I'm hoping to find a quick
>> alternative. SCP would be a slam dunk for me if I could construct the
>> source path correctly. The problem is that Jenkins is using an algorithm to
>> create a unique folder name, both for workspace names, and for branch job
>> names, but I'm not sure if that's consistent.
>>
>> E.g. to fetch artifacts from the corresponding branch of an upstream
>> multibranch pipeline job with Full project name of
>> "ProjectFolder/MyProject/feature%2Ffoo", in the downstream multibranch
>> pipeline, I would do something like:
>>
>> scp -r
>> jenkins-controller:<JENKINS_HOME>/jobs/ProjectFolder/jobs/MyProject/branches/
>> *<HOW_DO_I_COMPUTE_THE_BRANCH_PORTION_OF_PATH?>*
>> /lastSuccessfulBuild/artifact/<GLOB>
>> On Wednesday, April 29, 2015 at 7:02:09 AM UTC-7 [email protected] wrote:
>>
>>> I found that using that standard method is quite slow compared to scp.
>>> So I use that method to copy just a few small files, and one with GUIDs for
>>> fingerprinting, and for the big ones I do something like
>>>
>>> scp -v ${WORKSPACE}/bigfile.tar.gz
>>> user@jenkins_host_name:path_to_jenkins_root/jobs/${JOB_NAME}/builds/${BUILD_ID}/archive/
>>>
>>> 2>&1 | tail -n 5
>>>
>>> I think there's a ${JENKINS_HOME} or something for the path on the
>>> master. That copies a 2-3 GB file in roughly 40 seconds instead of
>>> something like 4 minutes. There was a fix put in recently for I think some
>>> Maven plugin where when copying files to the master, the master would poll
>>> the slave to send over the next packet with too many requests, and fixing
>>> that sped things up a ton, perhaps there's another fix coming for how other
>>> files are transferred.
>>>
>>> Since "big" can sometimes be > 8GB, it would choke the normal archiver
>>> which uses tar under the covers, or at least it did. In any case this is
>>> much faster, since pigz is multicore aware:
>>>
>>> tar cf ${WORKSPACE}/bigfile.tar.gz --use-compress-program=pigz [files to
>>> pack]
>>>
>>> YMMV
>>>
>>> --- Matt
>>>
>>>
>>> On Monday, April 27, 2015 at 1:27:43 AM UTC-7, [email protected]
>>> wrote:
>>>>
>>>> Are you using "Archive Artifacts" in the upstream job, and the "Copy
>>>> Artifact" plugin in the downstream job? This is the standard method.
>>>> If so, maybe the upstream job should produce a single zip file , which
>>>> the downstream job and get and unzip.
>>>> Matthew
>>>>
>>>> > -----Original Message-----
>>>> > From: [email protected] [mailto:[email protected]]
>>>> On Behalf Of Simon Richter
>>>> > Sent: 25 April 2015 01:03
>>>> > To: [email protected]
>>>> > Subject: Efficiently copying artifacts
>>>> >
>>>> > Hi,
>>>> >
>>>> > I have a project that outputs a few large files (compiled DLL and
>>>> static
>>>> > library) as well as a few hundred header files as artifacts for use
>>>> by
>>>> > the next project in the dependency chain. Copying these in and out of
>>>> > workspaces takes quite a long time, and the network link is not even
>>>> > near capacity, so presumably handling of multiple small files is not
>>>> > really efficient.
>>>> >
>>>> > Can this be optimized somehow, e.g. by packing and unpacking the
>>>> files
>>>> > for transfer? Manual inspection of artifacts is secondary, I think.
>>>> >
>>>> > Simon
>>>> >
>>>>
>>>> --
>>>> This e-mail and any attachments may contain confidential, copyright and
>>>> or privileged material, and are for the use of the intended addressee
>>>> only.
>>>> If you are not the intended addressee or an authorised recipient of the
>>>> addressee please notify us of receipt by returning the e-mail and do not
>>>> use, copy, retain, distribute or disclose the information in or attached
>>>> to
>>>> the e-mail.
>>>> Any opinions expressed within this e-mail are those of the individual
>>>> and not necessarily of Diamond Light Source Ltd.
>>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
>>>> attachments are free from viruses and we cannot accept liability for any
>>>> damage which you may sustain as a result of software viruses which may be
>>>> transmitted in or with the message.
>>>> Diamond Light Source Limited (company no. 4375679). Registered in
>>>> England and Wales with its registered office at Diamond House, Harwell
>>>> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United
>>>> Kingdom
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/8fa6845c-4cf7-4cd0-81c2-591f7ac9cc8fn%40googlegroups.com.
