Re: Unable to Run jenkins job on zos slave
Yes, please find the below output.
FSUM7343 cannot open □?EC□?¤IA for output : no such file or directory.
On Fri, 29 Oct 2021, 1:00 pm kuisathaverat, wrote:
> Did you test other commands in the Freestyle job? try this
>
> #!/bin/bash
> pwd
> echo foo
> date
>
>
>
> El vie, 29 oct 2021 a las 7:21, Adharsha sri ()
> escribió:
>
>> Hi,
>>
>> In /etc/profile added the below bash environment variables
>> export _CEE_RUNOPTS=“FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)”
>> export _BPXK_AUTOCVT=ON
>> export _TAG_REDIR_ERR=txt
>> export _TAG_REDIR_IN=txt
>> export _TAG_REDIR_OUT=txt
>>
>>
>> And then able to run the below pipeline job
>>
>> node('hostname')
>> {
>> stage('test')
>> {
>> sh "pwd"
>> sh " hostname"
>> sh "date"
>> }}
>>
>> But when running the below freestyle job , gettingerror
>>
>> Execute shell:
>> date
>>
>> Error:
>>
>> +□?A/EA
>> /tmp/jenkjns123456789.sh 1: FSUM7351 not found
>>
>>
>>
>>
>>
>>
>>
>> On Thu, 28 Oct 2021, 9:51 pm Ivan Fernandez Calvo, <
>> kuisathave...@gmail.com> wrote:
>>
>>> should not, the FreeStyle "Execute shell" step and the Pipeline sh step
>>> work more or less in the same way, I never see a fail one when the other
>>> works. In any case, it is difficult to help you if you do not provide more
>>> info about what you did, what does not work exactly, Why does not work,
>>> what error you see in the Jenkins logs (if any), what error you see in the
>>> build logs(if any), What did you change in the Agents, and so on.
>>>
>>> El jueves, 28 de octubre de 2021 a las 13:38:06 UTC+2,
>>> adharsh...@gmail.com escribió:
>>>
Now I am able run shell commands using pipeline job but unable to run
using freestyle job. Will there be any difference
On Wed, 27 Oct 2021, 10:10 pm Adharsha sri,
wrote:
> Thank you , will check on that part.
>
> On Wed, 27 Oct 2021, 6:45 pm Ivan Fernandez Calvo, <
> kuisat...@gmail.com> wrote:
>
>> the echo step is pure Groovy, however, the sh step uses bash, so
>> probably you do not have bash in the PATH of that user, check that bash
>> is
>> installed and that user can use it.
>>
>> El miércoles, 27 de octubre de 2021 a las 15:01:08 UTC+2,
>> adharsh...@gmail.com escribió:
>>
>>> Able to run the echo commands but not able to run the shell commands
>>>
>>> On Wed, 27 Oct 2021, 4:12 pm Adharsha sri,
>>> wrote:
>>>
Can anyone help on this error
On Wed, 27 Oct 2021, 2:38 pm Adharsha sri,
wrote:
> Thanks for the sample pipeline but getting error like " process
> apparently never started in path
> ( running jenkins temporarily with
> -Dorg.jenkinsci.plugins.durabletask.BourneshellScript.LAUNCH_DIAGNOSTICS=true
> might make the problem clearer).
>
> Thanks.
>
> On Wed, 27 Oct 2021, 1:18 pm Ivan Fernandez Calvo, <
> kuisat...@gmail.com> wrote:
>
>> This is a pretty simple pipeline to expose your environment
>> settings
>>
>> pipeline {
>> agent { label 'YOUR_AGENT_LABELS' }
>> stages {
>> stage('test') {
>> steps {
>> sh 'export'
>> sh 'set'
>> }
>> }
>> }
>> }
>>
>> El miércoles, 27 de octubre de 2021 a las 9:13:59 UTC+2,
>> adharsh...@gmail.com escribió:
>>
>>> The Configuration from agent is fine right, but it will be
>>> useful only when we run the jave script. So for shell commands we
>>> need to
>>> check .bashrc file in the agent server. Can you please provide the
>>> screenshot of the .bashrc file and please provide the script you
>>> are using
>>> and examples of export and set. This will be very helpful for us.
>>>
>>> Thanks.
>>>
>>> On Wed, 27 Oct 2021, 8:04 am Adharsha sri,
>>> wrote:
>>>
Can you please provide an example.
On Tue, 26 Oct 2021, 11:52 pm Ivan Fernandez Calvo, <
kuisat...@gmail.com> wrote:
> This is the output you see in the browser, isn't it? Did you
> try to download the "view as plain text" button output in a file
> and with
> an editor trying to find the correct encoding? the encoding you
> set in the
> Agent is only from Java process launched from the remoting
> thread, I mean
> is you launch a script in a job this script will use the
> encodings setting
> you have configured in the user you use to connect if in your
> .bashrc (or
> other shell configuration) you choose other encodings different
> than UTF-8
> you will have issues showing
回复： thread safety of scripted pipeline parallel and usage of nested parallel
sorry, pipeline code should be correct like this:
pipeline {
agent any
stages {
stage('Parallel Build') {
steps {
script {
def
builds = [:]
def
tests = [:]
stash name: 'src', include: 'src/**'
def
build_action = { platform, tests -
//build in some slave node.
node ("${platform}") {
//unstash need thread mutex lock?
unstash name: 'src'
sh "make ${platform}"
//archive need thread mutex lock?
archive name: "./${platform}"
}
//nested parallel, outside of node block
parallel tests
}
def
test_action = { platform -
//test in some slave node.
node ("${platform}") {
sh "make ${platform}-test"
}
}
tests["A1"] = { test_action("A1") }
tests["A2"] = { test_action("A2") }
tests["A3"] = { test_action("A3") }
builds["A"] = { build_action("A",tests) }
tests = [:]
tests["B1"] = { test_action("B1") }
tests["B2"] = { test_action("B2") }
builds["B"] = { build_action("B",tests) }
tests = [:]
tests["C1"] = { test_action("C1") }
tests["C2"] = { test_action("C2") }
builds["C"] = { build_action("B",tests) }
parallel builds
}
}
}
}
}
---原始邮件---
发件人: "'abstrakta' via Jenkins Users"https://groups.google.com/d/msgid/jenkinsci-users/0712b8c5-42dc-439f-a017-2a5ca45ad1e9n%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/fed01b0e-a04c-448b-9f32-3d2b3ffd6c40n%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/tencent_579508DE2429C25B92D7BFFF6AED09115C08%40qq.com.
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/tencent_E81808632457C23792EEBFC46A5A093E5C08%40qq.com.
?????? thread safety of scripted pipeline parallel and usage of nested parallel
OK,I know what you means. But I test these pipeline that data is not
synchronized many times( about more than 700 times,I use periodical build to
test it automatically). Shared data that is not synchronized is processed
correctly every time. Do you really encounter the situation that shared data is
corrupt in parallel task?
In some scenarios, I want to parallel build one more artifacts that used in
different platforms from the same source code. And then I parallel test all
artifacts in their corresponding platforms(One artifact might be used in one
more platforms).
build platform: A B C
test platform: A(A1 A2 A3), B(B1 B2), C(C1 C2)
- A1
- A - A2
| - A3
/
source code -- B - B1
\ - B2
|
- C - C1
- C2
- C3
Pipeline code like this:
pipeline {
agent any
stages {
stage('Parallel Build') {
steps {
script {
def builds = [:]
def tests = [:]
stash name: 'src', include: 'src/**'
def build_action = { platform, tests -
//build in some slave
node.
node ("${platform}") {
//unstash need thread mutex lock?
unstash name: 'src'
sh
"make ${platform}"
//archive need thread mutex lock?
archive name: "./${platform}"
}
//nested parallel,
outside of node block
parallel tests
}
def test_action = { platform -
//test in some slave node.
node ("${platform}") {
sh "make
${platform}-test"
}
}
tests["A1"] = test_action("A1")
tests["A2"] = test_action("A2")
tests["A3"] = test_action("A3")
builds["A"] = build_action("A",tests)
tests = [:]
tests["B1"] = test_action("B1")
tests["B2"] = test_action("B2")
builds["B"] = build_action("B",tests)
tests = [:]
tests["C1"] = test_action("C1")
tests["C2"] = test_action("C2")
builds["C"] = build_action("B",tests)
parallel builds
}
}
}
}
}
I have been running this pipeline for some time. It works well. But I worry
about the thread safety of unstash(and archive，or other commands). I want to
find some evidence that Jenkins garantee these parallel thread safety and
nested parallel is ok. Otherwise I must split these platforms into different
build jobs(which I think it's more difficult to manage projects.) Because
Jenkins must garantee parallel build jobs more safe? I just don't know the
underlying works that Jenkins do about parallel.I'm just curious about these
thread safety.If Jenkins take care of these synchronization,that is the best.
If such, I don't need to care about these thread things. If not,I want to find
correct ways to do parallel tasks.
--原始邮件--
发件人:
"jenkinsci-users"
https://groups.google.com/d/msgid/jenkinsci-users/0712b8c5-42dc-439f-a017-2a5ca45ad1e9n%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/fed01b0e-a04c-448b-9f32-3d2b3ffd6c40n%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/tencent_579508DE2429C25B92D7BFFF6AED09115C08%40qq.com.
Re: thread safety of scripted pipeline parallel and usage of nested parallel
The following example should work (I did not test it), I my case I have
used maps like the “result” variable, that it is a simple map not
synchronized and stores data from all task but it is not read from the
different tasks. The other two cases “data” and “mapSync” uses concurrent
classes, they are thread safe and synchronized so you can share data across
tasks, I dunno is they are in the allow list for pipeline, if not you have
to approve it use in pipelines in the Jenkins config. Finally, the last
part of the pipeline uses nested parallel task, from my experience is not a
good idea, the parallel explosion of task is a little incontrolable and
there are other solution like launch a job from those task and inside that
job launch parallel task, in this way you only have 1 parallel level that
is easy to control and understand when something goes wrong.
import groovy.transform.Field
import java.util.concurrent.atomic.AtomicInteger
import java.util.concurrent.ConcurrentHashMap
@Field def results = [:]
@Field AtomicInteger data = new AtomicInteger()
@Filed ConcurrentHashMap mapSync = new ConcurrentHashMap()
pipeline{
agent any
stages
{
stage(‘Parallel BuiLd’) {
steps {
script {
def i = 0
def builds = [:]
mapSync[“odd”] = 0
mapSync[“even”] = 0
stash name: ‘src’, include: ‘src/**’
//generate 1000 parallel block
for (i = 0; i<1000; i++) {
// make the Map of Closure
builds[“$i”] = {
results[“$i”] = 1
data++
if(i%2==0){
mapSync[“odd”] = mapSync[“odd”]++
} else {
mapSync[“even”] = mapSync[“even”]++
}
}
}
parallel builds
println results.toString()
println data
println mapSync
}
}
}
}
}
El sábado, 30 de octubre de 2021 a las 18:44:57 UTC+2, abstrakta escribió:
> Thanks for your reply.
> So parallel directive is like spawning some Java threads?Do you have any
> pipeline code can demo this thread safety issue and how to fix it
> using Java types that are thread safe ?
> I guess that some directive "stash unstash archive" should be thread
> safety internally.Because I find some articles use parallel unstash in
> different slave node without thread protection.Is my guess correct?I can't
> find any other articles that discuss this parallel thread safety issue.
> ---Original---
> *From:* "Ivan Fernandez Calvo"
> *Date:* Sun, Oct 31, 2021 00:03 AM
> *To:* "Jenkins Users";
> *Subject:* Re: thread safety of scripted pipeline parallel and usage of
> nested parallel
>
> No, if you plan to use shared variables across parallel steps you should
> use Java types that are thread safe, if not you will have random/weird
> results. I have several pipelines that uses a map to store results in
> parallel steps
>
> El sábado, 30 de octubre de 2021 a las 14:18:29 UTC+2, abstrakta escribió:
>
>> Hi, Jenkins friends.I wish that I'm in the right place that post these
>> Jenkins usage question.
>>
>> I find that the Scriped Pipeline parallel works like threads.Unlike the
>> Declarative Pipeline parallel,the Scriped Pipeline parallel just use one
>> executor.Closure in parallel parameters works like a thread.
>> My question is:
>>
>> 1.Does Jenkins garantee the data thread safety of parallel closure
>> internally?
>>
>> 2.Does I need to care about the thread safety of the commands that
>> executes in scriped parallel closure?
>>
>> 3.Is there any limit of usage in the commands that executes in
>> parallel?Can I use nested scripted parallel? Why the documentation of
>> Declarative Pipeline parallel in Pipeline Syntax reference says that "Note
>> that a stage must have one and only one of steps, stages, parallel, or
>> matrix. It is not possible to nest a parallel or matrix block within a
>> stage directive if that stage directive is nested within a parallel or
>> matrix block itself."
>>
>> I test some nested Pipeline code that might cause thread race condition
>> many times.Jekins always give the right answer that shared data is modified
>> correctly.Is this thread safety garanteed in the design of Jenkins parallel
>> directive?
>>
>> Pipeline code like this:
>>
>> pipeline{
>> agent any
>> stages
>> {
>> stage('Parallel BuiLd') {
>> steps {
>> script {
>> def i = 0
>> def data = 0
>> def builds = [:]
>> stash name: 'src', include: 'src/**'
>> //generate 1000 parallel
