Hi Ash,
Forgive me but can you tell me where I can find the agent process logs?
When the agent dies and a new one is spun up, I cant access the agent
status page...its only for the new one that spun up.
In the server logs I see this: Doesnt give me any reason on why it hung
and ultimately started a new pod. I want to stress that this is random
though. I have 30 pipelines and all are using the same agent profile I
have defined for each stage.
Some keep spawning in various stages and there is no predictive reason why.
When I check the Cluster, memory is low and CPU is low.
I'm using C6i.2xlarge in a 5 node cluster. I was using R4.xlarge and didn't
see this issue but can attribute any of these issues to the instance change
which I did a few weeks ago.
2022-04-12 13:57:05,540 WARN [ThreadPoolTaskScheduler-1]
ScheduleService:611 - Found hung job[id=JobInstance{stageId=184,
name='PLATFORM_DEPLOY_JOB', state=Building, result=Unknown,
agentUuid='3ff453d7-6a6b-413f-a845-728d96eec351', stateTransitions=[],
scheduledDate
=Tue Apr 12 13:46:05 UTC 2022,
timeProvider=com.thoughtworks.go.util.TimeProvider@715bdc39, ignored=false,
identifier=JobIdentifier[Platform-Deploy, 2, 2, platform_deploy_qa, 5,
PLATFORM_DEPLOY_JOB, 353], plan=null, runOnAllAgents=false,
runMultipleInstance=false}],
rescheduling it
On Tuesday, April 12, 2022 at 9:41:12 AM UTC-4 [email protected] wrote:
> This behaviour of GoCD usually points to Agent process dying mid-way and
> GoCD automatically re-assign the work to another agent and they would start
> from scratch. Can you check the agent process logs for the earlier runs to
> see if there are any exceptions that might have caused the GoCD Server to
> reassign the process to another agent?
>
> Sometimes it could be the pipeline itself that's killing the agent process
> for a variety of reasons.
>
> On Tue, 12 Apr 2022 at 19:02, Sifu Tian <[email protected]> wrote:
>
>> [image: Screen Shot 2022-04-12 at 9.21.28 AM.png]Hi all,
>>
>> I have some unusual behavior that is happening on random pipelines.
>> When the pipeline runs, it will run fine but the job will get to a
>> certain point and start all over again pulling materials and running the
>> same task. The first task appears to hang or just stops and a new but same
>> job is run. The pipeline never fails it just continues to run and it will
>> spawn the same job over and over. On the K8 cluster status page, it will
>> only show one pod but in the console, it will show a new pod was issued.
>>
>> I am using the Kubernetes elastic agent plugin
>> GoCD Server and agent are at 22.1
>>
>> Any thoughts or help would be greatly appreciated.[image: Screen Shot
>> 2022-04-12 at 9.23.20 AM.png]
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/go-cd/3bf59b24-31f1-4445-be9e-a2ba6606d396n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/go-cd/3bf59b24-31f1-4445-be9e-a2ba6606d396n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
>
> --
>
> Ashwanth Kumar / ashwanthkumar.in
>
>
--
You received this message because you are subscribed to the Google Groups
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/go-cd/abd08e77-15d9-484e-828f-a4f89feadb64n%40googlegroups.com.
