Hello Chad, What did you change around the time this behaviour started happening? -GoCD = no -Gitlab = upgrade from 16.2.0 to 16.2.3
Which container image for the server are you using? (only gocd-server-centos-9 <https://hub.docker.com/r/gocd/gocd-server-centos-9/> is built for ARM64 - if you are trying to run emulated you will probably have many problems) - I am using official gocd/gocd-server-centos-9:v23.1.0 What are the CPU requests and limits that you have assigned to the gocd server pod? And are you deploying with the standard GoCD helm chart? If the limits are too low, or the requests are low and other processes on the same node are using too much CPU you can still end up with CPU starvation, even if it looks like the pod isn't using much CPU, because it may be throttled. - I am using officail helm chart v 2.1.6. and no limit cpu and memory and set request CPU and Memory resources: requests: memory: 2048Mi cpu: 1000m PS. GoCD server is running on c6g.large instance type and this below is top node result. NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% ip-xx-xxx-xx-xx.ap-southeast-1.compute.internal 74m 3% 2570Mi 81% Best Regards, Komgrit On Tuesday, August 8, 2023 at 4:11:18 PM UTC+7 Chad Wilson wrote: > If you are getting logs like that, it sounds like the container is > experiencing CPU starvation. > > - What did you change around the time this behaviour started happening? > - Which container image for the server are you using? (only > gocd-server-centos-9 > <https://hub.docker.com/r/gocd/gocd-server-centos-9/> is built for > ARM64 - if you are trying to run emulated you will probably have many > problems) > - What are the CPU requests and limits that you have assigned to the > gocd server pod? And are you deploying with the standard GoCD helm chart? > If the limits are too low, or the requests are low and other processes on > the same node are using too much CPU you can still end up with CPU > starvation, even if it looks like the pod isn't using much CPU, because it > may be throttled. > > -Chad > > On Tue, Aug 8, 2023 at 4:06 PM Komgrit Aneksri <[email protected]> wrote: > >> Hello Chad, >> >> Thank you for your advice. >> >> I tried run git clone with debug environment variables command in gocd >> server pod. So the command run successfully and there is no any abnormal >> log. >> >> And CPU and memory are not high usage. >> >> We tried restart gocd server pods many time but it is not help. >> >> And I dig to see timeout logs. There is log messages. >> INFO | wrapper | 2023/08/08 07:43:48 | Wrapper Process has not >> received any CPU time for 22 seconds. Extending timeouts. >> >> Best Regards, >> Komgrit >> On Tuesday, August 8, 2023 at 12:16:45 PM UTC+7 Chad Wilson wrote: >> >>> That error message is not relevant to the issue, you can ignore it. Are >>> there other errors or timeouts in the logs? >>> >>> To refresh config repos, GoCD forks regular git processes to clone and >>> then fetches. You might want to exec into the container and see what these >>> processes are doing (high CPU? stuck somehow?) >>> >>> There are also a few general suggestions on similar issues around which >>> you might want to check: >>> >>> - https://github.com/gocd/gocd/issues/10480 >>> - https://github.com/gocd/gocd/issues/9588 >>> - https://github.com/gocd/gocd/issues/8565 >>> >>> If this is happening after a restart of the GoCD server, or due to some >>> other change you've made it's possible Gitlab is throttling the requests. >>> >>> Depending on whether you are using https or ssh connections, you may >>> want to use standard git environment variables to debug what's happening >>> (GIT_TRACE=1, >>> GIT_CURL_VERBOSE=1 etc). >>> >>> -Chad >>> >>> On Tue, Aug 8, 2023 at 12:29 PM Komgrit Aneksri <[email protected]> >>> wrote: >>> >>>> Hello GoCD team, >>>> >>>> I am facing issue in Config Repositories page. >>>> >>>> So All Config Repositories are always in fetching status for >>>> Gitlab(16.2.3) >>>> Our GoCD is 23.1.0 version on kubernetes and running ARM64. >>>> >>>> In server log, There is error message >>>> >>>> "jvm 1 | 2023-08-08 04:18:17,151 ERROR [qtp1962126505-38] >>>> VariableReplacer:385 - function ${escape:} type 'escape:' not a valid >>>> type" >>>> >>>> when i did refresh or open the Config Repositories page. >>>> [image: 1691468556134.jpg] >>>> >>>> Please help us for fix it. >>>> >>>> Best Regards, >>>> Komgrit >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "go-cd" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/af5f5dd4-6f27-47d3-8eb3-1a19c0675ae7n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/go-cd/af5f5dd4-6f27-47d3-8eb3-1a19c0675ae7n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "go-cd" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/f5b73449-c62b-426c-8bf6-1a4911a8d521n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/go-cd/f5b73449-c62b-426c-8bf6-1a4911a8d521n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/665e02e9-8b59-4573-8e3f-97a6366d180cn%40googlegroups.com.
