Re: [go-cd] Kubernetes Elastic Agents - Pods can no longer be created

[Chad Wilson]

Thanks for sharing. Sounds like it could be a network policy thing or
something like that? Did you double check the cluster profile config to see
if anything is amiss there?

I can't see why you'd get a different error message with 23.1.0 (and the
same plugin version) unless there was something different about the cluster
profile config (API server url etc), so that might be an avenue for
investigation?

-Chad

On Sat, 28 Oct 2023, 05:16 Kim Pham,  wrote:

> Hi,
>
> I just wanted to give an update.  We're still unsure what's preventing
> agents from spinning up on our old cluster.  The pods aren't getting
> created at all and there's no logs in Kubernetes side indicating activity.
> I've tried with a regular Debian agent and the logs are the same.  I did
> roll back to GoCD 23.1.0 to see what the log output was and it's showing a
> SocketTimeoutException error.  We're still investigating but it's possible
> some configuration changed that we're not aware of with that cluster.
>
> I was able to spin up a new cluster and it works fine there.  That cluster
> is running:
>
>- GoCD Server: 23.3.0
>- GoCD DinD Agent: 23.3.0
>- Elastic Agent Node Pools: 1.25.10-gk3.2700
>
> I'll update if there's anything relevant to share.  Thanks for all the
> help!
>
> On Thu, Oct 26, 2023 at 10:55 AM Kim Pham  wrote:
>
>> Hi Chad,
>>
>> I was just reading up on the changes for Kubernetes.  Looks 1.24 moves to
>> containerd runtime images and we are still using DIND for our elastic
>> agents.  That seems like it could be the culprit.  I'll do some testing and
>> change our elastic agent images.  Thanks for pointing that out.
>>
>> On Thu, Oct 26, 2023 at 10:46 AM Chad Wilson 
>> wrote:
>>
>>> Unfortunately the error message is a bit mysterious and useless. Which
>>> agent image you are using? Anything special in the elastic agent pod spec
>>> that might no longer work as expected on Kubernetes 1.24? (e.g use of
>>> docker dind images)
>>>
>>> Does the pod get created (and fail) if you look at the events on the
>>> kubernetes side, or does it never get that far?
>>>
>>> -Chad
>>>
>>> On Thu, Oct 26, 2023 at 11:37 PM Chad Wilson 
>>> wrote:
>>>
 Just curious - were the errors/stack traces on failure essentially
 identical before and after you upgraded your gocd and elastic agent plugin
 versions?

 On Thu, Oct 26, 2023 at 11:33 PM Kim Pham  wrote:

> Hi Ashwanth,
>
> I checked the clusterrole of the service account it's using and it
> basically has full access atm.
>
> PolicyRule:
>   Resources   Non-Resource URLs  Resource Names  Verbs
>   -   -  --  -
>   events  [] []  [*]
>   namespaces  [] []  [*]
>   nodes   [] []  [*]
>   pods/log[] []  [*]
>   pods[] []  [*]
>
> On Thu, Oct 26, 2023 at 10:21 AM 'Ashwanth Kumar' via go-cd <
> go-cd@googlegroups.com> wrote:
>
>> A wild guess, anything changed on the service account side or a
>> custom role being added as part of the upgrade that is probably not
>> allowing the gocd plugin to create the pod?
>>
>> Thanks,
>>
>>
>> On Thu, 26 Oct 2023 at 20:27, Kim Pham  wrote:
>>
>>> Hi All,
>>>
>>> We recently began to encounter issues where pods were unable to be
>>> created.  Nothing has changed in terms of GoCD server, agent, and
>>> Kubernetes elastic agent plugin versions.  However, we did notice that 
>>> the
>>> cluster went through an automatic upgrade and updated gke version to
>>> 1.24.14.  GoCD is able to see the node pools through the 'Status Report'
>>> button.
>>>
>>> When attempting to create an agent on those node pools, I do see a
>>> 500 in the plugin logs and gocd-server logs.  Attached are logs.
>>>
>>> I've tried updating GoCD and the plugins to latest release
>>> versions.  Our static agents that are running on older gke versions 
>>> aren't
>>> having any issues.
>>>
>>> Has anyone encountered this?
>>>
>>> Thanks in advance.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to go-cd+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/go-cd/a6b8e99d-f415-4c18-b67d-e86c3df16733n%40googlegroups.com
>>> 
>>> .
>>>
>>
>>
>> --
>>
>> Ashwanth Kumar / ashwanthkumar.in
>>
>> --
>> You received this message because you are subscribed to the Google
>> 

Re: [go-cd] Kubernetes Elastic Agents - Pods can no longer be created

[Kim Pham]

Hi,

I just wanted to give an update.  We're still unsure what's preventing
agents from spinning up on our old cluster.  The pods aren't getting
created at all and there's no logs in Kubernetes side indicating activity.
I've tried with a regular Debian agent and the logs are the same.  I did
roll back to GoCD 23.1.0 to see what the log output was and it's showing a
SocketTimeoutException error.  We're still investigating but it's possible
some configuration changed that we're not aware of with that cluster.

I was able to spin up a new cluster and it works fine there.  That cluster
is running:

   - GoCD Server: 23.3.0
   - GoCD DinD Agent: 23.3.0
   - Elastic Agent Node Pools: 1.25.10-gk3.2700

I'll update if there's anything relevant to share.  Thanks for all the help!

On Thu, Oct 26, 2023 at 10:55 AM Kim Pham  wrote:

> Hi Chad,
>
> I was just reading up on the changes for Kubernetes.  Looks 1.24 moves to
> containerd runtime images and we are still using DIND for our elastic
> agents.  That seems like it could be the culprit.  I'll do some testing and
> change our elastic agent images.  Thanks for pointing that out.
>
> On Thu, Oct 26, 2023 at 10:46 AM Chad Wilson 
> wrote:
>
>> Unfortunately the error message is a bit mysterious and useless. Which
>> agent image you are using? Anything special in the elastic agent pod spec
>> that might no longer work as expected on Kubernetes 1.24? (e.g use of
>> docker dind images)
>>
>> Does the pod get created (and fail) if you look at the events on the
>> kubernetes side, or does it never get that far?
>>
>> -Chad
>>
>> On Thu, Oct 26, 2023 at 11:37 PM Chad Wilson 
>> wrote:
>>
>>> Just curious - were the errors/stack traces on failure essentially
>>> identical before and after you upgraded your gocd and elastic agent plugin
>>> versions?
>>>
>>> On Thu, Oct 26, 2023 at 11:33 PM Kim Pham  wrote:
>>>
 Hi Ashwanth,

 I checked the clusterrole of the service account it's using and it
 basically has full access atm.

 PolicyRule:
   Resources   Non-Resource URLs  Resource Names  Verbs
   -   -  --  -
   events  [] []  [*]
   namespaces  [] []  [*]
   nodes   [] []  [*]
   pods/log[] []  [*]
   pods[] []  [*]

 On Thu, Oct 26, 2023 at 10:21 AM 'Ashwanth Kumar' via go-cd <
 go-cd@googlegroups.com> wrote:

> A wild guess, anything changed on the service account side or a custom
> role being added as part of the upgrade that is probably not allowing the
> gocd plugin to create the pod?
>
> Thanks,
>
>
> On Thu, 26 Oct 2023 at 20:27, Kim Pham  wrote:
>
>> Hi All,
>>
>> We recently began to encounter issues where pods were unable to be
>> created.  Nothing has changed in terms of GoCD server, agent, and
>> Kubernetes elastic agent plugin versions.  However, we did notice that 
>> the
>> cluster went through an automatic upgrade and updated gke version to
>> 1.24.14.  GoCD is able to see the node pools through the 'Status Report'
>> button.
>>
>> When attempting to create an agent on those node pools, I do see a
>> 500 in the plugin logs and gocd-server logs.  Attached are logs.
>>
>> I've tried updating GoCD and the plugins to latest release versions.
>> Our static agents that are running on older gke versions aren't having 
>> any
>> issues.
>>
>> Has anyone encountered this?
>>
>> Thanks in advance.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to go-cd+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/go-cd/a6b8e99d-f415-4c18-b67d-e86c3df16733n%40googlegroups.com
>> 
>> .
>>
>
>
> --
>
> Ashwanth Kumar / ashwanthkumar.in
>
> --
> You received this message because you are subscribed to the Google
> Groups "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/CAD9m7Cw2cK5yt_r9e5r1sxD%2B%2B%2Bjkd3%3DFTQe51vKqb081MunU%3DQ%40mail.gmail.com
> 
> .
>
 --
 You received this message because you are subscribed to the Google
 Groups 

Re: [go-cd] Kubernetes Elastic Agents - Pods can no longer be created

[Kim Pham]

Hi Chad,

I was just reading up on the changes for Kubernetes.  Looks 1.24 moves to
containerd runtime images and we are still using DIND for our elastic
agents.  That seems like it could be the culprit.  I'll do some testing and
change our elastic agent images.  Thanks for pointing that out.

On Thu, Oct 26, 2023 at 10:46 AM Chad Wilson  wrote:

> Unfortunately the error message is a bit mysterious and useless. Which
> agent image you are using? Anything special in the elastic agent pod spec
> that might no longer work as expected on Kubernetes 1.24? (e.g use of
> docker dind images)
>
> Does the pod get created (and fail) if you look at the events on the
> kubernetes side, or does it never get that far?
>
> -Chad
>
> On Thu, Oct 26, 2023 at 11:37 PM Chad Wilson 
> wrote:
>
>> Just curious - were the errors/stack traces on failure essentially
>> identical before and after you upgraded your gocd and elastic agent plugin
>> versions?
>>
>> On Thu, Oct 26, 2023 at 11:33 PM Kim Pham  wrote:
>>
>>> Hi Ashwanth,
>>>
>>> I checked the clusterrole of the service account it's using and it
>>> basically has full access atm.
>>>
>>> PolicyRule:
>>>   Resources   Non-Resource URLs  Resource Names  Verbs
>>>   -   -  --  -
>>>   events  [] []  [*]
>>>   namespaces  [] []  [*]
>>>   nodes   [] []  [*]
>>>   pods/log[] []  [*]
>>>   pods[] []  [*]
>>>
>>> On Thu, Oct 26, 2023 at 10:21 AM 'Ashwanth Kumar' via go-cd <
>>> go-cd@googlegroups.com> wrote:
>>>
 A wild guess, anything changed on the service account side or a custom
 role being added as part of the upgrade that is probably not allowing the
 gocd plugin to create the pod?

 Thanks,


 On Thu, 26 Oct 2023 at 20:27, Kim Pham  wrote:

> Hi All,
>
> We recently began to encounter issues where pods were unable to be
> created.  Nothing has changed in terms of GoCD server, agent, and
> Kubernetes elastic agent plugin versions.  However, we did notice that the
> cluster went through an automatic upgrade and updated gke version to
> 1.24.14.  GoCD is able to see the node pools through the 'Status Report'
> button.
>
> When attempting to create an agent on those node pools, I do see a 500
> in the plugin logs and gocd-server logs.  Attached are logs.
>
> I've tried updating GoCD and the plugins to latest release versions.
> Our static agents that are running on older gke versions aren't having any
> issues.
>
> Has anyone encountered this?
>
> Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google
> Groups "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/a6b8e99d-f415-4c18-b67d-e86c3df16733n%40googlegroups.com
> 
> .
>


 --

 Ashwanth Kumar / ashwanthkumar.in

 --
 You received this message because you are subscribed to the Google
 Groups "go-cd" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to go-cd+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/go-cd/CAD9m7Cw2cK5yt_r9e5r1sxD%2B%2B%2Bjkd3%3DFTQe51vKqb081MunU%3DQ%40mail.gmail.com
 
 .

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to go-cd+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/go-cd/CA%2BnJfx4-Zks6-FOr1bYSOroxG2o4e2e4ir0OFUm3TsggWsrYpA%40mail.gmail.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/CAA1RwH-tw%3D0MFCgWRQoT8ZwFYcyJsGT8T5pPxrBzzu46Nn960w%40mail.gmail.com
> 

Re: [go-cd] Kubernetes Elastic Agents - Pods can no longer be created

[Chad Wilson]

Unfortunately the error message is a bit mysterious and useless. Which
agent image you are using? Anything special in the elastic agent pod spec
that might no longer work as expected on Kubernetes 1.24? (e.g use of
docker dind images)

Does the pod get created (and fail) if you look at the events on the
kubernetes side, or does it never get that far?

-Chad

On Thu, Oct 26, 2023 at 11:37 PM Chad Wilson  wrote:

> Just curious - were the errors/stack traces on failure essentially
> identical before and after you upgraded your gocd and elastic agent plugin
> versions?
>
> On Thu, Oct 26, 2023 at 11:33 PM Kim Pham  wrote:
>
>> Hi Ashwanth,
>>
>> I checked the clusterrole of the service account it's using and it
>> basically has full access atm.
>>
>> PolicyRule:
>>   Resources   Non-Resource URLs  Resource Names  Verbs
>>   -   -  --  -
>>   events  [] []  [*]
>>   namespaces  [] []  [*]
>>   nodes   [] []  [*]
>>   pods/log[] []  [*]
>>   pods[] []  [*]
>>
>> On Thu, Oct 26, 2023 at 10:21 AM 'Ashwanth Kumar' via go-cd <
>> go-cd@googlegroups.com> wrote:
>>
>>> A wild guess, anything changed on the service account side or a custom
>>> role being added as part of the upgrade that is probably not allowing the
>>> gocd plugin to create the pod?
>>>
>>> Thanks,
>>>
>>>
>>> On Thu, 26 Oct 2023 at 20:27, Kim Pham  wrote:
>>>
 Hi All,

 We recently began to encounter issues where pods were unable to be
 created.  Nothing has changed in terms of GoCD server, agent, and
 Kubernetes elastic agent plugin versions.  However, we did notice that the
 cluster went through an automatic upgrade and updated gke version to
 1.24.14.  GoCD is able to see the node pools through the 'Status Report'
 button.

 When attempting to create an agent on those node pools, I do see a 500
 in the plugin logs and gocd-server logs.  Attached are logs.

 I've tried updating GoCD and the plugins to latest release versions.
 Our static agents that are running on older gke versions aren't having any
 issues.

 Has anyone encountered this?

 Thanks in advance.

 --
 You received this message because you are subscribed to the Google
 Groups "go-cd" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to go-cd+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/go-cd/a6b8e99d-f415-4c18-b67d-e86c3df16733n%40googlegroups.com
 
 .

>>>
>>>
>>> --
>>>
>>> Ashwanth Kumar / ashwanthkumar.in
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to go-cd+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/go-cd/CAD9m7Cw2cK5yt_r9e5r1sxD%2B%2B%2Bjkd3%3DFTQe51vKqb081MunU%3DQ%40mail.gmail.com
>>> 
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to go-cd+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/go-cd/CA%2BnJfx4-Zks6-FOr1bYSOroxG2o4e2e4ir0OFUm3TsggWsrYpA%40mail.gmail.com
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CAA1RwH-tw%3D0MFCgWRQoT8ZwFYcyJsGT8T5pPxrBzzu46Nn960w%40mail.gmail.com.

Re: [go-cd] Kubernetes Elastic Agents - Pods can no longer be created

[Chad Wilson]

Just curious - were the errors/stack traces on failure essentially
identical before and after you upgraded your gocd and elastic agent plugin
versions?

On Thu, Oct 26, 2023 at 11:33 PM Kim Pham  wrote:

> Hi Ashwanth,
>
> I checked the clusterrole of the service account it's using and it
> basically has full access atm.
>
> PolicyRule:
>   Resources   Non-Resource URLs  Resource Names  Verbs
>   -   -  --  -
>   events  [] []  [*]
>   namespaces  [] []  [*]
>   nodes   [] []  [*]
>   pods/log[] []  [*]
>   pods[] []  [*]
>
> On Thu, Oct 26, 2023 at 10:21 AM 'Ashwanth Kumar' via go-cd <
> go-cd@googlegroups.com> wrote:
>
>> A wild guess, anything changed on the service account side or a custom
>> role being added as part of the upgrade that is probably not allowing the
>> gocd plugin to create the pod?
>>
>> Thanks,
>>
>>
>> On Thu, 26 Oct 2023 at 20:27, Kim Pham  wrote:
>>
>>> Hi All,
>>>
>>> We recently began to encounter issues where pods were unable to be
>>> created.  Nothing has changed in terms of GoCD server, agent, and
>>> Kubernetes elastic agent plugin versions.  However, we did notice that the
>>> cluster went through an automatic upgrade and updated gke version to
>>> 1.24.14.  GoCD is able to see the node pools through the 'Status Report'
>>> button.
>>>
>>> When attempting to create an agent on those node pools, I do see a 500
>>> in the plugin logs and gocd-server logs.  Attached are logs.
>>>
>>> I've tried updating GoCD and the plugins to latest release versions.
>>> Our static agents that are running on older gke versions aren't having any
>>> issues.
>>>
>>> Has anyone encountered this?
>>>
>>> Thanks in advance.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "go-cd" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to go-cd+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/go-cd/a6b8e99d-f415-4c18-b67d-e86c3df16733n%40googlegroups.com
>>> 
>>> .
>>>
>>
>>
>> --
>>
>> Ashwanth Kumar / ashwanthkumar.in
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to go-cd+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/go-cd/CAD9m7Cw2cK5yt_r9e5r1sxD%2B%2B%2Bjkd3%3DFTQe51vKqb081MunU%3DQ%40mail.gmail.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/CA%2BnJfx4-Zks6-FOr1bYSOroxG2o4e2e4ir0OFUm3TsggWsrYpA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CAA1RwH9L%3DmBaJbvnXDfJ0d3J2_GuhC%2B38y4sMUiaMPDgAEd__Q%40mail.gmail.com.

Re: [go-cd] Kubernetes Elastic Agents - Pods can no longer be created

[Kim Pham]

Hi Ashwanth,

I checked the clusterrole of the service account it's using and it
basically has full access atm.

PolicyRule:
  Resources   Non-Resource URLs  Resource Names  Verbs
  -   -  --  -
  events  [] []  [*]
  namespaces  [] []  [*]
  nodes   [] []  [*]
  pods/log[] []  [*]
  pods[] []  [*]

On Thu, Oct 26, 2023 at 10:21 AM 'Ashwanth Kumar' via go-cd <
go-cd@googlegroups.com> wrote:

> A wild guess, anything changed on the service account side or a custom
> role being added as part of the upgrade that is probably not allowing the
> gocd plugin to create the pod?
>
> Thanks,
>
>
> On Thu, 26 Oct 2023 at 20:27, Kim Pham  wrote:
>
>> Hi All,
>>
>> We recently began to encounter issues where pods were unable to be
>> created.  Nothing has changed in terms of GoCD server, agent, and
>> Kubernetes elastic agent plugin versions.  However, we did notice that the
>> cluster went through an automatic upgrade and updated gke version to
>> 1.24.14.  GoCD is able to see the node pools through the 'Status Report'
>> button.
>>
>> When attempting to create an agent on those node pools, I do see a 500 in
>> the plugin logs and gocd-server logs.  Attached are logs.
>>
>> I've tried updating GoCD and the plugins to latest release versions.  Our
>> static agents that are running on older gke versions aren't having any
>> issues.
>>
>> Has anyone encountered this?
>>
>> Thanks in advance.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to go-cd+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/go-cd/a6b8e99d-f415-4c18-b67d-e86c3df16733n%40googlegroups.com
>> 
>> .
>>
>
>
> --
>
> Ashwanth Kumar / ashwanthkumar.in
>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/CAD9m7Cw2cK5yt_r9e5r1sxD%2B%2B%2Bjkd3%3DFTQe51vKqb081MunU%3DQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CA%2BnJfx4-Zks6-FOr1bYSOroxG2o4e2e4ir0OFUm3TsggWsrYpA%40mail.gmail.com.

Re: [go-cd] Kubernetes Elastic Agents - Pods can no longer be created

['Ashwanth Kumar' via go-cd]

A wild guess, anything changed on the service account side or a custom role
being added as part of the upgrade that is probably not allowing the gocd
plugin to create the pod?

Thanks,


On Thu, 26 Oct 2023 at 20:27, Kim Pham  wrote:

> Hi All,
>
> We recently began to encounter issues where pods were unable to be
> created.  Nothing has changed in terms of GoCD server, agent, and
> Kubernetes elastic agent plugin versions.  However, we did notice that the
> cluster went through an automatic upgrade and updated gke version to
> 1.24.14.  GoCD is able to see the node pools through the 'Status Report'
> button.
>
> When attempting to create an agent on those node pools, I do see a 500 in
> the plugin logs and gocd-server logs.  Attached are logs.
>
> I've tried updating GoCD and the plugins to latest release versions.  Our
> static agents that are running on older gke versions aren't having any
> issues.
>
> Has anyone encountered this?
>
> Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "go-cd" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to go-cd+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/go-cd/a6b8e99d-f415-4c18-b67d-e86c3df16733n%40googlegroups.com
> 
> .
>


-- 

Ashwanth Kumar / ashwanthkumar.in

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/CAD9m7Cw2cK5yt_r9e5r1sxD%2B%2B%2Bjkd3%3DFTQe51vKqb081MunU%3DQ%40mail.gmail.com.

[go-cd] Kubernetes Elastic Agents - Pods can no longer be created

[Kim Pham]

Hi All,

We recently began to encounter issues where pods were unable to be 
created.  Nothing has changed in terms of GoCD server, agent, and 
Kubernetes elastic agent plugin versions.  However, we did notice that the 
cluster went through an automatic upgrade and updated gke version to 
1.24.14.  GoCD is able to see the node pools through the 'Status Report' 
button.  

When attempting to create an agent on those node pools, I do see a 500 in 
the plugin logs and gocd-server logs.  Attached are logs.  

I've tried updating GoCD and the plugins to latest release versions.  Our 
static agents that are running on older gke versions aren't having any 
issues.

Has anyone encountered this?

Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/go-cd/a6b8e99d-f415-4c18-b67d-e86c3df16733n%40googlegroups.com.
java.lang.RuntimeException: The plugin sent a response that could not be 
understood by Go. Plugin returned with code '500' and the following response: 
'Failed to handle request cd.go.elastic-agent.create-agent'
at 
com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:58)
at 
com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.createAgent(ElasticAgentExtensionV5.java:142)
at 
com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.createAgent(ElasticAgentExtension.java:59)
at 
com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.createAgent(ElasticAgentPluginRegistry.java:42)
at 
com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:31)
at 
com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:21)
at 
com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83)
at 
com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63)
at java.base/java.lang.Thread.run(Unknown Source)2023-10-26 14:38:28,825 ERROR [237@MessageListener for CreateAgentListener] 
KubernetesPlugin:127 - Failed to handle request cd.go.elastic-agent.create-agent
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
at 
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129)
at 
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122)
at 
io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:44)
at 
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:1142)
at 
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:97)
at 
cd.go.contrib.elasticagent.KubernetesInstanceFactory.createKubernetesPod(KubernetesInstanceFactory.java:151)
at 
cd.go.contrib.elasticagent.KubernetesInstanceFactory.createUsingPodYaml(KubernetesInstanceFactory.java:246)
at 
cd.go.contrib.elasticagent.KubernetesInstanceFactory.create(KubernetesInstanceFactory.java:59)
at 
cd.go.contrib.elasticagent.KubernetesAgentInstances.createKubernetesInstance(KubernetesAgentInstances.java:91)
at 
cd.go.contrib.elasticagent.KubernetesAgentInstances.create(KubernetesAgentInstances.java:65)
at 
cd.go.contrib.elasticagent.KubernetesAgentInstances.create(KubernetesAgentInstances.java:36)
at 
cd.go.contrib.elasticagent.executors.CreateAgentRequestExecutor.execute(CreateAgentRequestExecutor.java:55)
at 
cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:82)
at 
com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:134)
at 
com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:205)
at 
com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:164)
at 
com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:131)
at 
com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
at 
com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.createAgent(ElasticAgentExtensionV5.java:142)
at 
com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.createAgent(ElasticAgentExtension.java:59)
at 
com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.createAgent(ElasticAgentPluginRegistry.java:42)
at