[JIRA] (JENKINS-59705) hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from IP/IP:58344 failed. The channel is closing down or has closed dow
Title: Message Title Karol Gil commented on JENKINS-59705 Re: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from IP/IP:58344 failed. The channel is closing down or has closed dow Hey Eddie Mashayev, thanks for the response. I've been fighting this one for a while now as well and can confirm that your "not related" section is correct - we did both changes and issues are still being observed once in a while. Our monitoring shows that root volumes are far from full in any of the nodes being used for running our jobs so I doubt it's related - maybe the symptom is similar? I think it may be related to autoscaling as you said - we're observing this mostly in jobs that are using specific autoscaling group that has default capacity set to 0 and in peaks scales up to 80 nodes - this is when issue is most common. What bugs be is the fact that I can't track the hostnames that are listed in build log - these machines are not defined in AWS nor can I see them in autoscaler logs. By any chance - did you manage to reproduce that effectively? Or it appears to be "random"? Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202406.1570547862000.17921.1571813880130%40Atlassian.JIRA.
[JIRA] (JENKINS-59705) hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from IP/IP:58344 failed. The channel is closing down or has closed dow
Title: Message Title Karol Gil commented on JENKINS-59705 Re: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from IP/IP:58344 failed. The channel is closing down or has closed dow Eddie Mashayev did this stop for you after you increased resources? We're observing these failures on a daily basis and in all cases it's trying to connect to non-existing node hostnames. We're running on demand worker nodes on EKS (no spots used). According to our monitoring JNLP container never uses more than 1.2 GB RAM and ~0.8 CPU, hence I doubt it's because of resources. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.202406.1570547862000.13175.1571752621928%40Atlassian.JIRA.
[JIRA] (JENKINS-58540) Console output of pipeline extra stars
Title: Message Title Karol Gil commented on JENKINS-58540 Re: Console output of pipeline extra stars Jesse Glick we have a set of secrets assigned to pods depending on the environment which they are testing. If given service does not exist on certain environment secret for it is empty. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.200689.1563433318000.2886.1564405980252%40Atlassian.JIRA.
[JIRA] (JENKINS-58540) Console output of pipeline extra stars
Title: Message Title Karol Gil commented on JENKINS-58540 Re: Console output of pipeline extra stars I've added a PR with fix proposal, feel free to comment/feedback there: https://github.com/jenkinsci/kubernetes-plugin/pull/558 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.200689.1563433318000.2649.1564398480197%40Atlassian.JIRA.
[JIRA] (JENKINS-55392) java.util.concurrent.RejectedExecutionException AGAIN
Title: Message Title Karol Gil commented on JENKINS-55392 Re: java.util.concurrent.RejectedExecutionException AGAIN I'm not sure if this is the case - the fact is that we saw RejectedExecutionException and log about increasing max connection, but of course it might be weird failure mode. I posted it here as maybe it'll be helpful to someone in similar environment. For now I won't open another issue as it seems to be resolved. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.196500.1546530449000.2003.1564163100919%40Atlassian.JIRA.
[JIRA] (JENKINS-55392) java.util.concurrent.RejectedExecutionException AGAIN
Title: Message Title
Karol Gil commented on JENKINS-55392
Re: java.util.concurrent.RejectedExecutionException AGAIN
I think I solved it in my case, maybe it'll help others*.* TL;DR; If your API tokens have limited lifetime then decrease cache expiry time. As documented in README of this plugin EKS (it's our case) clusters use aws-iam-authenticator and tokens grantend to kubernetes clients have limited lifetime (15 minutes). The default expiry time for clients in the plugin is 60 minutes which means that plugin can create and cache client, which will become unauthorized in 15 minutes. I've decrease cache expiry to 15 seconds and clients purge time to 120 seconds (to avoid keeping all those expired in memory) via system properties: {{ -Dorg.csanchez.jenkins.plugins.kubernetes.clients.expiredClientsPurgeTime=120}} {{ -Dorg.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=30}}
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.196500.1546530449000.1859.1564152602057%40Atlassian.JIRA.
[JIRA] (JENKINS-58540) Console output of pipeline extra stars
Title: Message Title Karol Gil commented on JENKINS-58540 Re: Console output of pipeline extra stars I meant using system properties - won't clutter the UI but will still be an option. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.200689.1563433318000.1813.1564152000627%40Atlassian.JIRA.
[JIRA] (JENKINS-58540) Console output of pipeline extra stars
Title: Message Title Karol Gil commented on JENKINS-58540 Re: Console output of pipeline extra stars I think it still might not be enough. The problem is IMO that your secret may be common for example `kubernetes` and then making it into `***` can easily give away your secret anyway - as you'll find many like that in the logs. But this is fixable on end user side of things, so I agree, having even length > 0 would be great or even better - to be configurable. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.200689.1563433318000.1763.1564147500175%40Atlassian.JIRA.
[JIRA] (JENKINS-58540) Console output of pipeline extra stars
Title: Message Title
Karol Gil edited a comment on JENKINS-58540
Re: Console output of pipeline extra stars
I think the problem lies in `SecretsMasker` class. Here, it tries to replace every value kept in `values` set with bunch of asterisks: [https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/SecretsMasker.java#L74]But what is not takes into account I assume, is that some of strings kept in `values` field may be empty string "" causing this line be like:{{ s = s.replace("", "");}}which actually gives away the secret value :) I'm not sure how to implement it properly, but this assumes that secrets are not obvious strings that will be replaced.
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.200689.1563433318000.1691.1564142100198%40Atlassian.JIRA.
[JIRA] (JENKINS-58540) Console output of pipeline extra stars
Title: Message Title
Karol Gil edited a comment on JENKINS-58540
Re: Console output of pipeline extra stars
I think the problem lies in `SecretsMasker` class. Here, it tries to replace every value kept in `values` set with bunch of asterisks: [ https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/SecretsMasker.java#L74 ] But what is not takes into account I assume, is that some of strings kept in `values` field may be empty string "" causing this line be like: ``` s = s.replace("", ""); ``` which actually gives away the secret value :)
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.200689.1563433318000.1677.1564141980961%40Atlassian.JIRA.
[JIRA] (JENKINS-58540) Console output of pipeline extra stars
Title: Message Title
Karol Gil edited a comment on JENKINS-58540
Re: Console output of pipeline extra stars
I think the problem lies in `SecretsMasker` class. Here, it tries to replace every value kept in `values` set with bunch of asterisks: [https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/SecretsMasker.java#L74]But what is not takes into account I assume, is that some of strings kept in `values` field may be empty string "" causing this line be like:{{ s = s.replace("", "");}}which actually gives away the secret value :)
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.200689.1563433318000.1685.1564141981200%40Atlassian.JIRA.
[JIRA] (JENKINS-58540) Console output of pipeline extra stars
Title: Message Title
Karol Gil edited a comment on JENKINS-58540
Re: Console output of pipeline extra stars
I think the problem lies in `SecretsMasker` class. Here, it tries to replace every value kept in `values` set with bunch of asterisks: [https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/SecretsMasker.java#L74]But what is not takes into account I assume, is that some of strings kept in `values` field may be empty string "" causing this line be like: {{ s = s.replace("", ""); }} which actually gives away the secret value :)
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.200689.1563433318000.1683.1564141981154%40Atlassian.JIRA.
[JIRA] (JENKINS-58540) Console output of pipeline extra stars
Title: Message Title
Karol Gil commented on JENKINS-58540
Re: Console output of pipeline extra stars
I think the problem lies in `SecretsMasker` class. Here, it tries to replace every value kept in `values` set with bunch of asterisks: https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/SecretsMasker.java#L74 But what is not takes into account I assume, is that some of strings kept in `values` field may be empty string "" causing this line be like: ``` s = s.replace("", ""); ``` which actually gives away the secret value
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.200689.1563433318000.1667.1564141921182%40Atlassian.JIRA.
[JIRA] (JENKINS-55392) java.util.concurrent.RejectedExecutionException AGAIN
Title: Message Title Karol Gil commented on JENKINS-55392 Re: java.util.concurrent.RejectedExecutionException AGAIN We are running into the same issue with version 1.17.2 with no `containerLog` steps used - every time is happens it's on `sh` step. Increasing max connections, even to absurd numbers (100k or more) does not help at all. If I can provide any logs or information that could be useful in finally solving that please let me know. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.196500.1546530449000.972.1564070401023%40Atlassian.JIRA.
[JIRA] (JENKINS-56735) Builds hanging after pod start in version 1.14.9
Title: Message Title Karol Gil commented on JENKINS-56735 Re: Builds hanging after pod start in version 1.14.9 We thought so and we tested fixed version 1.15.5 last week. Unfortunately, we still a lot of Interrupted while waiting for websocket connection, you should increase the Max connections to Kubernetes API issues in logs. We reverted to 1.14.9 again and everything works smoothly once again. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.198376.1553505622000.19735.1559584560381%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-56939) pipeline gets stuck: root cause unclear
Title: Message Title Karol Gil commented on JENKINS-56939 Re: pipeline gets stuck: root cause unclear Did you try upgrading the property that is mentioned in the error message? I've also encountered that for pipelines that are running ~100 pods but when we run Jenkins with -D org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator.websocketConnectionTimeout=60 the error no longer appears and pipelines are running normally. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.198624.1554752568000.14892.1559122740208%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-57213) Blue Ocean changes nodes from red to green after some time
Title: Message Title Karol Gil updated an issue Jenkins / JENKINS-57213 Blue Ocean changes nodes from red to green after some time Change By: Karol Gil We have a pipeline running one big `parallel` block in it. Each branch of the `parallel` uses a `build` step to call some downstream job. When one of the downstream builds fail it properly shows it as a failed stage in Blue Ocean UI (see "failed" in attachment). But then after some time (1 hour, a couple of hours, sometimes a day), it turns all red nodes into green (see "failed-but-green" attachment). The same thing happens for `parallel` without `build` step in it but for some reason, it occurs less often. It does not happen in all pipelines but happens sometimes for the largest of them. Status of the build itself is still displayed properly (as a failure) and as you can see in "stages" attachment substages of a given Parallel branch are still properly marked as failed. Only the Blue Ocean node becomes green for some reason.If I can provide any more details/logs/information, please let me know. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options,
[JIRA] (JENKINS-57213) Blue Ocean changes nodes from red to green after some time
Title: Message Title Karol Gil created an issue Jenkins / JENKINS-57213 Blue Ocean changes nodes from red to green after some time Issue Type: Bug Assignee: Unassigned Attachments: failed-but-green.png, failed.png, stages.png Components: blueocean-plugin Created: 2019-04-29 08:10 Environment: Jenkins 2.164.1 BlueOcean plugin 1.14.0 Labels: blueocean parallel downstream Priority: Major Reporter: Karol Gil We have a pipeline running one big `parallel` block in it. Each branch of the `parallel` uses a `build` step to call some downstream job. When one of the downstream builds fail it properly shows it as a failed stage in Blue Ocean UI (see "failed" in attachment). But then after some time (1 hour, a couple of hours, sometimes a day), it turns all red nodes into green (see "failed-but-green" attachment). It does not happen in all pipelines but happens sometimes for the largest of them. Status of the build itself is still displayed properly (as a failure) and as you can see in "stages" attachment substages of a given Parallel branch are still properly marked as failed. Only the Blue Ocean node becomes green for some reason. If I can provide any more details/logs/information, please let me know.
[JIRA] (JENKINS-56735) Builds hanging after pod start in version 1.14.9
Title: Message Title
Karol Gil updated an issue
Jenkins / JENKINS-56735
Builds hanging after pod start in version 1.14.9
Change By:
Karol Gil
We updated kubernetes plugin from version 1.14.3 to version 1.14.9. On version 1.14.3 everything was running smoothly and after upgrade to 1.14.9 when Jenkins is under heavy load (starting/running ~100 jobs/pods) we observe that builds are stuck right after pod start or after Git checkout (first step of our test pipelines).We have a step timeout after 30 minutes and those jobs that were stuck could not be killed and was stuck with the following log: {code:java} > git checkout -f 073a008e8e8fdad44c3d637a8b9e5995277724aeCancelling nested steps due to timeoutBody did not finish within grace period; terminating with extreme prejudice{code}Only hard kill with calling {{POST BUILD_URL/kill}} did stop the build.What is interesting, sometimes those builds did fail and asked for changing max connections to k8s API{code:java}Caused: java.io.IOException: Interrupted while waiting for websocket connection, you should increase the Max connections to Kubernetes API{code}We increased the setting gradually to up to 6 (sic!) and it did not solve the issue in any way. On other Jenkins instances running under not so heavy load there is no sign of the issue whatsoever. Can it be somehow related to expiring kubernetes clients introduced in 1.14.5?I'm guessing that may be the case, because everything is working smoothly after Jenkins restart for some time (~24 hours usually, but we had shorter time frames as well) and then everything for stuck. After killing all jobs and restarting Jenkins everything went back to normal - again for finite time. Downgrade to version 1.14.3 solved all the issues.Please let me know if any additional information should be provided.
Add Comment
[JIRA] (JENKINS-56735) Builds hanging after pod start in version 1.14.9
Title: Message Title Karol Gil created an issue Jenkins / JENKINS-56735 Builds hanging after pod start in version 1.14.9 Issue Type: Bug Assignee: Carlos Sanchez Components: kubernetes-plugin Created: 2019-03-25 09:20 Environment: EKS version 1.11.8 Jenkins version 2.164.1 Kubernetes Plugin version 1.14.9 Java version 8 Jenkins installation from Helm chart Jenkins running with following JVM args: -Dkubernetes.websocket.ping.interval=3 -Dkubernetes.websocket.timeout=1 These were introduced due to reoccurring socket timeouts and did solve the issue. Labels: kuberenetes-plugin kubernetes plugin jenkins Priority: Major Reporter: Karol Gil We updated kubernetes plugin from version 1.14.3 to version 1.14.9. On version 1.14.3 everything was running smoothly and after upgrade to 1.14.9 when Jenkins is under heavy load (starting/running ~100 jobs/pods) we observe that builds are stuck right after pod start or after Git checkout (first step of our test pipelines). We have a step timeout after 30 minutes and those jobs that were stuck could not be killed and was stuck with the following log: > git checkout -f 073a008e8e8fdad44c3d637a8b9e5995277724ae Cancelling nested steps due to timeout Body did not finish within grace period; terminating with extreme prejudice
