I'm assuming then that there's no network activity at the end of the 30 minute window. What is the master doing? Perhaps cleaning up old builds? How many do you keep? FYI...I normally keep about 100 per pipeline & branch.
Jeremy Mordkoff Director, Engineering Services [cid:d4bea858-87ac-488e-8153-7355689370c1]<https://dzsi.com> Headquarters: 5700 Tennyson Parkway, Plano, Texas, USA Mobile: +1 978.257.2183 Email: [email protected] [cid:bbb0ed23-5a42-4404-aa60-52d5ceab42c4]<https://dzsi.com> [cid:72ec9354-4241-4492-92dd-eb329950aabe]<https://www.linkedin.com/company/dzsi/>[cid:23de3289-25c5-485c-a4ba-627848162597]<https://twitter.com/dzs_innovation>[cid:99ecbcfa-4523-4d4c-b0af-c151665d5bb4]<https://www.youtube.com/channel/UCgr0fSzluyi0QYaoDgHhkBg> ________________________________ From: [email protected] <[email protected]> on behalf of rssouthw <[email protected]> Sent: Wednesday, December 22, 2021 1:29 PM To: Jenkins Users <[email protected]> Subject: Re: Jenkins jobs not completing CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Nothing like a TCP timeout I think. I forgot to include that during my manual check, I do the same exact ssh /path command, running as the same user as our Jenkins process, from the same host that we have Jenkins running on. It always completes very quickly. lf you click down into the console log of the job itself, you can see it completes (several of our scripts say "done" or "exiting" at the end). At the project page level shows it's complete, but the very top page shows Jenkins still thinking it's running. The job can't be cancelled with the "X". But yes, it has something to do with the job being on the other end of an ssh but it's not clear what. If the check is on the same host, Jenkins works great. Again, we've been using this methodology for years, and it's worked great. It's only been since, oh, 2-3 months the issue has shown up. On Tuesday, December 21, 2021 at 7:56:12 AM UTC-8 [email protected] wrote: 30 minutes sounds like a TCP timeout. Are there any firewalls or NAT devices in the path? Sometimes they close the connection when the first FIN is sent but before the last FIN-ACK and that can cause SSH to hang up. A second possibility is asymetric routes where the replies come in on a different interface then the outgoing packets. In either case, tcpdump can pinpoint the issue. On Monday, December 20, 2021 at 4:27:50 PM UTC-5 [email protected] wrote: We use Jenkins to check that various things are running on our infrastructure hosts. This is typically done with something like: ssh hostname /the/script/to/check If the check exits with a zero status, Jenkins goes on to mark the build as good. Sometime in the last couple months, the script and the ssh exits with a zero status but Jenkins hangs on and does not mark the job as complete for a long time, like close to 30 minutes. If we invoke these manually, there is no issue. The check runs very quickly (under a couple seconds). We've updated Jenkins several times (now on 2.325) in an attempt to fix the issue. Any ideas where to look? Thanks in advance. -- You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/01yoX_YnaGk/unsubscribe<https://groups.google.com/d/topic/jenkinsci-users/01yoX_YnaGk/unsubscribe>. To unsubscribe from this group and all its topics, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ffe78b3b-fb83-4fb6-9108-ff825b10abb5n%40googlegroups.com<https://groups.google.com/d/msgid/jenkinsci-users/ffe78b3b-fb83-4fb6-9108-ff825b10abb5n%40googlegroups.com?utm_medium=email&utm_source=footer>. Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/DM6PR12MB26827332009F972CDF74FD0D837D9%40DM6PR12MB2682.namprd12.prod.outlook.com.
