[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title Jeff Thompson closed an issue as Not A Defect Closing this again as it appears to be Not a Defect, Duplicate, or Cannot Reproduce and no further information has been provided or explanation beyond the existing workarounds. Jenkins / JENKINS-48106 java.io.EOFException continue occur when using health check with aws network load balancer Change By: Jeff Thompson Status: Reopened Closed Resolution: Not A Defect Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title Jeff Thompson commented on JENKINS-48106 Re: java.io.EOFException continue occur when using health check with aws network load balancer Looking at other reports, it's not clear that downgrading or ignoring this error is the correct thing to do in all environments. In some environments this message looks like it provides meaningful information. It sounds like Owen was correct initially, this is a duplicate of JENKINS-46893. Without further ideas I don't know what we could do here and sounds like there are some possibilities for addressing this in the system deployment configuration. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title Jeff Thompson commented on JENKINS-48106 Re: java.io.EOFException continue occur when using health check with aws network load balancer Interesting. Thanks for the analysis, Aaron Trout. If we could figure out the appropriate behavior, we could do something about it. Maybe just lowering the level for that log message would be the right thing to do. I've done that for a number of messages lately that were overly aggressive when they were first implemented. I don't know if there are any situations in which that message might be useful, though. Perhaps, log it only if some specific condition is met. Maybe ignore the EOFException on the attempt to read the header but not if it comes later. I'm inclined to do that to clean up these spammy messages in these environments. Any thoughts? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title Jeff Thompson reopened an issue Jenkins / JENKINS-48106 java.io.EOFException continue occur when using health check with aws network load balancer Change By: Jeff Thompson Resolution: Cannot Reproduce Status: Closed Reopened Assignee: Jeff Thompson Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title Aaron Trout commented on JENKINS-48106 Re: java.io.EOFException continue occur when using health check with aws network load balancer Sorry for posting three comments in a row, but to summarise; potentially not a jenkins bug. It doesn't break anything, just the log is super spammy in certain environments... Could raise the log level to work around that if it is a problem for you. Arguably though, the case where a TCP connection is established but no data is sent should be handled by the agent port listener so that we don't trigger this exception... Either way though, from jenkins point of view this is nothing to do with kubernetes or load balancers. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title chris vest commented on JENKINS-48106 Re: java.io.EOFException continue occur when using health check with aws network load balancer I am seeing the same issue as Aaron Trout Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title
Aaron Trout edited a comment on JENKINS-48106
Re: java.io.EOFException continue occur when using health check with aws network load balancer
By the way, a *MUCH* easier way to reproduce this is to just use netcat against the JNLP port: Run a local Jenkins: {code:java}$ docker run -p 5:5 jenkins/jenkins:lts{code}Wait for the "Jenkins is fully up and running" message, then in another terminal:{code:java}$ echo | nc my-jenkins-hostname 8090 localhost 5 {code}causes:{code:java} Nov 21, 2018 11:50:20 AM hudson.TcpSlaveAgentListener$ConnectionHandler run WARNING: Connection # 2644 1 failedjava.io.EOFExceptionat java.io.DataInputStream.readFully(DataInputStream.java:197)at java.io.DataInputStream.readFully(DataInputStream.java:169)at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:242){code}
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title Aaron Trout commented on JENKINS-48106 Re: java.io.EOFException continue occur when using health check with aws network load balancer By the way, a MUCH easier way to reproduce this is to just use netcat against the JNLP port: $ echo | nc jenkins-nm-jenkins 8090 causes: Nov 21, 2018 11:50:20 AM hudson.TcpSlaveAgentListener$ConnectionHandler run WARNING: Connection #2644 failed java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:242) Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title
Aaron Trout edited a comment on JENKINS-48106
Re: java.io.EOFException continue occur when using health check with aws network load balancer
By the way, a *MUCH* easier way to reproduce this is to just use netcat against the JNLP port:{code:java}$ echo | nc jenkins my - nm- jenkins -hostname 8090{code}causes:{code:java}Nov 21, 2018 11:50:20 AM hudson.TcpSlaveAgentListener$ConnectionHandler runWARNING: Connection #2644 failedjava.io.EOFExceptionat java.io.DataInputStream.readFully(DataInputStream.java:197)at java.io.DataInputStream.readFully(DataInputStream.java:169)at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:242){code}
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title
Aaron Trout edited a comment on JENKINS-48106
Re: java.io.EOFException continue occur when using health check with aws network load balancer
I'm also hitting this. I am guessing that Jenkins is unhappy about the TCP health check probes that the NLB is sending it. My limited understanding of Jenkins internals is that on this JNLP / agent port there are multiple protocols (HTTP for JNLP protocol discovery, various JNLP versions). Since the load balancer is just L4, it is doing a simple TCP health check. I have attached a pcap of the health checks coming from the AWS NLB; it just sets up a TCP connection and immediately tears it down again without sending any data over the connection. [^nlb-hc.pcap]During this time we are getting one instance of the below error for every new TCP connection received: {code:java}Nov 21, 2018 11:27:52 AM hudson.TcpSlaveAgentListener$ConnectionHandler run WARNING: Connection #2576 failed java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:242){code} I have a similar setup, running Jenkins in Kubernetes with a `Service` of type `LoadBalancer` and the annotation to make it use the new NLB (rather than ELB classic). Because of this, the problem gets worse the larger your cluster is, since the NLB will run the health check against every node in the cluster (and kube-proxy will forward *all* those requests to the single Jenkins instance).
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title
Aaron Trout edited a comment on JENKINS-48106
Re: java.io.EOFException continue occur when using health check with aws network load balancer
I'm also hitting this. I am guessing that Jenkins is unhappy about the TCP health check probes that the NLB is sending it. My limited understanding of Jenkins internals is that on this JNLP / agent port there are multiple protocols (HTTP for JNLP protocol discovery, various JNLP versions). Since the load balancer is just L4, it is doing a TCP health check. I have attached a pcap of the health checks coming from the AWS NLB; it just sets up a TCP connection and immediately tears it down again without sending any data over the connection. [^nlb-hc.pcap] ^ During this time we are getting one instance of the below error for every new TCP connection received: ^ { {^ code:java} Nov 21, 2018 11:27:52 AM hudson.TcpSlaveAgentListener$ConnectionHandler run ^ ^ WARNING: Connection #2576 failed ^ ^ java.io.EOFException ^ ^ at java.io.DataInputStream.readFully(DataInputStream.java:197) ^ ^ at java.io.DataInputStream.readFully(DataInputStream.java:169) ^ ^ at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:242) ^ {code } }
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title
Aaron Trout commented on JENKINS-48106
Re: java.io.EOFException continue occur when using health check with aws network load balancer
I'm also hitting this. I am guessing that Jenkins is unhappy about the TCP health check probes that the NLB is sending it. My limited understanding of Jenkins internals is that on this JNLP / agent port there are multiple protocols (HTTP for JNLP protocol discovery, various JNLP versions). Since the load balancer is just L4, it is doing a TCP health check. I have attached a pcap of the health checks coming from the AWS NLB; it just sets up a TCP connection and immediately tears it down again without sending any data over the connection. nlb-hc.pcap During this time we are getting one instance of the below error for every new TCP connection received: {{Nov 21, 2018 11:27:52 AM hudson.TcpSlaveAgentListener$ConnectionHandler run WARNING: Connection #2576 failed java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:242)}}
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title Aaron Trout updated an issue Jenkins / JENKINS-48106 java.io.EOFException continue occur when using health check with aws network load balancer Change By: Aaron Trout Attachment: nlb-hc.pcap Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title Jeff Thompson closed an issue as Cannot Reproduce Closing for lack of response providing sufficient reproduction or diagnostic information. Jenkins / JENKINS-48106 java.io.EOFException continue occur when using health check with aws network load balancer Change By: Jeff Thompson Status: Open Closed Resolution: Cannot Reproduce Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48106) java.io.EOFException continue occur when using health check with aws network load balancer
Title: Message Title Jeff Thompson commented on JENKINS-48106 Re: java.io.EOFException continue occur when using health check with aws network load balancer Are you still seeing this issue? Can you provide further information about what is happening? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
