We see this error a few times in a day on a highly busy production environment. Unfortunately there is too much traffic on the server to keep tcpdump/ngrep running and we cannot re-produce it on test environment :(
I have started tcpdump on a test environment of another implementation and will let you as soon as the issue gets triggerred again. On Fri, Mar 31, 2017 at 4:17 PM, Andrei <[email protected]> wrote: > Can you provide a tcpdump/ngrep of the requests between > Client/Varnish/Apache along with the varnishlog entry to see if that > uncovers anything? > > On Fri, Mar 31, 2017 at 7:25 AM, Hazar Güney <[email protected]> wrote: > >> Any idea? >> >> On Thu, Mar 30, 2017 at 3:41 PM, Hazar Güney <[email protected]> >> wrote: >> >>> It did not work either: >>> >>> * << BeReq >> 127418176 >>> - Begin bereq 127418175 fetch >>> - Timestamp Start: 1490877149.450124 0.000000 0.000000 >>> - BereqMethod GET >>> - BereqURL XXXX >>> - BereqProtocol HTTP/1.1 >>> - BereqHeader Accept: text/css,*/*;q=0.1 >>> - BereqHeader User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS >>> 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0 >>> Mobile/14C92 Safari/602.1 >>> - BereqHeader Accept-Language: tr-tr >>> - BereqHeader Referer: XXXX >>> - BereqHeader Host: XXXX >>> - BereqHeader RIP: XXXX >>> - BereqHeader X-Forwarded-For: XXXX >>> - BereqHeader Accept-Encoding: gzip >>> - BereqHeader X-Varnish: 127418176 >>> - VCL_call BACKEND_FETCH >>> - BereqHeader connection: Close >>> - VCL_return fetch >>> - BackendOpen 25 reload_2017-03-30T14:53:46.st2 10.35.78.11 80 >>> 172.17.0.2 59152 >>> - BackendStart 10.35.78.11 80 >>> - Timestamp Bereq: 1490877149.450594 0.000470 0.000470 >>> - FetchError http first read error: EOF >>> - BackendClose 25 reload_2017-03-30T14:53:46.st2 >>> - Timestamp Beresp: 1490877149.451184 0.001060 0.000590 >>> - Timestamp Error: 1490877149.451189 0.001065 0.000005 >>> - BerespProtocol HTTP/1.1 >>> - BerespStatus 503 >>> - BerespReason Service Unavailable >>> - BerespReason Backend fetch failed >>> - BerespHeader Date: Thu, 30 Mar 2017 12:32:29 GMT >>> - BerespHeader Server: Varnish >>> - VCL_call BACKEND_ERROR >>> - BereqHeader X-Varnish-Backend-5xx: 1 >>> - VCL_return retry >>> - Timestamp Retry: 1490877149.451205 0.001081 0.000016 >>> - Link bereq 127298071 retry >>> - End >>> >>> On Thu, Mar 30, 2017 at 2:34 PM, Guillaume Quintard < >>> [email protected]> wrote: >>> >>>> It does, I'm suspecting that the connection reuse is creating some >>>> issues, probably because Apache is doing some non-standard stuff (protip: >>>> always blame Apache). >>>> >>>> -- >>>> Guillaume Quintard >>>> >>>> On Thu, Mar 30, 2017 at 1:17 PM, Hazar Güney <[email protected]> >>>> wrote: >>>> >>>>> "Connection: close" supersedes keep-alive behavior, is that correct? >>>>> >>>>> On Thu, Mar 30, 2017 at 2:08 PM, Guillaume Quintard < >>>>> [email protected]> wrote: >>>>> >>>>>> Can you try something: add 'set bereq.http.connection = "Close"; ' at >>>>>> the beginning of vcl_backend_fetch and see if that helps? >>>>>> >>>>>> -- >>>>>> Guillaume Quintard >>>>>> >>>>>> On Thu, Mar 30, 2017 at 1:04 PM, Hazar Güney <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> MaxKeepAliveRequests 20 >>>>>>> KeepAliveTimeout 2 >>>>>>> >>>>>>> Version is "4.1.3 revision 5e3b6d2". We have also seen "straight >>>>>>> insufficient bytes" error with POST requests to a specific php script >>>>>>> hosted by another backend and fixed it by using "pipe" instead of "pass" >>>>>>> but this specific backend gives "http first read error: EOF" error. >>>>>>> Another >>>>>>> example from today: >>>>>>> >>>>>>> * << BeReq >> 126635444 >>>>>>> - Begin bereq 126635443 fetch >>>>>>> - Timestamp Start: 1490870598.921499 0.000000 0.000000 >>>>>>> - BereqMethod GET >>>>>>> - BereqURL XXXX >>>>>>> - BereqProtocol HTTP/1.1 >>>>>>> - BereqHeader Host: XXXX >>>>>>> - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; >>>>>>> x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 >>>>>>> Safari/537.36 >>>>>>> - BereqHeader Accept: image/webp,image/*,*/*;q=0.8 >>>>>>> - BereqHeader Referer: XXXX >>>>>>> - BereqHeader Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.6,en; >>>>>>> q=0.4 >>>>>>> - BereqHeader RIP: XXXX >>>>>>> - BereqHeader X-Forwarded-For: XXXX >>>>>>> - BereqHeader Accept-Encoding: gzip >>>>>>> - BereqHeader X-Varnish: 126635444 >>>>>>> - VCL_call BACKEND_FETCH >>>>>>> - VCL_return fetch >>>>>>> - BackendOpen 35 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 >>>>>>> 172.17.0.2 48896 >>>>>>> - BackendStart 10.35.78.11 80 >>>>>>> - Timestamp Bereq: 1490870598.922050 0.000552 0.000552 >>>>>>> *- FetchError http first read error: EOF* >>>>>>> - BackendClose 35 reload_2017-03-20T11:32:44.st2 >>>>>>> - Timestamp Beresp: 1490870598.922622 0.001124 0.000572 >>>>>>> - Timestamp Error: 1490870598.922627 0.001129 0.000005 >>>>>>> - BerespProtocol HTTP/1.1 >>>>>>> - BerespStatus 503 >>>>>>> - BerespReason Service Unavailable >>>>>>> - BerespReason Backend fetch failed >>>>>>> - BerespHeader Date: Thu, 30 Mar 2017 10:43:18 GMT >>>>>>> - BerespHeader Server: Varnish >>>>>>> - VCL_call BACKEND_ERROR >>>>>>> - BereqHeader X-Varnish-Backend-5xx: 1 >>>>>>> - VCL_return retry >>>>>>> - Timestamp Retry: 1490870598.922657 0.001159 0.000030 >>>>>>> - Link bereq 126832283 retry >>>>>>> - End >>>>>>> >>>>>>> On Wed, Mar 29, 2017 at 12:03 PM, Mattias Geniar <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> > Backend is Apache. >>>>>>>> >>>>>>>> In older Varnish versions, you could sometimes see a similar error; >>>>>>>> >>>>>>>> > 11 FetchError c straight insufficient bytes >>>>>>>> >>>>>>>> The error message you’re seeing might be related, as it mentions >>>>>>>> the EOF. >>>>>>>> >>>>>>>> This happens when the backend sends a Content-Length header that >>>>>>>> doesn’t match the _actual_ content length it’s sending. In Apache, >>>>>>>> this was >>>>>>>> commonly caused by a mod_deflate misconfiguration. >>>>>>>> >>>>>>>> For testing, could you try disabling Gzip either in your backend or >>>>>>>> strip the Accept-Encoding header in Varnish to force a plain text >>>>>>>> response? >>>>>>>> >>>>>>>> Mattias >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> _______________________________________________ >> varnish-misc mailing list >> [email protected] >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > >
_______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
