[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
This bug was fixed in the package linux - 3.13.0-39.66 --- linux (3.13.0-39.66) trusty; urgency=low [ Luis Henriques ] * Release Tracking Bug - LP: #1386629 [ Upstream Kernel Changes ] * KVM: x86: Check non-canonical addresses upon WRMSR - LP: #1384539 - CVE-2014-3610 * KVM: x86: Prevent host from panicking on shared MSR writes. - LP: #1384539 - CVE-2014-3610 * KVM: x86: Improve thread safety in pit - LP: #1384540 - CVE-2014-3611 * KVM: x86: Fix wrong masking on relative jump/call - LP: #1384545 - CVE-2014-3647 * KVM: x86: Warn if guest virtual address space is not 48-bits - LP: #1384545 - CVE-2014-3647 * KVM: x86: Emulator fixes for eip canonical checks on near branches - LP: #1384545 - CVE-2014-3647 * KVM: x86: emulating descriptor load misses long-mode case - LP: #1384545 - CVE-2014-3647 * KVM: x86: Handle errors when RIP is set during far jumps - LP: #1384545 - CVE-2014-3647 * kvm: vmx: handle invvpid vm exit gracefully - LP: #1384544 - CVE-2014-3646 * Input: synaptics - gate forcepad support by DMI check - LP: #1381815 linux (3.13.0-38.65) trusty; urgency=low [ Luis Henriques ] * Release Tracking Bug - LP: #1379244 [ Andy Whitcroft ] * Revert SAUCE: scsi: hyper-v storsvc switch up to SPC-3 - LP: #1354397 * [Config] linux-image-extra is additive to linux-image - LP: #1375310 * [Config] linux-image-extra postrm is not needed on purge - LP: #1375310 [ Upstream Kernel Changes ] * Revert KVM: x86: Increase the number of fixed MTRR regs to 10 - LP: #1377564 * Revert USB: option,zte_ev: move most ZTE CDMA devices to zte_ev - LP: #1377564 * aufs: bugfix, stop calling security_mmap_file() again - LP: #1371316 * ipvs: fix ipv6 hook registration for local replies - LP: #1349768 * Drivers: add blist flags - LP: #1354397 * sd: fix a bug in deriving the FLUSH_TIMEOUT from the basic I/O timeout - LP: #1354397 * drm/i915/bdw: Add 42ms delay for IPS disable - LP: #1374389 * drm/i915: add null render states for gen6, gen7 and gen8 - LP: #1374389 * drm/i915/bdw: 3D_CHICKEN3 has write mask bits - LP: #1374389 * drm/i915/bdw: Disable idle DOP clock gating - LP: #1374389 * drm/i915: call lpt_init_clock_gating on BDW too - LP: #1374389 * drm/i915: shuffle panel code - LP: #1374389 * drm/i915: extract backlight minimum brightness from VBT - LP: #1374389 * drm/i915: respect the VBT minimum backlight brightness - LP: #1374389 * drm/i915/bdw: Apply workarounds in render ring init function - LP: #1374389 * drm/i915/bdw: Cleanup pre prod workarounds - LP: #1374389 * drm/i915: Replace hardcoded cacheline size with macro - LP: #1374389 * drm/i915: Refactor Broadwell PIPE_CONTROL emission into a helper. - LP: #1374389 * drm/i915: Add the WaCsStallBeforeStateCacheInvalidate:bdw workaround. - LP: #1374389 * drm/i915/bdw: Remove BDW preproduction W/As until C stepping. - LP: #1374389 * mptfusion: enable no_write_same for vmware scsi disks - LP: #1371591 * iommu/amd: Fix cleanup_domain for mass device removal - LP: #1375266 * cifs: mask off top byte in get_rfc1002_length() - LP: #1372482 * Input: synaptics - add support for ForcePads - LP: #1377564 * ASoC: pxa-ssp: drop SNDRV_PCM_FMTBIT_S24_LE - LP: #1377564 * drm/radeon: add bapm module parameter - LP: #1377564 * drm/radeon: Add missing lines to ci_set_thermal_temperature_range - LP: #1377564 * drm/radeon: Add ability to get and change dpm state when radeon PX card is turned off - LP: #1377564 * ALSA: hda/realtek - Avoid setting wrong COEF on ALC269 co - LP: #1377564 * of/irq: Fix lookup to use 'interrupts-extended' property first - LP: #1377564 * Possible null ptr deref in SMB2_tcon - LP: #1377564 * CIFS: Fix SMB2 readdir error handling - LP: #1377564 * CIFS: Fix wrong directory attributes after rename - LP: #1377564 * md/raid6: avoid data corruption during recovery of double-degraded RAID6 - LP: #1377564 * ARM: dts: i.MX53: fix apparent bug in VPU clks - LP: #1377564 * pata_scc: propagate return value of scc_wait_after_reset - LP: #1377564 * libata: widen Crucial M550 blacklist matching - LP: #1377564 * ALSA: hda - restore the gpio led after resume - LP: #1358116, #1377564 * md/raid10: fix memory leak when reshaping a RAID10. - LP: #1377564 * md/raid10: Fix memory leak when raid10 reshape completes. - LP: #1377564 * MIPS: OCTEON: make get_system_type() thread-safe - LP: #1377564 * can: c_can: checking IS_ERR() instead of NULL - LP: #1377564 * HID: logitech: perform bounds checking on device_id early enough - LP: #1377564 * firmware: Do not use WARN_ON(!spin_is_locked()) - LP: #1377564 * drm/radeon: add new KV pci id - LP: #1377564 * drm/radeon: add
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
This bug was fixed in the package linux - 3.13.0-39.66 --- linux (3.13.0-39.66) trusty; urgency=low [ Luis Henriques ] * Release Tracking Bug - LP: #1386629 [ Upstream Kernel Changes ] * KVM: x86: Check non-canonical addresses upon WRMSR - LP: #1384539 - CVE-2014-3610 * KVM: x86: Prevent host from panicking on shared MSR writes. - LP: #1384539 - CVE-2014-3610 * KVM: x86: Improve thread safety in pit - LP: #1384540 - CVE-2014-3611 * KVM: x86: Fix wrong masking on relative jump/call - LP: #1384545 - CVE-2014-3647 * KVM: x86: Warn if guest virtual address space is not 48-bits - LP: #1384545 - CVE-2014-3647 * KVM: x86: Emulator fixes for eip canonical checks on near branches - LP: #1384545 - CVE-2014-3647 * KVM: x86: emulating descriptor load misses long-mode case - LP: #1384545 - CVE-2014-3647 * KVM: x86: Handle errors when RIP is set during far jumps - LP: #1384545 - CVE-2014-3647 * kvm: vmx: handle invvpid vm exit gracefully - LP: #1384544 - CVE-2014-3646 * Input: synaptics - gate forcepad support by DMI check - LP: #1381815 linux (3.13.0-38.65) trusty; urgency=low [ Luis Henriques ] * Release Tracking Bug - LP: #1379244 [ Andy Whitcroft ] * Revert SAUCE: scsi: hyper-v storsvc switch up to SPC-3 - LP: #1354397 * [Config] linux-image-extra is additive to linux-image - LP: #1375310 * [Config] linux-image-extra postrm is not needed on purge - LP: #1375310 [ Upstream Kernel Changes ] * Revert KVM: x86: Increase the number of fixed MTRR regs to 10 - LP: #1377564 * Revert USB: option,zte_ev: move most ZTE CDMA devices to zte_ev - LP: #1377564 * aufs: bugfix, stop calling security_mmap_file() again - LP: #1371316 * ipvs: fix ipv6 hook registration for local replies - LP: #1349768 * Drivers: add blist flags - LP: #1354397 * sd: fix a bug in deriving the FLUSH_TIMEOUT from the basic I/O timeout - LP: #1354397 * drm/i915/bdw: Add 42ms delay for IPS disable - LP: #1374389 * drm/i915: add null render states for gen6, gen7 and gen8 - LP: #1374389 * drm/i915/bdw: 3D_CHICKEN3 has write mask bits - LP: #1374389 * drm/i915/bdw: Disable idle DOP clock gating - LP: #1374389 * drm/i915: call lpt_init_clock_gating on BDW too - LP: #1374389 * drm/i915: shuffle panel code - LP: #1374389 * drm/i915: extract backlight minimum brightness from VBT - LP: #1374389 * drm/i915: respect the VBT minimum backlight brightness - LP: #1374389 * drm/i915/bdw: Apply workarounds in render ring init function - LP: #1374389 * drm/i915/bdw: Cleanup pre prod workarounds - LP: #1374389 * drm/i915: Replace hardcoded cacheline size with macro - LP: #1374389 * drm/i915: Refactor Broadwell PIPE_CONTROL emission into a helper. - LP: #1374389 * drm/i915: Add the WaCsStallBeforeStateCacheInvalidate:bdw workaround. - LP: #1374389 * drm/i915/bdw: Remove BDW preproduction W/As until C stepping. - LP: #1374389 * mptfusion: enable no_write_same for vmware scsi disks - LP: #1371591 * iommu/amd: Fix cleanup_domain for mass device removal - LP: #1375266 * cifs: mask off top byte in get_rfc1002_length() - LP: #1372482 * Input: synaptics - add support for ForcePads - LP: #1377564 * ASoC: pxa-ssp: drop SNDRV_PCM_FMTBIT_S24_LE - LP: #1377564 * drm/radeon: add bapm module parameter - LP: #1377564 * drm/radeon: Add missing lines to ci_set_thermal_temperature_range - LP: #1377564 * drm/radeon: Add ability to get and change dpm state when radeon PX card is turned off - LP: #1377564 * ALSA: hda/realtek - Avoid setting wrong COEF on ALC269 co - LP: #1377564 * of/irq: Fix lookup to use 'interrupts-extended' property first - LP: #1377564 * Possible null ptr deref in SMB2_tcon - LP: #1377564 * CIFS: Fix SMB2 readdir error handling - LP: #1377564 * CIFS: Fix wrong directory attributes after rename - LP: #1377564 * md/raid6: avoid data corruption during recovery of double-degraded RAID6 - LP: #1377564 * ARM: dts: i.MX53: fix apparent bug in VPU clks - LP: #1377564 * pata_scc: propagate return value of scc_wait_after_reset - LP: #1377564 * libata: widen Crucial M550 blacklist matching - LP: #1377564 * ALSA: hda - restore the gpio led after resume - LP: #1358116, #1377564 * md/raid10: fix memory leak when reshaping a RAID10. - LP: #1377564 * md/raid10: Fix memory leak when raid10 reshape completes. - LP: #1377564 * MIPS: OCTEON: make get_system_type() thread-safe - LP: #1377564 * can: c_can: checking IS_ERR() instead of NULL - LP: #1377564 * HID: logitech: perform bounds checking on device_id early enough - LP: #1377564 * firmware: Do not use WARN_ON(!spin_is_locked()) - LP: #1377564 * drm/radeon: add new KV pci id - LP: #1377564 * drm/radeon: add
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
This bug is awaiting verification that the kernel in -proposed solves
the problem. Please test the kernel and update this bug with the
results. If the problem is solved, change the tag 'verification-needed-
trusty' to 'verification-done-trusty'.
If verification is not done by 5 working days from today, this fix will
be dropped from the source code, and this bug will be closed.
See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how
to enable and use -proposed. Thank you!
** Tags added: verification-needed-trusty
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Fix Released
Status in “linux” source package in Trusty:
Fix Committed
Status in “linux” source package in Utopic:
Fix Released
Bug description:
SRU Justificaton:
[Impact]
Users of ipvs may encounter dropped packets and message such as:
[70325.516724] IPv6 header not found
[Fix]
Commit eb90b0c734ad793d5f5bf230a9e9a4dcc48df8aa.
[Regression Potential]
This changes NFPROTO_IPV4 to NFPROTO_IPV6 as we'd expect this nf_hook_ops
struct to be in the code.
--
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
Yes, I can confirm that the trusty-proposed 3.13.0-38 kernel resolves
this issue. The dnsmasq-tftp transfer succeeds in my test environment
without any of the symptoms that were present with the trusty-
updates/security kernels between -30 and -37, nor do I notice any
regressions in my (IPv4) ipvs ruleset.
$ uname -a
Linux catcp2-test 3.13.0-38-generic #65-Ubuntu SMP Thu Oct 9 11:36:50 UTC 2014
x86_64 x86_64 x86_64 GNU/Linux
$ apt-cache policy linux-image-generic
linux-image-generic:
Installed: 3.13.0.38.45
Candidate: 3.13.0.38.45
Version table:
*** 3.13.0.38.45 0
50 http://fi.archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64
Packages
100 /var/lib/dpkg/status
...
** Tags removed: verification-needed-trusty
** Tags added: verification-done-trusty
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Fix Released
Status in “linux” source package in Trusty:
Fix Committed
Status in “linux” source package in Utopic:
Fix Released
Bug description:
SRU Justificaton:
[Impact]
Users of ipvs may encounter dropped packets and message such as:
[70325.516724] IPv6 header not found
[Fix]
Commit eb90b0c734ad793d5f5bf230a9e9a4dcc48df8aa.
[Regression Potential]
This changes NFPROTO_IPV4 to NFPROTO_IPV6 as we'd expect this nf_hook_ops
struct to be in the code.
--
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
@terom-u
Thanks for verifying this fix!
--chris
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Fix Released
Status in “linux” source package in Trusty:
Fix Committed
Status in “linux” source package in Utopic:
Fix Released
Bug description:
SRU Justificaton:
[Impact]
Users of ipvs may encounter dropped packets and message such as:
[70325.516724] IPv6 header not found
[Fix]
Commit eb90b0c734ad793d5f5bf230a9e9a4dcc48df8aa.
[Regression Potential]
This changes NFPROTO_IPV4 to NFPROTO_IPV6 as we'd expect this nf_hook_ops
struct to be in the code.
--
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
** Branch linked: lp:ubuntu/trusty-proposed/linux-keystone
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Fix Released
Status in “linux” source package in Trusty:
Fix Committed
Status in “linux” source package in Utopic:
Fix Released
Bug description:
SRU Justificaton:
[Impact]
Users of ipvs may encounter dropped packets and message such as:
[70325.516724] IPv6 header not found
[Fix]
Commit eb90b0c734ad793d5f5bf230a9e9a4dcc48df8aa.
[Regression Potential]
This changes NFPROTO_IPV4 to NFPROTO_IPV6 as we'd expect this nf_hook_ops
struct to be in the code.
--
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
** Branch linked: lp:ubuntu/precise-proposed/linux-lts-trusty
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Fix Released
Status in “linux” source package in Trusty:
Fix Committed
Status in “linux” source package in Utopic:
Fix Released
Bug description:
SRU Justificaton:
[Impact]
Users of ipvs may encounter dropped packets and message such as:
[70325.516724] IPv6 header not found
[Fix]
Commit eb90b0c734ad793d5f5bf230a9e9a4dcc48df8aa.
[Regression Potential]
This changes NFPROTO_IPV4 to NFPROTO_IPV6 as we'd expect this nf_hook_ops
struct to be in the code.
--
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2]
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
This bug was fixed in the package linux - 3.16.0-18.25
---
linux (3.16.0-18.25) utopic; urgency=low
[ Tim Gardner ]
* Release Tracking Bug
- LP: #1373682
-- Tim Gardner tim.gard...@canonical.com Wed, 24 Sep 2014 19:23:23 -0600
** Changed in: linux (Ubuntu Utopic)
Status: Fix Committed = Fix Released
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Fix Released
Status in “linux” source package in Trusty:
Fix Committed
Status in “linux” source package in Utopic:
Fix Released
Bug description:
SRU Justificaton:
[Impact]
Users of ipvs may encounter dropped packets and message such as:
[70325.516724] IPv6 header not found
[Fix]
Commit eb90b0c734ad793d5f5bf230a9e9a4dcc48df8aa.
[Regression Potential]
This changes NFPROTO_IPV4 to NFPROTO_IPV6 as we'd expect this nf_hook_ops
struct to be in the code.
--
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
** Description changed:
+ SRU Justificaton:
+
+ [Impact]
+ Users of ipvs may encounter dropped packets and message such as:
+ [70325.516724] IPv6 header not found
+
+ [Fix]
+ Commit eb90b0c734ad793d5f5bf230a9e9a4dcc48df8aa.
+
+ [Regression Potential]
+ This changes NFPROTO_IPV4 to NFPROTO_IPV6 as we'd expect this nf_hook_ops
struct to be in the code.
+
+ --
+
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the Loading
/linux TFTP transfer, with the transfer stalling roughly ~1000
blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself - the
stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
- ---
+ ---
AlsaDevices:
- total 0
- crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
- crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
+ total 0
+ crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
+ crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
** Changed in: linux (Ubuntu Utopic)
Status: In Progress = Fix Committed
** Changed in: linux (Ubuntu Trusty)
Status: In Progress = Fix Committed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Fix Committed
Status in “linux” source package in Trusty:
Fix Committed
Status in “linux” source package in Utopic:
Fix Committed
Bug description:
SRU Justificaton:
[Impact]
Users of ipvs may encounter dropped packets and message such as:
[70325.516724] IPv6 header not found
[Fix]
Commit eb90b0c734ad793d5f5bf230a9e9a4dcc48df8aa.
[Regression Potential]
This changes NFPROTO_IPV4 to NFPROTO_IPV6 as we'd expect this nf_hook_ops
struct to be in the code.
--
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
Ok patch is upstream in v3.17-rc5:
eb90b0c734ad793d5f5bf230a9e9a4dcc48df8aa
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
In Progress
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
In Progress
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease:
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
ACK
3.13.0-35-generic #62~lp1349768v5v201408250842 appears to resolve the
issue completely on my actual test setup as well. No TFTP stalls,
dnsmasq EPERM errors or dmesg errors, and ftrace doesn't show any calls
to ipv6_find_hdr.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
In Progress
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
In Progress
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse:
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
NAK
Unfortunately no, the 3.13.0-35-generic #62~lp1349768v4v201408220857
from above continues to exhibit the same issues with my actual test
setup. Same thing with the ./lp1349768.sh testcase.
The 0041-ipvs-fix-ipv4-issues-in-ip_vs_out.patch in the lp1349768v4
build is a different patch, though?
The resulting ftrace that correlates with dnsmasq sendto() getting stuck
is still the same:
dnsmasq-1447 [000] 1272.430975: ipv6_find_hdr
-ip_vs_out_icmp_v6.isra.21
dnsmasq-1447 [000] 1272.430983: stack trace
= ip_vs_out.part.22
= ip_vs_local_reply6
= nf_iterate
= nf_hook_slow
= __ip_local_out
= ip_local_out
= ip_send_skb
= udp_send_skb
= udp_sendmsg
= inet_sendmsg
= sock_sendmsg
= SYSC_sendto
= SyS_sendto
= tracesys
Is there something I can use to confirm that the booted kernel image
actually has the correct nf_hook_ops.pf set? The fix seems pretty
obvious, so I was expecting it to work...
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
In Progress
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
In Progress
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs),
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
@terom,
Well, I posted the wrong kernel! I'll post the correct one soon after it
builds. Thanks
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
In Progress
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
In Progress
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
This should be the right kernel:
http://people.canonical.com/~arges/lp1349768v5/
I verified it myself and it works, once you verify, I'll go through the SRU
process and get this into Trusty.
Thanks,
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
In Progress
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
In Progress
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v',
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
@terom
Can you please test the following build:
http://people.canonical.com/~arges/lp1349768v4/
This has the patch identified by upstream; it would be good to have positive
testing on this.
This has VS_IPV6 enabled with the fix:
http://archive.linuxvirtualserver.org/html/lvs-devel/2014-08/msg00027.html
Thanks,
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
In Progress
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
In Progress
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
@terom
Ok, I have an approach here for fixing this here:
http://people.canonical.com/~arges/lp1349768v4/
I'd appreciate any testing on this before I send this upstream. I'll look at
this a bit more to see if I'm missing something.
Thanks,
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
In Progress
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
In Progress
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
E-mail sent upstream about this issue:
http://archive.linuxvirtualserver.org/html/lvs-devel/2014-08/msg00026.html
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
In Progress
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
In Progress
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
NAK
Linux 3.13.0-35-generic #62~lp1349768v3v201408191448 still exhibits the
same symptoms (stalled dnsmasq-tftp transfer and IPv6 header not found
messages). Unloading the ip_vs module clears the issue.
I'll attempt to work out a minimal reproducer.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Confirmed
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
Confirmed
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
Attached is a small script to setup dnsmasq-tftp and ipvs to reproduce
the dnsmasq-tftp stall. The strace at the end should start showing
sendto() = EPERM errors after running the curl tftp://.../ command given
as an example from a remote host.
The exact contents of the ipvs rules don't seem to matter.
The dnsmasq-tftp stall issue appears to happen with all the ubuntu-
installer/amd64/linux versions that I've tried, with the exact block
within the TFTP stream where it stalls changing betwen different
versions but remaining consistent between runs.
With the current netboot.tar.gz it appers to happen at block 600:
d4fc2ba26cad96f4360cb36f04c26b2f50dceee9413c206431148b87e4958909
/srv/tftp/ubuntu-installer/amd64/linux
With the original ubuntu-installer netboot.tar.gz version (from sometime
early June) I've been using it happens at block 311:
bdebc6c7bbd873bf5ad2480d90c2523da49cca8da5a0f4cb40adb7d1aeafb821
/srv/tftp/lp1349768/linux
Curiously, however, testing this setup on the 3.13.0-34-generic kernel
the dmesg error (IPv6 header not found) does NOT show up... may be
some other facters at play as well there?
** Attachment added: minimal testcase for LP#1349768 on Ubuntu 14.04
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1349768/+attachment/4182427/+files/lp1349768.sh
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Confirmed
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
Confirmed
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
@terom
Alright! I've been able to reproduce this issue, I'll start debugging
this today.
Regarding the 3.13.0-34 differences I can look at the patches in that
kernel to see if there was some changes to ipvs or around where that
message shows up.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Confirmed
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
Confirmed
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
** Tags added: bug-exists-upstream
** Changed in: linux (Ubuntu Utopic)
Assignee: (unassigned) = Chris J Arges (arges)
** Changed in: linux (Ubuntu Utopic)
Status: Confirmed = In Progress
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
In Progress
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
In Progress
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v',
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
@terom
Yes, a minimal reproducer would help so I can debug on this end. I really
appreciate your testing and analysis so far. : )
I did setup a quick netboot server using MAAS and it did seem to work; but I
did not do the additional steps of setting up the proper ipvs ruleset to
reproduce the issue. I also wonder if those rules can be set and something as
simple as a ping could be used to illustrate the bug?
Here is a patch that looks like it could fix this issue; but unfortunately this
was never placed upstream:
http://archive.linuxvirtualserver.org/cgi-bin/mesg.cgi?a=lvs-develi=20140219123150.GF20771%40telegraafnet.nl
I've built a test build with that patch applied for 3.13, let me know if this
is an ACK or NAK in the meantime:
http://people.canonical.com/~arges/lp1349768v3/
Thanks
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Confirmed
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
Confirmed
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
ACK
I can confirm that this issue is no longer present with the Linux
3.13.0-33-generic #58~lp1349768v2v201407300928 kernel as installed from
above. Rebooting into the given kernel results in dnsmasq-tftp transfers
working fine, and no related dmesg errors occur.
Apologies for the delay. I was away, and had to reproduce this in a test
environment first.
Do tell me if you would still like a minimal reproducer to track down
the issue further, and I can work on getting something like that up.
AFAICT it would involve dnsmasq with enable-tftp/tftp-root and a
libvirt-qemu domain with boot dev='network' , requesting the ubuntu-
installer amd64 linux kernel over TFTP, with the ip_vs modules loaded
and a trivial ipvs ruleset applied on the server.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Confirmed
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
Confirmed
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again,
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
The utopic kernel appears to be affected as well. Booting the same
Ubuntu 14.04 machine into the Linux 3.16.0-6-generic #11-Ubuntu kernel
installed from
https://launchpad.net/ubuntu/+source/linux/3.16.0-6.11/+build/6217368
gives identical symptoms, i.e. dnsmasq-tftp stalls, dmesg reports IPv6
header not found, stracing dnsmasq shows sendto() giving EPERM, and
unloading ip_vs allows the stalled tftp transfer to continue.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Confirmed
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
Confirmed
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
NACK
The same faulty behaviour persists with linux-
image-3.13.0-33-generic=3.13.0-33.58~lp1349768v201407290941 (uname
3.13.0-33-generic #58~lp1349768v201407290941).
I do not notice any difference; identical symptoms. `rmmod ip_vs_rr
ip_vs` continues to immediately resolve the fault.
Note that my configuration also includes openvswitch. The tftp stall
continues to happen when flushing the ipvs ruleset, only unloading the
ip_vs module resolves it. However, only re-loading the ip_vs_rr module
does not trigger the stall, but loading the ipvs ruleset does. My IPVS
ruleset consists of only four TCP Masq forwards.
Using ktrace to try and get a rough stack trace, I was able to observe
the following sequences:
dnsmasq-5012 [001] 2060.572946: ipv6_find_hdr
-ip_vs_out.part.23
dnsmasq-5012 [001] 2060.572955: stack trace
= ip_vs_local_reply6
= nf_iterate
= nf_hook_slow
= __ip_local_out
= ip_local_out
= ip_send_skb
= udp_send_skb
= udp_sendmsg
= inet_sendmsg
= sock_sendmsg
= SYSC_sendto
= SyS_sendto
= tracesys
dnsmasq-5012 [001] 2060.572956: ipv6_find_hdr
-ip_vs_out_icmp_v6.isra.22
dnsmasq-5012 [001] 2060.572963: stack trace
= ip_vs_out.part.23
= ip_vs_local_reply6
= nf_iterate
= nf_hook_slow
= __ip_local_out
= ip_local_out
= ip_send_skb
= udp_send_skb
= udp_sendmsg
= inet_sendmsg
= sock_sendmsg
= SYSC_sendto
= SyS_sendto
= tracesys
with the first sequence being repeated for every packet (?) and the
second, slightly different, sequence corresponding per the timestamp to
the dmesg:
[ 2060.572964] IPv6 header not found
This is with a my full iptables rulset loaded, but my OUTPUT DROP rules
counters do not indicate any matches. The ftrace output seems to be
identical with the iptables ruleset flushed.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Fix Released
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
Fix Released
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17,
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
I believe this is related to a CONFIG change:
UBUNTU: [Config] Enable CONFIG_IP_VS_IPV6=y
I'm building a 3.13 kernel with this CONFIG reverted I would like you to test
and confirm.
(I'll post a link in a few hours)
In the meantime can you verify if the utopic 3.16 series kernel is also
affected? (since it has the same CONFIG):
https://launchpad.net/ubuntu/+source/linux/3.16.0-6.11/+build/6217368
Thanks,
** Changed in: linux (Ubuntu Utopic)
Status: Fix Released = New
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
New
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
New
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
Ok here is the 3.13 kernel with CONFIG_IP_VS_IPV6=y reverted:
http://people.canonical.com/~arges/lp1349768v2/
Please let me know the results when you can.
In addition, if there is a way to explain a minimal reproducer of this
issue, I could set this up here and run experiments as well.
Thanks,
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Confirmed
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
Confirmed
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices:
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
apport information
** Tags added: apport-collected trusty
** Description changed:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the Loading
/linux TFTP transfer, with the transfer stalling roughly ~1000
blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself - the
stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
- This seems to happen reproducibly on boot with -32 and -30. This does
- NOT seem to happen with 3.13.0-29 which I was using up until now.
+ This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
+ ---
+ AlsaDevices:
+ total 0
+ crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
+ crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
+ AplayDevices: Error: [Errno 2] No such file or directory
+ ApportVersion: 2.14.1-0ubuntu3.2
+ Architecture: amd64
+ ArecordDevices: Error: [Errno 2] No such file or directory
+ AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
+ CRDA: Error: [Errno 2] No such file or directory
+ DistroRelease: Ubuntu 14.04
+ HibernationDevice: RESUME=/dev/mapper/catcp2-swap
+ InstallationDate: Installed on 2014-06-03 (56 days ago)
+ InstallationMedia: Ubuntu-Server 14.04 LTS Trusty Tahr - Release amd64
(20140416.2)
+ MachineType: Dell Inc. PowerEdge R410
+ Package:
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
Reported apport-collect information against package linux-
image-3.13.0-32-generic, reproducing the dnsmasq-tftp issue with a
remote client pxebooting and stalling.
The CurrentDmesg contains the symptom message at 1214.740931
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Confirmed
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
** Also affects: linux (Ubuntu Trusty)
Importance: Undecided
Status: New
** Also affects: linux (Ubuntu Utopic)
Importance: Undecided
Status: Confirmed
** Changed in: linux (Ubuntu Trusty)
Status: New = In Progress
** Changed in: linux (Ubuntu Trusty)
Assignee: (unassigned) = Chris J Arges (arges)
** Changed in: linux (Ubuntu Utopic)
Status: Confirmed = Fix Released
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Fix Released
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
Fix Released
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices:
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
@terom:
Can you test the following build:
http://people.canonical.com/~arges/lp1349768/
I've reverted commit: b05f2cbc367eae3ad8de17c65a832fd10e30f4b5 from
ubuntu-trusty which was a backport of
57a7744e09867ebcfa0ccf1d6d529caa7728d552.
Thanks,
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Fix Released
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
Fix Released
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
[Kernel-packages] [Bug 1349768] Re: kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket sendto() EPERM errors
** Changed in: linux (Ubuntu Trusty)
Importance: Undecided = Medium
** Changed in: linux (Ubuntu Utopic)
Importance: Undecided = Medium
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1349768
Title:
kernel 3.13.0-32 ipvs IPv6 header not found related to UDP socket
sendto() EPERM errors
Status in “linux” package in Ubuntu:
Fix Released
Status in “linux” source package in Trusty:
In Progress
Status in “linux” source package in Utopic:
Fix Released
Bug description:
I have an Ubuntu 14.04 host that I am using as both a keepalived/ipvs
loadbalancer and dnsmasq server for pxebooting servers.
After updating linux-image 3.13.0-29.53 - 3.13.0-32.57 I noticed that
dnsmasq-tftp stopped working. pxeboot clients would hang on the
Loading /linux TFTP transfer, with the transfer stalling roughly
~1000 blocks into the transfer:
10:30:51.011728 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.011924 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
10:30:51.012012 IP 10.1.1.2.43540 10.1.12.1.49165: UDP, length 1412
10:30:51.012183 IP 10.1.12.1.49165 10.1.1.2.43540: UDP, length 4
stracing dnsmasq I noticed something very odd: sendto() on the
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) would suddenly start
persistently returning EPERM in mid-transfer, even when dnsmasq
continued to periodically retry:
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249834})
recvfrom(17, \0\4\3\352, 4096, 0, NULL, NULL) = 4
lseek(16, 1410816, SEEK_SET) = 1410816
read(16,
\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32\221X+\v...,
1408) = 1408
sendto(17,
\0\3\3\353\25\306\345f\2{\r\4)W\276\32\336q\252_\230q\213\341U\354\25\374k7\243\32...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = 1412
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 1 (in [17],
left {0, 249839})
recvfrom(17, \0\4\3\353, 4096, 0, NULL, NULL) = 4
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
select(18, [4 5 6 7 8 9 10 11 12 15 17], [], [], {0, 25}) = 0 (Timeout)
lseek(16, 1412224, SEEK_SET) = 1412224
read(16, *\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277\221\\\307\372...,
1408) = 1408
sendto(17, \0\3\3\354*\360
C\363l\320:\256~\307\236\26P\323\274%\260\362\341\232\r\243\370\224\277...,
1412, 0, {sa_family=AF_INET, sin_port=htons(49165),
sin_addr=inet_addr(10.1.11.3)}, 16) = -1 EPERM (Operation not permitted)
This was with all iptables rules unloaded (so no OUTPUT -j DENY) and
apparmor profiles torn down.
I also noticed the following dmesgs appearing at roughly similar times
to the tftp transfers getting stuck (although not coinciding exactly
with the stall):
[70325.516724] IPv6 header not found
The error pointed to ipvs (which I am using on the same host as an IPv4 NAT
loadbalancer):
http://archive.linuxvirtualserver.org/html/lvs-devel/2012-08/msg00018.html
http://comments.gmane.org/gmane.comp.linux.lvs.devel/3614
I then tore down the ipvs rules (service keepalived stop) and unloaded
the modules (rmmod ip_vs_rr ip_vs), and the issue resolved itself -
the stalled dnsmasq-tftp transfer resumed!
This seems to be reproducible, i.e. modprobing ip_vs and starting
keepalived will cause dnsmasq-tftp to stall again, and
stopping/unloading will resume.
This seems to happen reproducibly on boot with -32 and -30. This does NOT
seem to happen with 3.13.0-29 which I was using up until now.
---
AlsaDevices:
total 0
crw-rw 1 root audio 116, 1 Jul 29 13:43 seq
crw-rw 1 root audio 116, 33 Jul 29 13:43 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code
