Hi Arkadiy, seems like you're right. I've just checked the relevant code for UDP inputs in Graylog and the size of UDP packets is indeed limited to 8192 bytes. Unfortunately there's currently no configuration option to change that but we'll address the issue in the next Graylog release.
Until then I unfortunately can only recommend either using TCP or using an intermediate log shipper like nxlog or logstash to get those messages into Graylog. :( Cheers, Jochen On Thursday, 7 May 2015 17:24:58 UTC+2, Arkadiy Shinkarev wrote: > > Jochen, thanks for you reply! > > As I mentioned in my first post, I can see with tcpdump that packet > lenght, that comes to Graylog node is more than 8192 bytes: > $ sudo tcpdump -n -i tunl0 port 12500 and udp > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on tunl0, link-type RAW (Raw IP), capture size 65535 bytes > 18:22:19.062304 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 54 > 18:22:19.079891 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 81 > 18:22:19.113119 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 92 > 18:22:19.117398 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 93 > 18:22:19.121636 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 81 > 18:22:19.123707 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 94 > 18:22:22.092734 IP 10.218.49.5.61843 > 10.218.50.20.12500: UDP, length 108 > 18:22:22.093300 IP 10.218.49.5.61843 > 10.218.50.20.12500: UDP, length 70 > 18:22:22.238882 IP 10.218.49.5.61843 > 10.218.50.20.12500: UDP, length 71 > 18:22:24.067068 IP 10.218.49.4.59298 > 10.218.50.20.12500: UDP, length 87 > 18:22:26.148394 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 85 > 18:22:27.477703 IP 10.218.49.4.59298 > 10.218.50.20.12500: UDP, length > 13642 > 18:22:31.158020 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 91 > 18:22:35.945376 IP 10.218.49.6.63104 > 10.218.50.20.12500: UDP, length 69 > 18:22:35.945489 IP 10.218.49.6.63104 > 10.218.50.20.12500: UDP, length 91 > 18:22:37.279499 IP 10.218.49.5.61843 > 10.218.50.20.12500: UDP, length 108 > > Also, I have another node with Graylog2 0.20, messages from same sources > comes to logstash UDP input without problems. > > So, I don't thinks that it is network related problems. > > On Thursday, May 7, 2015 at 4:04:26 PM UTC+3, Jochen Schalanda wrote: >> >> Hi Arkadiy, >> >> not all network devices support UDP packets bigger than 8KiB (8192 bytes) >> and this seems to be the case on your network somewhere. To circumvent this >> restriction, Graylog or more specifically the GELF format supports chunking >> which means splitting a large message into multiple UDP packets of a >> certain size (see https://www.graylog.org/resources/gelf for details on >> the GELF format). So in order to solve your problem you could collect the >> logs on the same machine which produces those raw log lines with a log >> shipper like nxlog or logstash and send them to Graylog via GELF. >> Alternatively you'll have to switch from UDP to TCP which doesn't suffer >> this kind of restriction. >> >> Cheers, >> Jochen >> >> On Thursday, 7 May 2015 13:56:25 UTC+2, Arkadiy Shinkarev wrote: >>> >>> Hi! >>> >>> I'm trying to send messages to raw udp input in Graylog 1.0.2. >>> The message size is 1k-20k, but Graylog only shown first 8k of message. >>> >>> I have configured "recv_buffer_size: 10485760" for input, also >>> set net.core.rmem_max = 26214400 in sysctl.conf. >>> >>> When I run tcpdump, I see that message len is ok (>8k). >>> When I run strace -e trace=network i see the following: >>> [pid 10539] recvfrom(365, >>> "\24\21\0\0\370\370\352\1778q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10539] recvfrom(365, >>> "\24\21\0\0\371\370\352\1779q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10537] <... recvfrom resumed> >>> "\24\21\0\0O\372\352\177\235q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10537] recvfrom(365, >>> "\24\21\0\0X\372\352\177\236q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10541] recvfrom(365, >>> "\24\21\0\0\200\372\352\177\240q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10541] recvfrom(365, >>> "\24\21\0\0\201\372\352\177\241q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10534] recvfrom(365, >>> "\24\21\0\0\302\372\352\177\330q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10534] recvfrom(365, >>> "\24\21\0\0\303\372\352\177\331q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10540] <... recvfrom resumed> >>> "\24\21\0\0$\373\352\177-r\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10540] <... recvfrom resumed> >>> "\24\21\0\0&\373\352\177/r\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10535] recvfrom(362, >>> "\24\21\0\0b\373\352\177jr\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10535] recvfrom(362, >>> "\24\21\0\0c\373\352\177kr\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10537] recvfrom(362, >>> "\24\21\0\0\315\375\352\177_s\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10537] recvfrom(362, >>> "\24\21\0\0\316\375\352\177`s\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10538] recvfrom(362, >>> "\24\21\0\0\v\376\352\177\224s\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10538] recvfrom(362, >>> "\24\21\0\0\f\376\352\177\225s\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10534] <... recvfrom resumed> >>> "\24\21\0\0\234\376\352\177\30t\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> [pid 10534] <... recvfrom resumed> >>> "\24\21\0\0\237\376\352\177\33t\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>> >>> 8192, 0, NULL, NULL) = 4372 >>> >>> I'm also test GELF UDP input, messages comes from logstash (to logstash >>> messages comes from UDP input with 32k buffer size), there is no problem - >>> message looks good, Graylog shows full message. >>> >>> Where is the problem? >>> >>> Some additional information: >>> OS: CentOS release 6.5 (Final) >>> Kernel: 2.6.32-431.29.2.el6.centos.plus.x86_64 >>> Graylog: 1.0.2 >>> >>> 2 graylog-server nodes behind load balancer (LVS) + 2 nodes >>> elasticsearch cluster. >>> >>> Thank you! >>> >> -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
