Ok, will be waiting for fix. Thank you! On Thursday, May 7, 2015 at 6:50:59 PM UTC+3, Jochen Schalanda wrote: > > Hi Arkadiy, > > seems like you're right. I've just checked the relevant code for UDP > inputs in Graylog and the size of UDP packets is indeed limited to 8192 > bytes. Unfortunately there's currently no configuration option to change > that but we'll address the issue in the next Graylog release. > > Until then I unfortunately can only recommend either using TCP or using an > intermediate log shipper like nxlog or logstash to get those messages into > Graylog. :( > > > Cheers, > Jochen > > On Thursday, 7 May 2015 17:24:58 UTC+2, Arkadiy Shinkarev wrote: >> >> Jochen, thanks for you reply! >> >> As I mentioned in my first post, I can see with tcpdump that packet >> lenght, that comes to Graylog node is more than 8192 bytes: >> $ sudo tcpdump -n -i tunl0 port 12500 and udp >> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode >> listening on tunl0, link-type RAW (Raw IP), capture size 65535 bytes >> 18:22:19.062304 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 54 >> 18:22:19.079891 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 81 >> 18:22:19.113119 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 92 >> 18:22:19.117398 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 93 >> 18:22:19.121636 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 81 >> 18:22:19.123707 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 94 >> 18:22:22.092734 IP 10.218.49.5.61843 > 10.218.50.20.12500: UDP, length 108 >> 18:22:22.093300 IP 10.218.49.5.61843 > 10.218.50.20.12500: UDP, length 70 >> 18:22:22.238882 IP 10.218.49.5.61843 > 10.218.50.20.12500: UDP, length 71 >> 18:22:24.067068 IP 10.218.49.4.59298 > 10.218.50.20.12500: UDP, length 87 >> 18:22:26.148394 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 85 >> 18:22:27.477703 IP 10.218.49.4.59298 > 10.218.50.20.12500: UDP, length >> 13642 >> 18:22:31.158020 IP 10.218.52.42.56745 > 10.218.50.20.12500: UDP, length 91 >> 18:22:35.945376 IP 10.218.49.6.63104 > 10.218.50.20.12500: UDP, length 69 >> 18:22:35.945489 IP 10.218.49.6.63104 > 10.218.50.20.12500: UDP, length 91 >> 18:22:37.279499 IP 10.218.49.5.61843 > 10.218.50.20.12500: UDP, length 108 >> >> Also, I have another node with Graylog2 0.20, messages from same sources >> comes to logstash UDP input without problems. >> >> So, I don't thinks that it is network related problems. >> >> On Thursday, May 7, 2015 at 4:04:26 PM UTC+3, Jochen Schalanda wrote: >>> >>> Hi Arkadiy, >>> >>> not all network devices support UDP packets bigger than 8KiB (8192 >>> bytes) and this seems to be the case on your network somewhere. To >>> circumvent this restriction, Graylog or more specifically the GELF format >>> supports chunking which means splitting a large message into multiple UDP >>> packets of a certain size (see https://www.graylog.org/resources/gelf >>> for details on the GELF format). So in order to solve your problem you >>> could collect the logs on the same machine which produces those raw log >>> lines with a log shipper like nxlog or logstash and send them to Graylog >>> via GELF. Alternatively you'll have to switch from UDP to TCP which doesn't >>> suffer this kind of restriction. >>> >>> Cheers, >>> Jochen >>> >>> On Thursday, 7 May 2015 13:56:25 UTC+2, Arkadiy Shinkarev wrote: >>>> >>>> Hi! >>>> >>>> I'm trying to send messages to raw udp input in Graylog 1.0.2. >>>> The message size is 1k-20k, but Graylog only shown first 8k of message. >>>> >>>> I have configured "recv_buffer_size: 10485760" for input, also >>>> set net.core.rmem_max = 26214400 in sysctl.conf. >>>> >>>> When I run tcpdump, I see that message len is ok (>8k). >>>> When I run strace -e trace=network i see the following: >>>> [pid 10539] recvfrom(365, >>>> "\24\21\0\0\370\370\352\1778q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10539] recvfrom(365, >>>> "\24\21\0\0\371\370\352\1779q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10537] <... recvfrom resumed> >>>> "\24\21\0\0O\372\352\177\235q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10537] recvfrom(365, >>>> "\24\21\0\0X\372\352\177\236q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10541] recvfrom(365, >>>> "\24\21\0\0\200\372\352\177\240q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10541] recvfrom(365, >>>> "\24\21\0\0\201\372\352\177\241q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10534] recvfrom(365, >>>> "\24\21\0\0\302\372\352\177\330q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10534] recvfrom(365, >>>> "\24\21\0\0\303\372\352\177\331q\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10540] <... recvfrom resumed> >>>> "\24\21\0\0$\373\352\177-r\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10540] <... recvfrom resumed> >>>> "\24\21\0\0&\373\352\177/r\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10535] recvfrom(362, >>>> "\24\21\0\0b\373\352\177jr\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10535] recvfrom(362, >>>> "\24\21\0\0c\373\352\177kr\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10537] recvfrom(362, >>>> "\24\21\0\0\315\375\352\177_s\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10537] recvfrom(362, >>>> "\24\21\0\0\316\375\352\177`s\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10538] recvfrom(362, >>>> "\24\21\0\0\v\376\352\177\224s\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10538] recvfrom(362, >>>> "\24\21\0\0\f\376\352\177\225s\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10534] <... recvfrom resumed> >>>> "\24\21\0\0\234\376\352\177\30t\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> [pid 10534] <... recvfrom resumed> >>>> "\24\21\0\0\237\376\352\177\33t\1\0\1\0\0\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >>>> >>>> 8192, 0, NULL, NULL) = 4372 >>>> >>>> I'm also test GELF UDP input, messages comes from logstash (to logstash >>>> messages comes from UDP input with 32k buffer size), there is no problem - >>>> message looks good, Graylog shows full message. >>>> >>>> Where is the problem? >>>> >>>> Some additional information: >>>> OS: CentOS release 6.5 (Final) >>>> Kernel: 2.6.32-431.29.2.el6.centos.plus.x86_64 >>>> Graylog: 1.0.2 >>>> >>>> 2 graylog-server nodes behind load balancer (LVS) + 2 nodes >>>> elasticsearch cluster. >>>> >>>> Thank you! >>>> >>>
-- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
