Re: [graylog2] Graylog 1.0 UDP process buffer performance
Johan, Henrik, I tried to track this problem down.The problem is that the JVM does not cache reverse DNS lookups. The available JVM DNS cache settings like networkaddress.cache.ttl only affect forward DNS lookups. The code for doing the reverse lookups in Graylog did not change in a long time, so this problem is not new in 1.0. I my test setup enabling force_rdns for a syslog input reduced the throughput from around 7000 msg/s to 300 msg/s. This was without a local DNS cache. Once I installed a DNS cache on the Graylog server, the throughput went up to around 3000 msg/s. We will investigate if there is a sane way to cache the reverse lookups ourselves. In the meantime I suggest to test with a DNS cache installed on the Graylog server nodes to see if that helps or to disable the force_rdns setting. Regards, Bernd On 25 February 2015 at 18:00, Bernd Ahlers be...@graylog.com wrote: Johan, Henrik, thanks for the details. I created an issue on GitHub and will investigate. https://github.com/Graylog2/graylog2-server/issues/999 Regards, Bernd On 25 February 2015 at 17:48, Henrik Johansen h...@myunix.dk wrote: Bernd, Correct - that issue started after 0.92.x. We are still seeing evaluated CPU utilisation but we are attributing that to the fact that 0.92 was loosing messages in our setup. On 25 Feb 2015, at 17:37, Bernd Ahlers be...@graylog.com wrote: Henrik, uh, okay. I suppose it worked for you in 0.92 as well? I will create an issue on GitHub for that. Bernd On 25 February 2015 at 17:14, Henrik Johansen h...@myunix.dk wrote: Bernd, We saw the exact same issue - here is a graph over the CPU idle percentage across a few of the cluster nodes during the upgrade : http://5.9.37.177/graylog_cluster_cpu_idle.png We went from ~20% CPU utilisation to ~100% CPU utilisation across ~200 cores and things only settled down after disabling force_rdns. On 25 Feb 2015, at 11:55, Bernd Ahlers be...@graylog.com wrote: Johan, the only thing that changed from 0.92 to 1.0 is that the DNS lookup is now done when the messages are read from the journal and not in the input path where the messages are received. Otherwise, nothing has changed in that regard. We do not do any manual caching of the DNS lookups, but the JVM caches them by default. Check http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html for networkaddress.cache.ttl and networkaddress.cache.negative.ttl. Regards, Bernd On 25 February 2015 at 08:56, sun...@sunner.com wrote: This is strange, I went through all of the settings for my reply, and we are indeed using rdns, and it seems to be the culprit. The strangeness is that it works fine on the old servers even though they're on the same networks, and using the same DNS's and resolver settings. Did something regarding reverse DNS change between 0.92 and 1.0? I'm thinking perhaps the server is trying to do one lookup per message instead of caching reverse lookups, seeing as the latter would result in very little DNS traffic since most of the logs will be coming from a small number of hosts. Regards Johan On Tuesday, February 24, 2015 at 5:08:54 PM UTC+1, Bernd Ahlers wrote: Johan, this sounds very strange indeed. Can you provide us with some more details? - What kind of messages are you pouring into Graylog via UDP? (GELF, raw, syslog?) - Do you have any extractors or grok filters running for the messages coming in via UDP? - Any other differences between the TCP and UDP messages? - Can you show us your input configuration? - Are you using reverse DNS lookups? Thank you! Regards, Bernd On 24 February 2015 at 16:45, sun...@sunner.com wrote: Well that could be a suspect if it wasn't for the fact that the old nodes running on old hardware handle it just fine, along with the fact that the traffic seems to reach the nodes just fine(i.e it actually fills the journal up just fine, and the input buffer never breaks a sweat). And it's really not that much traffic, even spread across four nodes those ~1000 messages per second will cause this whereas the old nodes are just two and can handle it just fine. About disk tuning, I haven't done much of that, and I realize I forgot to mention that the Elasticsearch cluster is on separate physical hardware so there's a minuscule amount of disk I/O happening on the Graylog nodes. It's really very strange since it seems like UDP itself isn't to blame, after all the messages get into Graylog just fine and fills up the journal rapidly. The screenshot from I linked was from after I had stopped sending logs, i.e there was no longer any ingress traffic so the Graylog process had nothing to do except emptying it's journal so it should all be internal processing and egress traffic to Elasticsearch. And as can be seen in the screenshot it seems like it's doing it in small bursts. In the exact same scenario(i.e when I just streamed a large file
Re: [graylog2] Logs from Cisco ASA with bad source field
Roberto, the Cisco ASA does not send valid Syslog, unfortunately. You have to create a Raw input and create extractors. There is a blog post about this here: http://spottedhyena.co.uk/2015/01/graylog2-cisco-asa-cisco-catalyst/ Hope that helps! Regards, Bernd On 27 February 2015 at 15:57, robertocarn...@gmail.com wrote: Dear, I have a Graylog2 version 0.20.6 as our syslog server of our company. I defined an INPUT Syslog UDP running on port UDP/10514, and after that we point several Windows and Linux servers to the Graylog2 with no problems. But in the case of the Cisco ASA firewalls, we have a problem because the source sometimes matches something like: :%ASA-session-6-302013: In the Cisco ASA's I setup: logging enable logging emblem logging trap informational logging history debugging logging asdm debugging logging device-id hostname logging host inside_Frontend 10.1.1.1 format emblem I want to have the original hostname in the source field, so what can I do??? Regards, Roberto -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Index size message count in web panel
Hello, In webpanel - only write-active indice has information about it's size and message count. Active (but not writeable) indexes has only information about time like: Contains messages up to an hour ago. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Problem generating/loading chunked Gelf message in graylog2
Hi, I'm trying to process with gawk a PHP log for loading it graylog2 (I
have many log lines really big). I'm not able of send the correct
information to graylog2 input UDP 12200
If I want to send the next log (is gelf formated) entry to graylog2 using
two chunks how could I do it? What information must have exactly each chunk?
{\n \version\: \1.1\,\n \host\:\phcaeproma01\,\n
\short_message\:\Chunked message\,\n \timestamp\: 123455134,\n
\level\:1,\n \_remote_addr\:\10.1.104.57\,\n
\_idf\:\987297342\,\n \_process\:\Process\,\n
\_uid\:\9798742.938292\,\n \_idcert\:\9386101233\ \n}
I'm able of loading this log line without using chunks (it's a simple log
line sample) I'm trying to send the next two chunks to graylog2:
1.
*\x1e\x0f000102*{\n \version\: \1.1\,\n
\host\:\phcaeproma01\,\n \short_message\:\%s\,\n \timestamp\:
%d,\n \level\:%d,\n \_remote_addr\:\%s\,\n \_idf\:\%s\,\n
\_process\:\%s\,\n
2.
*\x1e\x0f000112*\_uid\:\%s\,\n \_idcert\:\%s\ \n}
and I obtain the next trace in graylog2 server log
*2015-02-26 16:59:05,389 DEBUG:
org.graylog2.plugin.inputs.transports.NettyTransport - More chunks
necessary to complete this message*
*2015-02-26 16:59:05,390 DEBUG:
org.graylog2.inputs.codecs.GelfChunkAggregator - Dumping GELF chunk map
[chunks for 1 messages]:*
*Message 3030303030303031 Chunks:*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*not arrived yet*
*ID: 3030303030303031Sequence: 49/50 Arrival:
1424966345389 Data size: 212*
*not arrived yet*
*2015-02-26 16:59:05,390 DEBUG:
org.graylog2.plugin.inputs.transports.NettyTransport - More chunks
necessary to complete this message*
What I'm doing wrong?
I'm using the next sentences to send the information from gawk server to
graylog2 server:
printf \x1e\x0f%s%c%c%s,0001,48,50,substr(v_cad,1,200) |
/inet/udp/0/10.253.114.218/12200;
printf \x1e\x0f%s%c%c%s,0001,49,50,substr(v_cad,201) |
/inet/udp/0/10.253.114.218/12200;
Thank you very much for any help. It's very important to me be able of send
a long message in chunks
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Graylog 1.0 heap size
Hello, How to set heap size (Xms and Xmx values) in graylog 1.0? I've already set up elasticsearch heap size, by modifying es_heap_size in /etc/default/elasticsearch, but I don't see any similar variable in graylog config file. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Key value pair converter doesn't work? - logstash kv{} worked on this..
Hi, I am trying to convert from an ELK to Graylog, but I am not having much luck extracting useful information. I have a log in key value pair format, and have set up an extractor to copy input and then convert using key value pair. it does not work. the log entry looks like this: hname auditd: date=Feb 26 18:18:28 2015 UTC,fac=f_http_proxy,area=a_libproxycommon,type=t_nettraffic,pri=p_major,pid=2135,ruid=0,euid=0,pgid=2135,logid=0,cmd=httpp,domain=htpp,edomain=htpp,hostname=firewall.example.domain,event=proxy traffic end,service_name=http,netsessid=54ef6373000d8e32,srcip=1.2.3.4,srcport=23862,srcburb=internal,protocol=6,dstip=4.3.2.1,dstport=80,dstburb=external,bytes_written_to_client=246,bytes_written_to_server=528,rule_name=Netflix,cache_hit=0,request_status=0,start_time=Thu Feb 26 11:18:27 2015 As can be seen, this is straight key/value pair with comma delimited pairs and '=' to separate them. Some versions have quoted text, others do not. The logstash kv function worked fine against this. Is there any way to get graylog to do this? I tried grok but the log message can have different field names and I am nowhere near good enough at grok to do this. Appreciate any insight anyone can give for this, it's driving me nuts! -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Logs from Cisco ASA with bad source field
Dear, I have a Graylog2 version 0.20.6 as our syslog server of our company. I defined an INPUT Syslog UDP running on port UDP/10514, and after that we point several Windows and Linux servers to the Graylog2 with no problems. But in the case of the Cisco ASA firewalls, we have a problem because the source sometimes matches something like: :%ASA-session-6-302013: In the Cisco ASA's I setup: logging enable logging emblem logging trap informational logging history debugging logging asdm debugging logging device-id hostname logging host inside_Frontend 10.1.1.1 format emblem I want to have the original hostname in the source field, so what can I do??? Regards, Roberto -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Graylog 1.0 heap size
Previously, you had to define a value in the init script (i.e. shown below) and then add $HEAP_SIZE after the $JAVA variable in the actual line that starts the process. HEAP_SIZE='-Xms3072M -Xmx3072M' However, I noticed this broke in graylog 1.0. It's now inheriting the value from somewhere else. On Friday, February 27, 2015 at 12:32:46 PM UTC-5, Fisz wrote: Hello, How to set heap size (Xms and Xmx values) in graylog 1.0? I've already set up elasticsearch heap size, by modifying es_heap_size in /etc/default/elasticsearch, but I don't see any similar variable in graylog config file. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Journal settings
Thanks I will look into this and update my findings. On Thursday, February 26, 2015 at 12:00:15 AM UTC-7, Bernd Ahlers wrote: Hey, you can tweak the message_journal_max_age and message_journal_max_size settings in your graylog.conf. (see https://github.com/Graylog2/graylog2-server/blob/master/misc/graylog2.conf#L250-L254) That said, if you constantly writing more messages into the journal than you read from it, the journal will always fill up. This indicates that your Graylog server or Elasticsearch server is not able to keep up with the incoming message rate. The journal helps with temporary load spikes or if Elasticsearch is down for a short period. It does not help if you send more messages than you can process. Since your Elasticsearch seems to be bored, you might check the CPU usage of your Graylog server. Do you have lots of extractors? Reverse DNS lookups enabled? Regards, Bernd On 24 February 2015 at 17:16, da...@wildcatracing.com javascript: wrote: I have upgraded to 1.0, I am seeing errors regarding Journaling useage being too high and deletion of messages due to journaling params being too low. what params can I change? I only see on or off when it comes to journaling. I do have a cluster for ES, running two nodes 32gb of ram each, 4 cores. they seem bored looking at the metrics. I have increased the batch size to 1500, but journaling is still running at ~95% and higher. Thanks in advance for any suggestions. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
