Re: [graylog2] Re: graylog2-server receiveBufferSize
Thanks Arie. I have already tried that yesterday and did not help. I have removed unnecessary TCP input, and I don't have any NettyTransport exceptions now. I still have problem with RecvQ in peaks (as seen in netstat) which should be related to slow processing. Do you have any benchmark data with bare-metal vs VM, and different processors numbers / ring_size in graylog2.conf ? On Wed, Jan 28, 2015 at 11:49 AM, Arie satyava...@gmail.com wrote: Maybe this can be helpfull to you: https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html or this for more advanced network tuning: https://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php hth,, Arie On Tuesday, January 27, 2015 at 6:27:39 PM UTC+1, Petar Koraca wrote: Hello, I have some performance issues with graylog2-server 0.92.4 (cannot process more than 7-8k per second), and I think it may be related to UDP buffers. This is CentOS 6 virtual machine with 16 vCPU. $ netstat -ulptn|grep 12201 tcp0 0 :::12201:::* LISTEN 2311/java udp75960 0 :::12201:::* 2311/java I've noticed this in my logs: 16:31:06,719 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for [id: 0x537c78d9, /0:0:0:0:0:0:0:0:12201] should be 1048576 but is 43690. I've set udp_recvbuffer_sizes=1048576 but no luck. Also, I've set net.core.rmem_max from 124928 to 26214400. Any idea where did this 43690 come from? if you need additional information I am at your disposal. Kind regards, Petar Koraca -- You received this message because you are subscribed to a topic in the Google Groups graylog2 group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/graylog2/SR9sqDyZqrU/unsubscribe. To unsubscribe from this group and all its topics, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] search wildcard in quotes
Thanks for clarifications. I think I found a workaround for my issue: source:staging* AND message:(Missing AND assetId*) which is not beautiful but does it's job. Thanks for your time On Wednesday, January 28, 2015 at 1:45:37 PM UTC+1, Arie wrote: Marciej, THis is exactly as I told you. For this type of query you have to specify a default_field AND your contend* search query. The default field could be the input of your messages for example, or any other field that is relied to your search. On Wednesday, January 28, 2015 at 1:24:36 PM UTC+1, Maciej Strömich wrote: This is not exactly true, or I'm misreading something in the elasticsearch docs. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html analyze_wildcard - By default, wildcards terms in a query string are not analyzed. By setting this value to true, a best effort will be made to analyze those as well. So it looks like the query is just incomplete or maybe there are other unknown to me reasons behind this behaviour. On Wednesday, January 28, 2015 at 12:59:31 PM UTC+1, Edmundo Alvarez wrote: Hello, As far as I know, it is not possible to use an exact phrase (a search term enclosed in quotation marks) with wildcards inside in Elasticsearch. The wildcard will be simply ignored. If you only want to check that your query matches both Missing assetId and Missing assetIds, this is what I would do: message:Missing assetId OR message:Missing assetIds I hope that helps. Regards, Edmundo -- Developer Tel.: +49 (0)40 609 452 077 Mobile: +49 (0)171 27 22 181 Mobile (US): +1 (713) 321 8126 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) On 28 Jan 2015, at 11:41, Arie satya...@gmail.com wrote: An the second option I gave, does that work? We experience exactly the same thing. On Tuesday, January 27, 2015 at 2:37:50 PM UTC+1, Maciej Strömich wrote: Hi, I know that allow_leading_wildcard_searches and it's used to search for terms like *something, and I know that it can cause increased memory consumption. My question is strictly connected to the query language. when we query for Missing assetIds Misssing assetIds* the results are found but when we do a search for Missing assetId* there are no results found which is kind of strange because following the docs you could assume that this should search for all occurrences of Missing assetIds. Maybe we're missing something that's why I've asked about the options part :) On Monday, January 26, 2015 at 10:55:47 PM UTC+1, Arie wrote: Hi, such a parameter exist in graylog2.conf, but don't know if it is wise to use. allow_leading_wildcard_searches = false If we are using such searches and it is within an know source or other qualified field we use # source:hostname last acc* hth,, Arie. On Monday, January 26, 2015 at 5:28:02 PM UTC+1, Maciej Strömich wrote: Hi, can someone elaborate a bit on using wildcard searches inside double quotes in GL? We're running 0.92 and have a case where we need to search for an exact phrase with wildcard in the end and it doesn't work for us. e.g. something like message:Missing assetId* Maybe there's an option in graylog2-server conf which needs to be turned on like allow_leading_wildcard_searches? Digging a bit through a group I found only https://groups.google.com/forum/#!searchin/graylog2/wildcard/graylog2/4IQubA243-A/BCnBpW78wQkJ which can be somehow connected with our issue thanks. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] graylog2-server receiveBufferSize
I doubt that this is a graylog issue because it tries to have Netty set the correct size which fails. I don't know centos well enough to tell if there's anything special (I doubt it) but there's also tcp_rmem and tcp_moderate_rcvbuf to check. Pretty sure there are no ulimits in place since this is root, but doesn't hurt to check. On Jan 28, 2015 10:24 AM, Petar Koraca pkor...@gmail.com wrote: Hello Kay, graylog2-server is running as root, on CentOS 6 minimal that does not have additional limits. I've noticed that this exception is shown only with TCP input..so there may be TCP limitation. I haven't tweaked TCP since we use TCP only for Keepalived HTTP checks, and UDP for logs (GELF input). Both TCP and UDP socket are on same port (12001). Could this cause issue with receiveBufferSize on graylog2-server? Thanks On Tuesday, January 27, 2015 at 6:31:50 PM UTC+1, Kay Röpke wrote: Could this be an additional user account level? As long as Netty tries to set the correct value and then complains this sounds like a system limit. On Jan 27, 2015 6:27 PM, Petar Koraca pko...@gmail.com wrote: Hello, I have some performance issues with graylog2-server 0.92.4 (cannot process more than 7-8k per second), and I think it may be related to UDP buffers. This is CentOS 6 virtual machine with 16 vCPU. $ netstat -ulptn|grep 12201 tcp0 0 :::12201:::* LISTEN 2311/java udp75960 0 :::12201:::* 2311/java I've noticed this in my logs: 16:31:06,719 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for [id: 0x537c78d9, /0:0:0:0:0:0:0:0:12201] should be 1048576 but is 43690. I've set udp_recvbuffer_sizes=1048576 but no luck. Also, I've set net.core.rmem_max from 124928 to 26214400. Any idea where did this 43690 come from? if you need additional information I am at your disposal. Kind regards, Petar Koraca -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] graylog2-server receiveBufferSize
Hello Kay, graylog2-server is running as root, on CentOS 6 minimal that does not have additional limits. I've noticed that this exception is shown only with TCP input..so there may be TCP limitation. I haven't tweaked TCP since we use TCP only for Keepalived HTTP checks, and UDP for logs (GELF input). Both TCP and UDP socket are on same port (12001). Could this cause issue with receiveBufferSize on graylog2-server? Thanks On Tuesday, January 27, 2015 at 6:31:50 PM UTC+1, Kay Röpke wrote: Could this be an additional user account level? As long as Netty tries to set the correct value and then complains this sounds like a system limit. On Jan 27, 2015 6:27 PM, Petar Koraca pko...@gmail.com javascript: wrote: Hello, I have some performance issues with graylog2-server 0.92.4 (cannot process more than 7-8k per second), and I think it may be related to UDP buffers. This is CentOS 6 virtual machine with 16 vCPU. $ netstat -ulptn|grep 12201 tcp0 0 :::12201:::* LISTEN 2311/java udp75960 0 :::12201:::* 2311/java I've noticed this in my logs: 16:31:06,719 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for [id: 0x537c78d9, /0:0:0:0:0:0:0:0:12201] should be 1048576 but is 43690. I've set udp_recvbuffer_sizes=1048576 but no luck. Also, I've set net.core.rmem_max from 124928 to 26214400. Any idea where did this 43690 come from? if you need additional information I am at your disposal. Kind regards, Petar Koraca -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: search wildcard in quotes
Not really because when you do source:hostname Missing assetId* you are doing OR and not AND. So all messages containing a hostname OR a Missing assetId* is being searched. When you add AND then no messages are found when using a wildcard. On Wednesday, January 28, 2015 at 11:41:51 AM UTC+1, Arie wrote: An the second option I gave, does that work? We experience exactly the same thing. On Tuesday, January 27, 2015 at 2:37:50 PM UTC+1, Maciej Strömich wrote: Hi, I know that allow_leading_wildcard_searches and it's used to search for terms like *something, and I know that it can cause increased memory consumption. My question is strictly connected to the query language. when we query for Missing assetIds Misssing assetIds* the results are found but when we do a search for Missing assetId* there are no results found which is kind of strange because following the docs you could assume that this should search for all occurrences of Missing assetIds. Maybe we're missing something that's why I've asked about the options part :) On Monday, January 26, 2015 at 10:55:47 PM UTC+1, Arie wrote: Hi, such a parameter exist in graylog2.conf, but don't know if it is wise to use. allow_leading_wildcard_searches = false If we are using such searches and it is within an know source or other qualified field we use # source:hostname last acc* hth,, Arie. On Monday, January 26, 2015 at 5:28:02 PM UTC+1, Maciej Strömich wrote: Hi, can someone elaborate a bit on using wildcard searches inside double quotes in GL? We're running 0.92 and have a case where we need to search for an exact phrase with wildcard in the end and it doesn't work for us. e.g. something like message:Missing assetId* Maybe there's an option in graylog2-server conf which needs to be turned on like allow_leading_wildcard_searches? Digging a bit through a group I found only https://groups.google.com/forum/#!searchin/graylog2/wildcard/graylog2/4IQubA243-A/BCnBpW78wQkJ which can be somehow connected with our issue thanks. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] search wildcard in quotes
I don't think that information is related to the issue you reported, at least if I understood it correctly. As far as I know, we always use query string queries for searching. Anyway, when you type the term assetId* (without quotes), your wildcard gets analysed correctly or it should in most cases (things are more complicated than that). The moment when the wildcard is ignored is when you look for an exact phrase like Message assetId*. Looking around a little I could find this message that reports the same issue: http://elasticsearch-users.115913.n3.nabble.com/Wildcards-in-exact-phrase-in-query-string-search-td4020826.html Regards, Edmundo -- Developer Tel.: +49 (0)40 609 452 077 Mobile: +49 (0)171 27 22 181 Mobile (US): +1 (713) 321 8126 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) On 28 Jan 2015, at 13:24, Maciej Strömich maciej.strom...@gmail.com wrote: This is not exactly true, or I'm misreading something in the elasticsearch docs. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html analyze_wildcard - By default, wildcards terms in a query string are not analyzed. By setting this value to true, a best effort will be made to analyze those as well. So it looks like the query is just incomplete or maybe there are other unknown to me reasons behind this behaviour. On Wednesday, January 28, 2015 at 12:59:31 PM UTC+1, Edmundo Alvarez wrote: Hello, As far as I know, it is not possible to use an exact phrase (a search term enclosed in quotation marks) with wildcards inside in Elasticsearch. The wildcard will be simply ignored. If you only want to check that your query matches both Missing assetId and Missing assetIds, this is what I would do: message:Missing assetId OR message:Missing assetIds I hope that helps. Regards, Edmundo -- Developer Tel.: +49 (0)40 609 452 077 Mobile: +49 (0)171 27 22 181 Mobile (US): +1 (713) 321 8126 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) On 28 Jan 2015, at 11:41, Arie satya...@gmail.com wrote: An the second option I gave, does that work? We experience exactly the same thing. On Tuesday, January 27, 2015 at 2:37:50 PM UTC+1, Maciej Strömich wrote: Hi, I know that allow_leading_wildcard_searches and it's used to search for terms like *something, and I know that it can cause increased memory consumption. My question is strictly connected to the query language. when we query for Missing assetIds Misssing assetIds* the results are found but when we do a search for Missing assetId* there are no results found which is kind of strange because following the docs you could assume that this should search for all occurrences of Missing assetIds. Maybe we're missing something that's why I've asked about the options part :) On Monday, January 26, 2015 at 10:55:47 PM UTC+1, Arie wrote: Hi, such a parameter exist in graylog2.conf, but don't know if it is wise to use. allow_leading_wildcard_searches = false If we are using such searches and it is within an know source or other qualified field we use # source:hostname last acc* hth,, Arie. On Monday, January 26, 2015 at 5:28:02 PM UTC+1, Maciej Strömich wrote: Hi, can someone elaborate a bit on using wildcard searches inside double quotes in GL? We're running 0.92 and have a case where we need to search for an exact phrase with wildcard in the end and it doesn't work for us. e.g. something like message:Missing assetId* Maybe there's an option in graylog2-server conf which needs to be turned on like allow_leading_wildcard_searches? Digging a bit through a group I found only https://groups.google.com/forum/#!searchin/graylog2/wildcard/graylog2/4IQubA243-A/BCnBpW78wQkJ which can be somehow connected with our issue thanks. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups
Re: [graylog2] search wildcard in quotes
Hello, As far as I know, it is not possible to use an exact phrase (a search term enclosed in quotation marks) with wildcards inside in Elasticsearch. The wildcard will be simply ignored. If you only want to check that your query matches both Missing assetId and Missing assetIds, this is what I would do: message:Missing assetId OR message:Missing assetIds I hope that helps. Regards, Edmundo -- Developer Tel.: +49 (0)40 609 452 077 Mobile: +49 (0)171 27 22 181 Mobile (US): +1 (713) 321 8126 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) On 28 Jan 2015, at 11:41, Arie satyava...@gmail.com wrote: An the second option I gave, does that work? We experience exactly the same thing. On Tuesday, January 27, 2015 at 2:37:50 PM UTC+1, Maciej Strömich wrote: Hi, I know that allow_leading_wildcard_searches and it's used to search for terms like *something, and I know that it can cause increased memory consumption. My question is strictly connected to the query language. when we query for Missing assetIds Misssing assetIds* the results are found but when we do a search for Missing assetId* there are no results found which is kind of strange because following the docs you could assume that this should search for all occurrences of Missing assetIds. Maybe we're missing something that's why I've asked about the options part :) On Monday, January 26, 2015 at 10:55:47 PM UTC+1, Arie wrote: Hi, such a parameter exist in graylog2.conf, but don't know if it is wise to use. allow_leading_wildcard_searches = false If we are using such searches and it is within an know source or other qualified field we use # source:hostname last acc* hth,, Arie. On Monday, January 26, 2015 at 5:28:02 PM UTC+1, Maciej Strömich wrote: Hi, can someone elaborate a bit on using wildcard searches inside double quotes in GL? We're running 0.92 and have a case where we need to search for an exact phrase with wildcard in the end and it doesn't work for us. e.g. something like message:Missing assetId* Maybe there's an option in graylog2-server conf which needs to be turned on like allow_leading_wildcard_searches? Digging a bit through a group I found only https://groups.google.com/forum/#!searchin/graylog2/wildcard/graylog2/4IQubA243-A/BCnBpW78wQkJ which can be somehow connected with our issue thanks. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] search wildcard in quotes
Marciej, THis is exactly as I told you. For this type of query you have to specify a default_field AND your contend* search query. The default field could be the input of your messages for example, or any other field that is relied to your search. On Wednesday, January 28, 2015 at 1:24:36 PM UTC+1, Maciej Strömich wrote: This is not exactly true, or I'm misreading something in the elasticsearch docs. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html analyze_wildcard - By default, wildcards terms in a query string are not analyzed. By setting this value to true, a best effort will be made to analyze those as well. So it looks like the query is just incomplete or maybe there are other unknown to me reasons behind this behaviour. On Wednesday, January 28, 2015 at 12:59:31 PM UTC+1, Edmundo Alvarez wrote: Hello, As far as I know, it is not possible to use an exact phrase (a search term enclosed in quotation marks) with wildcards inside in Elasticsearch. The wildcard will be simply ignored. If you only want to check that your query matches both Missing assetId and Missing assetIds, this is what I would do: message:Missing assetId OR message:Missing assetIds I hope that helps. Regards, Edmundo -- Developer Tel.: +49 (0)40 609 452 077 Mobile: +49 (0)171 27 22 181 Mobile (US): +1 (713) 321 8126 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) On 28 Jan 2015, at 11:41, Arie satya...@gmail.com wrote: An the second option I gave, does that work? We experience exactly the same thing. On Tuesday, January 27, 2015 at 2:37:50 PM UTC+1, Maciej Strömich wrote: Hi, I know that allow_leading_wildcard_searches and it's used to search for terms like *something, and I know that it can cause increased memory consumption. My question is strictly connected to the query language. when we query for Missing assetIds Misssing assetIds* the results are found but when we do a search for Missing assetId* there are no results found which is kind of strange because following the docs you could assume that this should search for all occurrences of Missing assetIds. Maybe we're missing something that's why I've asked about the options part :) On Monday, January 26, 2015 at 10:55:47 PM UTC+1, Arie wrote: Hi, such a parameter exist in graylog2.conf, but don't know if it is wise to use. allow_leading_wildcard_searches = false If we are using such searches and it is within an know source or other qualified field we use # source:hostname last acc* hth,, Arie. On Monday, January 26, 2015 at 5:28:02 PM UTC+1, Maciej Strömich wrote: Hi, can someone elaborate a bit on using wildcard searches inside double quotes in GL? We're running 0.92 and have a case where we need to search for an exact phrase with wildcard in the end and it doesn't work for us. e.g. something like message:Missing assetId* Maybe there's an option in graylog2-server conf which needs to be turned on like allow_leading_wildcard_searches? Digging a bit through a group I found only https://groups.google.com/forum/#!searchin/graylog2/wildcard/graylog2/4IQubA243-A/BCnBpW78wQkJ which can be somehow connected with our issue thanks. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: search wildcard in quotes
An the second option I gave, does that work? We experience exactly the same thing. On Tuesday, January 27, 2015 at 2:37:50 PM UTC+1, Maciej Strömich wrote: Hi, I know that allow_leading_wildcard_searches and it's used to search for terms like *something, and I know that it can cause increased memory consumption. My question is strictly connected to the query language. when we query for Missing assetIds Misssing assetIds* the results are found but when we do a search for Missing assetId* there are no results found which is kind of strange because following the docs you could assume that this should search for all occurrences of Missing assetIds. Maybe we're missing something that's why I've asked about the options part :) On Monday, January 26, 2015 at 10:55:47 PM UTC+1, Arie wrote: Hi, such a parameter exist in graylog2.conf, but don't know if it is wise to use. allow_leading_wildcard_searches = false If we are using such searches and it is within an know source or other qualified field we use # source:hostname last acc* hth,, Arie. On Monday, January 26, 2015 at 5:28:02 PM UTC+1, Maciej Strömich wrote: Hi, can someone elaborate a bit on using wildcard searches inside double quotes in GL? We're running 0.92 and have a case where we need to search for an exact phrase with wildcard in the end and it doesn't work for us. e.g. something like message:Missing assetId* Maybe there's an option in graylog2-server conf which needs to be turned on like allow_leading_wildcard_searches? Digging a bit through a group I found only https://groups.google.com/forum/#!searchin/graylog2/wildcard/graylog2/4IQubA243-A/BCnBpW78wQkJ which can be somehow connected with our issue thanks. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: graylog2-server receiveBufferSize
Maybe this can be helpfull to you: https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html or this for more advanced network tuning: https://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php hth,, Arie On Tuesday, January 27, 2015 at 6:27:39 PM UTC+1, Petar Koraca wrote: Hello, I have some performance issues with graylog2-server 0.92.4 (cannot process more than 7-8k per second), and I think it may be related to UDP buffers. This is CentOS 6 virtual machine with 16 vCPU. $ netstat -ulptn|grep 12201 tcp0 0 :::12201:::* LISTEN 2311/java udp75960 0 :::12201:::* 2311/java I've noticed this in my logs: 16:31:06,719 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for [id: 0x537c78d9, /0:0:0:0:0:0:0:0:12201] should be 1048576 but is 43690. I've set udp_recvbuffer_sizes=1048576 but no luck. Also, I've set net.core.rmem_max from 124928 to 26214400. Any idea where did this 43690 come from? if you need additional information I am at your disposal. Kind regards, Petar Koraca -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] search wildcard in quotes
This is not exactly true, or I'm misreading something in the elasticsearch docs. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html analyze_wildcard - By default, wildcards terms in a query string are not analyzed. By setting this value to true, a best effort will be made to analyze those as well. So it looks like the query is just incomplete or maybe there are other unknown to me reasons behind this behaviour. On Wednesday, January 28, 2015 at 12:59:31 PM UTC+1, Edmundo Alvarez wrote: Hello, As far as I know, it is not possible to use an exact phrase (a search term enclosed in quotation marks) with wildcards inside in Elasticsearch. The wildcard will be simply ignored. If you only want to check that your query matches both Missing assetId and Missing assetIds, this is what I would do: message:Missing assetId OR message:Missing assetIds I hope that helps. Regards, Edmundo -- Developer Tel.: +49 (0)40 609 452 077 Mobile: +49 (0)171 27 22 181 Mobile (US): +1 (713) 321 8126 Fax.: +49 (0)40 609 452 078 TORCH GmbH Steckelhörn 11 20457 Hamburg Germany https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) On 28 Jan 2015, at 11:41, Arie satya...@gmail.com javascript: wrote: An the second option I gave, does that work? We experience exactly the same thing. On Tuesday, January 27, 2015 at 2:37:50 PM UTC+1, Maciej Strömich wrote: Hi, I know that allow_leading_wildcard_searches and it's used to search for terms like *something, and I know that it can cause increased memory consumption. My question is strictly connected to the query language. when we query for Missing assetIds Misssing assetIds* the results are found but when we do a search for Missing assetId* there are no results found which is kind of strange because following the docs you could assume that this should search for all occurrences of Missing assetIds. Maybe we're missing something that's why I've asked about the options part :) On Monday, January 26, 2015 at 10:55:47 PM UTC+1, Arie wrote: Hi, such a parameter exist in graylog2.conf, but don't know if it is wise to use. allow_leading_wildcard_searches = false If we are using such searches and it is within an know source or other qualified field we use # source:hostname last acc* hth,, Arie. On Monday, January 26, 2015 at 5:28:02 PM UTC+1, Maciej Strömich wrote: Hi, can someone elaborate a bit on using wildcard searches inside double quotes in GL? We're running 0.92 and have a case where we need to search for an exact phrase with wildcard in the end and it doesn't work for us. e.g. something like message:Missing assetId* Maybe there's an option in graylog2-server conf which needs to be turned on like allow_leading_wildcard_searches? Digging a bit through a group I found only https://groups.google.com/forum/#!searchin/graylog2/wildcard/graylog2/4IQubA243-A/BCnBpW78wQkJ which can be somehow connected with our issue thanks. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: graylog2-server receiveBufferSize
O I forgot (:- Are vmware-tools installed? We recently found some systems that where forgotten, and that has more impact than foreseen. Op woensdag 28 januari 2015 21:25:11 UTC+1 schreef Arie: Petar, we are running on bare metal, with a low load. Tested to 10k messages with the http test input, with everything on one (test)server and running well. I can tell you that in our production systems in our private/local cloud we are encountering severe network/disk related problems with our systems. All of your network is CPU bound. Sometimes there are delays that we can count in seconds. All VM Hosts is running @75% CPU. Must say that we had problems with sending windows eventlogs thru UDP/GELF with nxlog, those were gone when switching tot TCP. Have you already done some graylog2 performance tweaks already? Op woensdag 28 januari 2015 14:41:43 UTC+1 schreef Petar Koraca: Thanks Arie. I have already tried that yesterday and did not help. I have removed unnecessary TCP input, and I don't have any NettyTransport exceptions now. I still have problem with RecvQ in peaks (as seen in netstat) which should be related to slow processing. Do you have any benchmark data with bare-metal vs VM, and different processors numbers / ring_size in graylog2.conf ? On Wed, Jan 28, 2015 at 11:49 AM, Arie satya...@gmail.com wrote: Maybe this can be helpfull to you: https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html or this for more advanced network tuning: https://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php hth,, Arie On Tuesday, January 27, 2015 at 6:27:39 PM UTC+1, Petar Koraca wrote: Hello, I have some performance issues with graylog2-server 0.92.4 (cannot process more than 7-8k per second), and I think it may be related to UDP buffers. This is CentOS 6 virtual machine with 16 vCPU. $ netstat -ulptn|grep 12201 tcp0 0 :::12201:::* LISTEN 2311/java udp75960 0 :::12201:::* 2311/java I've noticed this in my logs: 16:31:06,719 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for [id: 0x537c78d9, /0:0:0:0:0:0:0:0:12201] should be 1048576 but is 43690. I've set udp_recvbuffer_sizes=1048576 but no luck. Also, I've set net.core.rmem_max from 124928 to 26214400. Any idea where did this 43690 come from? if you need additional information I am at your disposal. Kind regards, Petar Koraca -- You received this message because you are subscribed to a topic in the Google Groups graylog2 group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/graylog2/SR9sqDyZqrU/unsubscribe. To unsubscribe from this group and all its topics, send an email to graylog2+u...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: graylog2-server receiveBufferSize
Petar, we are running on bare metal, with a low load. Tested to 10k messages with the http test input, with everything on one (test)server and running well. I can tell you that in our production systems in our private/local cloud we are encountering severe network/disk related problems with our systems. All of your network is CPU bound. Sometimes there are delays that we can count in seconds. All VM Hosts is running @75% CPU. Must say that we had problems with sending windows eventlogs thru UDP/GELF with nxlog, those were gone when switching tot TCP. Have you already done some graylog2 performance tweaks already? Op woensdag 28 januari 2015 14:41:43 UTC+1 schreef Petar Koraca: Thanks Arie. I have already tried that yesterday and did not help. I have removed unnecessary TCP input, and I don't have any NettyTransport exceptions now. I still have problem with RecvQ in peaks (as seen in netstat) which should be related to slow processing. Do you have any benchmark data with bare-metal vs VM, and different processors numbers / ring_size in graylog2.conf ? On Wed, Jan 28, 2015 at 11:49 AM, Arie satya...@gmail.com javascript: wrote: Maybe this can be helpfull to you: https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html or this for more advanced network tuning: https://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php hth,, Arie On Tuesday, January 27, 2015 at 6:27:39 PM UTC+1, Petar Koraca wrote: Hello, I have some performance issues with graylog2-server 0.92.4 (cannot process more than 7-8k per second), and I think it may be related to UDP buffers. This is CentOS 6 virtual machine with 16 vCPU. $ netstat -ulptn|grep 12201 tcp0 0 :::12201:::* LISTEN 2311/java udp75960 0 :::12201:::* 2311/java I've noticed this in my logs: 16:31:06,719 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for [id: 0x537c78d9, /0:0:0:0:0:0:0:0:12201] should be 1048576 but is 43690. I've set udp_recvbuffer_sizes=1048576 but no luck. Also, I've set net.core.rmem_max from 124928 to 26214400. Any idea where did this 43690 come from? if you need additional information I am at your disposal. Kind regards, Petar Koraca -- You received this message because you are subscribed to a topic in the Google Groups graylog2 group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/graylog2/SR9sqDyZqrU/unsubscribe. To unsubscribe from this group and all its topics, send an email to graylog2+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
