[graylog2] Graylog Collector Sidecar Analysis

[Marvin Popyk]

Hello,

We are testing graylog to see if it fits our needs for a centralized 
logging system.  We've installed and setup graylog and we wanted to be able 
to import specific log files to graylog.  We read that graylog collector 
sidecar is an option.  We have setup a new beats input and tested an apache 
collection recommended by the graylog instructions.  That worked like a 
charm.  We setup a new collection to import authentication logs 
(/var/log/auth.log) but it seems like the host that has sidecar installed 
is not getting the updates for the 2nd configuration and is not pushing the 
auth log to graylog.

1. I looked in /etc/graylog/collector-sidecar/collector_sidecar.yml and i 
noticed the tags aren't updated with the new configuration tag
2. I also looked in /etc/graylog/collector-sidecar/generated/filebeat.yml 
and noticed the input_type doesn't match the new configuration file type. 
 I changed it to auth instead of log. 

However, if i edit these 2 yml files with the correct information, graylog 
with start pulling authentication logs. BUT, it will still say the input 
type is LOG instead of AUTH.  

Not sure why the host isn't getting the configuration updates of the 2nd 
configuration for the authentication logs.  I've restarted the service and 
that didn't work.

Also, would you recommend using NXLog instead of Beats?

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/5f4a1918-0fdb-46b7-819b-d70ca0bbeae9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Graylog Web Interface is disconnected

['Rodion Ovodnev' via Graylog Users]

Help me please! What's the problem, look at the log that correct?

1.> 

>

2.> 

>

3.> 

?



graylog/server.log: 
2016-11-29 15:48:11,503 ERROR: 
org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - 
Unhandled exception in REST resource
org.elasticsearch.ElasticsearchTimeoutException: Timeout waiting for task.
at 
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:73)
at 
org.graylog2.indexer.elasticsearch.GlobalTimeoutClient$GlobalTimeoutActionFuture.actionGet(GlobalTimeoutClient.java:163)
at org.graylog2.indexer.searches.Searches.count(Searches.java:190)
at 
org.graylog2.dashboards.widgets.SearchResultCountWidget.computeInternal(SearchResultCountWidget.java:73)
at 
org.graylog2.dashboards.widgets.SearchResultCountWidget.compute(SearchResultCountWidget.java:68)
at 
org.graylog2.dashboards.widgets.DashboardWidget$ComputationResultSupplier.get(DashboardWidget.java:155)
at 
org.graylog2.dashboards.widgets.DashboardWidget$ComputationResultSupplier.get(DashboardWidget.java:151)
at 
com.google.common.base.Suppliers$ExpiringMemoizingSupplier.get(Suppliers.java:192)
at 
org.graylog2.dashboards.widgets.DashboardWidget.getComputationResult(DashboardWidget.java:123)
at 
org.graylog2.rest.resources.dashboards.DashboardsResource.widgetValue(DashboardsResource.java:355)
at sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at 
org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
at 
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at 
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at 
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at 
org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
at 
org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at 
org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:606)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-11-29 15:48:38,980 WARN : org.graylog2.outputs.BlockingBatchedESOutput 
- Error while waiting for healthy Elasticsearch cluster. Not flushing.
java.util.concurrent.TimeoutException: Elasticsearch cluster didn't get 
healthy within timeout
at 

[graylog2] Re: Check activity on a define time range only

[Sébastien cieloch]

Hello Aykisn,

Thank you ! For the first part, I knew that, but you're right with the 2nd 
part ( with the timestamp must match regular expression : blabla ) it was 
exactly my need. I will adapt. 

But how did you find the regular expression with the timestamp ? I tried to 
find it, without real success...  ( 
*[0-9]{4}-[0-9]{2}-[0-9]{2}T[00]|[00-06]:[0-9]{2}:[0-9]+* )

Also sorry about my respond time, I was very busy with other stuff :) 


Le lundi 7 novembre 2016 08:34:19 UTC+1, Aykisn a écrit :

> Actually, there are several options avalable to you, depending on what you 
> want exactly.
>
> 1) If you just want to se if those logs actually exists just do this : 
>
> a) On the search page, just change the timeframe with the absolute 
> settings, and enter the corresponding timeframe, here's an example, which 
> will show you every logs between yesterday night and in the morning today :
>
>
> 
>
>
>
>
>
>
> b) Or you can just search on a one day timeframe, and look at the 
> histogram, which will show you exactly what you want, while not needing to 
> adapt the above settings every time. You can also put this graph in a 
> dashboard for easy acces/view.
>
>
> 
>
>
> 2) If you want to keep track of and see all the logs that are in the wrong 
> timeframe (not between 6am and 22pm), you will have to create a stream with 
> the following rules for example :
>
> - source:yourwindowsserver  ("source matches exactly yourwindowsserver")
> - timestamp must match regular expression : 
> *[0-9]{4}-[0-9]{2}-[0-9]{2}T[00]|[22-23]:[0-9]{2}:[0-9]+*
> - timestamp must match regular expression : 
> *[0-9]{4}-[0-9]{2}-[0-9]{2}T[00]|[00-06]:[0-9]{2}:[0-9]+*
> Not sure about the regexes, but you get the idea.
> All logs coming from your windows server will belong to this stream, but 
> only those who have a timestamp between 22pm to 23:59 pm, and between 0:00 
> to 6am.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/2c88171d-d76e-4290-87da-80b72cbf09a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Pipeline with multiple streams and data accumulation

[Evgeny Shepelyuk]

Hello, 

Is it possible to write a pipeline that will trigger alerts comparing data 
from 2 streams ?
For instance, one stream - is user registration, another stream is clicking 
confirmation emails
So, is it possible to create a pipeline that will trigger an alert when 
count from both streams ain't equal during the day ?

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e74e1636-49d2-4622-a132-65ed3eb2a7a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Source field population when reading from raw/plaintext kafka input

[Evgeny Shepelyuk]

Hello

I've setup Raw/Plaintext Kafka input and I'm able to receive my messages 
from Kafka topic.
My messages are serialized JSON string but not in GELF format.
Unfortunately source field is not populated at all.
Ho should I adjust my message to have source populated ?

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/61684a9c-07e3-4291-8ba1-daf63e4e6406%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Check activity on a define time range only

[Aykisn]

Hi,

My regexes and conditions in my previous post are wrong.

The steam should have two rules :
1) source must match the server(s) name
2) the timestamp should match either between 22 to 00 or between 00 to 06.

If you're not familiar with regexes, I hardly suggest you learn to 
create/use them. If you're going to play with graylog or logs in general, 
it's gonna be hard to work without them.
Here you want to make a regex for the field timestamp (*seen when you 
unwrap a log on the search page*). Should look like this: 
2016-11-30T07:30:10.255Z
The following regex should work to catch all logs between 10 pm to 6 am 
(quickly tested and worked)
  *[0-9]{4}-[0-9]{2}-[0-9]{2}((T0[0-6]:.+)|(T2[2-3]:.+))  * 

I'll add an explanation later if you don't understand it, when I have time.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/594c4395-e156-402d-84df-c7c6a44e88e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.