[graylog2] hostname missing in logs received from syslog-ng

[Li Li]

Hi, all,

A portion of logs received from syslog-ng is missing, for example, logs 
entries expected are:

Jan 12 17:04:22 Lab-PA5020.lab.hsc.net.ou.edu 1,2017/01/12 
17:04:21,0011C102743,TRAFFIC,start,1

But in graylog, "Jan 12 17:04:22 Lab-PA5020.lab.hsc.net.ou.edu 
1,2017/01/12" is missing, logs seen in graylog start with 
"17:04:21,0011C102743,TRAFFIC,start,1"

when I have graylog writing to a file, the logs appear to be correct, ie, 
nothing is missing.

My syslog-ng version is 3.7.3, graylog version is 2.0.3. 

Can anyone give some suggestions? Your help would be greatly appreciated!

Thanks,
-Li

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/baaa281f-5376-4168-8e2c-7a771ca38ce6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: SImple pipelene creation issues

[Jochen Schalanda]

Hi Eugene,

On Friday, 13 January 2017 17:39:50 UTC+1, Evgueni Gordienko wrote:
>
> I did manual message loading and applying the rule and it works as 
> intended.
> No clue how to debug.
> I generate message with create_message("metric:123").
>

Is the "metric" field also there if you search for these messages in 
Graylog?

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/5d6de254-7d8f-402b-aec2-17434c059db0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Incomplete write in php gelf library

[Jochen Schalanda]

Hi,

On Friday, 13 January 2017 12:50:53 UTC+1, Алексей Лашнев wrote:
>
> I'have aready done it. https://github.com/bzikarsky/gelf-php/issues/78 - 
> but there is no reply yet. So i don't know what's the problem there? In 
> graylog or in the library...
>

Since the error message originates from the PHP library, it's a problem 
with either the library, how you integrate the library in your PHP 
application, or with your PHP setup.

Cheers,
Jochen 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/21018e93-4ce4-4b31-b57e-5f1c299f3099%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Seperate Data from streams in defferent elastic nodes

[Jochen Schalanda]

Hi Richard,

On Friday, 13 January 2017 12:40:31 UTC+1, Richard S. Westmoreland wrote:
>
> Wow!  That is going to be an awesome feature in so many different ways. 
>  What kind of timeline do you have for this next release?
>

We're already in beta phase and will probably publish a release candidate 
within the next few weeks.

If you want to help us, feel free to give the beta versions a test drive.

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/d6f0dec4-9c1b-462a-978d-dab4fb6824bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Can I change dashboard source from input to stream?

[Jochen Schalanda]

Hi Joan,

On Friday, 13 January 2017 12:33:35 UTC+1, Joan wrote:
>
> I've seen that some people are exporting as a content pack and editing the 
> json, but is this the simplest way to achieve it?
>

Yes, that's currently the easiest way. Alternatively you can edit the 
dashboard definition in MongoDB (which is basically just another form of a 
JSON-like data structure).

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/912d4312-5a22-4507-9391-6e270b53c4ff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Splunk output plugin error

[Jochen Schalanda]

Hi Frank,

On Friday, 13 January 2017 14:49:56 UTC+1, Frank wrote:
>
> There is a grok filter %{SYSLOGBASE2} (from the default logstash grok 
> patterns) which should format the timestamp correctly.
>

Did you make sure that the "timestamp" field is an actual timestamp and not 
a string after using the Grok extractor?

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/55f3d8f4-3007-4c4a-8a37-1a99bf968972%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Splunk output plugin error

[Frank]

Hi,

these are syslog messages that get into Graylog by a syslog input.
There is a grok filter %{SYSLOGBASE2} (from the default logstash grok 
patterns) which should format the timestamp correctly.
Anyway, we decided to ditch the Splunk output completely, so I don't have 
the possibility to do anymore tests.

Thank you,
Frank

On Thursday, January 12, 2017 at 4:51:30 PM UTC+1, Jochen Schalanda wrote:
>
> Hi Frank,
>
> what's the content of your messages? How are you ingesting them?
>
> Cheers,
> Jochen
>
> On Thursday, 12 January 2017 14:37:52 UTC+1, Frank wrote:
>>
>> That's what I expected. I just added a converter to the timestamp field, 
>> but that didn't change anything.
>>
>> On Thursday, January 12, 2017 at 2:21:40 PM UTC+1, Jochen Schalanda wrote:
>>>
>>> Hi Frank,
>>>
>>> it looks like the "timestamp" message field in one (or more) of your 
>>> messages has the wrong type (String as opposed to being an actual 
>>> timestamp).
>>>
>>> This *shouldn't* happen, but maybe rotating indices (System / Indices / 
>>> Maintenance) will help.
>>>
>>> Cheers,
>>> Jochen
>>>
>>> On Thursday, 12 January 2017 11:55:05 UTC+1, Frank wrote:

 Hi,

 I installed and configured the Splunk output plugin, to forward one 
 stream to Splunk directly.
 But when new messages get routed to the stream, the plugin just logs 
 this error:

 ERROR [OutputBufferProcessor] Error in output [class 
 com.graylog.splunk.output.SplunkOutput].
 java.lang.ClassCastException: Cannot cast java.lang.String to 
 org.joda.time.DateTime
 at java.lang.Class.cast(Class.java:3369) ~[?:1.8.0_111]
 at org.graylog2.plugin.Message.getFieldAs(Message.java:380) 
 ~[graylog.jar:?]
 at org.graylog2.plugin.Message.getTimestamp(Message.java:178) 
 ~[graylog.jar:?]
 at com.graylog.splunk.output.senders.TCPSender.send(TCPSender.java:151) 
 ~[?:?]
 at com.graylog.splunk.output.SplunkOutput.write(SplunkOutput.java:87) 
 ~[?:?]
 at 
 org.graylog2.buffers.processors.OutputBufferProcessor$1.run(OutputBufferProcessor.java:189)
  
 [graylog.jar:?]
 at 
 com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
  
 [graylog.jar:?]
 at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
 [?:1.8.0_111]
 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
 [?:1.8.0_111]
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  
 [?:1.8.0_111]
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  
 [?:1.8.0_111]
 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

 Any ideas how to solve this?

 Frank

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a5cc500c-7d8e-44df-a1ab-05ec14f3b072%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Incomplete write in php gelf library

[Алексей Лашнев]

I'have aready done it. https://github.com/bzikarsky/gelf-php/issues/78 - 
but there is no reply yet. So i don't know what's the problem there? In 
graylog or in the library...
So, i wrote error here too.

On Thursday, January 12, 2017 at 4:29:00 PM UTC+3, Jochen Schalanda wrote:
>
> Hi,
>
> make sure that you're using the latest version of the gelf-php library 
> from https://github.com/bzikarsky/gelf-php. If the problem still occurs, 
> please create a bug report at 
> https://github.com/bzikarsky/gelf-php/issues/.
>
> Cheers,
> Jochen
>
> On Thursday, 12 January 2017 10:35:09 UTC+1, Алексей Лашнев wrote:
>>
>> Hello! I have the probmer with writing logs into gray log with gelf-php 
>> library from time to time.
>> I have 2 servers. Main - with high load and reserve (absolutely copy of 
>> the main server). On the reserve server is all ok. And when i use manually 
>> (with php-gelf) graylog on main server - all is ok! But sometimes i have 
>> the exception from the library:
>>
>> Incomplete write: Only 0 of 358 written in 
>> /data/home/projects/payprocessing/classes/vendor/graylog2/gelf-php/src/Gelf/Transport/StreamSocketClient.php:212
>>
>> Here is the message send to socket:
>>
>> {
>> "version": "1.0",
>> "host": "pay-1.reserve.lan",
>> "short_message": "Redirect to https://www.platron.ru/payment_params.;,
>> "full_message": "Redirect to 
>> https://www.platron.ru/payment_params.php?customer=5d44643437990b1774efb742ed1fb9a031005685\r\n(Process
>>  
>> number: 84073)",
>> "level": 6,
>> "timestamp": 1484144247.0146,
>> "facility": "paypocessing",
>> "file": "Platron::payment"
>> }
>>
>>
>> Some code from library:
>>
>>
>> $socket = $this->getSocket();
>>
>> $byteCount = @fwrite($socket, $buffer);
>>
>> $bufLen = strlen($buffer);
>>
>>
>> if ($byteCount === false) {
>>
>> throw new \RuntimeException("Failed to write to socket");
>>
>> }
>>
>>
>> if ($byteCount !== $bufLen) {
>>
>> throw new \RuntimeException("Incomplete write: Only $byteCount of $bufLen 
>> written");
>>
>> }
>>
>>
>> I try to send it manually. And all is ok. The socket is ok at this line - 
>> i checked that. I thy to call fwite 3 times (because of description fwrite 
>> php function could not to write some times) - and it's don't work.
>>
>>
>> One more comment. Log from GrayLog about this error:
>>
>> 2017-01-12T11:45:15.267+03:00 ERROR [NettyTransport] Error in Input
>> [GELF TCP/585a6742565c11041d194d7c] (channel [id: 0xb735c24e,
>> /10.1.1.10:45950 => /10.1.2.21:12201])
>> java.lang.IllegalStateException: GELF message is too short. Not even the
>> type header would fit.
>> at
>>
>> org.graylog2.inputs.codecs.gelf.GELFMessage.getGELFType(GELFMessage.java:46)
>> ~[graylog.jar:?]
>> at
>>
>> org.graylog2.inputs.codecs.GelfChunkAggregator.addChunk(GelfChunkAggregator.java:95)
>> ~[graylog.jar:?]
>> at
>>
>> org.graylog2.plugin.inputs.transports.NettyTransport$MessageAggregationHandler.messageReceived(NettyTransport.java:303)
>> ~[graylog.jar:?]
>> at
>>
>> org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
>> ~[graylog.jar:?]
>> at
>>
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> [graylog.jar:?]
>> at
>>
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>> [graylog.jar:?]
>> at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>> [graylog.jar:?]
>> at
>>
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
>> [graylog.jar:?]
>> at
>>
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
>> [graylog.jar:?]
>> at
>>
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
>> [graylog.jar:?]
>> at
>>
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>> [graylog.jar:?]
>> at
>>
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> [graylog.jar:?]
>> at
>>
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>> [graylog.jar:?]
>> at
>>
>> org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)
>> [graylog.jar:?]
>> at
>>
>> org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
>> [graylog.jar:?]
>> at
>>
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> [graylog.jar:?]
>> at
>>
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>> [graylog.jar:?]
>> at
>>
>> org.jboss.netty.handler.traffic.AbstractTrafficShapingHandler.messageReceived(AbstractTrafficShapingHandler.java:718)
>> [graylog.jar:?]
>> at
>>
>> 

Re: [graylog2] Re: Seperate Data from streams in defferent elastic nodes

[Richard S. Westmoreland]

Wow!  That is going to be an awesome feature in so many different ways.  What 
kind of timeline do you have for this next release?


> On Jan 13, 2017, at 7:05 PM, Jochen Schalanda  wrote:
> 
> Hi Till,
> 
>> On Friday, 13 January 2017 10:29:45 UTC+1, Till Brinkmann wrote:
>> So does anyone can give us a hint how we can delete the AD loggs by days or 
>> can seperate it in another database store on disk. 
> 
> This will be possible in Graylog 2.2.0 with index sets.
> 
> Cheers,
> Jochen
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/graylog2/b21adcd6-0ef0-4488-931c-3f016dd2fd97%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/354578CB-8201-465D-9DFA-6C36B37E2544%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Can I change dashboard source from input to stream?

[Joan]

I'm on graylog 2.1.2, and recently I started using the roles feature to 
create some read only users.
When we started with graylog we created all the dashboards using the input 
instead of a stream, but now this is an issue because the read only users 
can only see the dashboards but are not able to got to the results.
What would be the simplest approach to change the dashboard's source so 
they use the stream I newly created?
I've seen that some people are exporting as a content pack and editing the 
json, but is this the simplest way to achieve it?

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/23792592-e80b-4e09-bb80-2f681dd3da56%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Seperate Data from streams in defferent elastic nodes

[Till Brinkmann]

OK Thanks !

We will update and read the DOCS. 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e4d58081-d79b-4802-939a-ac0369987e7a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Seperate Data from streams in defferent elastic nodes

[Jochen Schalanda]

Hi Till,

On Friday, 13 January 2017 10:29:45 UTC+1, Till Brinkmann wrote:
>
> So does anyone can give us a hint how we can delete the AD loggs by days 
> or 
> can seperate it in another database store on disk. 
>

This will be possible in Graylog 2.2.0 with index sets.

Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b21adcd6-0ef0-4488-931c-3f016dd2fd97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Seperate Data from streams in defferent elastic nodes

[Till Brinkmann]

Hi there,

we need to seperate huge amound of data of the Windows AD Servers from all 
others. 

The Windows AD Servers are heavy bullshit talking systems ~300msg per 
seconds.  

We do not need to keep this information longer than 5 Days. 

Because of the heavy load from the AD Servers the Elastic/Graylog Database 
nodes will 
be overwirtten around 2 weeks. 

/var/lib/elasticsearch/graylog2/

So does anyone can give us a hint how we can delete the AD loggs by days or 
can seperate it in another database store on disk. 

Thanks Till

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/daf64dab-e6f2-42f2-b5ae-6dcc5d13e323%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: graylog REST: All messages from stream or from specific server

[Till Brinkmann]

THANKS !

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/291fea24-1b5e-4b61-b1cb-252277966a06%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: SImple pipelene creation issues

[Jochen Schalanda]

Hi Evgueni,

do the messages in Graylog, which have been processed by that rule, contain 
the "metric" message field?

Cheers,
Jochen

On Friday, 13 January 2017 03:10:42 UTC+1, Evgueni Gordienko wrote:
>
> Hi All,
>
> Need some help with creating simple test pipeline.
> I created pipeline Test and aded two connections:
> Default stream
> Test stream - which has configured syslog plain output to local host:
> syslog ID: 5878215e60eec31982e38194
> Type: com.wizecore.graylog2.plugin.SyslogOutput
> Edit Delete from stream Delete globally
> format: full
> host: localhost
> keystore: 
> keystorePassword: 
> maxlen: 
> port: 514
> protocol: udp
> truststore: 
> truststorePassword: 
> 
> Then I created simple rule 
> rule "Test"
> when
>   true
> then
>   let out_message = create_message("metric:123");
>   route_to_stream(name:"Test", message:out_message);
> end
>
> I don't use input data - just need to test output.
> The Test stream has rule:
> message must contain "metric"
>
> So I presume to see 
> metric:123
>
> in /var/log/messages but nothing is there.
>
> Questions:
> - Any step by step instructions how to create dummy pipeline with output 
> messages?
> - Any debugging hints for pipelines?
> - Any hints what could be wrong?
>
> Thanks,
> Eugene
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/111a6919-4584-430c-b248-d714395c2e84%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.