[graylog2] Re: Multiple nodes in a cluster

[Steve Kuntz]

As well some additional logs form the nodes are

[NodePingThread] Did not find meta info of this node. Re-registering. I 
have changed all IPs appropriately in the configuration of the 3rd node.

On Wednesday, August 24, 2016 at 4:15:02 PM UTC-4, Steve Kuntz wrote:
>
> I have 2 nodes running in a cluster, one master and one slave. When I look 
> at the nodes collection in Mongo I see the 2 nodes.
>
> I have added a 3rd node as a slave and when this node is running I end up 
> with an issue where the cluster is complaining that there isn't a master 
> node. When I look at the node collection in Mongo it looks like there are 
> many different states, sometimes I will see the all 3, but when I look a 
> second or 2 later, there is only one of the slave and no master or just the 
> master. This rotation only happens when the 2nd slave is added and returns 
> to normal when it is stopped. All the node-id files are different. I copied 
> the configuration from from the other slave but have I coped something that 
> I should have? What is causing the collection to be overwritten by the 
> second slave? I did start it as a master accidentally the first time, is 
> there a cache somewhere I should delete?
>
> All help is greatly appreciated
>
> Thanks in advance. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a1f7f8c0-dbac-423b-b90e-9c7bde16ded3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Multiple nodes in a cluster

[Steve Kuntz]

I have 2 nodes running in a cluster, one master and one slave. When I look 
at the nodes collection in Mongo I see the 2 nodes.

I have added a 3rd node as a slave and when this node is running I end up 
with an issue where the cluster is complaining that there isn't a master 
node. When I look at the node collection in Mongo it looks like there are 
many different states, sometimes I will see the all 3, but when I look a 
second or 2 later, there is only one of the slave and no master or just the 
master. This rotation only happens when the 2nd slave is added and returns 
to normal when it is stopped. All the node-id files are different. I copied 
the configuration from from the other slave but have I coped something that 
I should have? What is causing the collection to be overwritten by the 
second slave? I did start it as a master accidentally the first time, is 
there a cache somewhere I should delete?

All help is greatly appreciated

Thanks in advance. 

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/2aa40ef4-47b0-42a7-844b-05412dd9a489%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Removing a Graylog node from the cluster

[Steve Kuntz]

Hi,

I've been unable to find any documentation around this. How do I completely 
remove a graylog node from the cluster?

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/020aa6f0-b41f-49c5-b7c4-ef048ac61c68%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog Failing jvm Allocation Failure [jvm] [graylog-4e9a7285-48ce-468c-8604-6b2bf613eafd] [gc][old][501][37] duration [38.6s],

[Ricardo Ferreira]

Hey Jochen,
tried with Xmx=30Gb

changed the GC algo and i still have big pauses...

2016-08-24T17:06:30.042Z WARN  [GarbageCollectionWarningThread] Last GC run 
with G1 Young Generation took longer than 1 second (last duration=10443 
milliseconds)
2016-08-24T17:06:30.042Z WARN  [jvm] 
[graylog-4e9a7285-48ce-468c-8604-6b2bf613eafd] [gc][young][65][35] duration 
[10.4s], collections [1]/[10.6s], total [10.4s]/[36.9s], memory 
[29.2gb]->[29.3gb]/[30gb], all_pools {[young] 
[13.1gb]->[8mb]/[0b]}{[survivor] [56mb]->[152mb]/[0b]}{[old] 
[15.9gb]->[29.1gb]/[30gb]}

2016-08-24T17:06:30.391Z WARN  [NodePingThread] Did not find meta info of 
this node. Re-registering.
2016-08-24T17:06:51.065Z WARN  [NodePingThread] Did not find meta info of 
this node. Re-registering.
2016-08-24T17:06:51.065Z WARN  [jvm] 
[graylog-4e9a7285-48ce-468c-8604-6b2bf613eafd] [gc][old][66][2] duration 
[18.8s], collections [1]/[20.9s], total [18.8s]/[41.6s], memory 
[29.3gb]->[8.5gb]/[30gb], all_pools {[young] [8mb]->[0b]/[0b]}{[survivor] 
[152mb]->[0b]/[0b]}{[old] [29.1gb]->[8.5gb]/[30gb]}

2016-08-24T17:06:51.276Z WARN  [GarbageCollectionWarningThread] Last GC run 
with G1 Young Generation took longer than 1 second (last duration=1299 
milliseconds)
2016-08-24T17:06:51.284Z WARN  [GarbageCollectionWarningThread] Last GC run 
with G1 Old Generation took longer than 1 second (last duration=18865 
milliseconds)

Will let it run for a hour then ill post the GC logs here


On Wednesday, 24 August 2016 14:12:45 UTC+1, Jochen Schalanda wrote:
>
> Hi Ricardo,
>
> try configuring *less* heap memory for your JVM, ideally less than 32G. 
> See 
> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
>  
> for details.
>
> Cheers,
> Jochen
>
> On Wednesday, 24 August 2016 15:02:10 UTC+2, Ricardo Ferreira wrote:
>>
>> So, we have a 2 identical node graylog setup:
>>
>>
>>  ProLiant DL380 Gen9 (719061-B21) 
>>  32: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
>>  128GB RAM
>>  20MiB L3 cache
>>  2147479552 bytes (2.1 GB) copied, 12.8211 s, 167 MB/s
>>
>> 016-08-24T10:46:34.255Z INFO  [CmdLineTool] Running with JVM arguments: 
>> -Xms32g -Xmx32g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC 
>> -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC 
>> -XX:-OmitStackTraceInFastThrow -Xloggc:/tmp/memoryGC.log 
>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps 
>> -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml 
>> -Djava.library.path=/usr/share/graylog-server/lib/sigar 
>> -Dgraylog2.installation_source=deb
>>
>> different from default configs:
>>
>>
>> elasticsearch_shards = 5
>> elasticsearch_replicas = 1
>>
>> processbuffer_processors = 15
>> outputbuffer_processors = 12
>> ring_size = 4194304
>> inputbuffer_ring_size = 4194304
>> inputbuffer_processors = 6
>> inputbuffer_wait_strategy = blocking
>> message_journal_max_size = 30gb
>> async_eventbus_processors = 4
>>
>> So our environment does ~ 30-50GB per hour in this testing phase we are 
>> at ~5kmsg/s
>> If you look at the graphs we started doing a run just into one done and 
>> was changed to another node around 11:50UTC
>>
>> While it was running i noticed that the graylog node was de registering 
>> himself:
>>
>> 2016-08-24T10:46:57.254Z WARN  [jvm] 
>> [graylog-1743aa7d-c19e-48fa-89fa-f2e619ec0692] [gc][young][8][4] duration 
>> [3.1s], collections [1]/[3.9s], total [3.1s]/[3.8s], memory 
>> [9gb]->[2gb]/[30.4gb], all_pools {[young] 
>> [7.8gb]->[445.1mb]/[12.8gb]}{[survivor] [1.2gb]->[886.4mb]/[1.5gb]}{[old] 
>> [0b]->[770.9mb]/[16gb]}
>> 2016-08-24T10:46:57.374Z INFO  [connection] Opened connection 
>> [connectionId{localValue:22, serverValue:790}] to 172.24.9.59:27017
>> 2016-08-24T10:46:57.402Z INFO  [connection] Opened connection 
>> [connectionId{localValue:21, serverValue:789}] to 172.24.9.59:27017
>> 2016-08-24T10:46:57.416Z INFO  [connection] Opened connection 
>> [connectionId{localValue:23, serverValue:791}] to 172.24.9.59:27017
>> 2016-08-24T10:47:03.900Z INFO  [jvm] 
>> [graylog-1743aa7d-c19e-48fa-89fa-f2e619ec0692] [gc][young][13][9] duration 
>> [1.5s], collections [2]/[2.4s], total [1.5s]/[6s], memory 
>> [7.9gb]->[4.2gb]/[30.4gb], all_pools {[young] 
>> [5.2gb]->[1.5gb]/[12.8gb]}{[survivor] [1.4gb]->[599.3mb]/[1.5gb]}{[old] 
>> [1.1gb]->[2.9gb]/[16gb]}
>> 2016-08-24T10:47:08.949Z INFO  [jvm] 
>> [graylog-1743aa7d-c19e-48fa-89fa-f2e619ec0692] [gc][young][17][12] duration 
>> [1.5s], collections [2]/[2s], total [1.5s]/[7.7s], memory 
>> [13.1gb]->[5.8gb]/[30.4gb], all_pools {[young] 
>> [8.9gb]->[910.8mb]/[12.8gb]}{[survivor] [1.2gb]->[102.5mb]/[1.5gb]}{[old] 
>> [2.9gb]->[4.8gb]/[16gb]}
>> 2016-08-24T10:47:11.874Z INFO  [connection] Opened connection 
>> [connectionId{localValue:28, serverValue:794}] to 172.24.9.59:27017
>> 2016-08-24T10:47:11.870Z INFO  [connection] Opened connection 
>> [connectionId{localValue:29, serverValue:797}] to 172.24.9.59:27017
>> 2016-08-24T10:47:11.878Z 

[graylog2] Re: Graylog 2.0.3 recommended MongoDB version

[Aleksey Chudov]

Thank you!

On Wednesday, August 24, 2016 at 6:21:19 PM UTC+3, Jochen Schalanda wrote:
>
> Hi Aleksey,
>
> while we recommend using the latest stable version of MongoDB for Graylog 
> 2.x, MongoDB 2.6.x from EPEL should also work fine.
>
>
> Cheers,
> Jochen
>
> On Wednesday, 24 August 2016 16:00:33 UTC+2, Aleksey Chudov wrote:
>>
>> Hi,
>>
>> In accodrance with current documentation 
>> http://docs.graylog.org/en/2.0/pages/installation/operating_system_packages.html#prerequisites
>>  Graylog 
>> requires MongoDB (>= 2.4) and CentOS installation instruction 
>> http://docs.graylog.org/en/2.0/pages/installation/os/centos.html 
>> describes installing the official MongoDB 3.2.
>>
>> I find it easier not to add additional repositories. So, I have a couple 
>> of questions. Is there any issues with MongoDB 2.6.11 from EPEL 7 
>> repository? Which version is best to choose for multi-node production 
>> deployment?
>>
>> Regards,
>> Aleksey
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/398eeca2-7d52-4ba1-bd95-3c79081af93b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Issues parsing incomming fields in a good way

[Jochen Schalanda]

Hi,

looking at the bash snippet you've posted, it should be fairly easy to 
iterate over the scanned/infected files and create a separate GELF message 
for each.

If you need to know, which infected files were found by the same scan, you 
can simply add a unique identifier to the GELF messages (using the same 
identifier for all infected files found by the same scan).

See http://stackoverflow.com/a/19772067, 
https://unix.stackexchange.com/questions/39473/command-substitution-splitting-on-newline-but-not-space,
 
and http://wiki.bash-hackers.org/commands/builtin/mapfile for some 
inspiration.

Cheers,
Jochen

On Wednesday, 24 August 2016 13:13:52 UTC+2, ravedog wrote:
>
> Hi Jochen,  
>
> First of all, thanks again for taking your time. Its very highly 
> appreciated :)
>
> Ok sure, from a bash script, the GELF is generated like this:
>
> FILES=$(/bin/cat /var/log/avscanoutputfile | /bin/grep ^/)
>
> echo -e '{
>"version": "1.1",
>"host":"'${HOSTNAME}'", 
>"short_message":"Clam AV Scan", 
>"level":6, 
>... #Alot of other customrows that are not 
> regarding this matter.
>"_clamav_infeced_files":"'${FILES}'"
>"_clamav_infected":"'${number of infected files}'"
>}\0' | nc -w 1 theserver.company.net 12201
>
> The avscanoutputfile which is read into FILE var above, looks like this:
>
>
>
>
>
> ---
>
> /var/log/syslog.1: Eicar-Test-Signature FOUND
> /home/JIMBOB/Desktop/Untitled Document: Eicar-Test-Signature FOUND
>
> --- SCAN SUMMARY ---
> Known viruses: 4745906
> Engine version: 0.99
> Scanned directories: 39075
> Scanned files: 209569
> Infected files: 2
> Data scanned: 11207.61 MB
> Data read: 891244.70 MB (ratio 0.01:1)
> Time: 734.572 sec (12 m 14 s)
>
> The big issue here is that I need each line (such as /var/log/syslog.1: 
> Eicar-Test-Signature FOUND) to be a separate field in above GELF. 
>
> Something like:
> echo -e '{
>"version": "1.1",
>"host":"'${HOSTNAME}'", 
>"short_message":"Clam AV Scan", 
>"level":6, 
>... #Alot of other customrows that are not 
> regarding this matter.
>"_clamav_infeced_file1":"/var/log/syslog.1: 
> Eicar-Test-Signature FOUND"
>   
>  "_clamav_infeced_file2":"/home/JIMBOB/Desktop/Untitled Document: 
> Eicar-Test-Signature FOUND"
>"_clamav_infected":"'${number of infected files}'"
>}\0' | nc -w 1 theserver.company.net 12201
>
> The issue here is that i don't know if there is 0 lines that matched the 
> FILES grep above, or 20. 
>
> I guess this turns out to be more of a bash scripting issue now, but i 
> really need a reliable way to read each line of that file into a separate 
> field.
>
> Any suggestions are highly appreciated.
>
> Thanks,
> R
>
> (EDIT: all the previous text was included, removed and re added the answer)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/3896de79-b24f-4da0-b7a7-ef7bf6f3d97e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog 2.0.3 recommended MongoDB version

[Jochen Schalanda]

Hi Aleksey,

while we recommend using the latest stable version of MongoDB for Graylog 
2.x, MongoDB 2.6.x from EPEL should also work fine.


Cheers,
Jochen

On Wednesday, 24 August 2016 16:00:33 UTC+2, Aleksey Chudov wrote:
>
> Hi,
>
> In accodrance with current documentation 
> http://docs.graylog.org/en/2.0/pages/installation/operating_system_packages.html#prerequisites
>  Graylog 
> requires MongoDB (>= 2.4) and CentOS installation instruction 
> http://docs.graylog.org/en/2.0/pages/installation/os/centos.html 
> describes installing the official MongoDB 3.2.
>
> I find it easier not to add additional repositories. So, I have a couple 
> of questions. Is there any issues with MongoDB 2.6.11 from EPEL 7 
> repository? Which version is best to choose for multi-node production 
> deployment?
>
> Regards,
> Aleksey
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a8293c2d-74f9-47c4-a765-cb2f1f47de9b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Sidecar vs nxlog only?

[Nathan Mace]

Great, I thought that was the case but wasn't sure if there was more to 
Sidecar or not.

Thanks!

Nathan

On Wednesday, August 24, 2016 at 10:31:08 AM UTC-4, Marius Sturm wrote:
>
> Hi Nathan,
> it's basically a configuration layer. With Sidecar in between you can 
> control nxlog from within the Graylog web ui.
>
> Cheers,
> Marius
>
>
> On 24 August 2016 at 16:07, Nathan Mace  
> wrote:
>
>> I'm starting to roll out nxlog / Sidecar to replace our Splunk install.  
>> However the Windows Event Logs seem to make it into Graylog just fine 
>> without Sidecar being installed.  What does installing Sidecar add to the 
>> mix?
>>
>> Nathan
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Graylog Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to graylog2+u...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/graylog2/7ebc3980-5adf-42c3-83bf-9cb5992cc682%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Developer
>
> Tel.: +49 (0)40 609 452 077
> Fax.: +49 (0)40 609 452 078
>
> TORCH GmbH - A Graylog Company
> Poolstraße 21
> 20335 Hamburg
> Germany
>
> https://www.graylog.com 
>
> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
> Geschäftsführer: Lennart Koopmann (CEO)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e488c65f-0e62-490a-87b5-1aca7a5212f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Sidecar vs nxlog only?

[Marius Sturm]

Hi Nathan,
it's basically a configuration layer. With Sidecar in between you can
control nxlog from within the Graylog web ui.

Cheers,
Marius


On 24 August 2016 at 16:07, Nathan Mace  wrote:

> I'm starting to roll out nxlog / Sidecar to replace our Splunk install.
> However the Windows Event Logs seem to make it into Graylog just fine
> without Sidecar being installed.  What does installing Sidecar add to the
> mix?
>
> Nathan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/graylog2/7ebc3980-5adf-42c3-83bf-9cb5992cc682%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Developer

Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog Company
Poolstraße 21
20335 Hamburg
Germany

https://www.graylog.com 

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAMqbBbKZsfRqOnCMfdigtkZybQHDFTc--GnoDEXhWuST_rE%3DBA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Sidecar vs nxlog only?

[Nathan Mace]

I'm starting to roll out nxlog / Sidecar to replace our Splunk install. 
 However the Windows Event Logs seem to make it into Graylog just fine 
without Sidecar being installed.  What does installing Sidecar add to the 
mix?

Nathan

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/7ebc3980-5adf-42c3-83bf-9cb5992cc682%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Graylog 2.0.3 recommended MongoDB version

[Aleksey Chudov]

Hi,

In accodrance with current 
documentation 
http://docs.graylog.org/en/2.0/pages/installation/operating_system_packages.html#prerequisites
 Graylog 
requires MongoDB (>= 2.4) and CentOS installation 
instruction http://docs.graylog.org/en/2.0/pages/installation/os/centos.html 
describes installing the official MongoDB 3.2.

I find it easier not to add additional repositories. So, I have a couple of 
questions. Is there any issues with MongoDB 2.6.11 from EPEL 7 repository? 
Which version is best to choose for multi-node production deployment?

Regards,
Aleksey

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/7804b0c5-1511-48f9-bfc0-554a739f264b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Graylog 2.0 SSL issue

[Jochen Schalanda]

Hi Anant,

maybe Midori is using another certificate store than the other web browsers 
you've mentioned.

In any case, if you're using a self-signed certificate, you need to add 
this certificate to the list of trusted certificates in your web browser or 
system trust store.

On an additional note, you might want to replace the certificate and 
private key you've posted in your first mail for security reasons.

Cheers,
Jochen

On Wednesday, 24 August 2016 15:34:39 UTC+2, Anant Sawant wrote:
>
> Hi Dennis,
>
> I am looking for other details that I might have missed but there is this 
> strange thing happening, I tried to run Graylog on "Midori" without any 
> changes in the configuration and its running just fine, but its not on 
> Mozilla, IE or Chrome. I am attaching the screenshot. Please advise
>
> Thanks,
> Anant.
>
> On Wednesday, 24 August 2016 18:07:39 UTC+5:30, Dennis Oelkers wrote:
>>
>> Hey Anant, 
>>
>> it looks like https://172.16.0.78:12900/ 
>> 
>>  
>> is not reachable from your browser. Please make sure that your browser can 
>> connect to the REST API. For further information, please have a look at 
>> http://docs.graylog.org/en/2.0/pages/configuration/web_interface.html. 
>>
>> Kind regards, 
>> D. 
>>
>> > On 24.08.2016, at 13:47, Anant Sawant  wrote: 
>> > 
>> > Hi all, 
>> > 
>> > We have installed Graylog 2.0 manually  on ubuntu 14.04.1 and was 
>> running  fine on http. 
>> > We are now trying to run Graylog 2.0 web interface to run on "HTTPS". 
>> For this we followed the following documentation "
>> http://docs.graylog.org/en/2.0/pages/configuration/https.html#ssl-setup;. 
>>
>> > 
>> > After making the required changes we are unable to get the web 
>> interface on "https://x.x.x.x:9000;. 
>> > 
>> >  I have attached the screenshot of the error and also the certificates 
>> created following the documentations and the server.conf file. 
>> > 
>> > Please advise to overcome this. 
>> > 
>> > Thanks in advance. 
>> > Anant 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/91e8ff9c-e2d0-4599-9512-2a630765a0ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Graylog 2.0 SSL issue

[Anant Sawant]

Hi Dennis,

I am looking for other details that I might have missed but there is this 
strange thing happening, I tried to run Graylog on "Midori" without any 
changes in the configuration and its running just fine, but its not on 
Mozilla, IE or Chrome. I am attaching the screenshot. Please advise

Thanks,
Anant.

On Wednesday, 24 August 2016 18:07:39 UTC+5:30, Dennis Oelkers wrote:
>
> Hey Anant, 
>
> it looks like https://172.16.0.78:12900/ 
> 
>  
> is not reachable from your browser. Please make sure that your browser can 
> connect to the REST API. For further information, please have a look at 
> http://docs.graylog.org/en/2.0/pages/configuration/web_interface.html. 
>
> Kind regards, 
> D. 
>
> > On 24.08.2016, at 13:47, Anant Sawant  
> wrote: 
> > 
> > Hi all, 
> > 
> > We have installed Graylog 2.0 manually  on ubuntu 14.04.1 and was 
> running  fine on http. 
> > We are now trying to run Graylog 2.0 web interface to run on "HTTPS". 
> For this we followed the following documentation "
> http://docs.graylog.org/en/2.0/pages/configuration/https.html#ssl-setup;. 
> > 
> > After making the required changes we are unable to get the web interface 
> on "https://x.x.x.x:9000;. 
> > 
> >  I have attached the screenshot of the error and also the certificates 
> created following the documentations and the server.conf file. 
> > 
> > Please advise to overcome this. 
> > 
> > Thanks in advance. 
> > Anant 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Graylog Users" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to graylog2+u...@googlegroups.com . 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/graylog2/0429a7ba-defa-40c7-9f6d-aa3eb8b637b5%40googlegroups.com.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
> >  
>
> -- 
> Tel.: +49 (0)40 609 452 077 
> Fax.: +49 (0)40 609 452 078 
>
> TORCH GmbH - A Graylog company 
> Poolstrasse 21 
> 20355 Hamburg 
> Germany 
>
> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 
> Geschäftsführer: Lennart Koopmann (CEO) 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/30f43a11-7053-420c-903c-87c04b22b136%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog Failing jvm Allocation Failure [jvm] [graylog-4e9a7285-48ce-468c-8604-6b2bf613eafd] [gc][old][501][37] duration [38.6s],

[Jochen Schalanda]

Hi Ricardo,

try configuring *less* heap memory for your JVM, ideally less than 32G. 
See 
https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
 
for details.

Cheers,
Jochen

On Wednesday, 24 August 2016 15:02:10 UTC+2, Ricardo Ferreira wrote:
>
> So, we have a 2 identical node graylog setup:
>
>
>  ProLiant DL380 Gen9 (719061-B21) 
>  32: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
>  128GB RAM
>  20MiB L3 cache
>  2147479552 bytes (2.1 GB) copied, 12.8211 s, 167 MB/s
>
> 016-08-24T10:46:34.255Z INFO  [CmdLineTool] Running with JVM arguments: 
> -Xms32g -Xmx32g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC 
> -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC 
> -XX:-OmitStackTraceInFastThrow -Xloggc:/tmp/memoryGC.log 
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps 
> -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml 
> -Djava.library.path=/usr/share/graylog-server/lib/sigar 
> -Dgraylog2.installation_source=deb
>
> different from default configs:
>
>
> elasticsearch_shards = 5
> elasticsearch_replicas = 1
>
> processbuffer_processors = 15
> outputbuffer_processors = 12
> ring_size = 4194304
> inputbuffer_ring_size = 4194304
> inputbuffer_processors = 6
> inputbuffer_wait_strategy = blocking
> message_journal_max_size = 30gb
> async_eventbus_processors = 4
>
> So our environment does ~ 30-50GB per hour in this testing phase we are at 
> ~5kmsg/s
> If you look at the graphs we started doing a run just into one done and 
> was changed to another node around 11:50UTC
>
> While it was running i noticed that the graylog node was de registering 
> himself:
>
> 2016-08-24T10:46:57.254Z WARN  [jvm] 
> [graylog-1743aa7d-c19e-48fa-89fa-f2e619ec0692] [gc][young][8][4] duration 
> [3.1s], collections [1]/[3.9s], total [3.1s]/[3.8s], memory 
> [9gb]->[2gb]/[30.4gb], all_pools {[young] 
> [7.8gb]->[445.1mb]/[12.8gb]}{[survivor] [1.2gb]->[886.4mb]/[1.5gb]}{[old] 
> [0b]->[770.9mb]/[16gb]}
> 2016-08-24T10:46:57.374Z INFO  [connection] Opened connection 
> [connectionId{localValue:22, serverValue:790}] to 172.24.9.59:27017
> 2016-08-24T10:46:57.402Z INFO  [connection] Opened connection 
> [connectionId{localValue:21, serverValue:789}] to 172.24.9.59:27017
> 2016-08-24T10:46:57.416Z INFO  [connection] Opened connection 
> [connectionId{localValue:23, serverValue:791}] to 172.24.9.59:27017
> 2016-08-24T10:47:03.900Z INFO  [jvm] 
> [graylog-1743aa7d-c19e-48fa-89fa-f2e619ec0692] [gc][young][13][9] duration 
> [1.5s], collections [2]/[2.4s], total [1.5s]/[6s], memory 
> [7.9gb]->[4.2gb]/[30.4gb], all_pools {[young] 
> [5.2gb]->[1.5gb]/[12.8gb]}{[survivor] [1.4gb]->[599.3mb]/[1.5gb]}{[old] 
> [1.1gb]->[2.9gb]/[16gb]}
> 2016-08-24T10:47:08.949Z INFO  [jvm] 
> [graylog-1743aa7d-c19e-48fa-89fa-f2e619ec0692] [gc][young][17][12] duration 
> [1.5s], collections [2]/[2s], total [1.5s]/[7.7s], memory 
> [13.1gb]->[5.8gb]/[30.4gb], all_pools {[young] 
> [8.9gb]->[910.8mb]/[12.8gb]}{[survivor] [1.2gb]->[102.5mb]/[1.5gb]}{[old] 
> [2.9gb]->[4.8gb]/[16gb]}
> 2016-08-24T10:47:11.874Z INFO  [connection] Opened connection 
> [connectionId{localValue:28, serverValue:794}] to 172.24.9.59:27017
> 2016-08-24T10:47:11.870Z INFO  [connection] Opened connection 
> [connectionId{localValue:29, serverValue:797}] to 172.24.9.59:27017
> 2016-08-24T10:47:11.878Z INFO  [connection] Opened connection 
> [connectionId{localValue:24, serverValue:793}] to 172.24.9.59:27017
> 2016-08-24T10:47:11.878Z INFO  [connection] Opened connection 
> [connectionId{localValue:27, serverValue:796}] to 172.24.9.59:27017
> 2016-08-24T10:47:11.878Z INFO  [connection] Opened connection 
> [connectionId{localValue:25, serverValue:792}] to 172.24.9.59:27017
> 2016-08-24T10:47:11.876Z INFO  [connection] Opened connection 
> [connectionId{localValue:26, serverValue:795}] to 172.24.9.59:27017
> 2016-08-24T10:47:20.046Z WARN  [jvm] 
> [graylog-1743aa7d-c19e-48fa-89fa-f2e619ec0692] [gc][young][25][19] duration 
> [2s], collections [2]/[2.4s], total [2s]/[12.5s], memory 
> [18.8gb]->[11.3gb]/[30.4gb], all_pools {[young] 
> [9.6gb]->[360.1mb]/[12.8gb]}{[survivor] [1.5gb]->[813.1mb]/[1.5gb]}{[old] 
> [7.6gb]->[10.2gb]/[16gb]}
> 2016-08-24T10:47:25.065Z WARN  [jvm] 
> [graylog-1743aa7d-c19e-48fa-89fa-f2e619ec0692] [gc][young][29][21]* 
> duration [1.3s]*, collections [1]/[2s], total [1.3s]/[14.1s], memory 
> [20.3gb]->[13.5gb]/[30.4gb], all_pools {[young] 
> [8.4gb]->[161.9mb]/[12.8gb]}{[survivor] [1.5gb]->[1.5gb]/[1.5gb]}{[old] 
> [10.2gb]->[11.8gb]/[16gb]}
>
>
>
>
>
>
> *2016-08-24T10:48:02.521Z WARN  [NodePingThread] Did not find meta info of 
> this node. Re-registering.nevertheless i had other runs when this problem 
> manifests it self:*
> 2016-08-22T14:59:48.034Z INFO  [jvm] 
> [graylog-1743aa7d-c19e-48fa-89fa-f2e619ec0692] [gc][young][62][18] duration 
> [797ms], collections [1]/[1.5s], total [797ms]/[18.1s], memory [
> 24.7gb]->[15gb]/[30.4gb], all_pools {[young] 
> 

Re: [graylog2] Graylog 2.0 SSL issue

[Dennis Oelkers]

Hey Anant,

it looks like https://172.16.0.78:12900/ is not reachable from your browser. 
Please make sure that your browser can connect to the REST API. For further 
information, please have a look at 
http://docs.graylog.org/en/2.0/pages/configuration/web_interface.html.

Kind regards,
D.

> On 24.08.2016, at 13:47, Anant Sawant  wrote:
> 
> Hi all,
> 
> We have installed Graylog 2.0 manually  on ubuntu 14.04.1 and was running  
> fine on http. 
> We are now trying to run Graylog 2.0 web interface to run on "HTTPS". For 
> this we followed the following documentation 
> "http://docs.graylog.org/en/2.0/pages/configuration/https.html#ssl-setup;.
> 
> After making the required changes we are unable to get the web interface on 
> "https://x.x.x.x:9000;.
> 
>  I have attached the screenshot of the error and also the certificates 
> created following the documentations and the server.conf file. 
> 
> Please advise to overcome this.
> 
> Thanks in advance.
> Anant
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/graylog2/0429a7ba-defa-40c7-9f6d-aa3eb8b637b5%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> 

--
Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog company
Poolstrasse 21
20355 Hamburg
Germany

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/5CC85D2D-A2E9-49B5-896C-EF6C50980C83%40graylog.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Issues parsing incomming fields in a good way

[ravedog]

Hi Jochen,  

First of all, thanks again for taking your time. Its very highly 
appreciated :)

Ok sure, from a bash script, the GELF is generated like this:

FILES=$(/bin/cat /var/log/avscanoutputfile | /bin/grep ^/)

echo -e '{
   "version": "1.1",
   "host":"'${HOSTNAME}'", 
   "short_message":"Clam AV Scan", 
   "level":6, 
   ... #Alot of other customrows that are not regarding 
this matter.
   "_clamav_infeced_files":"'${FILES}'"
   "_clamav_infected":"'${number of infected files}'"
   }\0' | nc -w 1 theserver.company.net 12201

The avscanoutputfile which is read into FILE var above, looks like this:




---

/var/log/syslog.1: Eicar-Test-Signature FOUND
/home/JIMBOB/Desktop/Untitled Document: Eicar-Test-Signature FOUND

--- SCAN SUMMARY ---
Known viruses: 4745906
Engine version: 0.99
Scanned directories: 39075
Scanned files: 209569
Infected files: 2
Data scanned: 11207.61 MB
Data read: 891244.70 MB (ratio 0.01:1)
Time: 734.572 sec (12 m 14 s)

The big issue here is that I need each line (such as /var/log/syslog.1: 
Eicar-Test-Signature FOUND) to be a separate field in above GELF. 

Something like:
echo -e '{
   "version": "1.1",
   "host":"'${HOSTNAME}'", 
   "short_message":"Clam AV Scan", 
   "level":6, 
   ... #Alot of other customrows that are not regarding 
this matter.
   "_clamav_infeced_file1":"/var/log/syslog.1: 
Eicar-Test-Signature FOUND"
  
 "_clamav_infeced_file2":"/home/JIMBOB/Desktop/Untitled Document: 
Eicar-Test-Signature FOUND"
   "_clamav_infected":"'${number of infected files}'"
   }\0' | nc -w 1 theserver.company.net 12201

The issue here is that i don't know if there is 0 lines that matched the 
FILES grep above, or 20. 

I guess this turns out to be more of a bash scripting issue now, but i 
really need a reliable way to read each line of that file into a separate 
field.

Any suggestions are highly appreciated.

Thanks,
R

(EDIT: all the previous text was included, removed and re added the answer)

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/43630667-4290-41d7-8219-cd87a713b36d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Issues parsing incomming fields in a good way

[ravedog]

Hi Jochen,

First of all, thanks again for taking your time. Its very highly 
appreciated :)

Ok sure, from a bash script, the GELF is generated like this:

FILES=$(/bin/cat /var/log/avscanoutputfile | /bin/grep ^/)

echo -e '{
   "version": "1.1",
   "host":"'${HOSTNAME}'", 
   "short_message":"Clam AV Scan", 
   "level":6, 
   ... #Alot of other customrows that are not regarding 
this matter.
   "_clamav_infeced_files":"'${FILES}'"
   "_clamav_infected":"'${number of infected files}'"
   }\0' | nc -w 1 theserver.company.net 12201

The avscanoutputfile which is read into FILE var above, looks like this:




---

/var/log/syslog.1: Eicar-Test-Signature FOUND
/home/JIMBOB/Desktop/Untitled Document: Eicar-Test-Signature FOUND

--- SCAN SUMMARY ---
Known viruses: 4745906
Engine version: 0.99
Scanned directories: 39075
Scanned files: 209569
Infected files: 2
Data scanned: 11207.61 MB
Data read: 891244.70 MB (ratio 0.01:1)
Time: 734.572 sec (12 m 14 s)

The big issue here is that I need each line (such as /var/log/syslog.1: 
Eicar-Test-Signature FOUND) to be a separate field in above GELF. 

Something like:
echo -e '{
   "version": "1.1",
   "host":"'${HOSTNAME}'", 
   "short_message":"Clam AV Scan", 
   "level":6, 
   ... #Alot of other customrows that are not regarding 
this matter.
   "_clamav_infeced_file1":"/var/log/syslog.1: 
Eicar-Test-Signature FOUND"
  
 "_clamav_infeced_file2":"/home/JIMBOB/Desktop/Untitled Document: 
Eicar-Test-Signature FOUND"
   "_clamav_infected":"'${number of infected files}'"
   }\0' | nc -w 1 theserver.company.net 12201

The issue here is that i don't know if there is 0 lines that matched the 
FILES grep above, or 20. 

I guess this turns out to be more of a bash scripting issue now, but i 
really need a reliable way to read each line of that file into a separate 
field.

Any suggestions are highly appreciated.

Thanks,
R





Den onsdag 24 augusti 2016 kl. 11:30:21 UTC+2 skrev Jochen Schalanda:
>
> Hi,
>
> splitting a message into multiple messages according to the pattern you've 
> mentioned is kind of hard.
>
> I would (still) recommend changing the generation of the GELF messages at 
> the source and send one GELF message for each infected/found file. If you 
> tell us, how you generate the GELF messages and how the source for those 
> looks like, we might be able to give you some advice how and what to change.
>
> Cheers,
> Jochen
>
> On Tuesday, 23 August 2016 21:01:41 UTC+2, ravedog wrote:
>>
>> Hi,
>>
>> Just installed graylog server to handle my logging needs. However, I 
>> bumped into an issue id be grateful for some help with.
>>
>> So i'm creating GELF messages and sending them to the graylog server. 
>> The GELFS are containing variables, but expanded they could look like 
>> this inside the GELF:
>>
>> ...
>> "_clamav_infected":"1",
>> "_clamav_infected_files":"/path/to/files: name-of-virus FOUND",
>> ...
>>
>> So the field called "_clamav_infected_files" above will constantly 
>> contain the same three field pattern.
>>
>>- /path/to/file = Always unix format, beginning with "/" and ending 
>>with ":".
>>- name-of-virus = will in all cases i have seen be worded together 
>>without any space, using "." or "-" instead (example Very-Nasty.Virus-XZY)
>>- FOUND = Static word.
>>
>>
>> This is working great and exactly as I wanted. However, I have bumped in 
>> to issues when several of these patterns are in the same field. 
>> Say for example that two viruses has been found, in that case the above 
>> example will look like this:
>>  
>> ...
>> "_clamav_infected":"1",
>> "_clamav_infected_files":"/path/to/file1: name-of-virus1 FOUND 
>> /path/to/file2: name-of-virus2 FOUND",
>> ...
>>
>> I know this is sub-par as far as filling out a field goes, but I wonder 
>> if there is any way I can match this and get this sorted from inside 
>> graylog?
>> So that every pattern starting with "/" ending with "FOUND" would get a 
>> new field by an extractor, regardless if there is 1 pattern like this, or 
>> 10. 
>>
>> Any suggestions on if this is possible?
>>
>> Talked to the guys at graylog Feenode IRC today and they pointed me 
>> towards changing the source incoming, but i'm curious to see if there is 
>> any other way than that. 
>> I also got a suggestion as to use grok for this, i was succeeding in 
>> sorting the first pattern in to separate fields (so "path" into a new field 
>> and "virus name" into a new field) but never the the full pattern as one 
>> field and never more than the first 

Re: [graylog2] Re: Graylog2 sidecar and SSL

[Michael Anthon]

Thank you Marius!

Confirming that I didn't even consider that... I've just updated my copy of 
sidecar and it now functions correctly.

Cheers,
Michael

On Wednesday, 24 August 2016 18:38:31 UTC+10, Marius Sturm wrote:
>
> Hi Michael,
> usually the issue is a better place to ask related questions. In your case 
> I guess you updated the collector server plugin but didn't do the same for 
> the sidecar itself. After installing the latest sidecar version the fields 
> should be gone and the config should be valid again.
>
> Cheers,
> Marius
>
>
> On 24 August 2016 at 03:37, Michael Anthon  > wrote:
>
>> Thanks Marius,
>> Sorry it's taken me so long to repsond, I just had an unexpected week in 
>> the hospital.
>>
>> I've just tested the changes in beta 4 and I can now save those fields 
>> without a value however the generated config file still includes the 
>> entries like so...
>>
>> 
>>  Module om_ssl
>>  Host graylog2.example.org
>>  Port 12443
>>  OutputType GELF_TCP
>>  CAFile 
>>  CertFile 
>>  CertKeyFile 
>>  AllowUntrusted True
>>  Exec $short_message = $raw_event; # Avoids truncation of the 
>> short_message field.
>>  Exec $gl2_source_collector = '9e2660a6-b960-4daf-8d90-e37c3c0e1684';
>>  Exec $Hostname = hostname_fqdn();
>> 
>>
>> This doesn't work as nxlog doesn't like the missing values
>> 2016-08-24 11:34:58 INFO nxlog-ce-2.9.1716 started
>> 2016-08-24 11:34:58 ERROR SSL error, Failed to open certfile: ; The 
>> device does not recognize the command. 
>>
>> The config will need to be generated without those entries.
>>
>> As an aside, is the mailing list the best place to provide this type of 
>> feedback or should I be putting comments back against the issue in github?
>>
>> Thanks,
>> Michael
>>
>> On Monday, 15 August 2016 20:45:51 UTC+10, Marius Sturm wrote:
>>>
>>> Hi Michael,
>>> this was done here: 
>>> https://github.com/Graylog2/graylog-plugin-collector/issues/13
>>> Should be available in Graylog 2.1.0-RC1
>>>
>>> Cheers,
>>> Marius
>>>
>>>
>>> On 12 August 2016 at 13:20, Michael Anthon  
>>> wrote:
>>>
 Thanks Marius,
 I've just upgraded to the latest beta and it certainly is looking a lot 
 better, it also looks like it might solve another issue I was going to 
 raise with the verbatim configurations since we want to do some custom 
 processing on inputs to strip sensitive data before it's sent over to 
 graylog.

 I still do however have an issue getting this to work.  The 3 field for 
 the CA, certificate and key files are currently required before you can 
 save the output.  I have no need of client certificates and don't have any 
 since my goal is just to ensure that the data in transit is encrypted.  

 I've tried configuring these with just a space, a dot or a double 
 quoted empty string but the nxlog config always includes the values which 
 causes nxlog to reject the output configuration since the files don't 
 exist.

 I have tried shutting down sidecar, removing those 3 lines from the 
 config and running nxlog manually and this definitely works, it connects 
 and sends messages to graylog.

 If those 3 fields could be made optional and not add those entries to 
 the generated nxlog.conf then I think this would work perfectly.

 Cheers,
 Michael

 On Friday, 12 August 2016 01:59:50 UTC+10, Marius Sturm wrote:
>
> Ah ja ok, we shipped the SSL feature recently. So you will see it in 
> the next Graylog release or you test the beta version.
>
>
> -- 
 You received this message because you are subscribed to the Google 
 Groups "Graylog Users" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to graylog2+u...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/graylog2/2e018723-1fbc-42f9-8f43-097b9d6acee0%40googlegroups.com
  
 
 .

 For more options, visit https://groups.google.com/d/optout.

>>>
>>>
>>>
>>> -- 
>>> Developer
>>>
>>> Tel.: +49 (0)40 609 452 077
>>> Fax.: +49 (0)40 609 452 078
>>>
>>> TORCH GmbH - A Graylog Company
>>> Poolstraße 21
>>> 20335 Hamburg
>>> Germany
>>>
>>> https://www.graylog.com 
>>>
>>> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
>>> Geschäftsführer: Lennart Koopmann (CEO)
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Graylog Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to graylog2+u...@googlegroups.com .
>> To view this discussion on the web visit 
>> 

[graylog2] Re: CSV to field converter using whitespace delimiter

[julioqc47]

Oh I agree and have switched to Grok since I posted the original message. 
Yes those are IIS :)

However, Grok patterns takes much more time to configure where CSV 
literally takes 20 sec to setup. I'm just getting lazy I suppose haha 
Anyhow, CSV seems problematic for certain delimiters and poorly handles 
exceptions (Exchange especially) so I'm using Grok from now on. 

On Tuesday, 23 August 2016 22:19:54 UTC-4, Michael Anthon wrote:
>
> Going by the headers I'm guessing that's an IIS log?  As Jochen suggested 
> previously, Grok is your friend.
>
> These are the patterns I'm using for my IIS logs (one for entries with a 
> referer and one without)
>
> %{YEAR:year;int}-%{MONTHNUM:monthnum;int}-%{MONTHDAY:monthday;int}[T 
> ](?!<[0-9])%{HOUR:hour;int}:%{MINUTE:minute;int}(?::%{SECOND:second;int
> })(?![0-9]) %{IPORHOST:s_ip} %{WORD:cs_method} %{URIPATH:cs_uri_stem} %{
> NOTSPACE:cs_uri_query} %{NUMBER:s_port;int} %{NOTSPACE:cs_username} %{
> IPORHOST:c_ip} %{NOTSPACE:cs_user_agent} %{NUMBER:sc_status;int} %{NUMBER:
> sc_substatus;int} %{NUMBER:sc_win32_status;int} %{NUMBER:time_taken;long}
>
> %{YEAR:year;int}-%{MONTHNUM:monthnum;int}-%{MONTHDAY:monthday;int}[T 
> ](?!<[0-9])%{HOUR:hour;int}:%{MINUTE:minute;int}(?::%{SECOND:second;int
> })(?![0-9]) %{IPORHOST:s_ip} %{WORD:cs_method} %{URIPATH:cs_uri_stem} %{
> NOTSPACE:cs_uri_query} %{NUMBER:s_port;int} %{NOTSPACE:cs_username} %{
> IPORHOST:c_ip} %{NOTSPACE:cs_user_agent} %{NOTSPACE:cs_referer} %{NUMBER:
> sc_status;int} %{NUMBER:sc_substatus;int} %{NUMBER:sc_win32_status;int} %{
> NUMBER:sc_bytes;int} %{NUMBER:cs_bytes;int} %{NUMBER:time_taken;long}
>
>
>
> On Wednesday, 17 August 2016 01:28:21 UTC+10, juli...@gmail.com wrote:
>>
>> Hi,
>>
>>
>> So it seems the CSV to field converter doesn't work with whitespace 
>> delimiters?
>>
>> Sample log:
>> 2016-08-16 15:14:20 192.168.20.100 POST /Clients - 80 DOMAIN\user 
>> 192.168.30.171 
>> Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/52.0.2743.116+Safari/537.36
>>  
>> 302 0 0 187
>>
>> I've tried both an actual whitespace and \s in the 'Separator character' 
>> field but nothing does it.
>>
>>
>> 
>>
>>
>>
>> Any tips or more doc on the matter so I can achieve this?
>> I mean I can alternatively use GROK or do it from nxlog at the source but 
>> I'd like this to work as well :)
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/4e7b6ef2-ff7d-4d7a-bd3c-091da077bff2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Issues parsing incomming fields in a good way

[Jochen Schalanda]

Hi,

splitting a message into multiple messages according to the pattern you've 
mentioned is kind of hard.

I would (still) recommend changing the generation of the GELF messages at 
the source and send one GELF message for each infected/found file. If you 
tell us, how you generate the GELF messages and how the source for those 
looks like, we might be able to give you some advice how and what to change.

Cheers,
Jochen

On Tuesday, 23 August 2016 21:01:41 UTC+2, ravedog wrote:
>
> Hi,
>
> Just installed graylog server to handle my logging needs. However, I 
> bumped into an issue id be grateful for some help with.
>
> So i'm creating GELF messages and sending them to the graylog server. 
> The GELFS are containing variables, but expanded they could look like this 
> inside the GELF:
>
> ...
> "_clamav_infected":"1",
> "_clamav_infected_files":"/path/to/files: name-of-virus FOUND",
> ...
>
> So the field called "_clamav_infected_files" above will constantly contain 
> the same three field pattern.
>
>- /path/to/file = Always unix format, beginning with "/" and ending 
>with ":".
>- name-of-virus = will in all cases i have seen be worded together 
>without any space, using "." or "-" instead (example Very-Nasty.Virus-XZY)
>- FOUND = Static word.
>
>
> This is working great and exactly as I wanted. However, I have bumped in 
> to issues when several of these patterns are in the same field. 
> Say for example that two viruses has been found, in that case the above 
> example will look like this:
>  
> ...
> "_clamav_infected":"1",
> "_clamav_infected_files":"/path/to/file1: name-of-virus1 FOUND 
> /path/to/file2: name-of-virus2 FOUND",
> ...
>
> I know this is sub-par as far as filling out a field goes, but I wonder if 
> there is any way I can match this and get this sorted from inside graylog?
> So that every pattern starting with "/" ending with "FOUND" would get a 
> new field by an extractor, regardless if there is 1 pattern like this, or 
> 10. 
>
> Any suggestions on if this is possible?
>
> Talked to the guys at graylog Feenode IRC today and they pointed me 
> towards changing the source incoming, but i'm curious to see if there is 
> any other way than that. 
> I also got a suggestion as to use grok for this, i was succeeding in 
> sorting the first pattern in to separate fields (so "path" into a new field 
> and "virus name" into a new field) but never the the full pattern as one 
> field and never more than the first instance of the pattern.
>
> Thanks in advance!
>
> /R
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/8e7b9a74-8db7-4c80-bb64-9b7447a7d3c3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [graylog2] Re: Graylog2 sidecar and SSL

[Marius Sturm]

Hi Michael,
usually the issue is a better place to ask related questions. In your case
I guess you updated the collector server plugin but didn't do the same for
the sidecar itself. After installing the latest sidecar version the fields
should be gone and the config should be valid again.

Cheers,
Marius


On 24 August 2016 at 03:37, Michael Anthon 
wrote:

> Thanks Marius,
> Sorry it's taken me so long to repsond, I just had an unexpected week in
> the hospital.
>
> I've just tested the changes in beta 4 and I can now save those fields
> without a value however the generated config file still includes the
> entries like so...
>
> 
>  Module om_ssl
>  Host graylog2.example.org
>  Port 12443
>  OutputType GELF_TCP
>  CAFile
>  CertFile
>  CertKeyFile
>  AllowUntrusted True
>  Exec $short_message = $raw_event; # Avoids truncation of the
> short_message field.
>  Exec $gl2_source_collector = '9e2660a6-b960-4daf-8d90-e37c3c0e1684';
>  Exec $Hostname = hostname_fqdn();
> 
>
> This doesn't work as nxlog doesn't like the missing values
> 2016-08-24 11:34:58 INFO nxlog-ce-2.9.1716 started
> 2016-08-24 11:34:58 ERROR SSL error, Failed to open certfile: ; The
> device does not recognize the command.
>
> The config will need to be generated without those entries.
>
> As an aside, is the mailing list the best place to provide this type of
> feedback or should I be putting comments back against the issue in github?
>
> Thanks,
> Michael
>
> On Monday, 15 August 2016 20:45:51 UTC+10, Marius Sturm wrote:
>>
>> Hi Michael,
>> this was done here: https://github.com/Graylog2/gr
>> aylog-plugin-collector/issues/13
>> Should be available in Graylog 2.1.0-RC1
>>
>> Cheers,
>> Marius
>>
>>
>> On 12 August 2016 at 13:20, Michael Anthon 
>> wrote:
>>
>>> Thanks Marius,
>>> I've just upgraded to the latest beta and it certainly is looking a lot
>>> better, it also looks like it might solve another issue I was going to
>>> raise with the verbatim configurations since we want to do some custom
>>> processing on inputs to strip sensitive data before it's sent over to
>>> graylog.
>>>
>>> I still do however have an issue getting this to work.  The 3 field for
>>> the CA, certificate and key files are currently required before you can
>>> save the output.  I have no need of client certificates and don't have any
>>> since my goal is just to ensure that the data in transit is encrypted.
>>>
>>> I've tried configuring these with just a space, a dot or a double quoted
>>> empty string but the nxlog config always includes the values which causes
>>> nxlog to reject the output configuration since the files don't exist.
>>>
>>> I have tried shutting down sidecar, removing those 3 lines from the
>>> config and running nxlog manually and this definitely works, it connects
>>> and sends messages to graylog.
>>>
>>> If those 3 fields could be made optional and not add those entries to
>>> the generated nxlog.conf then I think this would work perfectly.
>>>
>>> Cheers,
>>> Michael
>>>
>>> On Friday, 12 August 2016 01:59:50 UTC+10, Marius Sturm wrote:

 Ah ja ok, we shipped the SSL feature recently. So you will see it in
 the next Graylog release or you test the beta version.


 --
>>> You received this message because you are subscribed to the Google
>>> Groups "Graylog Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to graylog2+u...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/graylog2/2e018723-1fbc-42f9-8f43-097b9d6acee0%40googlegroups.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Developer
>>
>> Tel.: +49 (0)40 609 452 077
>> Fax.: +49 (0)40 609 452 078
>>
>> TORCH GmbH - A Graylog Company
>> Poolstraße 21
>> 20335 Hamburg
>> Germany
>>
>> https://www.graylog.com 
>>
>> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
>> Geschäftsführer: Lennart Koopmann (CEO)
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/graylog2/f8ee9ddd-977d-4998-9af9-508a195f7e79%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Developer

Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog Company
Poolstraße 21
20335 Hamburg
Germany

https://www.graylog.com