Re: [graylog2] Re: buglet: broken link http://info.graylog.org/marketplace-requests
Wow - it must be something I have then. I have noscript, ublock origin - a bunch of stuff, but I disabled some of them to test before sending the email - I guess I missed one. Found it - it was Ghostery: you're using "Hubspot" which Ghostery classifies as a "customer tracker" and blocks. Sorry, false alarm :-) On Tue, Feb 7, 2017 at 8:57 PM, Jochen Schalanda <joc...@graylog.com> wrote: > Hi Jason, > > the link http://info.graylog.org/marketplace-requests > <http://www.google.com/url?q=http%3A%2F%2Finfo.graylog.org%2Fmarketplace-requests=D=1=AFQjCNFbQ3o8VPvOplJY7s6brajZc0YC0Q> > shows > a contact form for me: > > > <https://lh3.googleusercontent.com/-Du31wh0KAco/WJl902PkPnI/AEc/pg9OZk_ENLYIb85a5yOja3Ikk7UdaPNRQCLcB/s1600/Bildschirmfoto%2B2017-02-07%2Bum%2B08.55.50.png> > > Which browser are you using? Which extensions (possibly blocking content > like Javascript) are you using? > > Cheers, > Jochen > > > On Tuesday, 7 February 2017 05:59:37 UTC+1, Jason Haar wrote: >> >> Hi there >> >> I just did a search on the marketplace for Azure related plugins and >> found nothing. There was a link saying >> >> "Sorry, nothing matches your query. >> >> Not found what you are looking for? Let us know what you'd like to see in >> the Marketplace!" >> >> But the link goes nowhere >> >> http://info.graylog.org/marketplace-requests >> >> -- >> Cheers >> >> Jason Haar >> Information Security Manager, Trimble Navigation Ltd. >> Phone: +1 408 481 8171 <+1%20408-481-8171> >> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 >> > -- > You received this message because you are subscribed to the Google Groups > "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/graylog2/64e3af80-90ae-47ba-8858-606c0c9886c4%40googlegroups.com > <https://groups.google.com/d/msgid/graylog2/64e3af80-90ae-47ba-8858-606c0c9886c4%40googlegroups.com?utm_medium=email_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJZiY58a7xd8AAATVf7H_6pidM7FtRE8d8a2ZRYj4FL5w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Changing timestamps?
On Tue, Jan 31, 2017 at 11:10 AM, Jochen Schalanda <joc...@graylog.com> wrote: > do the syslog messages from SUSE Linux on "Input 2" contain any timezone > information? If not, Graylog automatically assumes UTC. > Yeah this is a common problem with centralized syslog environments. The old standard assumed everyone lived in one timezone: ah for life to be that easy :-) So Jochen is correct in that the best thing to do would be the fix the problem at source - but in practice that can be an immense task. Not only with timezone issues, but also with dumb devices that can't keep good time. I think the syslog INPUT channel could do with a new feature to help solved this problem at destination. Currently on syslog INPUT channels you can set "allow_override_date" to true/false. But "true" actually means "override date and set to current time *if you cannot parse the date from the message*". I think if that was to be changed to a checkbox of "false", "true-on-error" and "always" (ie throw away valid timestamps in message and replace with "now"), then that would solve the problem for a bunch of people. If you're using syslog, then your records are flowing into graylog within sub-second accuracy - so throwing away the perceived timestamp and put a proper one in doesn't change the accuracy. And for those where being off by 0.4sec matters - well continue to use 'false' :-) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJ_7nsHr-x2b76j7_m2LSeLcKkMkfo%2BR%2Bb5PFjvr8rWXw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: mongodb_uri doesn't like multiple server urls?
On Mon, Jan 16, 2017 at 10:24 PM, Jochen Schalanda <joc...@graylog.com> wrote: > please stick to the format explained in http://docs.graylog.org/en/ > 2.1/pages/configuration/multinode_setup.html#graylog-to-mongodb-connection > and https://github.com/Graylog2/graylog2-server/blob/ > 2.1.2/misc/graylog.conf#L434-L442 for the mongodb_uri setting. > > Groan - I now get it. That isn't a very nice format. For other people's education, the format is meant to be mongodb_uri = mongodb://username:password@ [host1:port,host2:port,host3:port]/database ...except that there are no brackets :-) I did not find that intuitive, but that's the mongodb standard, so I'd better get used to it :-) Thanks! -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJRxUdGJ_eL6K44o3ZNrajiiaykASj8Gan1oBH%2BpHKqfw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] mongodb_uri doesn't like multiple server urls?
Hi there
I just converted our standalone mongodb server into a cluster according to
the instructions on
https://docs.mongodb.com/manual/tutorial/deploy-replica-set/
Looks good, the original is the master and the rest shows the "graylog"
database got synched to them and "rs.status()" looks happy (I even
confirmed I could log into the mongodb "graylog" database with the same
username/password on each slave)
So then I reconfigured server.conf to tell graylog it had multiple servers
to point to. I went from this working entry
mongodb_uri = mongodb://graylog:pass@ho.st1:27017/graylog
to
mongodb_uri = mongodb://graylog:pass@ho.st1
:27017/graylog,mongodb://graylog:pass@ho.st2
:27017/graylog,mongodb://graylog:pass@ho.st3:27017/graylog
However that errors out and graylog-server crashes. I see the below error.
It's like graylog "likes" the url format, but comma-separated values spook
it? I even tried the format in
http://docs.graylog.org/en/2.2/pages/configuration/multinode_setup.html
ie
mongodb_uri = mongodb://graylog:pass@ho.st1
:27017/graylog,ho.st2:27017/graylog,ho.st3:27017/graylog
- that didn't work either.
This is graylog-server-2.2.0-6.beta.4.noarch under CentOS7
Thanks
Jason
com.github.joschi.jadconfig.ValidationException: Couldn't run validator
method
at
com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:227)
~[graylog.jar:?]
at
com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:100)
~[graylog.jar:?]
at
org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:351)
[graylog.jar:?]
at
org.graylog2.bootstrap.CmdLineTool.readConfiguration(CmdLineTool.java:344)
[graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:177)
[graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
~[?:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
~[?:1.8.0_111]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_111]
at
com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53)
~[graylog.jar:?]
at
com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221)
~[graylog.jar:?]
... 5 more
Caused by: java.lang.IllegalArgumentException: The connection string
contains invalid user information. If the username or password contains a
colon (:) or an at-sign (@) then it must be urlencoded
at com.mongodb.ConnectionString.(ConnectionString.java:262)
~[graylog.jar:?]
at com.mongodb.MongoClientURI.(MongoClientURI.java:201)
~[graylog.jar:?]
at
org.graylog2.configuration.MongoDbConfiguration.getMongoClientURI(MongoDbConfiguration.java:55)
~[graylog.jar:?]
at
org.graylog2.configuration.MongoDbConfiguration.validate(MongoDbConfiguration.java:60)
~[graylog.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
~[?:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
~[?:1.8.0_111]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_111]
at
com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53)
~[graylog.jar:?]
at
com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221)
~[graylog.jar:?]
... 5 more
--
Cheers
Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/CAFChrgKP-DwDPd6frYKmGu-8Q_mj90Y2aqQ39KmNLTrWiak9ig%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] is there a plugin for Azure/365live logging?
Hi there Our company is starting to get into Azure a bit and I'm wanting to ensure the standard activity logs/etc generated in that environment flow into graylog. We are already in AWS and there's an existing AWS logging plugin for graylog to poll AWS for such logging, so I was wondering if anyone had one for Azure/365? Splunk has one (https://splunkbase.splunk.com/app/3110/) from the looks of it if that helps -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgLsWMdkBw8k%3DbzyEF_RXG9J00iZS83X77p66KuseZnmNw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: unable to receive syslog/tls from Cisco devices
We got it working, so I thought I'd share the trick Ended up the network guys had syslog/tls pointing at the IP address instead of the hostname that is present in the cert. There is no way to disable cert validation on Ciscos, so the Cisco was dropping the session due to the name mismatch. Also it appears (on Cisco) you have to configure CRL for the CA used to sign the graylog TLS cert. That sounds weird to me - but that's what they did to make it work logging enable logging timestamp logging buffer-size 8196 logging monitor debugging logging buffered debugging logging trap debugging logging history debugging logging asdm informational logging queue 0 logging device-id hostname logging host outside fqdn 6/portNUM secure logging permit-hostdown crypto ca trustpoint CA-who-signed-graylogCert enrollment terminal crl configure whatever goes here for your CA crypto ca certificate chain CA-who-signed-graylogCert certificate ca xxx This will then enable the Cisco to create a syslog/TLS session to the graylog server, and then you'll have to add some extractors to actually glean the information you want - the Cisco's are bad at that too. But all working now :-) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgLjk0P6szsiJ9XdB6H3d%2B%2B1Rmi7hDvDu2pVRWNbJ%2Bmiag%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: unable to receive syslog/tls from Cisco devices
On Mon, Jan 9, 2017 at 9:20 PM, Jochen Schalanda <joc...@graylog.com> wrote: > if you're using TLS client certificates No - the cisco doesn't support that anyway. The java error says "javax.net.ssl.SSLException: Received fatal alert: certificate_unknown", and wireshark confirms the TLS alert comes from the *client* - not the server. It seems to me the Cisco ASA doesn't trust the server cert or the CA that signed it - but I'm not a Cisco engineer and apparently our ones have never done this before and don't know any better either :-) I was hoping someone on this list had been through the drama. -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgKDG5SqkiiS%2BPFzEG6m-%2Byre%3DdazZuH8%3Dufm%2B8LwLD54Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] unable to receive syslog/tls from Cisco devices
Hi there We have set up our first Cisco ASA (8.4) to send syslog (TLS) messages through to graylog via logging host outside ip.add.ress TCP/ secure We already have some Unix systems using rsyslog successfully doing the same thing, but the Cisco records aren't being accepted. A sniffer shows traffic coming in from the Cisco, but server.log reports the following. That sounds like the Cisco attempted to handshake TLS and then sent an alert to graylog stating the error was "certificate_unknown"? That would make sense, but our network group have no idea how to make the CA trusted. Can someone point me at something they need to read to do this properly? Thanks, Jason 2017-01-09T00:07:56.088Z ERROR [NettyTransport] Error in Input [Syslog TCP/570cc00b9cdbc22f13f5cecd] (channel [id: 0x525ae1a4, /1.2.3.4:56720 => / 4.3.2.1:]) javax.net.ssl.SSLException: Received fatal alert: certificate_unknown at sun.security.ssl.Alerts.getSSLException(Unknown Source) ~[?:1.8.0_77] at sun.security.ssl.SSLEngineImpl.fatal(Unknown Source) ~[?:1.8.0_77] at sun.security.ssl.SSLEngineImpl.fatal(Unknown Source) ~[?:1.8.0_77] at sun.security.ssl.SSLEngineImpl.recvAlert(Unknown Source) ~[?:1.8.0_77] at sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source) ~[?:1.8.0_77] at sun.security.ssl.SSLEngineImpl.readNetRecord(Unknown Source) ~[?:1.8.0_77] at sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source) ~[?:1.8.0_77] at javax.net.ssl.SSLEngine.unwrap(Unknown Source) ~[?:1.8.0_77] at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1219) ~[graylog.jar:?] at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852) ~[graylog.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425) ~[graylog.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) ~[graylog.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[graylog.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [graylog.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [graylog.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [graylog.jar:?] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [graylog.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [graylog.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [graylog.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [graylog.jar:?] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [graylog.jar:?] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [graylog.jar:?] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [graylog.jar:?] at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_77] at java.lang.Thread.run(Unknown Source) [?:1.8.0_77] -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgLt-7-4MWjn4rv7Uf-Z861oO7bo-KGCzhrmUGNTMsaeJQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] elasticsearch_max_number_of_indices change doesn't appear to be listened to?
Hi there I added some more disk last week to our graylog-server-2.1.2-1 system and expanded my elasticsearch_max_number_of_indices setting to allow our rotation_strategy=size system to expand However, I was just looking at the logs and noticed it was still set to the old value. I then went into the web interface /system/indices area and saw the old value was still there So is server.conf field elasticsearch_max_number_of_indices no longer being used? No sign in server.log of any errors associated with indices...? I assume I just need to change that value through the web interface to fix? -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJ9OqxvCiF1tRxS5KPVN7QBc0HnZUsKU6_xsnrG%3Ds8nMQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: hot-warm-architecture
On Sat, Dec 10, 2016 at 5:19 AM, Jason Close <jason.m.cl...@gmail.com> wrote: > As far as mods go, I'm still tinkering. I had to turn off firewalld, but > that's fine because it's in an internal network. That caught my eye. How did firewalling cause you grief? A busy graylog/ES environment is still nowhere near as busy as using (say) Linux as a router could be, so I'm worried by your comment because we use firewalling on our graylog systems too... Although being old school, I used iptables - not firewalld (not that it should make a difference?) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJWTZFPL2AGLkTwSA-RMxR1jW2%3DBtMDBvVTExYd5t6RGw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Whats Better for Graylog Udp or Tcp
On Tue, Nov 15, 2016 at 4:33 AM, Jochen Schalanda <joc...@graylog.com> wrote: > Use whatever is supported best by your network appliances. > Well I would add "it depends". UDP is absolutely fine over LANs - if you have near guarantees about zero packet loss - use UDP as its more efficient. But if WANs or the Internet is involved - use TCP. And in fact, use TLS over TCP - just because it's 2016 - not 1999 ;-) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJDkSoR8V5iCFC%2BzcDVF6mb6Y%2B8gg-5H-Qgpoc9PTx1Eg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Graylog2 send Alerts with SMS
On Fri, Oct 28, 2016 at 3:26 AM, Benbrahim Anass <bbrahim.an...@gmail.com> wrote: > yes i figured that, but is it free or you need to subscribe to an SMS > provider,? > If any of them are free, you've got to ask yourself, "how are they paying for their infrastructure and time for maintaining it?" ie if you're not paying for a product, YOU ARE THE PRODUCT ...although these days, it seems to be that even if you are paying, you're also the product :-/ (see Windows 10) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgK1HDcErBD7ktEjf9UsATeHBzwsHCnLPDraWOkBuJT5GQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: unable to figure out permissions using REST API
On Fri, Sep 23, 2016 at 6:48 PM, Jochen Schalanda <joc...@graylog.com> wrote: > the required permissions are: > >- > > ?? What are these "permissions" you talk about :-) ie looking at the Authentication GUI, it says setting permissions is deprecated and I should use Roles instead. Is that a mistake? Also I can't see how to add permissions against the account - is this action not supported through the GUI? Thanks -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJ19NrbSD%3DOhBde7siMnqG1n1sdr0Av_81%3DJRW7C0QW6A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] unable to figure out permissions using REST API
Hi there I'm wanting to create a "read only" admin account that can do any search query against graylog that we want. I created a local account (normally we use LDAP) which just had the "Reader" role - and it couldn't do anything. I then gave it the Admin role and it could indeed search for everything But I want a "read only" account. This is going to be in scripts - and I don't want scripts lying about with full admin privs. So I played around with other Roles - but they are all stream-specific. So can someone tell me how I can create a Role that allows universal search - but with no form of write access? Thanks -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgKQ%2BgagBGN1Q7xYG6CdCaf%3DxZZhptJRv_xk9-8Lefe%2BhQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: alerting plugins seem to lack all context?
On Wed, Sep 7, 2016 at 4:30 AM, Jochen Schalanda <joc...@graylog.com> wrote:
> You could also try to use netcat or Wireshark to record the request the
> HTTP Alarm Callback is sending.
>
Great idea. Now I see the problem. That POST is of a JSON blob - it's not a
normal "web form". That's why I can't find any POST variables - there
aren't any.
So now I'm using the following to get me an array of field->values - works
fine :-)
$json = file_get_contents('php://input');
$obj = json_decode($json);
--
Cheers
Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/CAFChrgJpZvvPHr6MYYbGy7THgMb3m38QFx1GCfd0Mp%3DJAM9%3DkA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] alerting plugins seem to lack all context?
Hi there I've been playing around with alerts. The native "email" alert works as expected, but the HTTP and "Execute command" alarm options lack all detail about the event that triggered the alert I've written a script that simply dumps the command line options and environment vars to a file - nothing related to the event shows up. Similarly, the HTTP does a POST - but contains no variables at all What am I missing? The alerting options are really not that sophisticated, so I'd rather dump "alerts" into my own workflow program - but nothing besides email appears to have any actual data?? -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrg%2BhQS7XRHP8sAOGDmHCKZYX%3DTjCfW6DZN0De6xVK9yXFQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Geo-Location Processor doesn't create _geolocation fields for custom fields created by pipeline rules
On Thu, Sep 1, 2016 at 2:12 AM, Jan <jan.lemmerm...@gmail.com> wrote: > Found the error. In my original pipeline-rule I used the "to_ip" function > to convert the pattern match to an IP. With this setting resolving the IP > to a geo location fails. > I changed the rule now to convert the pattern match to a string by using > the "to_string" function. Voila... geo location works for all custom fields > now. > (to Graylog devs). That's a bug isn't it? I mean, what's wrong with assuming an IP address is an IP address? Shouldn't the GeoIP processor support both string and "ip" fields types? -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJFzC5HFDnX2c1soixC_7LH5n%3D2-MEiymEp88GQeUHhuw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Indicators of Compromise (IOCs)
On Tue, Aug 30, 2016 at 3:03 AM, Jochen Schalanda <joc...@graylog.com> wrote: > there's currently no official integration of TAXII with Graylog. I guess > you would need to write a custom plugin for integrating TAXII or other IoC > feeds and check against them. > I've just been thinking about this myself. It should be handled in a similar way to the GeoIP processor IMHO. Let's call it the "Reputation" processor. it could load an external 'database' of 'name,field,value' and when the INPUT data stream contains 'field: value' then trigger a new 'reputation:name' record. eg TALOS, src_ip, 1.2.3.4 SPAMHAUS, email_ip, 3.2.1.2 Then your firewall logs involving src_ip == 1.2.3.4 would get a "reputation:TALO" record and your email logs (email_ip == 3.2.1.2) would get a "reputation:SPAMHAUS" record This would be a more generalised solution - could be abused in all sorts of ways :-) Hmm, I thought I added this to the Ideas site a few days ago - can't find it now? -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgK0n3A%2BWvFyvb1dCE60Eh0UyhVB-UNvHd9-Dnp-1mt8sQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Syslog severity mapper decorator
FYI I get 'permission denied' trying to access the decorator documentation on: http://docs.graylog.org/en/2.1/pages/decorators.html On Sat, Aug 20, 2016 at 1:58 AM, Jan Doberstein <j...@jalogisch.de> wrote: > Hej Marcus > > That whats what I hoped for, but to me it looks like nothing has changed at > > all. Everything is like it was with 2.0 and/or 2.1beta2. I must be kind of > > too blind to see ;) > > From my understanding I could still search for something like: > level:<4 AND message:foo > > But I would expect to see in the search window of my message a decorated > view > of my messages with ERROR, WARNING, FATAL instead of the kind of raw > numbers. > > that is what they are used for. > > You need to choose source and target field (when just use the severity > mapper). We are a little behind providing documentation but it will be > present as soon as possible. > > Give the new beta.4 a try, we had fixed many issues. > > > /jd > > -- > You received this message because you are subscribed to the Google Groups > "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/graylog2/etPan.57b71092.1d770716.5c18%40jalogisch.de > <https://groups.google.com/d/msgid/graylog2/etPan.57b71092.1d770716.5c18%40jalogisch.de?utm_medium=email_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgK6hdcWpRbhLkdP-JDoKsA8QNhsAbt4qLagE1s3gi-BFg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] How to configure graylog 2.0.3 to email using system mail agent ?
On Fri, Aug 12, 2016 at 9:59 PM, T.J. Yang <tjyang2...@gmail.com> wrote: > Question is how can I configure graylog-server to use system's mail agent > ? > Instead of following configure to drectly insteract with remote mail relay > server ? > I don't think you can. But if you're running postfix, then I'd suspect you'll find it's actually running on 127.0.0.1 port 25 (that's the default for RHEL/CentOS). So just configure it to use that and it will act identically to calling /usr/sbin/sendmail -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgLhkUz5Dm7bA8ReLRhD5O7drmh26cmuV1d59FEp8Fy1sA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: upgrading from graylog-v2.0 to v2.1
On Tue, Aug 9, 2016 at 12:06 AM, Jochen Schalanda <joc...@graylog.com> wrote: > please read the upgrade note for Graylog 2.1.x at https://github.com/ > Graylog2/graylog2-server/blob/2.1.0-beta.2/UPGRADING.rst. > Brilliant - so beyond the REST API changes I'll have to make (as we use a reverse proxy) that sounds like it's backwards compatible -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgKisP9%2B8vR3G28ta8Y_hMtjZaT%2BNJ1HGcVC-d%2BYEKpU3Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] has anyone got a plugin for reading Google Apps APIs?
Hi there We're using Google Apps and I can see a wide range of VERY interesting audit information I'd love to flow into graylog: successful/failed login events, gdrive transaction logs, admin events, etc. Sort of the Google Apps equivalent of AWS CloudTrails They have an API and with my poor programming skills I did manage to get a python demo script successfully pulling down admin login events - but that's about my limits - so I'm hoping someone has done a deeper integration and just hasn't got around to publishing it on the graylog market? :-) Thanks -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgK-vJWW%2Bp5_TriWpg1RnUzjv%2B4cwTMCvr-_TVK8QUoXmw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Backfilling graylog with past data
On Fri, Jul 15, 2016 at 2:50 AM, Jeremy Farr <jf...@noblebank.com> wrote: > Jason have you noticed any issues when adding to indices that are not the > currently active one? > No. My indices don't last more than an hour and I have shoved in data that was days old - so it definitely all went into "old" indices. Waitaminute - that's not how it works. Mustn't it always go into the current index, even if the timestamps are no longer vaguely related? I can't say I've thought much about it - it simply worked -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrg%2BYuLYWXUSX0BuqQWUf-Yo72AZrHiZF1TFviv-inu6kNg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Extractor help - domain name only
On Wed, Jul 13, 2016 at 10:57 PM, Zoizo <houssdecoue...@gmail.com> wrote: > Well I'm a moron and forgot domain names could have more than two words > too so, I'm kinda lost as to what I can do here ^^' > Try a regex like "GET [a-z]+?://[^\.]+\.([^/]+)/" On "facebook.com" that would match "com". Assuming that's what you want of course (I stick to "GET [a-z]+://([^/]+)/" myself) If you want the last two, then "GET [a-z]+://[^/]+\.([^/]+\.[^/]+)/" should do that I know "https?" would be better - but I'm not near graylog at the moment and I don't know if it's weird Java-based regex library supports that - so I stuck with [a-z]+ to match "http" and "https" (and I guess "ftp" too) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJLX-OvQtgu6t6M%3DSLusRU_WyhpOkDc6PA3MK_mf0v4Lg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] debugging pipelines is... difficult
On Mon, Jul 11, 2016 at 11:28 AM, Jason Haar <jason_h...@trimble.com> wrote: > If I take the regex I wrote in this rule (as per first email), replace > '\\' with '\', then the regex works fine via egrep. It's a simple "when, do > this" type statement: I can't see what's gone wrong in it > Oh - and thanks to your comment about the regex needing to match the entire line, I put ".*" at the beginning and end - but it made no difference. Still no Cisco syslog messages (as above) match -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJZng%2Bzc-iZ%2Bv73%2Bd8Q6YatVATaDtj2R%3Dd7sR9iXZfbHQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] debugging pipelines is... difficult
On Fri, Jul 8, 2016 at 10:32 PM, Edmundo Alvarez <edmu...@graylog.com> wrote: > > It's hard to tell what is wrong from here, since we can't exactly see how > your messages look like. Could you share a couple of messages with us? > > Please be aware that at the moment, the "regex" function needs to match > the whole string: > https://github.com/Graylog2/graylog-plugin-pipeline-processor/issues/35 > Sure thing So that rule is attempting to extract any TWO ipv4 addresses detected in any form of syslog message. So to give you some examples of when that occurs, we'll look to Cisco firewalls/routers %ASA-4-106023: Deny tcp src inside:192.168.3.79/57577 dst outside: 54.171.242.51/843 by access-group "acl_inside" [0x2923dc37, 0x0] %ASA-7-710006: EIGRP request discarded from 192.168.23.1 to inside:224.0.0.10 %ASA-4-106023: Deny tcp src inside:192.168.4.52/62508 dst outside: 21.125.185.18/5287 by access-group If I take the regex I wrote in this rule (as per first email), replace '\\' with '\', then the regex works fine via egrep. It's a simple "when, do this" type statement: I can't see what's gone wrong in it I have another pipeline with two rules and it's working just fine - it seems to be the regex in this that is at fault, but I can't see how -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgK8T6t_728ynFmH2ePHMx9dhsFYq4stfk1DVcyrdtCRPw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: debugging pipelines is... difficult
On Wed, Jul 6, 2016 at 9:50 PM, Jochen Schalanda <joc...@graylog.com> wrote: > there's something coming up in Graylog 2.1.0 which will vastly simplify > testing pipeline rules. > That's great to hear. Any suggestions as to what's wrong with my rule? Thanks -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgL7rcbe_rFpciwxs%3D5%3Dh%3D%3DXC7E3mdXWyO-skSP4ZjidCg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: first pipeline attempt not working
On Tue, Jul 5, 2016 at 5:13 PM, Kay Roepke <k...@torch.sh> wrote: > The static fields are not added by the input, even though the UI makes you > believe they are. In fact they are added during the filter chain execution, > which I from your description runs after the pipeline in your system. > Whoa! Thanks - that's good to know Perhaps the Input fiddling options should be mentioned in the "Message Processors Configuration" page - perhaps like a fake, uneditable "plugin 99" or something? -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgKvRaSPjXNZt4FqYhmu7md-nW-U75QiqwNa%3D%3DvR_-sQ3A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] first pipeline attempt not working
Hi there
I'm trying to get my first pipeline working - without any luck
I have checked and "pipeline processor" is #1 in "Message Processors
Configuration", and consists of one pipeline, with one stage which contains
one rule. The rule is
rule "My little pony"
when
has_field("dont_like_cricket")
then
drop_message();
end
I can see on the "Pipeline overview" page that it's processing the entire
incoming feed.
What I have is a Syslog Input channel which I have the Input adding the
field "dont_like_cricket" to every incoming record (ie tagging it as
different from other Inputs). If I search graylog, I can see the records
contain the field "dont_like_cricket". But this pipeline never triggers - I
still see the records that should have been dropped.
My end-game is obviously a little more complicated, but even this doesn't
work - so 'baby steps' :-)
Any ideas? Also, I really only want this pipeline on one Input channel - do
they have to be "universal"?
Thanks!
--
Cheers
Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/CAFChrgJNhTrS%3DToy6UhfrMpfoP6RwswNvVv0LWaq6ifhPCF0Fg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Having some difficulties with 3 node graylog cluster
On Thu, Jun 23, 2016 at 7:18 PM, Yiannis <k...@stoiximan.gr> wrote: > seems to me that my browser goes again from the login screen (to send > again the user credential) before rendering the results > This is a known bug - see https://github.com/Graylog2/graylog2-server/issues/2071 Seems to only affect Firefox (ie I see it every time I do a search in Firefox, but don't see it with Chrome) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJ9%3DOYCKvJmwgN4Ugc0JVmFzDtOFYrj7n%2BVbVcyCC8hCg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] what is the best way of creating fields in graylog?
On Thu, Jun 23, 2016 at 6:00 AM, Jan Doberstein <j...@jalogisch.de> wrote: > Pipeline is stored in the MongoDB and shared with all Servers. > As this (pipelines) is the future and extractors will become part of > the pipeline you should look into them. > OK, so to restore existing pipeline configs after a reinstall, would that just be restoring pipeline_processor_pipelines* from backup, or would more mongodb fiddling be required? -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgLFXMSwe6ecVTX5TOM1gfCFhVKFNtHgbwxZjo%3DbOmthCA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] what is the best way of creating fields in graylog?
Hi there I've been happily creating extractors in graylog, and have the problem of keeping them synced between my 3 Syslog INPUT channels (ie UDP, TCP and TCP/TLS). As we are moving from a single graylog server to two, keeping such things in sync becomes critical. So I'm thinking of migrating them to some "sharable" format - and want to ensure performance is optimized too of course So what is the "official" best way of creating fields out of data? Drools? Pipeline? The (experimental) latter appears to be database-based - is that automagically shared between graylog servers? With drools, the rules file would be trivial to share - but I guess you have to restart graylog to reload it? Thanks -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJwVn_UEj7rqzQP1SJZz%3DvzdkuR7WPu45sFjJtP4CmrnQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] does graylog support multi-value queries?
Hi there Can I create GELF formatted data streams that contain multi-valued fields? I want to insert data into graylog that contains records where a record has a field that contains multiple values. I need that so I can then search for "field:value1" and will get hits on "field:[value1,value2]" and "field:[value4,value1]" kind of thing This is supported by ES, but my feeble attempts at the same JSON format within GELF didn't work Conversely, I guess I could do this as a wildcard lookup? ie "field:*value1*" - but then that needs wildcarding on a field and support for left hand wildcards - not good things to have in graylog from a performance perspective? Thanks -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrg%2BMi49-xuj3ks8xLJBQnBMwf1MURni66%3D5r73Bh50kfFQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] extractor not always running?
Hi there I have a whole bunch of extractors and in general they work just fine. However, I was just doing a search and got a surprising result back, and when I dug into it discovered events that a month-old extractor should have run on had failed to do so. ie I have extractors that create new fields, and identical records do not trigger the same field creation. If I manually load such a "skipped" event into the extractor wizard, it demonstrates that it wants to extract the field - but in reality had failed to do so There's no evidence of a load problem, and the extractors are definitely working - just not 100% of time... What could be behind this inconsistency? Could it be previous extractors? Frankly I'm not even sure what happens - if "extractor 3" matches, does "extractor 7" still get compared? This is CentOS-7, with graylog-server-2.0.2-1/elasticsearch-2.3.3-1 Thanks -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJPYheNn3PEx-ojs-%2BxUs_HTiAdA9B79xo4UDjrXAeVvQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] recommendations for image/graphing for graylog?
Hi there I'm wanting to scrape some reports out of graylog as tables/images for viewing by a wider range of internal staff - who definitely don't need access via the web interface I have no idea where to start. I assume I'll need to do some ES calls to pull out the raw data and then throw through some kind of graphing package - any recommendations from the group how to do this? Thanks! -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJ6rJ2ikQEhN08tikcJKtTdH6yHPoCsHyBmMhbaabDdZQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: large searches kill ES - can graylog stop this?
On Sat, Jun 4, 2016 at 3:26 AM, Jochen Schalanda <joc...@graylog.com> wrote:
> you can restrict the time range users are allowed to run queries in since
> Graylog 2.0.0 (see System -> Configurations -> Searches configuration).
> Other than that, it would help to split your indices into more shards (and
> distribute them on more Elasticsearch nodes).
>
> This being said, the error you've mentioned (field data cache being full)
> most often occurs if the mentioned field ("message" in this case) is being
> used in an aggregation (e. g. Quick Values), so you might want to avoid
> those on analyzed fields like "message", "full_message", and "source".
>
Hi Jochen
You just reinforced my point. What you're saying is that *graylog users
have to change behaviour* in order to stop graylog/ES from crashing...
My intention is to make our graylog server available to 50+ people to use
*as they see fit*. They all know a lot less than I do about the limitations
of graylog/ES - and I know nearly nothing.
I appreciate this might be impossible to fix (I know nearly nothing, but
this sounds like an ES issue - not a graylog issue), but it doesn't change
the fact that a user making an honest mistake leading to the entire system
being broken isn't a good outcome. I was just wondering if there were any
kind of filters graylog could do to stop such "runaway" search queries from
being run, or some mechanism to pick up the fact that ES has indeed "run
away" and then do something about it to fix it? Otherwise, how do graylog
users solve this problem? It will happen again
--
Cheers
Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/CAFChrgKeoZROXSHKyt_Wh6uT7ZH2fYAWXNQKCg0MT9RsX1Bv8w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] large searches kill ES - can graylog stop this?
Hi there
I just did a simple search on 30 days of data and managed to trigger the
following ES error
[2016-06-01 00:12:53,525][WARN ][indices.breaker.fielddata] [fielddata] New
used memory 11273780309 [10.4gb] for data of [message] would be larger than
configured breaker: 10857952051 [10.1gb], breaking
According to what I can google, this means that ES would have had to
allocate more resources than available to fulfil it, and that condition
somehow triggers an epic fail: either ES becomes unresponsive or
graylog-server does - I can't tell the difference. All I know is right now
I have messages going into graylog and nothing coming out.
Within a minute, things went bad to worse, suddenly I'm getting shard
errors (first shard errors in ages - definitely related)
[2016-06-01 00:21:32,860][WARN ][indices.cluster ] [fantail]
[[graylog_488][0]] marking and sending shard failed due to [engine failure,
reason [already closed by tragic event on the index writer]]
[graylog_488][[graylog_488][0]] ShardNotFoundException[no such shard]
at org.elasticsearch.index.IndexService.shardSafe(IndexService.java:197)
[2016-06-01 00:21:32,962][WARN ][cluster.action.shard ] [fantail]
[graylog_488][0] received shard failed for target shard [[graylog_488][0],
node[Tjzmk9cFRuCke6JEuomb4g], [P], v[2], s[STARTED],
a[id=dgyATFPBQAywkydc2mxmPw]], indexUUID [jxF7U5fESqOzJu9CSDF3WA], message
[engine failure, reason [already closed by tragic event on the index
writer]], failure [OutOfMemoryError[Java heap space]]
[2016-06-01 00:21:32,974][WARN ][cluster.action.shard ] [fantail]
[graylog_488][0] received shard failed for target shard [[graylog_488][0],
node[Tjzmk9cFRuCke6JEuomb4g], [P], v[2], s[STARTED],
a[id=dgyATFPBQAywkydc2mxmPw]], indexUUID [jxF7U5fESqOzJu9CSDF3WA], message
[master {fantail}{Tjzmk9cFRuCke6JEuomb4g}{127.0.0.1}{127.0.0.1:9300} marked
shard as started, but shard has previous failed. resending shard failure.]
[2016-06-01 00:21:33,182][INFO ][cluster.routing.allocation] [fantail]
Cluster health status changed from [GREEN] to [RED] (reason: [shards failed
[[graylog_488][0], [graylog_488][0]] ...]).
Restarting graylog-server and ES (and cleaning up...) will solve this - but
this is lame. graylog is an end-user tool that *by design* will have people
doing actions that - on occasion - are beyond the reach of the backend:
there has to be some way this could be handled better. The ES people seem
to think this is a case of "you're doing it wrong", but graylog isn't some
programmed frontend where every ES call is tightly managed - it's something
that is meant to be used to "play" with data. Basically all I did was take
a previous search that worked and asked it to re-run with an hourly graph
instead of daily - enough to tip it over the edge. This will happen time
and time again - so causing service outages is an acceptable outcome?
How are others dealing with this? Could graylog capture the ES error and
mitigate (somehow)? I for one should have shut everything down before that
"breaker" error turned into the "shard" error.
This is graylog-server-2.0.2/elasticsearch-2.3.3 under CentOS-7
Thanks
Jason
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/b7a7b095-3b6d-47fb-8bb0-bc62b8b67011%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] anyone know how to merge data into the GeoIP2 mmdb files?
Hi there We're primarily using graylog for our internal logging needs, and we're on the 10/8 network - so the GeoIP data is irrelevant to us. However, I have lat/long for all our offices - and I know what 10.X.Y subnets are assigned to each office - so I can easily map subnets to lat/long. Has anyone gone through the same thought process and figured out how to merge their internal IP ranges into MMDB? Others have done this for the older ".dat" format - but none of that applies to this new format Conversely, could graylog support the older format - where all the tools still are? ;-) Thanks -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgKs-X_37StMj_1gsGf5vphF03MBMXZW4ew9-GZzsJFoWw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] fielddata error with search
Hi there I tried to do what I thought was a simple search across a week's worth of data on a single-box graylog server (ie it also has ES and mongodb on it) Basically I did a search for "fieldname:value1 OR fieldname:value2 OR fieldname:value3" over 7days and graylog just sat there spinning it's wheels (before hand I was happily doing searches that weren't causing any grief at all) The CPU on the graylog server went through the roof, graylog error file showed no problem, but ES logs showed a bunch of these indices.breaker.fielddata] [fielddata] New used memory 11155918063 [10.3gb] for data of [message] would be larger than configured breaker: 10857952051 [10.1gb], breaking After five minutes of graylog just sitting there, I restarted ES, but graylog was now borked. The input channels were still receiving data, but nothing was flowing out. So I restarted graylog and all was good again Is this expected behaviour, and if so, what is needed to stop it? I've seen other non-graylog related postings on the ES list about this happening with large clusters, so it seems to be an error case for ES, but I'm more concerned over how graylog reacted: ie why didn't it give up and give me an error page for starters. It looks to me like graylog didn't expect that ES search to error out and that caused it to block? (I'm assuming ES generated an error - the logs shows that WARN - I dunno what happens next) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrg%2BMzp5Cr_vQtQ1Gsjd5SyF5qrSxgmp-0SR%3DgW_5xP4gGg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] web interface with v2.0 appears to require direct REST access?
Hi there Under graylog-1.3.4 I had published graylog-web behind a WAF - which nicely mapped https://graylog.internet.domain to http://graylog.intranet.domain (notice the different domain names too) With v2.0 I can't get this to work. Now it appears graylog returns content with hardwired URLs that are defined by rest_listen_uri? That means we end up with browser errors as they are talking to the WAF over HTTPS and the content contains HTTP links - to port 12900. Bad. Am I correct that graylog-v2 requires browsers to talk to non-web ports (ie 12900)? That's quite a change. The comments say "Must be reachable by other Graylog server nodes if you run a cluster" - no mention of this being required by web browsers. I'm confused? -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgL8SC3ZoxkB2k4A_QsAsiy3a_rwO%2BamDYHdLJPK-4Ww_A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] best way to do a "read only" audit account?
Hi there I want to set up graylog (ldap auth) so that there's a group who have full READ access (audit team). ie instead of creating a Stream and giving them access to that, I want them to be able to access all the data - but not be admins. I have Roles working well for other groups - but this corner case doesn't sit well. I want to use the LDAP groups to do this - so I have a group and can map it to a Role. But the Role needs to be assigned to a Stream - and the "default" Stream of "all" doesn't exist. I could go through and assign all the Streams to the Role - but then I'd have to remember every time we added a new Stream to go and update the Role... Alternatively I could create a new Stream called "Read-Only" that has no filter - and assign that to the Role - but that seems excessive. These Streams are not cheap (in terms of resources) - so you shouldn't create more than are needed Wouldn't it be sensible to always have a "default" Stream named "All data" (would probably have to be hard-wired as readonly) - so that it can be allocated to Roles? It's really a "virtual" Stream, consisting of everything -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJ2YuaaAiuNOHoAHM8eT0T%3D-ppNajE9kkevQwKjQv_Fng%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: installed marvel - now seemed to have corrupted entire graylog db
Hi Jochen There wasn't any error that pointed at this explicitly. This is graylog-1.3.4 with ES-1.7 on CentOS7. There has been a general degradation as my (single) test box grew to 5TB in size - I think I'm simply tickling all the edge cases of performance issues and this is the consequence. I do think ES could do with some major improvements - I continually saw graylog reporting it couldn't push data into ES and yet ES reported no problems and showed "green". And yet if I restarted ES (ie stop/start) it immediately came up as "red" with 10,000+ UNASSIGNED shards - so there's a fairly serious bug in there IMHO. In any case I ended up deleting all the indexes that wouldn't move off "UNASSIGNED" and basically ended up deleting most of my data, so I've thrown the entire thing away (hey, it is a test box :-) and restarted with the Beta graylog-2 with ES-2.3.1 and we'll see how that goes. This test box is almost at an end - new hardware has finally arrived - so I think the root cause of the issues will soon be resolved by actually having a cluster/etc. Thanks Jason On Wed, Apr 13, 2016 at 12:55 AM, Jochen Schalanda <joc...@graylog.com> wrote: > Hi Jason, > > what's your Graylog and Elasticsearch configuration? Are there any error > messages in the logs of either Graylog or Elasticsearch? > > Cheers, > Jochen > > > On Monday, 11 April 2016 07:15:17 UTC+2, Jason Haar wrote: >> >> Hi there >> >> Over the weekend I installed the ES marvel diagnostics package and the >> following day noticed that graylog was broken. Restarting ES showed 20,000 >> shards in an UNASSIGNED state. I disabled graylog-server (so there was no >> new data flowing in) and watched over the next couple of hours as that >> UNASSIGNED number dropped down to 0 (and GREEN). I then restarted ES and - >> bam - back to 20,000 UNASSIGNED shards again >> >> I've now done three iterations of that - looks like it's completely >> borked. There's over 5TB of data in there - how can I regain it? >> >> PS: I don't know if this has anything to do with marvel - it's just the >> last change I made. The reason I installed it was because I have had ES >> continually doing this kind of thing - but previously stopping graylog, >> restarting ES and waiting would lead to a happy ES - but no longer. >> >> -- >> Cheers >> >> Jason Haar >> Information Security Manager, Trimble Navigation Ltd. >> Phone: +1 408 481 8171 >> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 >> > -- > You received this message because you are subscribed to the Google Groups > "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/507dbfa3-0239-45ca-9799-2cae7db65418%40googlegroups.com > <https://groups.google.com/d/msgid/graylog2/507dbfa3-0239-45ca-9799-2cae7db65418%40googlegroups.com?utm_medium=email_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrg%2BL5zOhKu%2BHLJVtQjpZyjSQTe8HtsmS_%3DDJyBU%2B44UpTg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] installed marvel - now seemed to have corrupted entire graylog db
Hi there Over the weekend I installed the ES marvel diagnostics package and the following day noticed that graylog was broken. Restarting ES showed 20,000 shards in an UNASSIGNED state. I disabled graylog-server (so there was no new data flowing in) and watched over the next couple of hours as that UNASSIGNED number dropped down to 0 (and GREEN). I then restarted ES and - bam - back to 20,000 UNASSIGNED shards again I've now done three iterations of that - looks like it's completely borked. There's over 5TB of data in there - how can I regain it? PS: I don't know if this has anything to do with marvel - it's just the last change I made. The reason I installed it was because I have had ES continually doing this kind of thing - but previously stopping graylog, restarting ES and waiting would lead to a happy ES - but no longer. -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgLkJYm6PnHRLVstu%3Dva%2BDLfRsG4qGHcy9piu2BZf_oWPA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: [ANNOUNCE] Graylog v2.0-beta.1 has been released
On 25/03/16 10:53, Arie wrote: Super, Are there some guidelines on upgrading from 1.3.4 > 2.0? Would be useful. One is "don't have dots in your fieldnames" as I discovered :-( -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/56F479A8.2070605%40trimble.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] elasticsearch-1.7's "/_cat/shards" not reliable?
Hi there I'm having severe issues keeping graylog-server working and it looks like it's ES at fault. ie graylog-server starts reporting timeouts and basically gives up, but the ES logfile shows absolutely no problem. It could be a load issue with ES, but when you run the following commands curl http://localhost/_cluster/health curl -s -XGET http://localhost:9200/_cat/shards|grep -c UNASSIGNED ...and the first returns "green" and the second returns "0", you'd think that means that ES is happy and there are zero unassigned shards. But if I shutdown graylog-server, then restart ES, it immediately afterwards starts reporting "red" and that there are 14900 unassigned shards! So it smells like graylog-server realises through it's own logic that ES is not happy - but ES is not telling me that at all Is this expected behaviour? What should I be doing to actually pick up ES errors, when the logfiles and commands like this totally fail to mention there's a problem? I can't fix something that I can't detect :-( Thanks! -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrg%2BtR4Zq15dLxwnowZ2J6EL9_ws5FLxVLQW3q2%3Ds50YDZQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] relationship between graylog-server problems and elasticsearch
Hi there I've got ongoing issues with graylog stopping working and it doesn't look like it's elasticsearch related Right now graylog-web is reporting the following error: Some messages were deleted from the Graylog journal before they could be written to Elasticsearch. Please verify that your Elasticsearch cluster is healthy and fast enough According to "http://127.0.0.1:9200/_cluster/health;, elasticsearch is "green" with no unassigned shards/etc. I acknowledge I'm running an underspec'ed environment (waiting on hardware), but I'm trying to understand the failure modes here, so can someone confirm my logic?. graylog-server is writing to elasticsearch, ES is happy and reports no errors, but I guess it's taking too long to do individual transactions, and so graylog-server has to queue them up in the journal until it can. And if the journal fills up, then it triggers the above error? So we might be able to increase the journal size, but if the average throughput "load" of graylog-server is such that this is always happening, then you have to have a "faster" ES? Could it be that graylog-server itself is overloaded? ie that ES is fine - but I need more graylog-servers feeding into the same ES? How can I differentiate between those two - what does that error case look like? Thanks! -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgKRhcVKn44ELZYT-uUC%2Br5PMecf3eVtn-tdMfmEs2GnTA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] should there be a "replace extractors" option?
Hi there I have a bunch of extractor rules for our syslog INPUT. That means I actually have a UDP, TCP and TCP/TLS INPUT channel for syslog - and obviously want the extractors to be the same over the three of them Currently I have to edit one of them, get new changes working, and then delete all the extractor rules from the other two INPUT channels (which means clicking on DELETE-OK dozens of times) - so that I can then do a "export->import" of the updated set. during this process it means I've got incoming data that isn't having those extractors applied to them I'm sure there's some way I could do the same thing with curl/etc, but adding a "replace extractors" to the "import extractors/export extractors" dropdown list would be much easier? -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgK-Tyz%2BD03V36KtS8Xq4-%3DR9P4EPhahpfgFZX4XKLF5mQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] anyone written a nessus parser for GELF?
Hi there We use nessus for vulnerability scans and I'd like to import the nessus reports into graylog so that I can better cross-correlate events involving internal hosts I can see others in the ELK camp have done this, but can't find any evidence of anyone doing it for graylog? Has someone already done it, or conversely, is there a way of changing a program that outputs directly to Elasticsearch to output to GELF? (so as to make it work with graylog) If not, I guess I'll have a go at it :-) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJiwOfm3WwwuVrjhAFzrdHpFuGwGqmgGaW3Y4NRVAEiMg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] why do open files increase over time?
Hi there I've got a graylog-1.1.3 instance (web/server+elasticsearch) running (CentOS-7) that I haven't changed INPUTs on for some months (ie I have one incoming syslog feed and 'n' GELF feeds). From what I know, graylog-server takes that data and pushes it into elasticsearch according to the sharding/etc settings, with auto-expiring of old data according to settings As such, I would expect it to get into a "steady state" where the fundamental OS characteristics are fairly stable? ie it would use just about "this much" RAM, "this many open files", etc. Anyhow, two days it totally went down as it ran out of open file descriptors. Ended up corrupting over 9000 indexes before I noticed - a real mess. I increased nofiles, rebooted and then used that very nice script referred to below to re-absorb the borked indexes https://github.com/elastic/elasticsearch/issues/4206 So the thing I don't understand is why this happened (or why didn't this happen sooner)? In a steady state environment, why would the number of open files be increasing over time? eg only one index is open for write at any moment, and indexes are only open for read during searches, so why would this increase? More importantly, if this increase is meant to happen, doesn't that imply running out of file descriptors is inevitable? The other thing is why didn't graylog-server exit when this situation occurred? It seems to me that when elasticsearch started erroring, it should have exited (I mean, you don't recover from running out of file descriptors), but as it didn't, then why didn't graylog-server? Under what situation is it better to end up with 9000 corrupt indexes rather than a total outage? I'm still waiting for elasticsearch to finish re-assigning the unassigned_shards created by the above recovery process - it's working, but it's been 8 hours so far and it's still plodding along (so it's a two day outage for me so far). If graylog-server figured out elasticsearch was status "RED", why not shut down entirely so as to not make the situation any worse, and cause an easier to notice outage? Also, there's a bug with the elasticsearch rpm's. /etc/sysconfig/elasticsearch states to not set MAX_OPEN_FILES when using systemd (which you would be with CentOS7) and to instead set LimitNOFILE in /usr/lib/systemd/system/elasticsearch.service. However, /usr/lib/systemd/system/elasticsearch.service is replaced every time you upgrade elasticsearch. So either their documentation is wrong and /etc/sysconfig/elasticsearch is what "wins", or their rpm installer is broken. I'll open a bug report for them (not a graylog issue - but a FYI for others) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgKZYE0bQOfKJfwYAONjYZh%2BrO4R5ir85gTj55m-RffdTA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] logarithmic scaling on graphs?
Hi there I just had an event where I was looking at some data of interest via the Search Histogram and all I could see was this big spike - looked like there was nothing except for this spike. Threw me for a bit but I figured out that the size of the spike was so large that the other data points faded to near nothing. A logarithmic scale would be really useful in this situation - would allow you to see the secondary features without having to pre-filter the noisy one. Is there any intention of adding Y-axis changes as an option - eg logarithmic scale? I assume others would find that useful too? Thanks for the work! -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/56A9DE66.7060905%40trimble.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: GeoIp lookup plugin
On 10/12/15 23:03, Arie wrote: > As far as I know it is not there yet, but kind of work in > progres:https://graylog.ideas.aha.io/ideas/GL2E-I-364 That's not the case: the graylog staff have said there are no plans to implement this - and that ticket should actually be closed :-( (see "what can I do to prepare for geoip support?") -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/566A17FF.2040603%40trimble.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] confused how extractor fields work
On 23/11/15 10:58, Joi Owen wrote: > > I think your best bet is to split that one big text section you're > currently using into separate extracted fields. Use the original > regexp as the control test, the one that decides if a particular log > line is going to be further inspected for the matching field. > That really doesn't scale. eg I'm inserting our proxy server logs into graylog, and I want to search to find all urls that end in "/73.exe" - eg "/one/73.exe" and "/two/73.exe?id=112". Separating all urls into single-word fields would add nearly infinite numbers of fields - not really an option As you mentioned earlier, the documentation says "Also note that message, full_message, and source are the only fields that can be searched via wildcard by default". So that implies I should be able to make other fields act like the message field So how can I make my "url" field properly searchable? (ie to handle wildcards) Thanks -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/56634E6D.3060404%40trimble.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Graylog Best Practices
On 03/12/15 07:17, Joi Owen wrote: > One benefit of having separated inputs is that you can isolate unique > extractors to only the input that provides the fields of interest, > reducing the load of having to parse for those fields on log data > arriving from unrelated sources. > The way I look at it is that you have two ways of massaging the data. One is to do it on the "client" end (eg via graylog-collector, pygelf, etc) and one is on the "server" end via extractors. So what I have ended up with is all our syslog data goes in via the syslog connector and relies on extractors to create the fields, and all other forms of data input go via pygelf scripts - and I code into it how I want the fields to be defined - and they go in via a GELF/TLS connector The extractor method has the advantage that you can centralize all your massaging, but the client-based massaging has the advantage that you remove workload from the graylog-servers (and is way more powerful of course) Jason > > On Wed, Dec 2, 2015 at 10:40 AM, Sean McGurk <mymonkeyan...@gmail.com > <mailto:mymonkeyan...@gmail.com>> wrote: > > Thanks, Jochen, > > I perhaps didn't make myself clear in my question - I have a > number of Graylog collectors running on different instances and my > question was more whether I should create a separate input on a > distinct port for each of these collectors or just create one > input and have all the collectors send to that one input. > > In the end, I went with the second approach, so I have one GELF > TCP input started on port 12201, which aggregates the data from > all the collectors. > > I then have created a number of streams to route the incoming > data, where they can be separated by log (and application) type. > > I preferred this approach as it meant from a security point of > view, I only had to open one more port and the 'Streams' concept > allowed me to segment my log messages. > > Seán > > On Tuesday, 1 December 2015 16:41:40 UTC, Sean McGurk wrote: > > Hi there, > > I have set up a Graylog server with a number (7) of input sources. > > My question is, when configuring Graylog, is it better to open > a number of ports on the Graylog server and have each port > receive messages from a particular source or is it better to > only open 1 port and receive all inputs via this 1 port? > > Thanks, > > Seán > > > -- > You received this message because you are subscribed to the Google > Groups "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to graylog2+unsubscr...@googlegroups.com > <mailto:graylog2+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > > https://groups.google.com/d/msgid/graylog2/7c662b78-65ee-4eb0-8186-551b63147d5f%40googlegroups.com > > <https://groups.google.com/d/msgid/graylog2/7c662b78-65ee-4eb0-8186-551b63147d5f%40googlegroups.com?utm_medium=email_source=footer>. > > > For more options, visit https://groups.google.com/d/optout. > > > > > -- > > No matter what we think of Linux versus FreeBSD, etc., the one thing I > really like about Linux is that it has Microsoft worried. Anything > that kicks a monopoly in the pants has got to be good for something. > - Chris Johnson > > -- > You received this message because you are subscribed to the Google > Groups "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to graylog2+unsubscr...@googlegroups.com > <mailto:graylog2+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/CAL5rfGViaTCAQw4iCOgYH96Ghpq_sDoo7uVBzCb49LaSxmU9xA%40mail.gmail.com > <https://groups.google.com/d/msgid/graylog2/CAL5rfGViaTCAQw4iCOgYH96Ghpq_sDoo7uVBzCb49LaSxmU9xA%40mail.gmail.com?utm_medium=email_source=footer>. > For more options, visit https://groups.google.com/d/optout. -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/565F3F92.2010707%40trimble.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] confused how extractor fields work
Hi there I created an extractor to extract part of a record and associate it with a new fieldname. That works fine - expanding affected records inside graylog-web shows the new field shows up with the correct sub-section of the original message in it. However, if I search for a word inside that matched field, I get no hits. When I do the same search using the default "message" field - I get the hits expected. So what I have I missed? When you create new fields using extractors, how do you ensure they are searchable using (I guess) substring matches? It was a "word" that showed up in the middle of the "sentence" that was associated with the new fieldname - ie pretty simple. And yet I couldn't search for a word? The converter on the extractor was the default "Numeric" - should it be something else? This is graylog-1.2.2 Thanks -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/5651814D.703%40trimble.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Google Apps logging/reports support for graylog?
Hi there Has anyone figured out how to get Google Apps logging into graylog? (sort of the Google equivalent to the AWS CloudTrails plugin) I see there's a Splunk connector for it https://splunkbase.splunk.com/app/2714/ -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/5629635A.50105%40trimble.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: Upgrade to 1.2.1 Graylog Journal does not flush messages, index range issue
On 16/10/15 09:17, Andrei S wrote: > So, it seems the issue was caused by a corrupt journal file. It was a > coincidence that i upgraded to the new version just a day before. Perhaps a naive question, but shouldn't there be a way to detect corrupt files and skip them? I've been hit by similar issues in the past: ran out of disk space, too much incoming data for backend, etc - tonnes of ways that such problems can occur. To fix them I've always had to go in and delete "corrupt" journal files before graylog (elasticsearch?) would start working again. Surely it should be able to self-heal in these situations? I mean a corrupt file is useless - so why tolerate them? -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/56200C60.2010703%40trimble.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] question about interpreting graylog stats
Hi there I'm still running a test instance of graylog/elasticsearch and I know I have too much data coming in for the sizing, but it is showing me problems that my understanding better now should help in the future with "real" systems :-) Anyway, I have more incoming data via graylog-server that that elasticsearch backend can comfortably deal with - but that seems to manifest itself in an odd manner. The "throughput" stats shows figures like this Processing 475 incoming and 3,500 outgoing msg/s. 3,954,288 unprocessed messages are currently in the journal, in 52 segments. 476 messages have been appended to, and 403 messages have been read from the journal in the last second. What I don't understand is how there can be more outgoing messages than incoming. This isn't some "spike" - it's continuous. Surely if elasticsearch couldn't keep up, you'd see more incoming than outgoing? -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/56156671.1090207%40trimble.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: upgrading graylog-server from 1.16 to 1.2rc4 totally broke all LDAP access
On 09/09/15 20:41, Kay Roepke wrote:
> Could you please turn the log level
> of org.graylog2.security.ldap.LdapConnector to TRACE?
> The easiest way to do so is via the System/Logging section in the API
> browser (port 12900 of your graylog server).
>
Err - humor me - this is all new to me. I can't see a System/Logging
section - I can see a System/Loggers section - but I can't see how that
relates to LDAP settings - nothing shouts out as being related
Anyway, I simply cranked all graylog-server logging up to TRACE via the
"Logging" page on graylog-web and I assume that does the same thing (in
a noisier manner!)
I don't see any new errors (but they wouldn't be in TRACE?), but I see
vast amounts of LDAP data being recorded - so that looks fine (" egrep
-i 'UserServiceImpl|ldap' "). There's a lot of binary data in there -
I'd guess the login event pulls all fields? (BTW really shouldn't - that
slows things down - especially if there's a WAN involved). So you get
ones like "msexchrecordedname" - which is a 4K binary blob - and one
I'm looking at right now isn't even mine. I'm the only user on the
system, I would have thought graylog would only pull back details from
my account?
How does this new LDAP group-role mapping work? Is graylog trying to
suck out all groups from LDAP to populate the mapping page? The Global
Catalog of our AD forest is over 300MB in size if you were to try to
scrape the lot... I know I can put a filter in there - but as it's not
working with "(objectClass=group)" I don't think there's much point in
making it less likely to work ;-)
Anyway, these TRACE logs might mean something...?
015-09-09T05:46:52.776-04:00 TRACE [DelegatingSubject] attempting to get
session; create = false; session is null = true; session has id = false
2015-09-09T05:46:52.776-04:00 TRACE [DefaultSubjectDAO] Session storage
of subject state for Subject
[org.apache.shiro.subject.support.DelegatingSubject@7685d279] has been
disabled: identity and authentication state are expected to be
initialized on every request or invocation.
2015-09-09T05:46:52.776-04:00 TRACE [DefaultSecurityManager] This
org.apache.shiro.mgt.DefaultSecurityManager instance does not have a
[org.apache.shiro.mgt.RememberMeManager] instance configured.
RememberMe services will not be performed for account [ja...@nz.our.domain].
2015-09-09T05:46:52.776-04:00 TRACE [DelegatingSubject] attempting to
get session; create = false; session is null = true; session has id = false
--
Cheers
Jason Haar
Corporate Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/55F0062D.3010303%40trimble.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: upgrading graylog-server from 1.16 to 1.2rc4 totally broke all LDAP access
On 10/09/15 00:29, Kay Roepke wrote: > Would you be willing to give a snapshot build a try once I have it up? Sure thing - I'm still only got a single host test box - so it's no big thing if it breaks ;-) > > Not really, the interesting ones come from LdapConnector. OK, well here's what the following grep reports, after I remove what I assume are normal lines grep LdapConnector server.log.4|egrep -v ' DN CN=.* member\?$|Group Entry: Entry$' , manager=.., msexchrecordedname= 2015-09-09T05:46:52.504-04:00 TRACE [LdapConnector] Re-binding with DN CN= using password 2015-09-09T05:46:52.749-04:00 TRACE [LdapConnector] Binding DN CN= did not throw, connection authenticated: true So that binary blob shows up on it's own line (ie doesn't begin with a timestamp) - so there must be a carriage return in there - could that cause issues? -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/55F0B9A5.5020404%40trimble.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] upgrading graylog-server from 1.16 to 1.2rc4 totally broke all LDAP access
Hi there
Says it all really. After upgrading from 1.16 to 1.2rc4, none of the LDAP
(actually ActiveDirectory) accounts work - even the Admin ones (thankfully
the standard backdoor "admin" account still works)
I tried logging in with a new LDAP account - it also fails (default user
mode: Reader). But refreshing the "user" area shows the new account - so
it's been created even though I can't log in with it. The login page error
says "sorry those creds aren't valid"
I didn't change the LDAP User Mapping area [ which is set to
"(&(objectClass=user)(userPrincipalName={0}))" ], but changed the new Group
Mapping to
(&(objectClass=group)(cn=*))
with "Group Name Attribute" set to "cn". I also used ldapsearch to test
that filter - it works fine, returning a bunch of groups
However, after filling in that section I go to "LDAP Group Mapping" and it
says there are no LDAP groups - so something's wrong in the group section
of the "LDAP Settings". We are running an AD forest and I'm logging in
using an account from a child domain (we don't have user accounts in the
parent) - so could this be a recursion problem? However, the logs do show
evidence of the LDAP query bringing back groups from the child domains - so
it all looks good as far as I can see
I've turned up the Authentication logging to "debug" and this shows up on
any LDAP login event. That "ERR_04486_VALUE_ALREADY_EXISTS" is the only
thing that looks like an error?
2015-09-08T20:56:25.519-04:00 DEBUG [ModularRealmAuthenticator] Realm
[org.graylog2.security.realm.SessionAuthenticator@79ea39fc] does not
support token org.apache.shiro.authc.UsernamePasswordToken -
usern...@domain.name, rememberMe=false. Skipping realm.
2015-09-08T20:56:25.520-04:00 DEBUG [ModularRealmAuthenticator] Realm
[org.graylog2.security.realm.AccessTokenAuthenticator@5d75e8f0] does not
support token org.apache.shiro.authc.UsernamePasswordToken -
usern...@domain.name, rememberMe=false. Skipping realm.
2015-09-08T20:56:40.614-04:00 ERROR [DefaultAttribute]
ERR_04486_VALUE_ALREADY_EXISTS The value '20150728213900.0Z' already exists
in the attribute (dSCorePropagationData)
2015-09-08T20:56:41.964-04:00 WARN [UserServiceImpl] User
usern...@domain.name: No group mapping for ldap group
2015-09-08T20:56:41.969-04:00 WARN [UserServiceImpl] User
usern...@domain.name: No group mapping for ldap group
2015-09-08T20:56:41.969-04:00 WARN [UserServiceImpl] User
usern...@domain.name: No group mapping for ldap group
2015-09-08T20:56:41.971-04:00 DEBUG [AuthenticatingRealm] Looked up
AuthenticationInfo [usern...@domain.name] from doGetAuthenticationInfo
2015-09-08T20:56:41.971-04:00 DEBUG [AuthenticatingRealm]
AuthenticationInfo caching is disabled for info [usern...@domain.name].
Submitted token: [org.apache.shiro.authc.UsernamePasswordToken -
usern...@domain.name, rememberMe=false].
2015-09-08T20:56:41.973-04:00 DEBUG [AuthenticatingRealm] Looked up
AuthenticationInfo [null] from doGetAuthenticationInfo
2015-09-08T20:56:41.973-04:00 DEBUG [AuthenticatingRealm] No
AuthenticationInfo found for submitted AuthenticationToken
[org.apache.shiro.authc.UsernamePasswordToken - usern...@domain.name,
rememberMe=false]. Returning null.
2015-09-08T20:56:41.973-04:00 DEBUG [AuthenticatingRealm] Looked up
AuthenticationInfo [null] from doGetAuthenticationInfo
2015-09-08T20:56:41.973-04:00 DEBUG [AuthenticatingRealm] No
AuthenticationInfo found for submitted AuthenticationToken
[org.apache.shiro.authc.UsernamePasswordToken - usern...@domain.name,
rememberMe=false]. Returning null.
2015-09-08T20:56:41.973-04:00 DEBUG [AbstractAuthenticator] Authentication
successful for token [org.apache.shiro.authc.UsernamePasswordToken -
usern...@domain.name, rememberMe=false]. Returned account
[usern...@domain.name]
2015-09-08T20:56:41.973-04:00 DEBUG [DefaultSubjectContext] No
SecurityManager available in subject context map. Falling back to
SecurityUtils.getSecurityManager() lookup.
2015-09-08T20:56:41.973-04:00 DEBUG [DefaultSubjectContext] No
SecurityManager available in subject context map. Falling back to
SecurityUtils.getSecurityManager() lookup.
2015-09-08T20:56:41.976-04:00 DEBUG [DefaultSessionManager] Creating new
EIS record for new session instance
[org.apache.shiro.session.mgt.SimpleSession,id=null]
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/203aaf72-7700-4e9a-b881-2b4e1280f227%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: upgrading graylog-server from 1.16 to 1.2rc4 totally broke all LDAP access
Whoops - forgot to mention this was LDAPS to our Global Catalog LDAP service (that's the trick Microsoft uses to emulate "flattening" an AD hierarchy Also I just changed from LDAPS to LDAP so that I could sniff what's going on. According to wireshark the group search filter was working - returning data. However, if I removed the filter entirely, then I can log in via LDAP! So it's now back to the way it was before the upgrade. However, I need to figure out how to do the filter so as to get the LDAP mappings to Roles working. Also, if I even try the example filter "(objectClass=group)" - that breaks it again Thanks Jason -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/abaa8a07-0caf-41e0-a2bb-562938657321%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] is there a GELF over HTTPS option?
Hi there I just brought up a GELF HTTP Input channel and was surprised to see no HTTPS support (graylog-server 1.1.6). Is there some way of doing that? Secondly, is this HTTP channel limited to one JSON record per request? ie can it be used to "stream" GELF - like the TCP and UDP channel can (eg using HTTP Keepalive) - or is it really designed for once-in-a-while web application transactions Thanks -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/55E4F256.50607%40trimble.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Users and access on Global search
On 26/08/15 02:42, Juan Andres Ramirez wrote: Hello Guys, Read's users can't access on global search, this is a problem, because if you have a dashboard with a chart about any process, some time you need to see more details about this search. Anyone knows if in next versions the global search will be available for read's users?. Have you tried creating dashboards based on streams? ie don't expose dashboards to users who can't use them: instead create streams they are allowed to use/search, and then create dashboards limited to that stream - that way they CAN search on it -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups Graylog Users group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/55DCDCBA.8040102%40trimble.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] AWS Cloudtrail plugin issue with multiple regions?
Hi there I went through the documentation for setting up the Cloudtrail plugin for US-WEST-2 and US-EAST-1 and got it all working - the Cloudtrail records showed up in graylog/elasticsearch Then I configured the same S3 bucket for all the other regions - and AWS console shows a new hierarchy of directories (or whatever AWS calls such things) showed up under the S3 bucket - demonstrating to me that all regions are now logging to the same bucket As I had already got US-WEST-2 and US-EAST-1 working, I simply created a bunch of new Inputs using the same creds - with only the region being different. But it didn't work: their data isn't being logged into graylog and server.log is full of the appended error. Yes I checked and they are all configured the same way :-) I'm not that knowledgeable about AWS, but I can't see how it's not working. It all looks like it's dumping to the one S3 bucket correctly within the AWS console, and graylog is working for two regions - so how can the other ones not be working? Do you have to set more permissions for some reason? (I can't see how - if it was a perms issue, surely US-WEST2/US-EAST-1 wouldn't work either?). I've tried deliberately using bad creds and get a different error - so that's not it Any ideas what I've screwed up? Thanks Jason 2015-08-24T05:21:45.578-04:00 ERROR [CloudTrailSubscriber] Could not read messages from SNS. This is most likely a misconfiguration of the plugin. Going into sleep loop and retrying. com.amazonaws.services.sqs.model.QueueDoesNotExistException: The specified queue does not exist for this wsdl version. (Service: AmazonSQS; Status Code: 400; Error Code: AWS.SimpleQueueService.NonExistentQueue; Request ID: cf8f9757-6553-5c24-b2cf-d903813a3862) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1077) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:725) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295) at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2339) at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1072) at com.graylog2.input.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:41) at com.graylog2.input.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:80) -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups Graylog Users group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/55DAE709.9030100%40trimble.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] are there standard field names?
Hi there I've been testing graylog for a few weeks now and I think I'm noticing a few gotchas I've whacked myself with. eg I saw an example extractor that created a src_ip field - that mapped to an IP address. Well, I thought, that's no good - it's only matches IPv4 addresses, so since then I've decided to use src_ipv4 instead as it's more explicit But then I found some unrelated kibana documentation regarding elasticsearch referring to src_ip - which started making me feel like that change was a bad idea... Maybe src_ip is sort of the standard field name for an IP address?? So are there any standard field naming conventions that should be abided by? Obviously elasticsearch is meant to be arbitrarily extendable, but I'm concerned I'll try some plugin later and it won't work because I've fiddled with fieldnames when I shouldn't have, etc -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups Graylog Users group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/55DADF60.9020701%40trimble.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: what can I do to prepare for geoip support?
On 13/08/15 09:06, Jochen Schalanda wrote: Hi Jason, I hear that some form of geoip support is expected in graylog-1.2? That's currently not planned. Pity. You might want to update GL2E-I-364 - it says it's expected in 1.2 -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups Graylog Users group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/55CBC899.3020305%40trimble.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] what can I do to prepare for geoip support?
Hi there I hear that some form of geoip support is expected in graylog-1.2? As such, what can I do to my data today that will help it flick into beautiful life when that feature shows up? eg do I need to add a _location field containing latitude/longitude? Or what is the tag for IP address: _ip? Or _src_ip (or as some examples show, src_ip. Does the underscore make a difference?). I'd also like to track our internal 10.0.0.0/8 IPs (I have latitude/longitude for all our office subnets), so that kind of information wouldn't be in maxmind geoip feeds/etc. Any ideas how to integrate that? Fun times :-) -- Cheers Jason Haar Corporate Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups Graylog Users group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/55CBB434.4050004%40trimble.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Very slow output msg/s read from journal
I can imagine that impacting many of us. Does anyone know of a way of measuring the impact of extractors? ie metrics on how long it takes extractors to process messages? Then this sort of issue could be diagnosed a lot quicker -- You received this message because you are subscribed to the Google Groups Graylog Users group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/78bb7ee3-aead-4bcd-bf1e-98e3677cc6b8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] wildcard searches on fields besides messages?
Hi there I've used extractors to create a bunch of fields to apply over incoming data to search against. Similarly I have other data coming in via GELF with extra fields too. Search works when I do fieldName:full-value, but doesn't work for fieldName:full or fieldName:full*. It's as if they have to always explicitly match? Is there a way to apply simple wildcard rules to them? Thanks -- You received this message because you are subscribed to the Google Groups Graylog Users group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/64746d9c-04b0-4bde-a220-c7b2ea2adea4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: elasticsearch crashed and now graylog-server broken?
It just happened again and this time elasticsearch is hosed. The out of memory error was system-wide - dmesg confirmed it. So the system ran out of memory, elasticsearch crashed, and now graylog-server cannot talk to it any more. When graylog-server attempts to connect to elasticsearch, it now reports [2015-07-31 22:50:25,943][WARN ][indices.cluster ] [Kate Neville] [graylog2_1][0] failed to mark shard as failed (because of [failed recovery]) I have no idea how to fix it (I never used graylog or elasticsearch until last week). Is there some form of recovery process I can run to get this working again? Either fix it or throw the broken bit away and start working again are outcomes I'd be happy with. At the moment the entire system is completely broken :-( Thanks -- You received this message because you are subscribed to the Google Groups Graylog Users group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/898f1c2d-2adc-41dc-82f5-f1d2743f7409%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: GrayLog 1.1.4 - ActiveDirectory connection
You've got a bad filter. There is no uid field in AD, you either need to
use sAMAccountName or userPrincipalName
Use sAMAccountName is you are a single domain, or userPrincipalName if
you have multiple domains in a single forest (and use the GC ldap port
[3268] instead of standard ldap)
Jason
On Friday, July 31, 2015 at 4:25:01 AM UTC+12, Aram Mirzadeh wrote:
Hi,
I'm trying to connect GrayLog2 to our local ActiveDirectory server and
having no luck.
I'm getting a :-| User Check and {check mark} Login Check when
configuring and testing the user of a user.
*Base DN is*: dc=company,dc=local
*Search Pattern is*: ((objectClass=person)(uid={0}))
I have also tried:
=organizationalPerson
=Users
=user
*Display Name attribute*: displayName
I'm 100% sure of the Base DN and displayName being valid, however the
search is an open issue.
Any thoughts?
--
You received this message because you are subscribed to the Google Groups
Graylog Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/5bc70058-745b-4bdc-8773-46e579ca76b0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: GrayLog 1.1.4 - ActiveDirectory connection
We have the following format and it works for us
Search Base DN: dc=xxx,dc=yyy
User Search Pattern: ((objectClass=user)(userPrincipalName={0}))
Display Name attribute: displayName
(so did you remember to put brackets around the filter?)
Jason
--
You received this message because you are subscribed to the Google Groups
Graylog Users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/0bccb5b5-f90a-47d5-80bc-d66e7c97893d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] TCP Syslog input channel restarting all the time?
Hi there I'm using syslog-ng to feed in data via a syslog/TCP channel and it's continually (every 10 seconds) dropping the TCP channel - forcing syslog-ng to restart it 2015-07-29T02:26:31+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection broken; fd='408', server='AF_INET(192.168.6.3:1514)', time_reopen='10' 2015-07-29T02:26:41+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection established; fd='465', server='AF_INET(192.168.6.3:1514)', local='AF_INET(0.0.0.0:0)' 2015-07-29T02:26:41+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection broken; fd='465', server='AF_INET(192.168.6.3:1514)', time_reopen='10' 2015-07-29T02:26:51+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection established; fd='379', server='AF_INET(192.168.6.3:1514)', local='AF_INET(0.0.0.0:0)' 2015-07-29T02:26:51+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection broken; fd='379', server='AF_INET(192.168.6.3:1514)', time_reopen='10' 2015-07-29T02:27:01+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection established; fd='476', server='AF_INET(192.168.6.3:1514)', local='AF_INET(0.0.0.0:0)' 2015-07-29T02:27:02+00:00 syslog.server syslog notice syslog-ng[30512]: Syslog connection broken; fd='476', server='AF_INET(192.168.6.3:1514)', time_reopen='10' tcpdump shows normal data flow followed by two TCP resets coming back from the graylog-1.1.5 server - so it's definitely graylog that's borking. BTW, this system *is working*: I'm seeing these syslogs flowing in - can do searches/etc - but I assume I'm losing some records due to this issue. I even created a xinetd.d based tcp service on the graylog server that just logged what it received to a file, configured the syslog server to send to both tcp channels - and it's running fine with no restarts (ie tcpdump of both ports only shows TCP resets on the graylog port not the xinetd port). So I think that implies it isn't the OS (CentOS-7) Whatever the root cause is should be logged somewhere - can someone point out to me how the debug this? Thanks Jason -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Does GELF over UDP support timestamp field?
I don't believe - how STUPID of me!!! Spot on - I was testing this by working on an apache access_log file I copied over on Saturday and by the time I stopped fiddling and started testing, it was days old. And then didn't even think to expand the search to multiple days. D'oh!!! Working fine now :-) -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] resource cost of adding fields to graylog
Hi there I've been adding extractors to our incoming syslog Input and it's been great - fantastic feature :-) However, I'm starting to get worried about the longer-term impact of me going all hell-for-leather on this: how much of a performance impact does adding new fields have? I've added about 10 new fields so far, and let's assume I'll add another 10-20 in the next 6 months, what would that really mean from an overhead perspective. Telling me to just accept the overhead is an acceptable answer ;-) Jason -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Does GELF over UDP support timestamp field?
Hi there
I'm trying to feed data from files in via GELF over UDP and hit a wall. The
following sample code works fine (note xtimestamp)
echo '{version: 1.1,host: example.org,xtimestamp:
1437290906.000,short_message: A3 short message that helps you identify
what is going on,full_message: Backtrace here\n\nmore stuff,level:
1,_user_id: 9001,_some_info: foo,_some_env_var: bar}' | nc -w 1
-u localhost 12201
...but if I change xtimestamp to timestamp - no record shows up in
graylog at all! I have tried 1437290906.000 as well as 1437290906 - and
put it in quotes and without. It seems to me like GELF rejects messages
that contain a timestamp?
That can't be right? Thanks
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Graylog web interface 1.1.4 - change default port
Just a FYI but if you wanted to expose graylog on the normal ports (port 80 (http) or port 443 (https)) then it doesn't work because graylog runs as an unprivileged user. I fake it via iptables trickery (Linux - I expect other OSes have the same feature) ie *nat -A PREROUTING -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 9001 That would make incoming tcp/443 connections get redirected at the IP layer to 9001 (where I have https running) Works great :-) -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] rewrite incoming syslog stream to fix borked content?
Hi there I have an incoming syslog stream that amongst working data also contains borked syslog records from Snare - which is a Windows EventLog to syslog service The problem is the application_id ends up as MSWinEventLogtab0tabSecurity and message: begins with Sun Jul 26 03:36:53 2015 4769 Microsoft-Windows-Security-Auditing hostname blah blah ie application_id should be Microsoft-Windows-Security-Auditing and message should begin with blah (ie the extra timestamp/etc should be thrown away). I have a tonne of these servers - which I don't control - so want to fix up this data as it enters into graylog. The extractor option looks like it doesn't allow you to rewrite some fields - such as message - so is there some other way of me achieving my goal? Thanks -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] how does graylog-collector detect change?
Hi there
I'm wanting to feed our (multiple) squid server logs into graylog and want
to simply rsync the logs into a staging directory on the server, and have
the collector pipe them in via the GELF connector. (ie I don't want them
put into syslog, nor do I want to install java on the proxies just so I can
run the collector)
For testing I'm running it in a shell with the output going to stdout.
inputs {
squid {
type = file
path-glob-root = /var/spool/squid-logs
path-glob-pattern = *access.log
}
}
I have a server1-access.log file in there, and if I echo squidline
server1-access.log it triggers graylog-collector nicely and I see the
GELF. However, rsync doesn't work like that: it creates a new file with a
temporary filename, copies the original file to that, appends the new data
and then renames it over the original file. End result is the file is
updated, but has a new inode. It appears graylog-collector doesn't notice
that change, nor the fact the file is now a different size? Also, I'm using
globbing, but if I rsync a totally new filename into that directory (eg 2nd
squid server), then nothing happens - it doesn't pick it up either. In
fact, if I create a new file with one line of data, nothing happens; but if
I then append a new line to that file, that is picked up???
Am I doing it wrong?
Jason
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] thanks, and hint for those dealing with AD forests (LDAP authentication)
Hi there I just learnt about graylog2 yesterday: what a revelation! The developers have done a WONDERFUL job. I have our central syslog server forwarding 1000+ syslog records/sec into a single virtual CentOS7 server and it's humming along. Obviously not much data in it yet, but so far I'm stunned :-) Anyway, I had a few gotchas during the process and I thought I'd feed them back First off, anyone wanting to do CentOS/Redhat based installs really needs to read this howto on ITzGeek - it really helped http://www.itzgeek.com/how-tos/linux/centos-how-tos/how-to-install-graylog2-on-centos-7-rhel-7.html Secondly, we have multiple AD forests, so people should be aware that is not an environment conducive to LDAP. LDAP is all about one LDAP - and forests can have many LDAP domains even within one forest. So my hint in this case is that you can get graylog to support a single forest by configuring LDAP to point at a top-forest level Global Catalog server (ldap port=3268 - ldaps=3269) - as that flattens all the child domains into one namespace - then all the users in that forest can log in, instead of just those in one domain As far as handling multiple forests - you can't. That would require graylog to support multiple LDAP connectors. I rather see graylog gain SAML support - as we've already been through the exercise of flattening all our forests for SAML - so I'd rather just reuse that ;-) Thanks again for all the hard work Jason -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] how about abstracting out authentication via reverse-proxy?
Hi there
Let me confess that I have an agenda of wanting graylog-web to support
SAML, but from my google-ing about I can also see others have asked for
Kerberos, Basic, etc authentication to be added to graylog-web.
That is actually sounding like a whole lot of work... I was wondering if an
alternative would be to make it somebody else's problem. ie put a web
server (like apache) in front of graylog-web, and configure it with any of
the thousand authentication systems it supports - and configure apache to
reflect that user metadata via HTTP headers - so that graylog-web can
just use that instead
eg the following pseudo-code would make apache support SAML (via
mod_auth_mellon) and push some of the SAML-gained account data to
graylog-web via HTTP headers
LocationMatch ^/
AuthType Mellon
Require valid-user
MellonUser uid
MellonSetEnv e-mail mail
RequestHeader set X-Apache-UID %{MELLON_uid}e
RequestHeader set X-Apache-Email %{MELLON_e-mail}e
RequestHeader set X-Apache-Name %{MELLON_cn}e
ProxyPass http://127.0.0.1:9000/
ProxyPassReverse http://127.0.0.1:9000/
/LocationMatch
Then all that would remain to do would be to secure graylog-web by running
it only on 127.0.0.1:9000 and bringing up apache over HTTPS on 443. And the
only new code for graylog-web would be that it could be configured in
reverse proxy mode and to map arbitrary headers to authentication details
it allows to be set (hence my choice of username/email/name, as the current
LDAP implementation uses them too)
Just an idea.. :-)
Jason
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
