[graylog2] Re: Graylog2 setup, how to send data now?

2015-09-28 Thread Mark Estridge 
Typically no changes are made for Cisco ASA Firewall changes as long as 
they are sent in Emblem format.
Copy/paste the following into your EXTRACTOR to work with the 
routers/switches.  I've setup mine as follows:
logging origin-id hostname 
logging host "ip add" vrf "skip if unneeded" transport udp port 10514

*start*
{
  "extractors": [
{
  "condition_type": "none",
  "condition_value": "",
  "converters": [
{
  "config": {},
  "type": "syslog_pri_facility"
}
  ],
  "cursor_strategy": "copy",
  "extractor_config": {
"regex_value": "^<(\\d.+)>"
  },
  "extractor_type": "regex",
  "order": 0,
  "source_field": "message",
  "target_field": "facility",
  "title": "Facility"
},
{
  "condition_type": "none",
  "condition_value": "",
  "converters": [
{
  "config": {},
  "type": "syslog_pri_level"
}
  ],
  "cursor_strategy": "copy",
  "extractor_config": {
"regex_value": "^<(\\d.+)>"
  },
  "extractor_type": "regex",
  "order": 1,
  "source_field": "message",
  "target_field": "level",
  "title": "Level"
},
{
  "condition_type": "none",
  "condition_value": "",
  "converters": [
{
  "config": {},
  "type": "flexdate"
}
  ],
  "cursor_strategy": "copy",
  "extractor_config": {
"regex_value": ">:\\s.+:\\s(.+?):\\s%"
  },
  "extractor_type": "regex",
  "order": 3,
  "source_field": "message",
  "target_field": "timestamp",
  "title": "Timestamp"
},
{
  "condition_type": "none",
  "condition_value": "",
  "converters": [
{
  "config": {},
  "type": "lowercase"
}
  ],
  "cursor_strategy": "copy",
  "extractor_config": {
"regex_value": "%(.+?)-"
  },
  "extractor_type": "regex",
  "order": 4,
  "source_field": "message",
  "target_field": "local_facility",
  "title": "Local facility"
},
{
  "condition_type": "none",
  "condition_value": "",
  "converters": [
{
  "config": {},
  "type": "numeric"
}
  ],
  "cursor_strategy": "copy",
  "extractor_config": {
"regex_value": "%.+-(\\d?)-"
  },
  "extractor_type": "regex",
  "order": 5,
  "source_field": "message",
  "target_field": "local_level",
  "title": "Local level"
},
{
  "condition_type": "none",
  "condition_value": "",
  "converters": [],
  "cursor_strategy": "copy",
  "extractor_config": {
"regex_value": "%.+-\\d+-.+: (.*)$"
  },
  "extractor_type": "regex",
  "order": 7,
  "source_field": "message",
  "target_field": "message",
  "title": "Message"
},
{
  "condition_type": "none",
  "condition_value": "",
  "converters": [
{
  "config": {},
  "type": "lowercase"
}
  ],
  "cursor_strategy": "copy",
  "extractor_config": {
"regex_value": "%.+-\\d-(.+?):"
  },
  "extractor_type": "regex",
  "order": 6,
  "source_field": "message",
  "target_field": "mnemonic",
  "title": "Mnemonic"
},
{
  "condition_type": "none",
  "condition_value": "",
  "converters": [],
  "cursor_strategy": "copy",
  "extractor_config": {
"index": 2,
"split_by": ":"
  },
  "extractor_type": "split_and_index",
  "order": 0,
  "source_field": "full_message",
  "target_field": "source",
  "title": "Source_Cisco_Catalyst"
}
  ],
  "version": "1.2.1 (c301e97)"
}
***end**

On Sunday, September 27, 2015 at 6:57:10 AM UTC-5, Jochen Schalanda wrote:
>
> Hi Anthony,
>
> you can create a Syslog UDP or Syslog TCP input for Graylog in the web 
> interface at System -> Inputs. Also see 
> http://docs.graylog.org/en/1.2/pages/sending_data.html#syslog for a 
> description how to configure different syslog daemons to work smoothly with 
> Graylog.
>
> As for Cisco devices (or generally networking appliances), they sometimes 
> pretend to support syslog but actually don't emit any standardized format. 
> In this case, you'll need to create a Raw/Plaintext UDP/TCP input and 
> extract the required information via some extractors. Also check the 
> Graylog Marketplace for some existing content packs for Cisco devices: 
> https://marketplace.graylog.org/addons?search=cisco
>
>
> Cheers,
> Jochen
>
> On Friday, 25 September 2015 23:22:14 UTC+2, Anthony Srdar wrote:
>>
>> I follow this guide:
>>
>>
>> http://www.itzgeek.com/how-tos/linux/centos-how-tos/how-to-install-graylog2-on-centos-7-rhel-7.html
>>
>> I have graylog up and running, but how do I send my cisco ASA data to it 
>> to start logging? How do I create a syslog listener? 
>>
>

-- 
You received this message because you are subscribed

Re: [graylog2] Graylog inputs stopped yet still receiving syslogs

2015-09-28 Thread Mark Estridge 
In all honesty, I didn't think to check the message journal; however, the 
alerts that were coming in were (at the time) current date/time.  If the 
journal were playing catch up, then there would be a delta in the time.

On Friday, September 25, 2015 at 7:33:40 PM UTC-5, lennart wrote:
>
> Could it be that you have a message journal that is full of messages 
> that Graylog keeps processing? You can see the journal size in the 
> nodes overview and node details pages. It should be at 0. 
>
> On Thu, Sep 24, 2015 at 7:41 PM, Mark Estridge  > wrote: 
> > Graylog 1.2.1 setup and all inputs are stopped, yet I am continuing to 
> see 
> > current syslogs with a global search.  It is as if the STOP feature 
> doesn't 
> > work.  System Overview indicates that there are no running inputs...yet 
> I'm 
> > receiving on the order of ~12K messages per minute. 
> > 
> > Anyone else noting this behavior. 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "Graylog Users" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to graylog2+u...@googlegroups.com . 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/graylog2/ba8dffc3-99b0-4957-9f1f-f5454394dea6%40googlegroups.com.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/7cba0f13-d02e-4e94-b6bd-703d4d19a069%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog.-server service doesn't start after remove /var/lib/graylog-server/journal/* files

2015-09-28 Thread Jochen Schalanda 
Hi Roberto,

your Elasticsearch cluster health status is RED, which means that ES can't 
index documents anymore. Check the logs of your Elasticsearch nodes for the 
reason.

Additionally I'd recommend upgrading to Graylog 1.2.1, which also starts 
when Elasticsearch is unhealthy or not accessible.


Cheers,
Jochen

On Monday, 28 September 2015 17:21:59 UTC+2, roberto...@gmail.com wrote:
>
> Dear, I have Graylog 1.1 and today I have to remove all the files under 
> /var/lib/graylog-server/journal/.
>
> I remove all the files without stopping any service (elasticsearch, 
> graylog-web and graylog-server).
>
> After that, I reboot the server but the graylog-server doesn't start at 
> all, and I can see this error log message.can you help please ??? 
> Thanks a lot.
>
> 2015-09-28T12:00:42.414-03:00 INFO  [CmdLineTool] Loaded plugins: 
> [Anonymous Usage Statistics 1.0.5 
> [org.graylog.plugins.usagestatistics.UsageStatsPlugin]]
> 2015-09-28T12:00:42.469-03:00 INFO  [MongoDbConfiguration] You're using 
> deprecated configuration options for MongoDB. Please use mongodb_uri.
> 2015-09-28T12:00:42.504-03:00 INFO  [MongoDbConfiguration] Suggested value 
> for mongodb_uri = mongodb://graylog2:GrayPnet@127.0.0.1:27017/graylog2
> 2015-09-28T12:00:42.533-03:00 INFO  [CmdLineTool] Running with JVM 
> arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:PermSize=128m 
> -XX:MaxPermSize=256m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC 
> -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC 
> -XX:-OmitStackTraceInFastThrow 
> -Dlog4j.configuration=file:///etc/graylog/server/log4j.xml 
> -Djava.library.path=/usr/share/graylog-server/lib/sigar
> 2015-09-28T12:00:45.834-03:00 INFO  [InputBufferImpl] Message journal is 
> enabled.
> 2015-09-28T12:00:46.083-03:00 INFO  [LogManager] Found clean shutdown 
> file. Skipping recovery for all logs in data directory 
> '/var/lib/graylog-server/journal'
> 2015-09-28T12:00:46.084-03:00 INFO  [LogManager] Loading log 
> 'messagejournal-0'
> 2015-09-28T12:00:46.113-03:00 INFO  [Log] Completed load of log 
> messagejournal-0 with log end offset 0
> 2015-09-28T12:00:46.125-03:00 INFO  [KafkaJournal] Initialized Kafka based 
> journal at /var/lib/graylog-server/journal
> 2015-09-28T12:00:46.138-03:00 INFO  [InputBufferImpl] Initialized 
> InputBufferImpl with ring size <65536> and wait strategy 
> , running 2 parallel message handlers.
> 2015-09-28T12:00:46.318-03:00 INFO  [NodeId] Node ID: 
> b7b62947-250e-473b-b8df-7083d6df9886
> 2015-09-28T12:00:46.486-03:00 INFO  [node] [graylog2-server] 
> version[1.5.2], pid[3720], build[62ff986/2015-04-27T09:21:06Z]
> 2015-09-28T12:00:46.487-03:00 INFO  [node] [graylog2-server] initializing 
> ...
> 2015-09-28T12:00:46.496-03:00 INFO  [plugins] [graylog2-server] loaded 
> [graylog2-monitor], sites []
> 2015-09-28T12:00:48.786-03:00 INFO  [node] [graylog2-server] initialized
> 2015-09-28T12:00:48.796-03:00 INFO  [ProcessBuffer] Initialized 
> ProcessBuffer with ring size <65536> and wait strategy 
> .
> 2015-09-28T12:00:50.543-03:00 INFO  [RulesEngineProvider] No static rules 
> file loaded.
> 2015-09-28T12:00:50.741-03:00 INFO  [OutputBuffer] Initialized 
> OutputBuffer with ring size <65536> and wait strategy 
> .
> 2015-09-28T12:00:51.221-03:00 INFO  [Version] HV01: Hibernate 
> Validator 5.1.3.Final
> 2015-09-28T12:00:51.630-03:00 INFO  [ServerBootstrap] Graylog server 1.1.1 
> (893e8e7) starting up. (JRE: Oracle Corporation 1.7.0_79 on Linux 
> 3.2.0-4-amd64)
> 2015-09-28T12:00:51.645-03:00 INFO  [PeriodicalsService] Starting 21 
> periodicals ...
> 2015-09-28T12:00:51.729-03:00 INFO  [Periodicals] Starting 
> [org.graylog2.periodical.ThroughputCounterManagerThread] periodical in 
> [0s], polling every [1s].
> 2015-09-28T12:00:51.711-03:00 INFO  [node] [graylog2-server] starting ...
> 2015-09-28T12:00:51.780-03:00 INFO  [Periodicals] Starting 
> [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling 
> every [1s].
> 2015-09-28T12:00:51.787-03:00 INFO  [Periodicals] Starting 
> [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling 
> every [60s].
> 2015-09-28T12:00:51.793-03:00 INFO  [Periodicals] Starting 
> [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical 
> in [0s], polling every [1s].
> 2015-09-28T12:00:51.796-03:00 INFO  [Periodicals] Starting 
> [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s], 
> polling every [20s].
> 2015-09-28T12:00:51.806-03:00 INFO  [Periodicals] Starting 
> [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running 
> forever.
> 2015-09-28T12:00:51.817-03:00 INFO  [Periodicals] Starting 
> [org.graylog2.periodical.DeadLetterThread] periodical, running forever.
> 2015-09-28T12:00:51.818-03:00 INFO  [Periodicals] Starting 
> [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, 
> running forever.
> 2015-09-28T12:00:51.825-03:00 INFO  [Periodicals] Starting 
>

[graylog2] Graylog.-server service doesn't start after remove /var/lib/graylog-server/journal/* files

2015-09-28 Thread robertocarna36 
Dear, I have Graylog 1.1 and today I have to remove all the files under 
/var/lib/graylog-server/journal/.

I remove all the files without stopping any service (elasticsearch, 
graylog-web and graylog-server).

After that, I reboot the server but the graylog-server doesn't start at 
all, and I can see this error log message.can you help please ??? 
Thanks a lot.

2015-09-28T12:00:42.414-03:00 INFO  [CmdLineTool] Loaded plugins: 
[Anonymous Usage Statistics 1.0.5 
[org.graylog.plugins.usagestatistics.UsageStatsPlugin]]
2015-09-28T12:00:42.469-03:00 INFO  [MongoDbConfiguration] You're using 
deprecated configuration options for MongoDB. Please use mongodb_uri.
2015-09-28T12:00:42.504-03:00 INFO  [MongoDbConfiguration] Suggested value 
for mongodb_uri = mongodb://graylog2:GrayPnet@127.0.0.1:27017/graylog2
2015-09-28T12:00:42.533-03:00 INFO  [CmdLineTool] Running with JVM 
arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:PermSize=128m 
-XX:MaxPermSize=256m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC 
-XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC 
-XX:-OmitStackTraceInFastThrow 
-Dlog4j.configuration=file:///etc/graylog/server/log4j.xml 
-Djava.library.path=/usr/share/graylog-server/lib/sigar
2015-09-28T12:00:45.834-03:00 INFO  [InputBufferImpl] Message journal is 
enabled.
2015-09-28T12:00:46.083-03:00 INFO  [LogManager] Found clean shutdown file. 
Skipping recovery for all logs in data directory 
'/var/lib/graylog-server/journal'
2015-09-28T12:00:46.084-03:00 INFO  [LogManager] Loading log 
'messagejournal-0'
2015-09-28T12:00:46.113-03:00 INFO  [Log] Completed load of log 
messagejournal-0 with log end offset 0
2015-09-28T12:00:46.125-03:00 INFO  [KafkaJournal] Initialized Kafka based 
journal at /var/lib/graylog-server/journal
2015-09-28T12:00:46.138-03:00 INFO  [InputBufferImpl] Initialized 
InputBufferImpl with ring size <65536> and wait strategy 
, running 2 parallel message handlers.
2015-09-28T12:00:46.318-03:00 INFO  [NodeId] Node ID: 
b7b62947-250e-473b-b8df-7083d6df9886
2015-09-28T12:00:46.486-03:00 INFO  [node] [graylog2-server] 
version[1.5.2], pid[3720], build[62ff986/2015-04-27T09:21:06Z]
2015-09-28T12:00:46.487-03:00 INFO  [node] [graylog2-server] initializing 
...
2015-09-28T12:00:46.496-03:00 INFO  [plugins] [graylog2-server] loaded 
[graylog2-monitor], sites []
2015-09-28T12:00:48.786-03:00 INFO  [node] [graylog2-server] initialized
2015-09-28T12:00:48.796-03:00 INFO  [ProcessBuffer] Initialized 
ProcessBuffer with ring size <65536> and wait strategy 
.
2015-09-28T12:00:50.543-03:00 INFO  [RulesEngineProvider] No static rules 
file loaded.
2015-09-28T12:00:50.741-03:00 INFO  [OutputBuffer] Initialized OutputBuffer 
with ring size <65536> and wait strategy .
2015-09-28T12:00:51.221-03:00 INFO  [Version] HV01: Hibernate Validator 
5.1.3.Final
2015-09-28T12:00:51.630-03:00 INFO  [ServerBootstrap] Graylog server 1.1.1 
(893e8e7) starting up. (JRE: Oracle Corporation 1.7.0_79 on Linux 
3.2.0-4-amd64)
2015-09-28T12:00:51.645-03:00 INFO  [PeriodicalsService] Starting 21 
periodicals ...
2015-09-28T12:00:51.729-03:00 INFO  [Periodicals] Starting 
[org.graylog2.periodical.ThroughputCounterManagerThread] periodical in 
[0s], polling every [1s].
2015-09-28T12:00:51.711-03:00 INFO  [node] [graylog2-server] starting ...
2015-09-28T12:00:51.780-03:00 INFO  [Periodicals] Starting 
[org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling 
every [1s].
2015-09-28T12:00:51.787-03:00 INFO  [Periodicals] Starting 
[org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling 
every [60s].
2015-09-28T12:00:51.793-03:00 INFO  [Periodicals] Starting 
[org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical 
in [0s], polling every [1s].
2015-09-28T12:00:51.796-03:00 INFO  [Periodicals] Starting 
[org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s], 
polling every [20s].
2015-09-28T12:00:51.806-03:00 INFO  [Periodicals] Starting 
[org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running 
forever.
2015-09-28T12:00:51.817-03:00 INFO  [Periodicals] Starting 
[org.graylog2.periodical.DeadLetterThread] periodical, running forever.
2015-09-28T12:00:51.818-03:00 INFO  [Periodicals] Starting 
[org.graylog2.periodical.GarbageCollectionWarningThread] periodical, 
running forever.
2015-09-28T12:00:51.825-03:00 INFO  [Periodicals] Starting 
[org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], 
polling every [30s].
2015-09-28T12:00:51.845-03:00 INFO  [Periodicals] Starting 
[org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling 
every [300s].
2015-09-28T12:00:51.846-03:00 INFO  [IndexRetentionThread] Elasticsearch 
cluster not available, skipping index retention checks.
2015-09-28T12:00:51.846-03:00 INFO  [Periodicals] Starting 
[org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling 
every [10s].
2015-09-28T12:00:51.848-03:00 INFO  [Periodicals] Starting

[graylog2] Disk at 100% (linux newbie)

2015-09-28 Thread Drew Miranda 
This is actually something I'm somewhat wondering myself. My instinct says to 
use something like logrotate but I haven't tested. So far what I did that is 
working is to put the log on a different volume so it can't fill up the volume 
with the graylog message journal.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/f0ce6885-d4d1-4e38-b9f6-092917c10a8d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.