Hi Roberto,
About your osseccall you wrote this in the mail
But the file "template =>" /etc/logstash/elastic-ossec-template2.json
"I modified the lines 3 and 8.
Line 3: from "template", "ossec *" to "template", "ossecall *"
Line 8: from "ossec": to "ossecall":
You have an space between ossec, ossecall and the wildcard?, if you have,
you should not. And with the curl procedure:
$ Cd ~ / ossec_tmp / ossec-wazuh / extensions / ElasticSearch / &&
curl -XPUT "http: // localhost: 9200 / _template / ossec /" -d "@
elastic-ossec-template.json"
You need to apply the templates for both index.
For your last question, in this mail you have a bash script to reindex the
index. Please use carefully and check with curl
'localhost:9200/_cat/indices?v' after every step that the script is doing
well.
This script has 4 steps:
1. We move all index without mapping applied to a backup index, we do
that with the option reindex to apply the new template.
2. After the reindex is has finished we can delete the old index.
3. Now we can move the backup index to the original name.
4. When the step 3 has finished we can delete the backup index.
Pleas take a look the lines 72, 73 and 76, 77 in order to change the index
name from ossec-$index_elastic_name and ossec-$index_elastic_name by
ossecall-$index_elastic_name and ossecall-$index_elastic_name because
probably you need to run this script for your two index.
This one of a few utils that wazuh will release soon.
#!/bin/bash
# Copyright (C) 2015-2016 Wazuh, Inc.All rights reserved.
# Wazuh.com
#
# This program is a free software; you can redistribute it
# and/or modify it under the terms of the GNU General Public
# License (version 2) as published by the FSF - Free Software
# Foundation.
# Elasticsearch Reindexing
# Requires:
# Elasticsearch 2.3 or superior
if [ $# -ne 4 ]
then
echo "Usage: ./wazuh_elastic_reindex_index.sh date_from date_to
elasticsearch_ip step"
echo -e "\tDate format: YYYY-MM-DD"
echo -e "\tStep: 1|2|3|4"
echo -e "\tExample: ./wazuh_elastic_reindex_index.sh 20160826
20160901 10.0.0.20 1"
echo -e "\tNote: Each step takes its time to perform the actions
required. Review: tail -f /var/log/elasticsearch/ossec.log"
exit 0
fi
## Arguments
FROM=$1
TO=$2
ELASTIC_IP=$3
STEP=$4
## Main
startdate=$(date -d $FROM +"%Y%m%d")
enddate=$(date -d $TO +"%Y%m%d")
if [ $startdate -ge $enddate ];
then
echo "The date_from $startdate is bigger than date_to $enddate,
please review this arguments";
exit 1
fi
startdate=$(date -I -d "$FROM") || exit -1
enddate=$(date -I -d "$TO") || exit -1
echo -e "\n### Start reindexing [STEP $STEP], from $startdate to
$enddate are you sure? please confirm with YES/NO?"
read ADDRANSWER
exist_index () {
request="$ELASTIC_IP:9200/$1"
exist=`curl -s -XHEAD -i $request | head -n 1 | cut -d' ' -f2`
}
reindex () {
request="$ELASTIC_IP:9200/_reindex"
request_body='{ "source": { "index": "'"$1"'" }, "dest": {
"index": "'"$2"'" }}'
curl_result=`curl -s -XPOST $request -d "$request_body"`
echo $curl_result
}
delete_index () {
request="$ELASTIC_IP:9200/$1"
curl_result=`curl -s -XDELETE $request`
echo $curl_result
}
if [ $ADDRANSWER == 'YES' ]
then
d="$FROM"
while [ "$d" != "$enddate" ]; do
index_elastic_name=` echo $d | sed 's/-/\./g'`
if [ $STEP == '1' ] || [ $STEP == '2' ]; then
src_index="ossec-$index_elastic_name"
dst_index="ossec-$index_elastic_name-b"
exist_index $src_index
elif [ $STEP == '3' ] || [ $STEP == '4' ]; then
src_index="ossec-$index_elastic_name-b"
dst_index="ossec-$index_elastic_name"
exist_index $src_index
else
echo "Bad argument: step: $STEP"
exit 1
fi
if [ $exist != '404' ]; then
if [ $STEP == '1' ]; then
echo "### 1. Reindexing: $src_index -> $dst_index"
reindex $src_index $dst_index
elif [ $STEP == '2' ]; then
echo "### 2. Deleting old index: $src_index"
delete_index $src_index
elif [ $STEP == '3' ]; then
echo "### 3. Reindexing: $src_index"
reindex $src_index $dst_index
elif [ $STEP == '4' ]; then
echo "### 4. Deleting intemediate index: $src_index"
delete_index $src_index
fi
else
echo "### Index $src_index doest not exist. Skipping."
fi
# Update date.
d=$(date -I -d "$d + 1 day")
done
echo -e "\nPlease check 'curl -XGET
${ELASTIC_IP}:9200/_cat/indices' to re-check the indices"
echo "Reindexing ended [STEP $STEP]."
else
echo "This script is finished because you don't confirm with YES"
fi
i hope this helps.
Regards
-----------------------
Jose Luis Ruiz
Wazuh Inc.
[email protected]
On September 29, 2016 at 7:25:09 AM,
[email protected] (
[email protected]) wrote:
Hi Jose, thanks for reply!
Indeed, today the index is in template format. But only ossec index, the
index ossecall did not work, the fields still appear as "Analyzed Field".
I did not do the procedure:
$ Cd ~ / ossec_tmp / ossec-wazuh / extensions / ElasticSearch / && curl
-XPUT "http: // localhost: 9200 / _template / ossec /" -d "@ elastic-ossec-
template.json"
Just put the logstash output that I said.
But the file "template =>" /etc/logstash/elastic-ossec-template*2*.json "I
modified the lines 3 and 8.
Line 3: *from* "template", "ossec *" *to* "template", "ossecall *"
Line 8: *from* "ossec": *to* "ossecall":
I do not know if it was really necessary to do this. I did this because I
decided to create a separate index for logs archives.json file. Where ossec are
logging all.
About "After that, probably you will need to reindex all your index to
apply the new template."
Do you have any procedure to do this?
Em quarta-feira, 28 de setembro de 2016 18:01:12 UTC-3, jose escreveu:
>
> Hi Roberto,
>
> Have you applied the custom mapping?
>
> http://documentation.wazuh.com/en/latest/ossec_elk_
> elasticsearch.html#ossec-alerts-template
>
> If you have the custom mapping applied, and the template in Logstash, you
> need to wait until next day, when the next index is created with the new
> mapping and template.
>
> After that, probably you will need to reindex all your index to apply the
> new template.
>
>
> Regards
> -----------------------
> Jose Luis Ruiz
> Wazuh Inc.
> [email protected] <javascript:>
>
> On September 28, 2016 at 3:26:38 PM, [email protected]
> <javascript:> ([email protected] <javascript:>) wrote:
>
> Hi Pedro!
>
> I am using the ossec wazuh, I have a question about indexes.
> I had implemented the logstash without using the file "elastic-ossec-
> template.json". But I saw it would be good to use it. I am wanting use
> some indexes and Kibana shows "Analyzed Field", like "AgentName".
>
> I put the template in the configuration of logstash and the index has not
> changed to "not analized".
>
>
> My logstash output :
>
> output {
>
> #for archives.json log
> if [type] == "ossecall" {
> elasticsearch {
> hosts => "127.0.0.1:9200"
> index => "ossecall-%{+YYYY.MM.dd}"
> document_type => "ossecall"
> template => "/etc/logstash/elastic-ossec-template2.json"
> template_name => "ossecall"
> template_overwrite => true
> }
> }
> #for alerts.json log
> else {
> elasticsearch {
> hosts => "127.0.0.1:9200"
> index => "ossec-%{+YYYY.MM.dd}"
> document_type => "ossec"
> template => "/etc/logstash/elastic-ossec-template.json"
> template_name => "ossec"
> template_overwrite => true
> }
> }
> }
>
> Can you help me?
>
>
>
> Em quinta-feira, 2 de junho de 2016 08:25:09 UTC-3, Pedro S escreveu:
>>
>> Hi Maxim,
>>
>> How are you forwarding the alerts/archives to Kibana?
>>
>> I think you will need the archives JSON output setting, if you are using
>> Wazuh <http://wazuh.com/>, edit *ossec.conf* and add the following
>> setting:
>>
>> <global>
>>> *<logall_json>yes</logall_json>*
>>> </global>
>>
>>
>>
>> Once you do it, you will find new archives.json events files at:
>>
>> /var/ossec/logs/archives/archives.json
>>
>>
>>
>> The next step is forward these archives events to Elasticsearch, in order
>> to do it we need to edit Logstash configuration.
>>
>> My personal advice to index archives events is to create a dedicated
>> index pattern just for them, so you will be able to distinguish between
>> events and alerts, adding inside "output" section the following
>> configuration:
>>
>> output {
>> if [type] == "ossec-alerts" {
>> elasticsearch {
>> hosts => ["127.0.0.1:9200"]
>> index => "ossec-%{+YYYY.MM.dd}"
>> document_type => "ossec"
>> template => "/etc/logstash/elastic-ossec-template.json"
>> template_name => "ossec"
>> template_overwrite => true
>> }
>> }
>> if [type] == "ossec-archives" {
>> elasticsearch {
>> hosts => ["127.0.0.1:9200"]
>> index => "ossec-archives-%{+YYYY.MM.dd}"
>> document_type => "ossec"
>> template => "/etc/logstash/elastic-ossec-template.json"
>> template_name => "ossec"
>> template_overwrite => true
>> }
>> }
>> }
>>
>>
>> Later in Kibana you will need to create a new index pattern
>> (Settings->indices) matching for "ossec-archives-*".
>>
>> If you need to "reindex" or read the a log file from the beginning using
>> Logstash, you can use the file input with option *start_position* set to
>> *beginning* (+ info)
>> <https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-start_position>
>>
>>
>>
>> On Monday, May 30, 2016 at 4:53:10 PM UTC+2, Maxim Surdu wrote:
>>>
>>> i have this archives files with logs but in kibana i can not see them
>>> can i reindex this files?
>>> if i can, please help me step by step
>>>
>>> joi, 19 mai 2016, 10:17:51 UTC+3, Maxim Surdu a scris:
>>>>
>>>> Hi dear community,
>>>>
>>>> i had a problem with logstash, after i resolve it i saw what in kibana
>>>> are missing logs, how can i resolve the problem and reindexing all my logs
>>>> to kibana
>>>> I will be thankful if someone will help me step by step
>>>>
>>>>
>>>> i appreciate your help, and a lot of respect for developers and
>>>> community!
>>>>
>>> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "ossec-list" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected] <javascript:>.
> For more options, visit https://groups.google.com/d/optout.
>
> --
---
You received this message because you are subscribed to the Google Groups
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [email protected].
For more options, visit https://groups.google.com/d/optout.
--
---
You received this message because you are subscribed to the Google Groups
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
