Re: [heka] Uploading To Elasticsearch a non-analysed field

[Abhiman]

Thanks. What would the mapping be, if I want to make all strings non-analysed.


Regards,

-

Abhiman | Analyst

M: +91 8004274160 | www.tcg-digital.com


From: Xavier Lange <xrla...@tureus.com>
Sent: 21 April 2016 12:17:07
To: Abhiman
Cc: heka@mozilla.org
Subject: Re: [heka] Uploading To Elasticsearch a non-analysed field

Heka is not responsible for mappings in Elasticsearch. Heka doesn't create the 
mappings, that's a property of the dynamic schema in Elasticsearch. I highly 
recommend using an Elasticsearch index template to solve your problem. Here's a 
simple example:

curl -XPOST "$ES_URL/_template/event" -d '
{
  "order": 0,
  "template": "event-*",
  "settings": {
"index.number_of_shards": "5"
  },
  "mappings": {
"_default_": {
  "properties": {
"nameOfMyUnanalyzedProperty": {
  "type": "string",
  "index": "not_analyzed"
}
  }
}
  },
  "aliases": {}
}
'

You can find more information here: 
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
 . Note, you will need to drop your existing indexes because their schema is 
already determine to be analyzed.

Here's a more complicated example... something I wish I was given a while ago! 
This example disables analysis on all strings fields but turns on geo_point for 
one named property.

curl -XPOST "$ES_URL/_template/event" -d '
{
  "order": 0,
  "template": "event-*",
  "settings": {
"index.number_of_shards": "1"
  },
  "mappings": {
"_default_": {
  "dynamic_templates": [
{
  "strings": {
"match_mapping_type": "string",
"mapping": {
  "index": "not_analyzed",
      "type": "string"
}
  }
}
  ],
  "properties": {
"abcGeoCoords": {
  "type": "geo_point",
  "geohash_prefix": true,
  "geohash_precision": "1m"
}
  }
}
  },
  "aliases": {}
}
'

Hope this helps.

Xavier


On Apr 20, 2016, at 11:36 PM, Abhiman Talwar 
<abhi...@tcg-digital.com<mailto:abhi...@tcg-digital.com>> wrote:

I want to upload data to ES-server using HEKA. Everything was going fine till I
realized Heka uploads every field as 'analysed' and I want one field to be non-
analysed. How can I do that ?

Thanks

___
Heka mailing list
Heka@mozilla.org<mailto:Heka@mozilla.org>
https://mail.mozilla.org/listinfo/heka

___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

[heka] Uploading To Elasticsearch a non-analysed field

[Abhiman Talwar]

I want to upload data to ES-server using HEKA. Everything was going fine till I 
realized Heka uploads every field as 'analysed' and I want one field to be non-
analysed. How can I do that ?

Thanks

___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] multiple file upload using HEKA

[Abhiman]

> If I understand what you're saying, you're using LogstreamerInput 
> incorrectly. A single LogstreamerInput is meant to map to either a 
> single log stream, or possibly multiple logstreams of identical 
> structure, such as a set of access logs for a number of different 
domains.
> 
> You seem to be trying to use a single LogstreamerInput to track a 
bunch 
> of conceptually unrelated log files, which won't work. You'll want to 
> use a separate LogstreamerInput for each file.
> 
> -r
> 

Thaks Rob, I tried with separate LogstreamerInput for each file and it 
is working. But i have 50 files and it comes a tedious job to do it for 
all. Is there any short way to do the same i.e upload 50 log files 
without having to write the LogstreamerInput for each one.

Thanks




___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

[heka] multiple file upload using HEKA

[Abhiman]

I want to upload multiple files(which are non sequential) simultaneously using 
Heka. The files are conn.log, http.log, dhcp.log, dnp3.log, ftp.log, dns.log. I 
am trying with the following code

[networklogs]
Type = "LogstreamerInput"
log_directory = "/opt/bro/logs/current"
file_match= '(?P\d+)\.log'
priority=["Year"]
decoder="Json"

[networklogs.translation.Year]
conn = 1
dhcp = 2
dnp3 = 3
dns = 4
ftp = 5
http = 6

[Json]
type = "SandboxDecoder"
filename = "lua_decoders/json.lua"

[Json.config]
type = "raw.bro"
debug = false


[ESJsonEncoder]
index = "test_2"
type_name = "one"


[ElasticSearchOutput]
message_matcher = "TRUE"
server = "http://localhost:9200;
encoder = "ESJsonEncoder"


but it's not working. When I am trying with only one file, I am succeeding but 
for multiple files, no luck.

Regards,

-

Abhiman | Analyst

M: +91 8004274160 | www.tcg-digital.com
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka