On Wed 04 Jun 2014 03:13:41 AM PDT, Victor Castell wrote:
Good to know, thank you.
I'm using the "payload" format for ES output plugin, how this change
will affect me?
Good question. Every record that is sent to ElasticSearch needs to
contain some BulkAPI JSON that precedes the document JSON. This tells
ElasticSearch how the document is to be indexed, and typically looks
something like this:
{"index": {"_index": "YOUR_INDEX", "_type": "DOCUMENT_TYPE"}}
It is the encoder's job to provide this. So to replicate the v0.5.2
payload format, you'd create a very simple SandboxEncoder that
generates this JSON and spits it out w/ the message payload immediately
following. We've got a ticket open
(https://github.com/mozilla-services/heka/issues/875) that I plan on
working on today that will help w/ this. Once that's done, I'll send
another message out w/ an example 0.6 config replacing the use of
payload format.
-r
On Wed, Jun 4, 2014 at 2:18 AM, Rob Miller <[email protected]
<mailto:[email protected]>> wrote:
Hi,
Just want to let everyone know that I've been working on
converting the ElasticSearchOutput from using its custom 'format'
configuration setting to instead using the new encoder plugin
infrastructure that will be in the upcoming 0.6 release.
In v0.5.2, the two most (only?) used formats were "clean"
(generating a nicely formatted JSON representation of each
message) and "logstash_v0" (generating a JSON representation
matching the original "v0" schema used by Logstash). In 0.6, there
will be an ESJsonEncoder and an ESLogstashV0Encoder which generate
the same output, delta a few bug fixes.
In the process, the config options for the ElasticSearchOutput
itself were cleaned up a bit.
To give you an idea about the changes to expect, a v0.5.2 config
section like this:
[ElasticSearchOutput]
message_matcher = "Type == 'nginx.access'"
index = "%{Type}-%{2006.01.02}"
type_name = "%{Type}"
server = "http://es-server:9200";
format = "clean"
esindexfromtimestamp = true
flush_interval = 5000
flush_count = 10
Would in v0.6 look like this:
[ESJsonEncoder]
index = "%{Type}-%{2006.01.02}"
es_index_from_timestamp = true
type_name = "%{Type}"
[ElasticSearchOutput]
message_matcher = "Type == 'nginx.access'"
encoder = "ESJsonEncoder"
server = "http://es-server:9200";
flush_interval = 5000
flush_count = 10
This change will be merged to dev momentarily. As always, feel
free to ask questions on this mailing list or in one of our #heka
IRC channels (on irc.freenode.net <http://irc.freenode.net> or
irc.mozilla.org <http://irc.mozilla.org>) if you have any issues.
Thanks for Heka-ing!
-r
_________________________________________________
Heka mailing list
[email protected] <mailto:[email protected]>
https://mail.mozilla.org/__listinfo/heka
<https://mail.mozilla.org/listinfo/heka>
--
V
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
