Rob, Thank you, that is very helpful. I've enjoyed the talks you've given that have found their way onto the internet. Steve
-----Original Message----- From: Rob Miller [mailto:[email protected]] Sent: Tuesday, June 30, 2015 2:57 PM To: Lewis, Steven Cc: [email protected] Subject: Re: [heka] Question, going outside message schema on output, possible with lua? Encoders don't emit message structs, they emit byte streams. If a Lua encoder uses the `inject_message` API call then it will emit the protobuf encoding for the full message, but if the `inject_payload` API is used then it will emit the string contents of the provided payload. In other words, you're free to generate and emit any arbitrary JSON you want. Your code should do the following: - Call read_message to extract any needed data from the incoming message - Create a Lua table matching the JSON structure you want to generate - Use the `cjson` module to encode the table to a JSON string - Call zero or more `add_to_payload` calls, followed by an `inject_payload` call to emit whatever strings you want to emit Here's the source code for the ESPayloadEncoder: https://github.com/mozilla-services/heka/blob/versions/0.9/sandbox/lua/encoders/es_payload.lua That doesn't do any JSON encoding, b/c it assumes that the payload coming in will already contain the raw JSON that ElasticSearch expects. It does, however, show the use of read_message, add_to_payload, and inject_payload. As an added bonus it shows you how to read data from the config file, and how to use the `elasticsearch` Lua module we provide to generate the ES Bulk API index headers. -r On 06/30/2015 11:10 AM, Lewis, Steven wrote: > Hello heka people! > We have a kibana/elasticsearch stack already running (with custom json > formats), and I’d like to massage the message coming out of some JSON > inputs we have to not exactly follow the Heka message format. > Specifically, I’d like the JSON contained in the “@message” value to > move up a level, so that _source directly contains my relevant keys, > like so: > {“_source”: { > “@uuid”: <guid here>, > “@timestamp”: <timestr here>, > “@ID”: <my custom guid>, > “@status”: <custom status> > } > I was thinking at first that I could create a lua encoder, but looking > at the API I don’t see anything where I can modify the output directly. > I can modify the payload, or I can modify the message, but I can’t go > outside the schema of the Heka message (at least that’s my impression). > Do I need to create a Sandbox Output to do this? Or copy the > ElasticSearch go output and tweak it? Hopefully my question makes sense. > We can get what we want if we use logstash, but we love go and heka > seems awesome. > Thanks for any guidance! > Steve Lewis > Sr. Software Engineer > PETNET Solutions, Inc. > This message and any attachments are solely for the use of intended > recipients. The information contained herein may include trade > secrets, protected health or personal information, privileged or > otherwise confidential information. Unauthorized review, forwarding, > printing, copying, distributing, or using such information is strictly > prohibited and may be unlawful. If you are not an intended recipient, > you are hereby notified that you received this email in error, and > that any review, dissemination, distribution or copying of this email > and any attachment is strictly prohibited. If you have received this > email in error, please contact the sender and delete the message and > any attachment from your system. Thank you for your cooperation > > > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka > This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
