But how should I then handle modifying fields in subsequent decoders in multi-decoder? And actually how multi-decoder should be used then? Only with non-lua decoders?
I think in all other places I’ve used the same technique with Lua decoder being the first in the line but not PayloadRegexDecoder as in this case. Could that be the reason for the issue? One more point to add here: I’ve seen some Heka panics with this setup. These happened when there were some more lua plugins added to the setup. They are also stable in my other setups but again I don’t use PayloadRegexDecoder often. I could not reproduce that reliably so no stacktrace from me, sorry. Could that also be related to this issue? re: modifying lua_sandbox.go Many thanks, I’ll definitely try this for debugging setup! Thanks, Timur On 12 Oct 2015 at 23:03:51, Rob Miller ([email protected]) wrote: Simon was right in that the problem here is your use of `read_message("raw")`, which should almost never be used in a decoder. `read_message("raw")` returns the pack.MsgBytes value. Once a message hits the router, this is guaranteed to have an accurate protobuf encoding of the message, but before the router this is not the case. If you want to use `print` in your Lua scripts you can do so by removing `print` from the sandbox template's `remove_entries` value in the lua_sandbox.go.in file: https://github.com/mozilla-services/heka/blob/dev/sandbox/lua/lua_sandbox.go.in#L60 -r On 10/12/2015 07:17 AM, Timur Batyrshin wrote: > Hi, > > I’ve been writing a decoder for myself and have hit the following issue > which I can’t understand. > > When I start Heka it produces the following error message in logs: > > 2015/10/12 13:54:47 SubDecoder > ‘zerogw-zerogw_decoder-stdout-zerogw_rotate_fields’ error: FATAL: > process_message() /usr/share/heka/lua_decoders/rotate_fields.lua:30: > bad argument #0 to ‘decode_message’ (must have one string argument) > > At the same time the code for decoder is the following: > > |-- the only lines above are comments which are skipped metric_field = > read_config("metric_field") or "metric" value_field = > read_config("value_field") or "value" function process_message() local > fields = {} raw = read_message("raw”) # line 29 msg = > decode_message(raw) # line 30 -- other part of code is probably > irrelevant as crash is seen in the above line | > > (I’ve tried writing that as |decode_message(read_message("raw”))| with > the same effect) > > What’s really weird is the exactly the same decoder works fine on other > hosts. > > I’m using the following Heka config: > > |[zerogw] type = "ProcessInput" ticker_interval = 0 splitter = > "on_newline" decoder = "zerogw_decoder" stdout = true stderr = false > [zerogw.command.0] bin = "/usr/local/bin/zerogw_collector.py" args = > ["-s", "tcp://127.0.0.1:5111 <http://127.0.0.1:5111>"] [on_newline] type > = "TokenSplitter" delimiter = "\n" [estp_decoder] type = > "PayloadRegexDecoder" match_regex = '^(?P<Name>[^\s]+) > (?P<Timestamp>\d+) (?P<Value>\d+)' timestamp_layout = "Epoch" > [estp_decoder.message_fields] Service = "Zerogw" Metric = "%Name%" Value > = "%Value%" [zerogw_decoder] type = "MultiDecoder" subs = > ["estp_decoder", "zerogw_rotate_fields"] cascade_strategy = "all" > [zerogw_rotate_fields] type = "SandboxDecoder" filename = > "lua_decoders/rotate_fields.lua" [zerogw_rotate_fields.config] > metric_field = "Metric" value_field = "Value" | > > zerogw_collector.py produces about a dozen of lines to stdout every 5 > seconds in the format as seen in message payload (see below). > > As MultiDecoder has |cascade_strategy = "all"| Heka dumps messages > processed by the first decoder in the chain to stdout which are the > following: > > |2015/10/12 14:04:22 :Timestamp: 2015-10-12 14:04:22 +0000 UTC :Type: > ProcessInput :Hostname: t-eu-zgw :Pid: 5212 :Uuid: > 2c7deb23-7961-49dc-8f57-da716d851439 :Logger: zerogw :Payload: > zerogw.connections.total 1444658662 4 :EnvVersion: :Severity: 7 :Fields: > | name:"ProcessInputName" type:string value:"zerogw.stdout" | > name:"ExitStatus" type:integer value:0 | name:"Value" type:string > value:"4" | name:"Service" type:string value:"Zerogw" | name:"Metric" > type:string value:"zerogw.connections.total" | > > In plain Lua I’d dump the result of |read_message("raw")| to stdout, add > some prints everywhere and see what happens inside but don’t know how to do. > > Any clues on how I should debug such cases? > > Thanks, > Timur > > > > > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
