Hello Jochen,
I have nospecial extractors running. Maybe I can send you the real (not
anonymized) XML on a private way? To me it seems like it is a special
sequence in that XML that is causing this. I will try to reproduce it first
by myself.
The message that shows the shortened XML is still there.
I found this out because I am now tracking sizes of all messages sent to
Graylog to see big sized messages that should be shortened before sending
them. I investigated further and this one was a very large message but in
my framework internally shortened to 64kb (the full_message field) and ".."
appended. So the XML got cut. Nevertheless the message in graylog is
showing closing tags that were never sent and a full_message field of
~200byte. Very weired because I do no XML processing in the logging
framework as well just plain strings.
Thanks
Robby
Am Mittwoch, 19. Oktober 2016 12:41:16 UTC+2 schrieb Jochen Schalanda:
>
> Hi Robby,
>
> I cannot reproduce your problem.
>
> I've created a GELF UDP input on an installation of Graylog 2.1.1 and sent
> a GELF message to it with the XML example from your first email in the
> full_text message field.
>
> Everything has been ingested and is being shown as intended (i. e.
> unparsed and the verbatim XML). Maybe some extractor is interfering with
> your payload or even the client already sends the modified GELF message.
>
> Graylog and Elasticsearch itself do not parse XML from message fields by
> default.
>
> For reproduction, here's what I did:
>
> Create GELF UDP input with the following configuration and no extractors:
>
> bind_address: 127.0.0.1
> decompress_size_limit: 8388608
> override_source: <empty>
> port: 12201
> recv_buffer_size: 16384
>
>
> Send message to Graylog:
>
>
> $ echo '{"version":"1.1", "message":"XML TEST", "full_message":"<?xml
> version=\"1.0\" encoding=\"utf-8\"?>\n <Nest1>\n
> <Nest2>Success</Nest2>\n <Nest3>DoSomething</Nest4>\n
> <Nest4>8</Nest4>\n <Nest5>\n <Nest6>0</Nest6>\n <Nest7>\n
> <Nest8.."}' | nc -w1 -u 127.0.0.1 12201
>
>
> Message in Graylog:
>
> $ curl -i -u admin:admin
> http://127.0.0.1:9000/api/messages/graylog2_27/43c3aca0-95e7-11e6-8fc7-6c4008b8fc28?pretty=true
> HTTP/1.1 200 OK
> X-Graylog-Node-ID: cd03ee44-b2a7-4824-be16-bb7456149dbd
> X-Runtime-Microseconds: 1887
> Content-Type: application/json
> Date: Wed, 19 Oct 2016 10:38:40 GMT
> Content-Length: 1975
>
> {
> "message" : {
> "fields" : {
> "full_message" : "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n
> <Nest1>\n <Nest2>Success</Nest2>\n <Nest3>DoSomething</Nest4>\n
> <Nest4>8</Nest4>\n <Nest5>\n <Nest6>0</Nest6>\n <Nest7>\n
> <Nest8..",
> "streams" : [ "552b92b2e4b0c055e41ffb8e", "000000000000000000000001"
> ],
> "gl2_source_node" : "cd03ee44-b2a7-4824-be16-bb7456149dbd",
> "_id" : "43c3aca0-95e7-11e6-8fc7-6c4008b8fc28",
> "source" : "unknown",
> "message" : "XML TEST",
> "gl2_source_input" : "5592913ec026f261b8d5ce9a",
> "timestamp" : "2016-10-19T10:31:58.824Z"
> },
> "streams" : [ ],
> "source_input_id" : null,
> "filter_out" : false,
> "journal_offset" : -9223372036854775808,
> "message" : "XML TEST",
> "id" : "43c3aca0-95e7-11e6-8fc7-6c4008b8fc28",
> "source" : "unknown",
> "field_names" : [ "full_message", "streams", "gl2_source_node", "_id",
> "source", "message", "gl2_source_input", "timestamp" ],
> "complete" : true,
> "timestamp" : "2016-10-19T10:31:58.824Z",
> "validation_errors" : "",
> "fields_entries" : [ {
> "full_message" : "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n
> <Nest1>\n <Nest2>Success</Nest2>\n <Nest3>DoSomething</Nest4>\n
> <Nest4>8</Nest4>\n <Nest5>\n <Nest6>0</Nest6>\n <Nest7>\n
> <Nest8.."
> }, {
> "streams" : [ "552b92b2e4b0c055e41ffb8e", "000000000000000000000001"
> ]
> }, {
> "gl2_source_node" : "cd03ee44-b2a7-4824-be16-bb7456149dbd"
> }, {
> "_id" : "43c3aca0-95e7-11e6-8fc7-6c4008b8fc28"
> }, {
> "source" : "unknown"
> }, {
> "message" : "XML TEST"
> }, {
> "gl2_source_input" : "5592913ec026f261b8d5ce9a"
> }, {
> "timestamp" : "2016-10-19T10:31:58.824Z"
> } ],
> "field_count" : 8,
> "stream_ids" : [ "552b92b2e4b0c055e41ffb8e",
> "000000000000000000000001" ],
> "is_source_inet_address" : false,
> "inet_address" : null
> },
> "index" : "graylog2_27"
> }
>
>
> Screenshot:
>
>
> <https://lh3.googleusercontent.com/-TWdOiWfXpkA/WAdNxOL9P2I/AAAAAAAAADg/vNXFDvoveUEv1upqRotic88ikuTGQHwzgCLcB/s1600/Bildschirmfoto%2B2016-10-19%2Bum%2B12.40.12.png>
>
>
> Cheers,
> Jochen
>
> On Tuesday, 18 October 2016 14:10:11 UTC+2, 4BRobby wrote:
>>
>> Hell Jochen,
>>
>> Enclosed my configuration if this helps. It is the OVA appliance, latest
>> graylog version.
>>
>> A new log worked b.t.w. so it may be the problem that the .. at one time
>> breaks the xml tag itself.
>> Nevertheless it is an unexpected behaviour that the xml even gets parsed
>> out of the message. XML is of course base64 coded like all other log
>> message bodies (full_message) that are send to Graylog.
>>
>> I have found no information about this behaviour so far.
>>
>> I had to anonymize the message (That is why it reads nested). One tag is
>> called result - but only numbers and usual text in the tag values, Nothing
>> special really.
>>
>> Thanks
>> Robby
>>
>>
>> Am Dienstag, 18. Oktober 2016 12:45:50 UTC+2 schrieb Jochen Schalanda:
>>>
>>> Hi,
>>>
>>> On Tuesday, 18 October 2016 12:27:45 UTC+2, 4BRobby wrote:
>>>>
>>>> Configuraiton: There is no transformation for the message processing,
>>>> all to default.
>>>>
>>>
>>> That's not what I've asked for. Please provide the requested
>>> information, otherwise we might not be able to help you.
>>>
>>> Cheers,
>>> Jochen
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/1c8d255d-2552-437a-8a63-c772cd91c463%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
