[
https://issues.apache.org/jira/browse/NIFI-11341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Kimberley updated NIFI-11341:
-----------------------------------
Description:
In our environment, we use {{ListenUDPRecord}} to collect Syslog messages. This
processor is followed by a {{PartitionRecord}} processor that populates an
attribute for routing.In release {*}1.19.0{*}, this flow worked without issue.
In *1.20.0* though, I am seeing intermittent message truncation in
{{{}PartitionRecord{}}}, with bulletin messages like the following appearing
regularly:
{code:java}
PartitionRecord[id=03ea67a7-0b9c-1c9f-ffff-ffff8d7e5185] Failed to partition
FlowFile[filename=ca9c3e11-9365-4ff9-9499-29522fc0cab7]:
com.fasterxml.jackson.core.JsonParseException: Unexpected character (',' (code
44)): expected a value
at [Source: (org.apache.nifi.stream.io.NonCloseableInputStream); line: 1,
column: 381]{code}
An example message (note the absence of a Syslog header):
{noformat}
itor] [Unit test] Alarm check cfg warning threshold=75 critical threshold=85
warning alarm <...>{noformat}
{{ListenUDPRecord}} properties are attached.
h3. Reproduction
The attached minimal flow illustrates this setup.
To reproduce this issue, generate improperly-formatted syslog and send to
{{{}ListenUDPRecord{}}}.
In my environment, I have two syslog sources feeding this test cluster.
Scenario is as follows:
# First source (compliant Syslog format) feeds in.
# Flow is OK - no bulletins.
# Activate second source, which is of an invalid Syslog format and flows to
the {{parse.failure}} relationship of {{{{}}{{ListenUDPRecord{}}}{}}}. This is
expected - I deal with this gracefully.
# Bulletins start firing in {{PartitionRecord}} and the first source starts
getting truncated randomly.
Overall, the majority of messages from the well-formed source make it through.
However I'm seeing roughly 1 bulletin every few seconds, which indicates a
small proportion of messages are getting truncated.
was:
In our environment, we use {{ListenUDPRecord}} to collect Syslog messages. This
processor is followed by a {{PartitionRecord}} processor that populates an
attribute for routing.In release {*}1.19.0{*}, this flow worked without issue.
In *1.20.0* though, I am seeing intermittent message truncation in
{{{}PartitionRecord{}}}, with bulletin messages like the following appearing
regularly:
PartitionRecord[id=03ea67a7-0b9c-1c9f-ffff-ffff8d7e5185] Failed to partition
FlowFile[filename=ca9c3e11-9365-4ff9-9499-29522fc0cab7]:
com.fasterxml.jackson.core.JsonParseException: Unexpected character (',' (code
44)): expected a value
at [Source: (org.apache.nifi.stream.io.NonCloseableInputStream); line: 1,
column: 381]
An example message (note the absence of a Syslog header):
itor] [Unit test] Alarm check cfg warning threshold=75 critical threshold=85
warning alarm <...>
{{ListenUDPRecord}} properties are attached.
h3. Reproduction
The attached minimal flow illustrates this setup.
To reproduce this issue, generate improperly-formatted syslog and send to
{{{}ListenUDPRecord{}}}.
In my environment, I have two syslog sources feeding this test cluster.
Scenario is as follows: # First source (compliant Syslog format) feeds in.
# Flow is OK - no bulletins.
# Activate second source, which is of an invalid Syslog format and flows to
the {{parse.failure}} relationship of {{{}{{ListenUDPRecord}}{}}}. This is
expected - I deal with this gracefully.
# Bulletins start firing in {{PartitionRecord}} and the first source starts
getting truncated randomly.
Overall, the majority of messages from the well-formed source make it through.
However I'm seeing roughly 1 bulletin every few seconds, which indicates a
small proportion of messages are getting truncated.
> ListenUDPRecord truncating data
> -------------------------------
>
> Key: NIFI-11341
> URL: https://issues.apache.org/jira/browse/NIFI-11341
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.20.0
> Reporter: Peter Kimberley
> Priority: Major
> Attachments: NiFi_Flow.json, image.png
>
>
> In our environment, we use {{ListenUDPRecord}} to collect Syslog messages.
> This processor is followed by a {{PartitionRecord}} processor that populates
> an attribute for routing.In release {*}1.19.0{*}, this flow worked without
> issue. In *1.20.0* though, I am seeing intermittent message truncation in
> {{{}PartitionRecord{}}}, with bulletin messages like the following appearing
> regularly:
> {code:java}
> PartitionRecord[id=03ea67a7-0b9c-1c9f-ffff-ffff8d7e5185] Failed to partition
> FlowFile[filename=ca9c3e11-9365-4ff9-9499-29522fc0cab7]:
> com.fasterxml.jackson.core.JsonParseException: Unexpected character (','
> (code 44)): expected a value
> at [Source: (org.apache.nifi.stream.io.NonCloseableInputStream); line: 1,
> column: 381]{code}
>
> An example message (note the absence of a Syslog header):
> {noformat}
> itor] [Unit test] Alarm check cfg warning threshold=75 critical threshold=85
> warning alarm <...>{noformat}
> {{ListenUDPRecord}} properties are attached.
>
> h3. Reproduction
> The attached minimal flow illustrates this setup.
>
> To reproduce this issue, generate improperly-formatted syslog and send to
> {{{}ListenUDPRecord{}}}.
>
> In my environment, I have two syslog sources feeding this test cluster.
> Scenario is as follows:
> # First source (compliant Syslog format) feeds in.
> # Flow is OK - no bulletins.
> # Activate second source, which is of an invalid Syslog format and flows to
> the {{parse.failure}} relationship of {{{{}}{{ListenUDPRecord{}}}{}}}. This
> is expected - I deal with this gracefully.
> # Bulletins start firing in {{PartitionRecord}} and the first source starts
> getting truncated randomly.
> Overall, the majority of messages from the well-formed source make it
> through. However I'm seeing roughly 1 bulletin every few seconds, which
> indicates a small proportion of messages are getting truncated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
