Hao Yu created FLINK-41265:
------------------------------
Summary: Add a metric for JSON messages with ignored parse errors
Key: FLINK-41265
URL: https://issues.apache.org/jira/browse/FLINK-41265
Project: Flink
Issue Type: Improvement
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Runtime
/ Metrics
Affects Versions: 2.3.0
Reporter: Hao Yu
Problem
When json.ignore-parse-errors is enabled, JSON parsing or field conversion
errors can be ignored without a metric indicating how many input messages were
affected.
There are two relevant cases:
- Invalid JSON causes a message to be skipped.
- A field conversion fails and the field is set to null, while the remaining
row is still emitted.
For example, with an INT field named "id", the input \{"id":"abc"} may produce
a row whose id is null. Counting only errors caught by the outer
deserialization method would miss this case.
Motivation
In production, this option may have been enabled long before an incident, and
the users investigating the issue may no longer be aware of it. The job
continues running without failing, but some expected records are missing or
some fields unexpectedly become null. Users may therefore suspect data loss in
Flink, without realizing that parsing errors are being ignored according to the
configured behavior.
Users also need visibility into when invalid input starts appearing. A metric
exposed on a monitoring dashboard would help them:
- Identify when ignored parsing errors first occur.
- Observe how their frequency changes over time.
- Determine how long the problem persists and whether it is still ongoing.
- Correlate the incident with upstream data or schema changes.
- Configure alerts when ignored parsing errors occur or increase.
The metric would make these errors observable while preserving the existing
error-tolerant behavior.
Proposed change
Add a counter named numMessagesWithParseErrors to the plain JSON format,
registered through DeserializationSchema.InitializationContext when
json.ignore-parse-errors is enabled.
The counter should:
- Cover both the JsonNode and streaming JsonParser decoders.
- Include ignored JSON parsing errors and ignored field conversion errors.
- Increment at most once per input message, even if multiple fields fail.
- Treat a top-level JSON array as one input message, regardless of how many
rows it produces.
- Exclude accepted null field values, permitted missing fields, and empty
inputs that do not cause parsing exceptions.
- Preserve existing deserialization behavior.
The counter represents affected input messages, including messages that still
produce rows with null fields. It should not be interpreted as the number of
dropped output rows.
Scope
This issue covers the plain JSON format.
CDC formats and connector-specific metric context changes are outside its
scope. File-source metric reporting and Kafka key/value metric-group isolation
require separate follow-ups.
Related work
FLINK-20789 tracks the broader request for metrics in formats supporting
ignore-parse-errors. This issue proposes the JSON-specific implementation.
https://issues.apache.org/jira/browse/FLINK-20789
FLINK-38819 added logging for ignored JSON parsing errors. This proposal adds a
counter that can be used directly by monitoring dashboards and alerts.
https://issues.apache.org/jira/browse/FLINK-38819
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
