danzhewuju opened a new pull request, #29670: URL: https://github.com/apache/flink/pull/29670
## What is the purpose of the change This PR addresses [FLINK-41265](https://issues.apache.org/jira/browse/FLINK-41265) by exposing a metric for input messages affected by ignored JSON parsing or field conversion errors. In production, `json.ignore-parse-errors` may have been enabled long before an incident. Users investigating missing records or unexpected null fields may no longer be aware of this configuration and suspect data loss in Flink while the job continues running. The new counter makes these events visible through monitoring dashboards and alerts. Users can identify when errors start occurring, observe their frequency over time, and determine whether the problem is still ongoing. ## Brief change log - Register `json.numMessagesWithParseErrors` through the deserialization initialization context when the plain JSON format uses `json.ignore-parse-errors=true`. - Track ignored errors in both the JsonNode and streaming JsonParser decoders, including errors handled inside field converters. - Increment the counter at most once per input message. A top-level JSON array counts as one message, even if multiple elements contain errors. - Include messages that still produce rows with null fields while preserving existing deserialization behavior. - Add focused tests and document the metric in the English and Chinese JSON format documentation. The scope is limited to the plain JSON format. CDC formats do not report this metric. Connector-specific metric context propagation and key/value metric-group isolation are outside this PR. ## Verifying this change Added `JsonParseErrorMetricsTest`, parameterized for both JSON decoders, covering: - Counting malformed messages and field conversion failures, with at most one increment per input message. - Keeping the counter unchanged for valid messages, accepted null or missing fields, and empty inputs. - Preserving strict-mode exception behavior without registering the ignored-error counter. Updated `JsonFormatFactoryTest` to account for metric activation by the format factory. Local test execution and the full `./mvnw clean verify` have not been completed. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): No. - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: No. - The serializers: Yes, the JSON deserialization implementation is updated. No state serializer or checkpoint format is changed. - The runtime per-record code paths (performance sensitive): Yes, error tracking and message-level counter updates are added to JSON deserialization. - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: No. - The S3 file system connector: No. ## Documentation - Does this pull request introduce a new feature? Yes, a JSON format metric. - If yes, how is the feature documented? In the English and Chinese SQL JSON format documentation. The documentation explains that the counter measures affected input messages, including messages that still emit rows with null fields, rather than the number of dropped output rows. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: GPT-6 Astra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
