[
https://issues.apache.org/jira/browse/FLINK-9155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444080#comment-16444080
]
Tzu-Li (Gordon) Tai commented on FLINK-9155:
--------------------------------------------
I think we can implement both logging and metrics to track this.
For metrics, we should be able to use the user variable functionality to have
skip counters that can be grouped by topic / partition. This should allow more
prompt alerts of skipped messages, and users should direct to logs for more
details on the erroring record.
For logging, the information should contain topic, partition, and offset for
debugging.
> Provide message context information in DeserializationSchema
> ------------------------------------------------------------
>
> Key: FLINK-9155
> URL: https://issues.apache.org/jira/browse/FLINK-9155
> Project: Flink
> Issue Type: Improvement
> Reporter: Alex Smirnov
> Priority: Minor
>
> There's no way to retrieve more information about corrupted message in the
> DeserializationSchema class. It is only possible to return null, which is a
> signal to skip the message, and to throw an exception, which will cause job
> failure.
> For investigation purposes it would be good to have more information, like:
> * kafka topic from which the message came from
> ** in Flink 1.4, it is possible to subscribe using Pattern, so topic name is
> not always evident
> * kafka topic offset
> The idea is to write this information into the log file for further analysis.
> Having topic name and offset allows to retrieve the message and investigate
> it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
