Hi jing, It sounds good to me, we can add an option for it
Best, Shammon On Fri, Feb 17, 2023 at 3:13 PM Jing Ge <[email protected]> wrote: > Hi, > > It makes sense to offer this feature of catching and ignoring exp with > config on/off, when we put ourselves in users' shoes. WDYT? I will create a > ticket if most of you consider it as a good feature to help users. > > Best regards, > Jing > > On Fri, Feb 17, 2023 at 6:01 AM Shammon FY <[email protected]> wrote: > >> Hi Hatem >> >> As mentioned above, you can extend the KafkaSink or create a udf and >> process the record before sink >> >> Best, >> Shammon >> >> On Fri, Feb 17, 2023 at 9:54 AM yuxia <[email protected]> >> wrote: >> >>> Hi, Hatem. >>> I think there is no way to catch the exception and then ignore it in >>> current implementation for KafkaSink. You may also need to extend the >>> KafkaSink. >>> >>> Best regards, >>> Yuxia >>> >>> ------------------------------ >>> *发件人: *"Hatem Mostafa" <[email protected]> >>> *收件人: *"User" <[email protected]> >>> *发送时间: *星期四, 2023年 2 月 16日 下午 9:32:44 >>> *主题: *KafkaSink handling message size produce errors >>> >>> Hello, >>> I am writing a flink job that reads and writes into kafka, it is using a >>> window operator and eventually writing the result of the window into a >>> kafka topic. The accumulated data can exceed the maximum message size after >>> compression on the producer level. I want to be able to catch the exception >>> coming from the producer and ignore this window. I could not find a way to >>> do that in KafkaSink >>> <https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/#kafka-sink>, >>> is there a way to do so? >>> >>> I attached here an example of an error that I would like to handle >>> gracefully. >>> >>> [image: image.png] >>> >>> >>> This question is similar to one that was asked on stackoverflow here >>> <https://stackoverflow.com/questions/52308911/how-to-handle-exceptions-in-kafka-sink> >>> but >>> the answer is relevant for older versions of flink. >>> >>> Regards, >>> Hatem >>> >>>
