Sorry for the late reply, If a batch has 10 messages, but users only want to filter out parts of them such as 3 messages, only 7 messages should be processed at the consumer side. So if the proposal is for the entry filter, I think we should have the EntryFitler interface, not MessageFilter?
Actually, I also have some doubts about the entry filter, not sure if it can be used in the real world. Or we should disable the batch when using the filter or deserialize the single message metadata to decide if the consumer should skip this message, looks both of them will bring greater overhead to the broker. But I am not against the pluggable filter, not all users consider the performance first, If they are more think about it at a functional perspective, the pluggable filter will help them. We should clarify it in the proposal, let users know how to make trade-offs. Thanks, Penghui On Fri, Sep 10, 2021 at 4:34 PM Lin Lin <lin...@apache.org> wrote: > > > On 2021/09/10 06:22:09, PengHui Li <peng...@apache.org> wrote: > > Looks good overall, > > > > Do we need to consider the batch filter since the API is defined as > > `Message`, not `Entry`, so parts of the batch need to filter out, does it > > should be handled by Filter, or the consumer side, or the producer side? > > > > Thanks, > > Penghui > > > > Hi Penghui: > > In my opinion > 1) It should be the Entry dimension, and the execution should be before > AbstractBaseDispatcher#filterEntriesForConsumer, so that custom filtering > can be performed before the data is affected > 2) It should be filtered on the Broker side, which can reduce the > transmission of unnecessary data > > Thanks >
