[
https://issues.apache.org/jira/browse/KAFKA-6020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16597415#comment-16597415
]
Geoffray Adde commented on KAFKA-6020:
--------------------------------------
Hello.
I strongly support the idea on message filtering on the broker side.
I am using kafka as a message streaming system. I am not committing anything
and I am not even keeping track of the offsets.
On the other hands, I have to deal with a vast variety of distinct objects.
Hundreds of thousands, up to millions.
Obviously, I cannot have that many topics, but with broker side filtering, I
could get get some sort of sub-topics.
To me, the idea would be, on a fetch request to send a mask to test a custom
header.
If either the custom header is not present or if it does not match the mask,
the message is not sent as part of the reply to the fetch request.
The concept seems simple but I have no clue how much work it is to implement.
> Broker side filtering
> ---------------------
>
> Key: KAFKA-6020
> URL: https://issues.apache.org/jira/browse/KAFKA-6020
> Project: Kafka
> Issue Type: New Feature
> Components: consumer
> Reporter: Pavel Micka
> Priority: Major
> Labels: needs-kip
>
> Currently, it is not possible to filter messages on broker side. Filtering
> messages on broker side is convenient for filter with very low selectivity
> (one message in few thousands). In my case it means to transfer several GB of
> data to consumer, throw it away, take one message and do it again...
> While I understand that filtering by message body is not feasible (for
> performance reasons), I propose to filter just by message key prefix. This
> can be achieved even without any deserialization, as the prefix to be matched
> can be passed as an array (hence the broker would do just array prefix
> compare).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
