something like dynamic filtering that can be updated at runtime or deny all
but allow a certain set of topics that cannot be specified easily by regex
On Thu, Mar 12, 2015 at 9:06 PM, Guozhang Wang wangg...@gmail.com wrote:
Hmm, what kind of customized filtering do you have in mind? I thought
A little more context about my needs: I have a requirement that I need to
start/stop a topic at runtime based on a event sent to MM. at the moment I
need to bounce the MM and find a way to exclude the topic from whitelist
which is not an easy job with regex. If I can pass in a combination of
Thank you Guozhang for your advice. A dynamic topic filter is what I need
so that I can stop a topic consumption when I need to at runtime.
On Thu, Mar 12, 2015 at 9:21 PM, Guozhang Wang wangg...@gmail.com wrote:
1. Dynamic: yeah that is sth. we could think of, this could be useful
Hmm, what kind of customized filtering do you have in mind? I thought with
--whitelist you could already specify regex to do filtering.
On Thu, Mar 12, 2015 at 5:56 AM, tao xiao xiaotao...@gmail.com wrote:
Hi Guozhang,
I was meant to be topicfilter not topic-count. sorry for the confusion.
Note that with filtering in message handler, records from the source
cluster are still considered as consumed since the offsets will be
committed. If you change the filtering dynamically back to whilelist these
topics, you will lose the data that gets consumed during the period of the
blacklist.
Yes, that will work. message handle can filter out message sent from
certain topics
On Fri, Mar 13, 2015 at 6:30 AM, Jiangjie Qin j...@linkedin.com.invalid
wrote:
No sure if it is an option. But does filtering out topics with message
handler works for you? Are you going to resume consuming
I am not sure how MM is going to be rewritten. Based on the current
implementation in trunk offset is not committed unless it is produced to
destination. With assumption that this logic remains MM will not
acknowledge the offset back to source for filtered message. So I think it
is safe to filter
Yes, you are right. a dynamic topicfilter is more appropriate where I can
filter topics at runtime via some kind of interface e.g. JMX
On Thu, Mar 12, 2015 at 11:03 PM, Guozhang Wang wangg...@gmail.com wrote:
Tao,
Based on your description I think the combination of whitelist / blacklist
Tao,
Based on your description I think the combination of whitelist / blacklist
will not achieve your goal, since it is still static.
Guozhang
On Thu, Mar 12, 2015 at 6:30 AM, tao xiao xiaotao...@gmail.com wrote:
Thank you Guozhang for your advice. A dynamic topic filter is what I need
so
Hi Guozhang,
I was meant to be topicfilter not topic-count. sorry for the confusion.
What I want to achieve is to pass my own customized topicfilter to MM so
that I can filter out topics what ever I like. I know MM doesn't support
this now. I am just thinking if this is a good feature to add in
No sure if it is an option. But does filtering out topics with message
handler works for you? Are you going to resume consuming from a topic
after you stop consuming from it?
Jiangjie (Becket) Qin
On 3/12/15, 8:05 AM, tao xiao xiaotao...@gmail.com wrote:
Yes, you are right. a dynamic
Hi Tao,
Sorry I was mistaken before, yes in MM you can only directly specify
--whitelist, --blacklist, and the number of streams you want to create
via --num.streams, but cannot set specific topic-count. This is because
MM is mainly used for cross DC replication, and hence usually will pipe all
The topic list is not specified in consumer.properties and I don't think
there is any property in consumer config that allows us to specify what
topics we want to consume. Can you point me to the property if there is any?
On Thu, Mar 12, 2015 at 12:14 AM, Guozhang Wang wangg...@gmail.com wrote:
Tao,
In MM people can pass in consumer configs, in which people can specify
consumption topics, either in regular topic list format or whitelist /
blacklist. So I think it already does what you need?
Guozhang
On Tue, Mar 10, 2015 at 10:09 PM, tao xiao xiaotao...@gmail.com wrote:
Thank you
Hi,
I have an user case where I need to consume a list topics with name that
matches pattern topic.* except for one that is topic.10. Is there a way
that I can combine the use of whitelist and blacklist so that I can achieve
something like accept all topics with regex topic.* but exclude
I actually mean if we can achieve this in mirror maker.
On Tue, Mar 10, 2015 at 10:52 PM, tao xiao xiaotao...@gmail.com wrote:
Hi,
I have an user case where I need to consume a list topics with name that
matches pattern topic.* except for one that is topic.10. Is there a way
that I can
16 matches
Mail list logo
