[
https://issues.apache.org/jira/browse/STORM-817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345162#comment-15345162
]
Mike Treadway commented on STORM-817:
-------------------------------------
Hi there, I'm seeing an issue with this implementation and I'm wondering how it
ever worked with Trident.
The OpaquePartitionedTridentSpoutExecutor assumes that the partition state
directories are direct children of the transaction directory. When wildcard
support is enabled, the state directories are grandchildren of the transaction
directory. For example, without wildcard support, it all works and the
directory structure looks like this:
{stream}/user/partition_0
With wildcard support enabled, the partition id goes from 'partition_0' to
'{topic}/partition_0', so the directory structure looks like this:
{stream}/user/{topic}/partition_0
The implementation inside OpaquePartitionedTridentSpoutExecutor.commit()
assumes that the partition state is directly under user because it does
'_state.list("")', which is incorrect.
Is anyone else seeing this?
> Kafka Wildcard Topic Support
> ----------------------------
>
> Key: STORM-817
> URL: https://issues.apache.org/jira/browse/STORM-817
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-kafka
> Reporter: Sumit Chawla
> Assignee: Sumit Chawla
>
> Creating a feature request for supporting Wildcard Topic's for Kafka Spout.
> We want to be able to run a aggregation stream for data coming from all
> tenants. Tenants get added dynamically. So new kafka topics get created. All
> the topics will be matching a regex pattern.
> example:
> clickstream:tenant1:log
> clickstream:tenant2:log
> clickstream:tenant3:log
> Storm code should be able to perform auto-discovery, and should be able to to
> fetch from newly created topics in run time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
