[
https://issues.apache.org/jira/browse/FLINK-8190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276642#comment-16276642
]
ASF GitHub Bot commented on FLINK-8190:
---------------------------------------
GitHub user tzulitai opened a pull request:
https://github.com/apache/flink/pull/5117
[FLINK-8190] [kafka] Add constructors to expose topic pattern-based
subscription
## What is the purpose of the change
This PR simply exposes functionality that was already implemented.
There is no new code adding any new functionality.
Partition and topic regex-pattern-based subscription was implemented in
[FLINK-4022](https://issues.apache.org/jira/browse/FLINK-4022).
However, the functionality for pattern-based topic subscription was never
exposed in the API.
This PR exposes the feature through the following new constructors on all
version-specific Kafka consumers:
```
public FlinkKafkaConsumerXX(java.util.regex.Pattern subscriptionPattern,
DeserializationSchema<T> schema, Properties props);
public FlinkKafkaConsumerXX(java.util.regex.Pattern subscriptionPattern,
KeyedDeserializationSchema<T> schema, Properties props);
```
This allows the consumer to pick up all matching topics on startup.
To continuously pick up matching topics on the fly when they are created
after the job has already started running, users should additionally set the
`KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS` property, as they would do for
partition discovery.
## Brief change log
- Add new constructors for pattern-based topic subscription to
`FlinkKafkaConsumer08`, `..09`, `..010`, and `..011`.
- Update Kafka connector docs to document the feature.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs + Javadocs updated
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tzulitai/flink FLINK-8190
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5117.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5117
----
commit 7f7599108c06bbc046b5067bef71dfa859ab01b0
Author: Tzu-Li (Gordon) Tai <[email protected]>
Date: 2017-12-04T07:05:46Z
[FLINK-8190] [kafka] Add constructors to expose topic pattern-based
subscription
----
> Add extra FlinkKafkaConsumer constructors to expose pattern-based topic
> subscription functionality
> --------------------------------------------------------------------------------------------------
>
> Key: FLINK-8190
> URL: https://issues.apache.org/jira/browse/FLINK-8190
> Project: Flink
> Issue Type: New Feature
> Components: Kafka Connector
> Reporter: Tzu-Li (Gordon) Tai
> Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.4.0, 1.5.0
>
>
> The required internals for pattern-based topic discovery was implemented as
> part of FLINK-4022 (along with partition discovery). However, the
> functionality for pattern-based topic discovery was not yet exposed via any
> visible user API on the version-specific subclasses of
> {{FlinkKafkaConsumerBase}}.
> I propose to add two more constructors for this:
> {code}
> public FlinkKafkaConsumerXX(java.util.regex.Pattern subscriptionPattern,
> DeserializationSchema<T> schema, Properties props);
> public FlinkKafkaConsumerXX(java.util.regex.Pattern subscriptionPattern,
> KeyedDeserializationSchema<T> schema, Properties props);
> {code}
> This allows the consumer to pick up all matching topics on startup.
> To continuously pick up matching topics on the fly when they are created
> after the job has already started running, users should additionally set the
> {{KEY_PARTITION_DISCOVERY_INTERVAL_MILLIS}} property, as they would do for
> partition discovery.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
