[
https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=94761&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-94761
]
ASF GitHub Bot logged work on BEAM-3925:
----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Apr/18 19:06
Start Date: 24/Apr/18 19:06
Worklog Time Spent: 10m
Work Description: pupamanyu commented on a change in pull request #5141:
[BEAM-3925] Allow ValueProvider for KafkaIO so that we can create Beam
Templates using KafkaIO
URL: https://github.com/apache/beam/pull/5141#discussion_r183846804
##########
File path:
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
##########
@@ -359,20 +386,129 @@
* of how the partitions are distributed among the splits.
*/
public Read<K, V> withTopicPartitions(List<TopicPartition>
topicPartitions) {
- checkState(getTopics().isEmpty(), "Only topics or topicPartitions can be
set, not both");
- return
toBuilder().setTopicPartitions(ImmutableList.copyOf(topicPartitions)).build();
+ return withTopicPartitions(
+
ValueProvider.StaticValueProvider.of(Joiner.on(',').join(topicPartitions)));
+ }
+
+ /**
+ * Sets a list of partitions to read from. Partitions are provided as a
comma separated list of
+ * Strings in the format: topic1-partition1,topic1-partition2...
+ * This allows reading only a subset of partitions for one or more topics
when (if ever) needed.
+ *
+ * <p>See {@link KafkaUnboundedSource#split(int, PipelineOptions)} for
description
+ * of how the partitions are distributed among the splits.
+ */
+ public Read<K, V> withTopicPartitions(String topicPartitions) {
Review comment:
I have removed the code which handles the parsing of String representation
of Topic Partition for now and added
withTopoicPartitions(ValueProvider<List<TopicPartition>> partitions) so that we
can use this for templates
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 94761)
Time Spent: 2h 10m (was: 2h)
> Allow ValueProvider for KafkaIO
> -------------------------------
>
> Key: BEAM-3925
> URL: https://issues.apache.org/jira/browse/BEAM-3925
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
> Reporter: Sameer Abhyankar
> Assignee: Pramod Upamanyu
> Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Add ValueProvider support for the various methods in KafkaIO. This would
> allow us to use KafkaIO in reusable pipeline templates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
