[
https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=95798&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95798
]
ASF GitHub Bot logged work on BEAM-3925:
----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Apr/18 00:14
Start Date: 27/Apr/18 00:14
Worklog Time Spent: 10m
Work Description: pupamanyu commented on issue #5141: [BEAM-3925] Allow
ValueProvider for KafkaIO so that we can create Beam Templates using KafkaIO
URL: https://github.com/apache/beam/pull/5141#issuecomment-384825979
I think it is good idea to try this out. I was planning to do this once the
review is completed. With no significant changes pending, I will try the
template.
On Thu, Apr 26, 2018 at 3:09 PM, Raghu Angadi <[email protected]>
wrote:
> Let's step back a bit and see how a KafkaIO source would work in a
> template:
> AFAIK, the driver part of launching a Dataflow templated pipeline runs
> only once while installing the template. Each run of a template executes a
> serialized job that was stored and does not run the driver that builds the
> graph of computations.
>
> This does not work well for Beam unbounded sources. E.g. split()1
>
<https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java#L68>
> on an unbounded source like KafkaIO is invoked on the driver, this also
> decides parallelism. In KafkaIO, how many splits should we return? One
> option is to just return a fixed number like 100 during template
> preparation, and distribute the actual partitions among these 100 at
> template runtime.
>
> In order to iron out such issues early, it might be a good idea to try a
> template before finalizing this PR. WDYT?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/5141#issuecomment-384804212>, or mute
> the thread
>
<https://github.com/notifications/unsubscribe-auth/AB2QCLSNGqv1bmKTO0pL81Ui0EO_WhItks5tskV8gaJpZM4TXOv5>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 95798)
Time Spent: 5h 40m (was: 5.5h)
> Allow ValueProvider for KafkaIO
> -------------------------------
>
> Key: BEAM-3925
> URL: https://issues.apache.org/jira/browse/BEAM-3925
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
> Reporter: Sameer Abhyankar
> Assignee: Pramod Upamanyu
> Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> Add ValueProvider support for the various methods in KafkaIO. This would
> allow us to use KafkaIO in reusable pipeline templates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
