[
https://issues.apache.org/jira/browse/GOBBLIN-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanghang Liu closed GOBBLIN-1740.
---------------------------------
Resolution: Duplicate
> Split Workunit for Kafka Streaming jobs
> ---------------------------------------
>
> Key: GOBBLIN-1740
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1740
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-kafka
> Reporter: Hanghang Liu
> Assignee: Shirshanka Das
> Priority: Major
>
> When adding task from a running Helix streaming job, we need the capability
> to split an existing workunit.
> We can reuse most of the attributes within the old workunit, and just need to
> update following props:
> task.id
> writer.output.dir
> partition.id
> gobblin.kafka.streaming.numPartitions
> Since we get the partition watermark from state store, so we don't need to
> re-calculate that within the workunit. All kafka related properties can also
> be reused.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
