[
https://issues.apache.org/jira/browse/GOBBLIN-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthew Ho updated GOBBLIN-1687:
--------------------------------
Description:
When a workunit is computed, it is currently static and requires a restart of
the pipeline to be changed. This means the system cannot handle spikes in
traffic without a restart. This feature allows the task runners to send
messages when there is a spike in traffic.
Currently, the behavior is for the container to restart, but it doesn't change
the kafka topic partitions it is currently ingesting. New desired behavior is
for the producer to send a message when it is lagging
> Helix Dynamic Work Unit Change Event Producer Implementation
> ------------------------------------------------------------
>
> Key: GOBBLIN-1687
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1687
> Project: Apache Gobblin
> Issue Type: New Feature
> Reporter: Matthew Ho
> Priority: Major
>
> When a workunit is computed, it is currently static and requires a restart of
> the pipeline to be changed. This means the system cannot handle spikes in
> traffic without a restart. This feature allows the task runners to send
> messages when there is a spike in traffic.
> Currently, the behavior is for the container to restart, but it doesn't
> change the kafka topic partitions it is currently ingesting. New desired
> behavior is for the producer to send a message when it is lagging
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
