[
https://issues.apache.org/jira/browse/KAFKA-21525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Holzbach resolved KAFKA-21525.
-------------------------------------
Resolution: Duplicate
Duplicate of https://issues.apache.org/jira/browse/KAFKA-17719
> Connect tasks fail on startup with NullPointerException after a resubmitted
> connector config is compacted out of order in the config topic
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-21525
> URL: https://issues.apache.org/jira/browse/KAFKA-21525
> Project: Kafka
> Issue Type: Bug
> Components: connect
> Reporter: Andrew Holzbach
> Assignee: Andrew Holzbach
> Priority: Minor
>
> When a connector's config is written to the config topic again with no change
> to its task configs, the latest connector config ends up _after_ that
> connector's latest task configs and task commit record. This happens, for
> example, when the exact same config is sent again to \{{PUT
> /connectors/\{name\}/config}}. The connector produces identical task configs
> and the applied connector config matches the current one, so
> \{{AbstractHerder::taskConfigsChanged}} returns false. As a result, no new
> task configs or commit record are written after the new connector config.
> Once log compaction removes the older, superseded connector config record,
> the commit record is the first record a worker reads for that connector when
> it replays the topic.
> When \{{KafkaConfigBackingStore::processTasksCommitRecord}} reads a commit
> for a connector it has no config for, it assumes the connector was deleted
> and that compaction has removed both its config and its tombstone. It calls
> \{{processConnectorRemoval}} and throws away the deferred task configs. The
> connector config record arrives right after, so the worker ends up with a
> connector config but no task configs. Other workers read the topic before
> compaction and still have the full state, including the leader. The leader
> keeps assigning that connector's tasks, and any worker that rebuilt its state
> from the compacted topic fails to start them:
> \{{configState.taskConfig(taskId)}} returns \{{null}} and startup throws an
> NPE. Workers that started before compaction, and the worker running the
> connector, are not affected. The worker running the connector sees a task
> count mismatch and republishes, which fixes the problem. That makes the
> failure intermittent and hard to diagnose. Tasks that fail during startup
> also don't show up in the usual failed-task metrics.
> Example config topic contents for connector \{{foo}} before compaction:
> {noformat}
> offset key value
> 0 connector-foo \{"connector.class": "...", "tasks.max": "2", ...}
> 1 task-foo-0 \{...}
> 2 task-foo-1 \{...}
> 3 commit-foo \{"tasks": 2}
> 4 connector-foo \{"connector.class": "...", "tasks.max": "2", ...}
> <- identical config resubmitted; no new task configs/commit follow
> {noformat}
> After compaction removes offset 0, a worker that reads the topic from the
> start sees:
> {noformat}
> 1 task-foo-0 \{...} -> deferred
> 2 task-foo-1 \{...} -> deferred
> 3 commit-foo \{"tasks": 2} -> no config for "foo" yet; treated
> as deleted, task configs discarded
> 4 connector-foo \{...} -> connector config applied, but
> task count is now 0
> {noformat}
> *Steps to reproduce:*
> # Create a connector that generates at least one task.
> # Send the identical connector config to the REST API again, so it is written
> after the task configs.
> # Lower {{[segment.ms|http://segment.ms/]}} (e.g. 60000) and
> \{{min.cleanable.dirty.ratio}} (e.g. 0.01) on the config topic so compaction
> runs aggressively.
> # Wait for the segment to roll, then write an unrelated record to the config
> topic so the older segment becomes eligible for compaction.
> # Restart the workers running the connector's tasks, but not the worker
> running the connector itself. The tasks fail on startup with an NPE.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)