[ 
https://issues.apache.org/jira/browse/KAFKA-21525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Holzbach resolved KAFKA-21525.
-------------------------------------
    Resolution: Duplicate

Duplicate of https://issues.apache.org/jira/browse/KAFKA-17719

> Connect tasks fail on startup with NullPointerException after a resubmitted 
> connector config is compacted out of order in the config topic
> ------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-21525
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21525
>             Project: Kafka
>          Issue Type: Bug
>          Components: connect
>            Reporter: Andrew Holzbach
>            Assignee: Andrew Holzbach
>            Priority: Minor
>
> When a connector's config is written to the config topic again with no change 
> to its task configs, the latest connector config ends up _after_ that 
> connector's latest task configs and task commit record. This happens, for 
> example, when the exact same config is sent again to \{{PUT 
> /connectors/\{name\}/config}}. The connector produces identical task configs 
> and the applied connector config matches the current one, so 
> \{{AbstractHerder::taskConfigsChanged}} returns false. As a result, no new 
> task configs or commit record are written after the new connector config. 
> Once log compaction removes the older, superseded connector config record, 
> the commit record is the first record a worker reads for that connector when 
> it replays the topic.
> When \{{KafkaConfigBackingStore::processTasksCommitRecord}} reads a commit 
> for a connector it has no config for, it assumes the connector was deleted 
> and that compaction has removed both its config and its tombstone. It calls 
> \{{processConnectorRemoval}} and throws away the deferred task configs. The 
> connector config record arrives right after, so the worker ends up with a 
> connector config but no task configs. Other workers read the topic before 
> compaction and still have the full state, including the leader. The leader 
> keeps assigning that connector's tasks, and any worker that rebuilt its state 
> from the compacted topic fails to start them: 
> \{{configState.taskConfig(taskId)}} returns \{{null}} and startup throws an 
> NPE. Workers that started before compaction, and the worker running the 
> connector, are not affected. The worker running the connector sees a task 
> count mismatch and republishes, which fixes the problem. That makes the 
> failure intermittent and hard to diagnose. Tasks that fail during startup 
> also don't show up in the usual failed-task metrics.
> Example config topic contents for connector \{{foo}} before compaction:
> {noformat}
> offset  key             value
> 0       connector-foo   \{"connector.class": "...", "tasks.max": "2", ...}
> 1       task-foo-0      \{...}
> 2       task-foo-1      \{...}
> 3       commit-foo      \{"tasks": 2}
> 4       connector-foo   \{"connector.class": "...", "tasks.max": "2", ...}   
> <- identical config resubmitted; no new task configs/commit follow
> {noformat}
> After compaction removes offset 0, a worker that reads the topic from the 
> start sees:
> {noformat}
> 1       task-foo-0      \{...}            -> deferred
> 2       task-foo-1      \{...}            -> deferred
> 3       commit-foo      \{"tasks": 2}     -> no config for "foo" yet; treated 
> as deleted, task configs discarded
> 4       connector-foo   \{...}            -> connector config applied, but 
> task count is now 0
> {noformat}
> *Steps to reproduce:*
> # Create a connector that generates at least one task.
> # Send the identical connector config to the REST API again, so it is written 
> after the task configs.
> # Lower {{[segment.ms|http://segment.ms/]}} (e.g. 60000) and 
> \{{min.cleanable.dirty.ratio}} (e.g. 0.01) on the config topic so compaction 
> runs aggressively.
> # Wait for the segment to roll, then write an unrelated record to the config 
> topic so the older segment becomes eligible for compaction.
> # Restart the workers running the connector's tasks, but not the worker 
> running the connector itself. The tasks fail on startup with an NPE.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to