Andrew Holzbach created KAFKA-21525:
---------------------------------------

             Summary: Connect tasks fail on startup with NullPointerException 
after a resubmitted connector config is compacted out of order in the config 
topic
                 Key: KAFKA-21525
                 URL: https://issues.apache.org/jira/browse/KAFKA-21525
             Project: Kafka
          Issue Type: Bug
          Components: connect
            Reporter: Andrew Holzbach
            Assignee: Andrew Holzbach


When a connector's config is written to the config topic again with no change 
to its task configs, the latest connector config ends up _after_ that 
connector's latest task configs and task commit record. This happens, for 
example, when the exact same config is sent again to \{{PUT 
/connectors/\{name\}/config}}. The connector produces identical task configs 
and the applied connector config matches the current one, so 
\{{AbstractHerder::taskConfigsChanged}} returns false. As a result, no new task 
configs or commit record are written after the new connector config. Once log 
compaction removes the older, superseded connector config record, the commit 
record is the first record a worker reads for that connector when it replays 
the topic.

When \{{KafkaConfigBackingStore::processTasksCommitRecord}} reads a commit for 
a connector it has no config for, it assumes the connector was deleted and that 
compaction has removed both its config and its tombstone. It calls 
\{{processConnectorRemoval}} and throws away the deferred task configs. The 
connector config record arrives right after, so the worker ends up with a 
connector config but no task configs. Other workers read the topic before 
compaction and still have the full state, including the leader. The leader 
keeps assigning that connector's tasks, and any worker that rebuilt its state 
from the compacted topic fails to start them: 
\{{configState.taskConfig(taskId)}} returns \{{null}} and startup throws an 
NPE. Workers that started before compaction, and the worker running the 
connector, are not affected. The worker running the connector sees a task count 
mismatch and republishes, which fixes the problem. That makes the failure 
intermittent and hard to diagnose. Tasks that fail during startup also don't 
show up in the usual failed-task metrics.

Example config topic contents for connector \{{foo}} before compaction:

{noformat}
offset  key             value
0       connector-foo   \{"connector.class": "...", "tasks.max": "2", ...}
1       task-foo-0      \{...}
2       task-foo-1      \{...}
3       commit-foo      \{"tasks": 2}
4       connector-foo   \{"connector.class": "...", "tasks.max": "2", ...}   <- 
identical config resubmitted; no new task configs/commit follow
{noformat}

After compaction removes offset 0, a worker that reads the topic from the start 
sees:

{noformat}
1       task-foo-0      \{...}            -> deferred
2       task-foo-1      \{...}            -> deferred
3       commit-foo      \{"tasks": 2}     -> no config for "foo" yet; treated 
as deleted, task configs discarded
4       connector-foo   \{...}            -> connector config applied, but task 
count is now 0
{noformat}

*Steps to reproduce:*
# Create a connector that generates at least one task.
# Send the identical connector config to the REST API again, so it is written 
after the task configs.
# Lower {{[segment.ms|http://segment.ms/]}} (e.g. 60000) and 
\{{min.cleanable.dirty.ratio}} (e.g. 0.01) on the config topic so compaction 
runs aggressively.
# Wait for the segment to roll, then write an unrelated record to the config 
topic so the older segment becomes eligible for compaction.
# Restart the workers running the connector's tasks, but not the worker running 
the connector itself. The tasks fail on startup with an NPE.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to