Andrew Holzbach created KAFKA-21525:
---------------------------------------
Summary: Connect tasks fail on startup with NullPointerException
after a resubmitted connector config is compacted out of order in the config
topic
Key: KAFKA-21525
URL: https://issues.apache.org/jira/browse/KAFKA-21525
Project: Kafka
Issue Type: Bug
Components: connect
Reporter: Andrew Holzbach
Assignee: Andrew Holzbach
When a connector's config is written to the config topic again with no change
to its task configs, the latest connector config ends up _after_ that
connector's latest task configs and task commit record. This happens, for
example, when the exact same config is sent again to \{{PUT
/connectors/\{name\}/config}}. The connector produces identical task configs
and the applied connector config matches the current one, so
\{{AbstractHerder::taskConfigsChanged}} returns false. As a result, no new task
configs or commit record are written after the new connector config. Once log
compaction removes the older, superseded connector config record, the commit
record is the first record a worker reads for that connector when it replays
the topic.
When \{{KafkaConfigBackingStore::processTasksCommitRecord}} reads a commit for
a connector it has no config for, it assumes the connector was deleted and that
compaction has removed both its config and its tombstone. It calls
\{{processConnectorRemoval}} and throws away the deferred task configs. The
connector config record arrives right after, so the worker ends up with a
connector config but no task configs. Other workers read the topic before
compaction and still have the full state, including the leader. The leader
keeps assigning that connector's tasks, and any worker that rebuilt its state
from the compacted topic fails to start them:
\{{configState.taskConfig(taskId)}} returns \{{null}} and startup throws an
NPE. Workers that started before compaction, and the worker running the
connector, are not affected. The worker running the connector sees a task count
mismatch and republishes, which fixes the problem. That makes the failure
intermittent and hard to diagnose. Tasks that fail during startup also don't
show up in the usual failed-task metrics.
Example config topic contents for connector \{{foo}} before compaction:
{noformat}
offset key value
0 connector-foo \{"connector.class": "...", "tasks.max": "2", ...}
1 task-foo-0 \{...}
2 task-foo-1 \{...}
3 commit-foo \{"tasks": 2}
4 connector-foo \{"connector.class": "...", "tasks.max": "2", ...} <-
identical config resubmitted; no new task configs/commit follow
{noformat}
After compaction removes offset 0, a worker that reads the topic from the start
sees:
{noformat}
1 task-foo-0 \{...} -> deferred
2 task-foo-1 \{...} -> deferred
3 commit-foo \{"tasks": 2} -> no config for "foo" yet; treated
as deleted, task configs discarded
4 connector-foo \{...} -> connector config applied, but task
count is now 0
{noformat}
*Steps to reproduce:*
# Create a connector that generates at least one task.
# Send the identical connector config to the REST API again, so it is written
after the task configs.
# Lower {{[segment.ms|http://segment.ms/]}} (e.g. 60000) and
\{{min.cleanable.dirty.ratio}} (e.g. 0.01) on the config topic so compaction
runs aggressively.
# Wait for the segment to roll, then write an unrelated record to the config
topic so the older segment becomes eligible for compaction.
# Restart the workers running the connector's tasks, but not the worker running
the connector itself. The tasks fail on startup with an NPE.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)