[jira] [Resolved] (KAFKA-13165) Validate node id, process role and quorum voters
[
https://issues.apache.org/jira/browse/KAFKA-13165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantine Karantasis resolved KAFKA-13165.
Resolution: Fixed
> Validate node id, process role and quorum voters
>
>
> Key: KAFKA-13165
> URL: https://issues.apache.org/jira/browse/KAFKA-13165
> Project: Kafka
> Issue Type: Sub-task
> Components: kraft
>Reporter: Jose Armando Garcia Sancio
>Assignee: Ryan Dielhenn
>Priority: Blocker
> Labels: kip-500
> Fix For: 3.0.0
>
>
> Under certain configuration is possible for the Kafka Server to boot up as a
> broker only but be the cluster metadata quorum leader. We should validate the
> configuration to avoid this case.
> # If the {{process.roles}} contains {{controller}} then the {{node.id}}
> needs to be in the {{controller.quorum.voters}}
> # If the {{process.roles}} doesn't contain {{controller}} then the
> {{node.id}} cannot be in the {{controller.quorum.voters}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-13194) LogCleaner may clean past highwatermark
Lucas Bradstreet created KAFKA-13194:
Summary: LogCleaner may clean past highwatermark
Key: KAFKA-13194
URL: https://issues.apache.org/jira/browse/KAFKA-13194
Project: Kafka
Issue Type: Bug
Reporter: Lucas Bradstreet
Here we have the cleaning point being bounded to the active segment base offset
and the first unstable offset. Which makes sense:
// find first segment that cannot be cleaned
// neither the active segment, nor segments with any messages closer to the
head of the log than the minimum compaction lag time
// may be cleaned
val firstUncleanableDirtyOffset: Long = Seq(
// we do not clean beyond the first unstable offset
log.firstUnstableOffset,
// the active segment is always uncleanable
Option(log.activeSegment.baseOffset),
// the first segment whose largest message timestamp is within a minimum
time lag from now
if (minCompactionLagMs > 0) \{
// dirty log segments
val dirtyNonActiveSegments =
log.localNonActiveLogSegmentsFrom(firstDirtyOffset)
dirtyNonActiveSegments.find { s =>
val isUncleanable = s.largestTimestamp > now - minCompactionLagMs
debug(s"Checking if log segment may be cleaned: log='${log.name}'
segment.baseOffset=${s.baseOffset} " +
s"segment.largestTimestamp=${s.largestTimestamp}; now -
compactionLag=${now - minCompactionLagMs}; " +
s"is uncleanable=$isUncleanable")
isUncleanable
}.map(_.baseOffset)
} else None
).flatten.min
But LSO starts out as None.
@volatile private var firstUnstableOffsetMetadata: Option[LogOffsetMetadata]
= None
private[log] def firstUnstableOffset: Option[Long] =
firstUnstableOffsetMetadata.map(_.messageOffset)
For most code depending on the LSO, fetchLastStableOffsetMetadata is used to
default it to the hwm if it's not set.
private def fetchLastStableOffsetMetadata: LogOffsetMetadata = \{
checkIfMemoryMappedBufferClosed()
// cache the current high watermark to avoid a concurrent update
invalidating the range check
val highWatermarkMetadata = fetchHighWatermarkMetadata
firstUnstableOffsetMetadata match {
case Some(offsetMetadata) if offsetMetadata.messageOffset <
highWatermarkMetadata.messageOffset =>
if (offsetMetadata.messageOffsetOnly) {
lock synchronized {
val fullOffset =
convertToOffsetMetadataOrThrow(offsetMetadata.messageOffset)
if (firstUnstableOffsetMetadata.contains(offsetMetadata))
firstUnstableOffsetMetadata = Some(fullOffset)
fullOffset
}
} else \{
offsetMetadata
}
case _ => highWatermarkMetadata
}
}
This means that in the case where the hwm is prior to the active segment base,
the log cleaner may clean past the hwm. This is most likely to occur after a
broker restart when the log cleaner may start cleaning prior to replication
becoming active.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-13193) Replica manager doesn't update partition state when transitioning from leader to follower with unknown leader
Jose Armando Garcia Sancio created KAFKA-13193: -- Summary: Replica manager doesn't update partition state when transitioning from leader to follower with unknown leader Key: KAFKA-13193 URL: https://issues.apache.org/jira/browse/KAFKA-13193 Project: Kafka Issue Type: Bug Components: kraft, replication Reporter: Jose Armando Garcia Sancio Assignee: Jose Armando Garcia Sancio This issue applies to both the ZK and KRaft implementation of the replica manager. In the rare case when a replica transition from leader to follower with no leader the partition state is not updated. This is because when handling makeFollowers the ReplicaManager only updates the partition state if the leader is alive. The solution is to always transition to follower but not start the fetcher thread if the leader is unknown or not alive. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-13192) broker.id and node.id can be specified inconsistently
Ron Dagostino created KAFKA-13192: - Summary: broker.id and node.id can be specified inconsistently Key: KAFKA-13192 URL: https://issues.apache.org/jira/browse/KAFKA-13192 Project: Kafka Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ron Dagostino If both broker.id and node.id are set, and they are set inconsistently (e.g.broker.id=0, node.id=1) then the value of node.id is used and the broker.id value is left at the original value. The server should detect this inconsistency, throw a ConfigException, and fail to start. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-13191) Kafka 2.8 - simultaneous restarts of Kafka and zookeeper result in broken cluster
CS User created KAFKA-13191:
---
Summary: Kafka 2.8 - simultaneous restarts of Kafka and zookeeper
result in broken cluster
Key: KAFKA-13191
URL: https://issues.apache.org/jira/browse/KAFKA-13191
Project: Kafka
Issue Type: Bug
Components: protocol
Affects Versions: 2.8.0
Reporter: CS User
We're using confluent platform 6.2, running in a Kubernetes environment. The
cluster has been running for a couple of years with zero issues, starting from
version 1.1, 2.5 and now 2.8.
We've very recently upgraded to kafka 2.8 from kafka 2.5.
Since upgraded, we have seen issues when kafka and zookeeper pods restart
concurrently.
We can replicate the issue by restarting either the zookeeper statefulset first
or the kafka statefulset first, either way appears to result with the same
failure scenario.
We've attempted to mitigate by preventing the kafka pods from stopping if any
zookeeper pods are being restarted, or a rolling restart of the zookeeper
cluster is underway.
We've also added a check to stop the kafka pods from starting until all
zookeeper pods are ready, however under the following scenario we still see the
issue:
In a 3 node kafka cluster with 5 zookeeper servers
# kafka-2 starts to terminate - all zookeeper pods are running, so it proceeds
# zookeeper-4 terminates
# kafka-2 starts-up, and waits until the zookeeper rollout completes
# kafka-2 eventually fully starts, kafka comes up and we see the errors below
on other pods in the cluster.
Without mitigation and in the above scenario we see errors on pods
kafka-0/kafka-1:
{noformat}
Broker had a stale broker epoch (635655171205), retrying.
(kafka.server.DefaultAlterIsrManager) {noformat}
This never appears to recover.
If you then restart kafka-2, you'll see these errors:
{noformat}
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication
factor: 3 larger than available brokers: 0. {noformat}
This seems to completely break the cluster, partitions do not failover as
expected.
Checking zookeeper, and getting the values of the brokers look fine
{noformat}
get /brokers/ids/0{noformat}
etc, all looks fine there, each broker is present.
This error message appears to have been added to kafka in the last 11 months
{noformat}
Broker had a stale broker epoch {noformat}
Via this PR:
[https://github.com/apache/kafka/pull/9100]
I see also this comment around the leader getting stuck:
https://github.com/apache/kafka/pull/9100/files#r494480847
Recovery is possible by continuing to restart the remaining brokers in the
cluster. Once all have been restarted, everything looks fine.
Has anyone else come across this? It seems very simple to replicate in our
environment, simply start a simultaneous rolling restart of both kafka and
zookeeper.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
