[
https://issues.apache.org/jira/browse/KAFKA-9962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jose Armando Garcia Sancio updated KAFKA-9962:
----------------------------------------------
Description:
Users are getting this error when using a client version 2.5 against a 1.1.0
cluster/broker.
{code:java}
[2020-04-28 01:09:10,663] ERROR Failed to start KSQL
(io.confluent.ksql.rest.server.KsqlServerMain:63)
io.confluent.ksql.util.KsqlServerException: Could not get Kafka authorized
operations!
at
io.confluent.ksql.services.KafkaClusterUtil.isAuthorizedOperationsSupported(KafkaClusterUtil.java:51)
at
io.confluent.ksql.security.KsqlAuthorizationValidatorFactory.create(KsqlAuthorizationValidatorFactory.java:52)
at
io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:639)
at
io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:567)
at
io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:100)
at
io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:59)
Caused by: java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to write
a non-default includeClusterAuthorizedOperations at version 5
at
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at
io.confluent.ksql.services.KafkaClusterUtil.isAuthorizedOperationsSupported(KafkaClusterUtil.java:49)
... 5 more
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
Attempted to write a non-default includeClusterAuthorizedOperations at version 5
{code}
Looking at KIP-430, it mentions that the client is supposed to handle this case:
# Existing clients using older versions will not request authorized operations
in Describe requests since the default is to disable this feature. This keeps
older clients compatible with newer brokers.
# Newer clients connecting to older brokers will use the older protocol
version and hence will not request authorized operations.
# When the AdminClient is talking to a broker which does not support KIP-430,
it will fill in either null or UnsupportedVersionException for the returned ACL
operations fields in objects. For example,
`ConsumerGroupDescription#authorizedOperations` will be null if the broker did
not supply this information. DescribeClusterResult#authorizedOperations will
throw an `UnsupportedVersionException` if the broker did not supply this
information.
# When new operations are added, newer brokers may return operations that are
not known to older clients. AdminClient will ignore any bit that is set in
authorized_operations that is not known to the client. The Set<AclOperation>
created by the client from the bits returned by the broker will only include
operations that the client client knows about.
I assume that this deployment environment falls under case 2, we have this in
the serialization code:
{code:java}
if (_version >= 8) {
_writable.writeByte(includeClusterAuthorizedOperations ? (byte) 1 :
(byte) 0);
} else {
if (includeClusterAuthorizedOperations) {
throw new UnsupportedVersionException("Attempted to write a
non-default includeClusterAuthorizedOperations at version " + _version);
}
}
{code}
I also looks like we blindly set the version independent of the Broker’s
supported version:
{code:java}
MetadataRequest.Builder createRequest(int timeoutMs) {
// Since this only requests node information, it's safe to pass
true for allowAutoTopicCreation (and it
// simplifies communication with older brokers)
return new MetadataRequest.Builder(new MetadataRequestData()
.setTopics(Collections.emptyList())
.setAllowAutoTopicCreation(true)
.setIncludeClusterAuthorizedOperations(options.includeAuthorizedOperations()));
}
{code}
To implement 2. we need to make these properties ignorable.
was:
Users are getting this error when using a client version `2.5` against a
`1.1.0` cluster/broker.
{code}
[2020-04-28 01:09:10,663] ERROR Failed to start KSQL
(io.confluent.ksql.rest.server.KsqlServerMain:63)
io.confluent.ksql.util.KsqlServerException: Could not get Kafka authorized
operations!
at
io.confluent.ksql.services.KafkaClusterUtil.isAuthorizedOperationsSupported(KafkaClusterUtil.java:51)
at
io.confluent.ksql.security.KsqlAuthorizationValidatorFactory.create(KsqlAuthorizationValidatorFactory.java:52)
at
io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:639)
at
io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:567)
at
io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:100)
at
io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:59)
Caused by: java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to write
a non-default includeClusterAuthorizedOperations at version 5
at
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at
io.confluent.ksql.services.KafkaClusterUtil.isAuthorizedOperationsSupported(KafkaClusterUtil.java:49)
... 5 more
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
Attempted to write a non-default includeClusterAuthorizedOperations at version 5
{code}
Looking at `KIP-430`, it mentions that the client is supposed to handle this
case:
# Existing clients using older versions will not request authorized operations
in Describe requests since the default is to disable this feature. This keeps
older clients compatible with newer brokers.
# Newer clients connecting to older brokers will use the older protocol version
and hence will not request authorized operations.
# When the AdminClient is talking to a broker which does not support KIP-430,
it will fill in either null or UnsupportedVersionException for the returned ACL
operations fields in objects. For example,
`ConsumerGroupDescription#authorizedOperations` will be null if the broker did
not supply this information. DescribeClusterResult#authorizedOperations will
throw an `UnsupportedVersionException` if the broker did not supply this
information.
# When new operations are added, newer brokers may return operations that are
not known to older clients. AdminClient will ignore any bit that is set in
authorized_operations that is not known to the client. The Set<AclOperation>
created by the client from the bits returned by the broker will only include
operations that the client client knows about.
I assume that this deployment environment falls under case 2, we have this in
the serialization code:
{code}
if (_version >= 8) {
_writable.writeByte(includeClusterAuthorizedOperations ? (byte) 1 :
(byte) 0);
} else {
if (includeClusterAuthorizedOperations) {
throw new UnsupportedVersionException("Attempted to write a
non-default includeClusterAuthorizedOperations at version " + _version);
}
}
{code}
I also looks like we blindly set the version independent of the Broker’s
supported version:
{code}
MetadataRequest.Builder createRequest(int timeoutMs) {
// Since this only requests node information, it's safe to pass
true for allowAutoTopicCreation (and it
// simplifies communication with older brokers)
return new MetadataRequest.Builder(new MetadataRequestData()
.setTopics(Collections.emptyList())
.setAllowAutoTopicCreation(true)
.setIncludeClusterAuthorizedOperations(options.includeAuthorizedOperations()));
}
{code}
To implement 2. we need to make these properties ignorable.
> Admin client throws UnsupportedVersion exception when talking to old broker
> ---------------------------------------------------------------------------
>
> Key: KAFKA-9962
> URL: https://issues.apache.org/jira/browse/KAFKA-9962
> Project: Kafka
> Issue Type: Task
> Components: clients
> Affects Versions: 2.3.1, 2.5.0, 2.4.1
> Reporter: Jose Armando Garcia Sancio
> Assignee: Jose Armando Garcia Sancio
> Priority: Critical
>
> Users are getting this error when using a client version 2.5 against a 1.1.0
> cluster/broker.
> {code:java}
> [2020-04-28 01:09:10,663] ERROR Failed to start KSQL
> (io.confluent.ksql.rest.server.KsqlServerMain:63)
> io.confluent.ksql.util.KsqlServerException: Could not get Kafka authorized
> operations!
> at
> io.confluent.ksql.services.KafkaClusterUtil.isAuthorizedOperationsSupported(KafkaClusterUtil.java:51)
> at
> io.confluent.ksql.security.KsqlAuthorizationValidatorFactory.create(KsqlAuthorizationValidatorFactory.java:52)
> at
> io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:639)
> at
> io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:567)
> at
> io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:100)
> at
> io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:59)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to
> write a non-default includeClusterAuthorizedOperations at version 5
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> io.confluent.ksql.services.KafkaClusterUtil.isAuthorizedOperationsSupported(KafkaClusterUtil.java:49)
> ... 5 more
> Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
> Attempted to write a non-default includeClusterAuthorizedOperations at
> version 5
> {code}
> Looking at KIP-430, it mentions that the client is supposed to handle this
> case:
> # Existing clients using older versions will not request authorized
> operations in Describe requests since the default is to disable this feature.
> This keeps older clients compatible with newer brokers.
> # Newer clients connecting to older brokers will use the older protocol
> version and hence will not request authorized operations.
> # When the AdminClient is talking to a broker which does not support
> KIP-430, it will fill in either null or UnsupportedVersionException for the
> returned ACL operations fields in objects. For example,
> `ConsumerGroupDescription#authorizedOperations` will be null if the broker
> did not supply this information. DescribeClusterResult#authorizedOperations
> will throw an `UnsupportedVersionException` if the broker did not supply this
> information.
> # When new operations are added, newer brokers may return operations that
> are not known to older clients. AdminClient will ignore any bit that is set
> in authorized_operations that is not known to the client. The
> Set<AclOperation> created by the client from the bits returned by the broker
> will only include operations that the client client knows about.
> I assume that this deployment environment falls under case 2, we have this in
> the serialization code:
> {code:java}
> if (_version >= 8) {
> _writable.writeByte(includeClusterAuthorizedOperations ? (byte) 1
> : (byte) 0);
> } else {
> if (includeClusterAuthorizedOperations) {
> throw new UnsupportedVersionException("Attempted to write a
> non-default includeClusterAuthorizedOperations at version " + _version);
> }
> }
> {code}
> I also looks like we blindly set the version independent of the Broker’s
> supported version:
> {code:java}
> MetadataRequest.Builder createRequest(int timeoutMs) {
> // Since this only requests node information, it's safe to
> pass true for allowAutoTopicCreation (and it
> // simplifies communication with older brokers)
> return new MetadataRequest.Builder(new MetadataRequestData()
> .setTopics(Collections.emptyList())
> .setAllowAutoTopicCreation(true)
>
> .setIncludeClusterAuthorizedOperations(options.includeAuthorizedOperations()));
> }
> {code}
> To implement 2. we need to make these properties ignorable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
