[
https://issues.apache.org/jira/browse/KAFKA-17490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jirar updated KAFKA-17490:
--------------------------
Description:
I build a kafka cluster with kafka-2.8.1, use ranger as the authorization
platform. After the installation, when I firstly start the cluster, and then I
got ClusterAuthorizationException in kafka-server log files. In the same time,
SocketServer was started, but we cannot create topic with kafka-topics.sh tool,
at last, the creation would get an timeout exception.
There are some important logs bellow
[2024-08-31 15:38:04,904] [INFO ] [controller-event-thread:2605]
[kafka.controller.KafkaController] [Logging.scala:66] - [Controller id=1] 1
successfully elected as the controller. Epoch incremented to 1 and epoch zk
version is now
[2024-08-31 15:38:05,540] [INFO ] [main:3241]
[org.apache.ranger.plugin.service.RangerBasePlugin] [RangerBasePlugin.java:227]
- Created PolicyRefresher
Thread(PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1)-66)
// load roles to local file
[2024-08-31 15:38:06,056] [INFO ] [main:3757]
[org.apache.ranger.plugin.util.RangerRolesProvider]
[RangerRolesProvider.java:190] -
RangerRolesProvider(serviceName=kafka-abcd-rmgwrqt1): found updated version.
lastKnownRoleVersion=-1; newVersion=1
[2024-08-31 15:38:06,108] [WARN ] [main:3809]
[org.apache.ranger.plugin.util.PolicyRefresher] [PolicyRefresher.java:393] -
cache file does not exist or not readable
'/data/emr/kafka/policycache/kafka_kafka-abcd-rmgwrqt1.json'
[2024-08-31 15:38:06,411] [ERROR] [data-plane-kafka-request-handler-0:4112]
[kafka.server.RequestHandlerHelper] [Logging.scala:76] - [KafkaApi-1] Error
when handling request: clientId=1, correlationId=0, api=UPDATE_METADATA,
version=7, body=UpdateMetadataRequestData(controllerId=1, controllerEpoch=1,
brokerEpoch=4294967593, ungroupedPartitionStates=[], topicStates=[],
liveBrokers=[UpdateMetadataBroker(id=2, v0Host='', v0Port=0,
endpoints=[UpdateMetadataEndpoint(port=9092, host='10.0.0.22',
listener='SASL_PLAINTEXT', securityProtocol=2)], rack=null),
UpdateMetadataBroker(id=3, v0Host='', v0Port=0,
endpoints=[UpdateMetadataEndpoint(port=9092, host='10.0.0.29',
listener='SASL_PLAINTEXT', securityProtocol=2)], rack=null),
UpdateMetadataBroker(id=1, v0Host='', v0Port=0,
endpoints=[UpdateMetadataEndpoint(port=9092, host='10.0.0.45',
listener='SASL_PLAINTEXT', securityProtocol=2)], rack=null)])
org.apache.kafka.common.errors.ClusterAuthorizationException: Request
Request(processor=0, connectionId=10.0.0.45:9092-10.0.0.45:34308-0,
session=Session(User:hadoop,/10.0.0.45),
listenerName=ListenerName(SASL_PLAINTEXT), securityProtocol=SASL_PLAINTEXT,
buffer=null, envelope=None) is not authorized.
// load policy to local file
[2024-08-31 15:38:09,405] [INFO ]
[PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1)-66:7106]
[org.apache.ranger.plugin.util.PolicyRefresher] [PolicyRefresher.java:321] -
PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1): found updated version.
lastKnownVersion=-1; newVersion=6
[2024-08-31 15:38:09,406] [INFO ]
[PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1)-66:7107]
[org.apache.ranger.plugin.policyengine.PolicyEngine] [PolicyEngine.java:202] -
Policy engine will not perform in place update while processing policy-deltas.
[2024-01-31 15:38:09,419] [INFO ]
[PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1)-66:7120]
[org.apache.ranger.plugin.policyengine.RangerPolicyRepository]
[RangerPolicyRepository.java:950] - This policy engine contains 5 policy
evaluators
[2024-01-31 16:27:09,840] [INFO ] [data-plane-kafka-request-handler-5:2947541]
[kafka.server.ZkAdminManager] [Logging.scala:68] - [Admin Manager on Broker 1]:
Error processing create topic request
CreatableTopic(name='test-producer-consumer', numPartitions=1,
replicationFactor=2, assignments=[], configs=[])
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication
factor: 2 larger than available brokers: 0.
If I restart the kafka cluster, everything works well.
was:
I build a kafka cluster with kafka-2.8.1, use ranger as the authorization
platform. After the installation, when I firstly start the cluster, and then I
got ClusterAuthorizationException in kafka-server log files. In the same time,
SocketServer was started, but we cannot create topic with kafka-topics.sh tool,
at last, the creation would get an timeout exception.
There arr some important logs bellow
[2024-01-31 15:38:04,904] [INFO ] [controller-event-thread:2605]
[kafka.controller.KafkaController] [Logging.scala:66] - [Controller id=1] 1
successfully elected as the controller. Epoch incremented to 1 and epoch zk
version is now
[2024-01-31 15:38:05,540] [INFO ] [main:3241]
[org.apache.ranger.plugin.service.RangerBasePlugin] [RangerBasePlugin.java:227]
- Created PolicyRefresher
Thread(PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1)-66)
// load roles to local file
[2024-01-31 15:38:06,056] [INFO ] [main:3757]
[org.apache.ranger.plugin.util.RangerRolesProvider]
[RangerRolesProvider.java:190] -
RangerRolesProvider(serviceName=kafka-abcd-rmgwrqt1): found updated version.
lastKnownRoleVersion=-1; newVersion=1
[2024-01-31 15:38:06,108] [WARN ] [main:3809]
[org.apache.ranger.plugin.util.PolicyRefresher] [PolicyRefresher.java:393] -
cache file does not exist or not readable
'/data/emr/kafka/policycache/kafka_kafka-abcd-rmgwrqt1.json'
[2024-01-31 15:38:06,411] [ERROR] [data-plane-kafka-request-handler-0:4112]
[kafka.server.RequestHandlerHelper] [Logging.scala:76] - [KafkaApi-1] Error
when handling request: clientId=1, correlationId=0, api=UPDATE_METADATA,
version=7, body=UpdateMetadataRequestData(controllerId=1, controllerEpoch=1,
brokerEpoch=4294967593, ungroupedPartitionStates=[], topicStates=[],
liveBrokers=[UpdateMetadataBroker(id=2, v0Host='', v0Port=0,
endpoints=[UpdateMetadataEndpoint(port=9092, host='10.0.0.22',
listener='SASL_PLAINTEXT', securityProtocol=2)], rack=null),
UpdateMetadataBroker(id=3, v0Host='', v0Port=0,
endpoints=[UpdateMetadataEndpoint(port=9092, host='10.0.0.29',
listener='SASL_PLAINTEXT', securityProtocol=2)], rack=null),
UpdateMetadataBroker(id=1, v0Host='', v0Port=0,
endpoints=[UpdateMetadataEndpoint(port=9092, host='10.0.0.45',
listener='SASL_PLAINTEXT', securityProtocol=2)], rack=null)])
org.apache.kafka.common.errors.ClusterAuthorizationException: Request
Request(processor=0, connectionId=10.0.0.45:9092-10.0.0.45:34308-0,
session=Session(User:hadoop,/10.0.0.45),
listenerName=ListenerName(SASL_PLAINTEXT), securityProtocol=SASL_PLAINTEXT,
buffer=null, envelope=None) is not authorized.
// load policy to local file
[2024-01-31 15:38:09,405] [INFO ]
[PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1)-66:7106]
[org.apache.ranger.plugin.util.PolicyRefresher] [PolicyRefresher.java:321] -
PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1): found updated version.
lastKnownVersion=-1; newVersion=6
[2024-01-31 15:38:09,406] [INFO ]
[PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1)-66:7107]
[org.apache.ranger.plugin.policyengine.PolicyEngine] [PolicyEngine.java:202] -
Policy engine will not perform in place update while processing policy-deltas.
[2024-01-31 15:38:09,419] [INFO ]
[PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1)-66:7120]
[org.apache.ranger.plugin.policyengine.RangerPolicyRepository]
[RangerPolicyRepository.java:950] - This policy engine contains 5 policy
evaluators
[2024-01-31 16:27:09,840] [INFO ] [data-plane-kafka-request-handler-5:2947541]
[kafka.server.ZkAdminManager] [Logging.scala:68] - [Admin Manager on Broker 1]:
Error processing create topic request
CreatableTopic(name='test-producer-consumer', numPartitions=1,
replicationFactor=2, assignments=[], configs=[])
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication
factor: 2 larger than available brokers: 0.
If I restart the kafka cluster, everything works well.
> UpdateMetadataRequest failed because of failed authorization
> ------------------------------------------------------------
>
> Key: KAFKA-17490
> URL: https://issues.apache.org/jira/browse/KAFKA-17490
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 2.8.1
> Environment: kafka 2.8.1 + Ranger 2.3.0
> Reporter: jirar
> Assignee: jirar
> Priority: Major
> Fix For: 4.0.0
>
>
> I build a kafka cluster with kafka-2.8.1, use ranger as the authorization
> platform. After the installation, when I firstly start the cluster, and then
> I got ClusterAuthorizationException in kafka-server log files. In the same
> time, SocketServer was started, but we cannot create topic with
> kafka-topics.sh tool, at last, the creation would get an timeout exception.
>
> There are some important logs bellow
> [2024-08-31 15:38:04,904] [INFO ] [controller-event-thread:2605]
> [kafka.controller.KafkaController] [Logging.scala:66] - [Controller id=1] 1
> successfully elected as the controller. Epoch incremented to 1 and epoch zk
> version is now
> [2024-08-31 15:38:05,540] [INFO ] [main:3241]
> [org.apache.ranger.plugin.service.RangerBasePlugin]
> [RangerBasePlugin.java:227] - Created PolicyRefresher
> Thread(PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1)-66)
> // load roles to local file
> [2024-08-31 15:38:06,056] [INFO ] [main:3757]
> [org.apache.ranger.plugin.util.RangerRolesProvider]
> [RangerRolesProvider.java:190] -
> RangerRolesProvider(serviceName=kafka-abcd-rmgwrqt1): found updated version.
> lastKnownRoleVersion=-1; newVersion=1
> [2024-08-31 15:38:06,108] [WARN ] [main:3809]
> [org.apache.ranger.plugin.util.PolicyRefresher] [PolicyRefresher.java:393] -
> cache file does not exist or not readable
> '/data/emr/kafka/policycache/kafka_kafka-abcd-rmgwrqt1.json'
> [2024-08-31 15:38:06,411] [ERROR] [data-plane-kafka-request-handler-0:4112]
> [kafka.server.RequestHandlerHelper] [Logging.scala:76] - [KafkaApi-1] Error
> when handling request: clientId=1, correlationId=0, api=UPDATE_METADATA,
> version=7, body=UpdateMetadataRequestData(controllerId=1, controllerEpoch=1,
> brokerEpoch=4294967593, ungroupedPartitionStates=[], topicStates=[],
> liveBrokers=[UpdateMetadataBroker(id=2, v0Host='', v0Port=0,
> endpoints=[UpdateMetadataEndpoint(port=9092, host='10.0.0.22',
> listener='SASL_PLAINTEXT', securityProtocol=2)], rack=null),
> UpdateMetadataBroker(id=3, v0Host='', v0Port=0,
> endpoints=[UpdateMetadataEndpoint(port=9092, host='10.0.0.29',
> listener='SASL_PLAINTEXT', securityProtocol=2)], rack=null),
> UpdateMetadataBroker(id=1, v0Host='', v0Port=0,
> endpoints=[UpdateMetadataEndpoint(port=9092, host='10.0.0.45',
> listener='SASL_PLAINTEXT', securityProtocol=2)], rack=null)])
> org.apache.kafka.common.errors.ClusterAuthorizationException: Request
> Request(processor=0, connectionId=10.0.0.45:9092-10.0.0.45:34308-0,
> session=Session(User:hadoop,/10.0.0.45),
> listenerName=ListenerName(SASL_PLAINTEXT), securityProtocol=SASL_PLAINTEXT,
> buffer=null, envelope=None) is not authorized.
> // load policy to local file
> [2024-08-31 15:38:09,405] [INFO ]
> [PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1)-66:7106]
> [org.apache.ranger.plugin.util.PolicyRefresher] [PolicyRefresher.java:321] -
> PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1): found updated version.
> lastKnownVersion=-1; newVersion=6
> [2024-08-31 15:38:09,406] [INFO ]
> [PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1)-66:7107]
> [org.apache.ranger.plugin.policyengine.PolicyEngine] [PolicyEngine.java:202]
> - Policy engine will not perform in place update while processing
> policy-deltas.
> [2024-01-31 15:38:09,419] [INFO ]
> [PolicyRefresher(serviceName=kafka-abcd-rmgwrqt1)-66:7120]
> [org.apache.ranger.plugin.policyengine.RangerPolicyRepository]
> [RangerPolicyRepository.java:950] - This policy engine contains 5 policy
> evaluators
> [2024-01-31 16:27:09,840] [INFO ]
> [data-plane-kafka-request-handler-5:2947541] [kafka.server.ZkAdminManager]
> [Logging.scala:68] - [Admin Manager on Broker 1]: Error processing create
> topic request CreatableTopic(name='test-producer-consumer', numPartitions=1,
> replicationFactor=2, assignments=[], configs=[])
> org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication
> factor: 2 larger than available brokers: 0.
>
> If I restart the kafka cluster, everything works well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
