[jira] [Commented] (KAFKA-7119) Intermittent test failure with GSSAPI authentication failure
[
https://issues.apache.org/jira/browse/KAFKA-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582135#comment-16582135
]
ASF GitHub Bot commented on KAFKA-7119:
---
rajinisivaram closed pull request #5509: KAFKA-7119: Handle transient Kerberos
errors on server side
URL: https://github.com/apache/kafka/pull/5509
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java
b/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java
index 8d6549d867c..8934e8e5487 100644
---
a/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java
+++
b/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java
@@ -42,7 +42,7 @@
import org.apache.kafka.common.requests.SaslHandshakeResponse;
import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
import org.apache.kafka.common.security.auth.KafkaPrincipal;
-import org.apache.kafka.common.utils.Java;
+import org.apache.kafka.common.security.kerberos.KerberosError;
import org.apache.kafka.common.utils.Utils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -52,7 +52,6 @@
import javax.security.sasl.SaslClient;
import javax.security.sasl.SaslException;
import java.io.IOException;
-import java.lang.reflect.Method;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.security.Principal;
@@ -376,7 +375,7 @@ public void close() throws IOException {
Throwable cause = e.getCause();
// Treat transient Kerberos errors as non-fatal SaslExceptions
that are processed as I/O exceptions
// and all other failures as fatal SaslAuthenticationException.
-if (kerberosError != null && kerberosError.retriable)
+if (kerberosError != null && kerberosError.retriable())
throw new SaslException(error, cause);
else
throw new SaslAuthenticationException(error, cause);
@@ -443,73 +442,4 @@ static final String firstPrincipal(Subject subject) {
}
}
-/**
- * Kerberos exceptions that may require special handling. The standard
Kerberos error codes
- * for these errors are retrieved using KrbException#errorCode() from the
underlying Kerberos
- * exception thrown during {@link SaslClient#evaluateChallenge(byte[])}.
- */
-private enum KerberosError {
-// (Mechanism level: Server not found in Kerberos database (7) -
UNKNOWN_SERVER)
-// This is retriable, but included here to add extra logging for this
case.
-SERVER_NOT_FOUND(7, false),
-// (Mechanism level: Client not yet valid - try again later (21))
-CLIENT_NOT_YET_VALID(21, true),
-// (Mechanism level: Ticket not yet valid (33) - Ticket not yet
valid)])
-// This could be a small timing window.
-TICKET_NOT_YET_VALID(33, true),
-// (Mechanism level: Request is a replay (34) - Request is a replay)
-// Replay detection used to prevent DoS attacks can result in false
positives, so retry on error.
-REPLAY(34, true);
-
-
-private static final Class KRB_EXCEPTION_CLASS;
-private static final Method KRB_EXCEPTION_RETURN_CODE_METHOD;
-
-static {
-try {
-if (Java.isIbmJdk()) {
-KRB_EXCEPTION_CLASS =
Class.forName("com.ibm.security.krb5.internal.KrbException");
-} else {
-KRB_EXCEPTION_CLASS =
Class.forName("sun.security.krb5.KrbException");
-}
-KRB_EXCEPTION_RETURN_CODE_METHOD =
KRB_EXCEPTION_CLASS.getMethod("returnCode");
-} catch (Exception e) {
-throw new KafkaException("Kerberos exceptions could not be
initialized", e);
-}
-}
-
-private final int errorCode;
-private final boolean retriable;
-
-KerberosError(int errorCode, boolean retriable) {
-this.errorCode = errorCode;
-this.retriable = retriable;
-}
-
-private static KerberosError fromException(Exception exception) {
-Throwable cause = exception.getCause();
-while (cause != null && !KRB_EXCEPTION_CLASS.isInstance(cause)) {
-cause = cause.getCause();
-}
-if (cause == null)
-return null;
-else {
-try {
-Integer errorCode = (Integer)
KRB_EXCEPTION_RETURN_CODE_METHOD.invoke(cause);
-return
[jira] [Commented] (KAFKA-7119) Intermittent test failure with GSSAPI authentication failure
[
https://issues.apache.org/jira/browse/KAFKA-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580865#comment-16580865
]
ASF GitHub Bot commented on KAFKA-7119:
---
rajinisivaram opened a new pull request #5509: KAFKA-7119: Handle transient
Kerberos errors on server side
URL: https://github.com/apache/kafka/pull/5509
Don't report retriable Kerberos errors on the server-side as authentication
failures to clients.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Intermittent test failure with GSSAPI authentication failure
>
>
> Key: KAFKA-7119
> URL: https://issues.apache.org/jira/browse/KAFKA-7119
> Project: Kafka
> Issue Type: Bug
> Components: security
>Affects Versions: 2.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 1.0.3, 1.1.2, 2.0.1, 2.1.0
>
>
> I have seen this failure a couple of times in builds (e.g.
> [https://builds.apache.org/job/kafka-pr-jdk10-scala2.12/2412/testReport/junit/kafka.api/SaslSslAdminClientIntegrationTest/testLogStartOffsetCheckpoint/)]
> {quote}
> org.apache.kafka.common.errors.SaslAuthenticationException: An error:
> (java.security.PrivilegedActionException: javax.security.sasl.SaslException:
> GSS initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Request is a replay (34) - Request is a replay)]) occurred
> when evaluating SASL token received from the Kafka Broker. Kafka Client will
> go to AUTHENTICATION_FAILED state. Caused by:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - Request is a replay)] at
> jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$2.run(SaslClientAuthenticator.java:358)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$2.run(SaslClientAuthenticator.java:356)
> at java.base/java.security.AccessController.doPrivileged(Native Method) at
> java.base/javax.security.auth.Subject.doAs(Subject.java:423) at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:356)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:268)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:205)
> at
> org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:127)
> at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:425) at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510) at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:271)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:218)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:231)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:314)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1181)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
> at
> kafka.api.AdminClientIntegrationTest.$anonfun$subscribeAndWaitForAssignment$2(AdminClientIntegrationTest.scala:980)
> at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:781) at
> kafka.api.AdminClientIntegrationTest.subscribeAndWaitForAssignment(AdminClientIntegrationTest.scala:979)
> at
> kafka.api.AdminClientIntegrationTest.testLogStartOffsetCheckpoint(AdminClientIntegrationTest.scala:755)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
>
[jira] [Commented] (KAFKA-7119) Intermittent test failure with GSSAPI authentication failure
[
https://issues.apache.org/jira/browse/KAFKA-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580070#comment-16580070
]
ASF GitHub Bot commented on KAFKA-7119:
---
rajinisivaram closed pull request #5487: KAFKA-7119: Handle transient Kerberos
errors as non-fatal exceptions
URL: https://github.com/apache/kafka/pull/5487
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java
b/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java
index 2ef6d77f13f..8d6549d867c 100644
---
a/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java
+++
b/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java
@@ -42,6 +42,7 @@
import org.apache.kafka.common.requests.SaslHandshakeResponse;
import org.apache.kafka.common.security.auth.AuthenticateCallbackHandler;
import org.apache.kafka.common.security.auth.KafkaPrincipal;
+import org.apache.kafka.common.utils.Java;
import org.apache.kafka.common.utils.Utils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -51,6 +52,7 @@
import javax.security.sasl.SaslClient;
import javax.security.sasl.SaslException;
import java.io.IOException;
+import java.lang.reflect.Method;
import java.nio.ByteBuffer;
import java.nio.channels.SelectionKey;
import java.security.Principal;
@@ -360,11 +362,9 @@ public void close() throws IOException {
});
} catch (PrivilegedActionException e) {
String error = "An error: (" + e + ") occurred when evaluating
SASL token received from the Kafka Broker.";
+KerberosError kerberosError = KerberosError.fromException(e);
// Try to provide hints to use about what went wrong so they can
fix their configuration.
-// TODO: introspect about e: look for GSS information.
-final String unknownServerErrorText =
-"(Mechanism level: Server not found in Kerberos database (7) -
UNKNOWN_SERVER)";
-if (e.toString().contains(unknownServerErrorText)) {
+if (kerberosError == KerberosError.SERVER_NOT_FOUND) {
error += " This may be caused by Java's being unable to
resolve the Kafka Broker's" +
" hostname correctly. You may want to try to adding" +
" '-Dsun.net.spi.nameservice.provider.1=dns,sun' to your
client's JVMFLAGS environment." +
@@ -373,7 +373,13 @@ public void close() throws IOException {
}
error += " Kafka Client will go to AUTHENTICATION_FAILED state.";
//Unwrap the SaslException inside `PrivilegedActionException`
-throw new SaslAuthenticationException(error, e.getCause());
+Throwable cause = e.getCause();
+// Treat transient Kerberos errors as non-fatal SaslExceptions
that are processed as I/O exceptions
+// and all other failures as fatal SaslAuthenticationException.
+if (kerberosError != null && kerberosError.retriable)
+throw new SaslException(error, cause);
+else
+throw new SaslAuthenticationException(error, cause);
}
}
@@ -436,4 +442,74 @@ static final String firstPrincipal(Subject subject) {
throw new KafkaException("Principal could not be determined
from Subject, this may be a transient failure due to Kerberos re-login");
}
}
+
+/**
+ * Kerberos exceptions that may require special handling. The standard
Kerberos error codes
+ * for these errors are retrieved using KrbException#errorCode() from the
underlying Kerberos
+ * exception thrown during {@link SaslClient#evaluateChallenge(byte[])}.
+ */
+private enum KerberosError {
+// (Mechanism level: Server not found in Kerberos database (7) -
UNKNOWN_SERVER)
+// This is retriable, but included here to add extra logging for this
case.
+SERVER_NOT_FOUND(7, false),
+// (Mechanism level: Client not yet valid - try again later (21))
+CLIENT_NOT_YET_VALID(21, true),
+// (Mechanism level: Ticket not yet valid (33) - Ticket not yet
valid)])
+// This could be a small timing window.
+TICKET_NOT_YET_VALID(33, true),
+// (Mechanism level: Request is a replay (34) - Request is a replay)
+// Replay detection used to prevent DoS attacks can result in false
positives, so retry on error.
+REPLAY(34, true);
+
+
+private static final Class KRB_EXCEPTION_CLASS;
+private static
[jira] [Commented] (KAFKA-7119) Intermittent test failure with GSSAPI authentication failure
[
https://issues.apache.org/jira/browse/KAFKA-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576023#comment-16576023
]
ASF GitHub Bot commented on KAFKA-7119:
---
rajinisivaram opened a new pull request #5487: KAFKA-7119: Handle transient
Kerberos errors as non-fatal exceptions
URL: https://github.com/apache/kafka/pull/5487
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Intermittent test failure with GSSAPI authentication failure
>
>
> Key: KAFKA-7119
> URL: https://issues.apache.org/jira/browse/KAFKA-7119
> Project: Kafka
> Issue Type: Bug
> Components: security
>Affects Versions: 2.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Major
>
> I have seen this failure a couple of times in builds (e.g.
> [https://builds.apache.org/job/kafka-pr-jdk10-scala2.12/2412/testReport/junit/kafka.api/SaslSslAdminClientIntegrationTest/testLogStartOffsetCheckpoint/)]
> {quote}
> org.apache.kafka.common.errors.SaslAuthenticationException: An error:
> (java.security.PrivilegedActionException: javax.security.sasl.SaslException:
> GSS initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Request is a replay (34) - Request is a replay)]) occurred
> when evaluating SASL token received from the Kafka Broker. Kafka Client will
> go to AUTHENTICATION_FAILED state. Caused by:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - Request is a replay)] at
> jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$2.run(SaslClientAuthenticator.java:358)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$2.run(SaslClientAuthenticator.java:356)
> at java.base/java.security.AccessController.doPrivileged(Native Method) at
> java.base/javax.security.auth.Subject.doAs(Subject.java:423) at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:356)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:268)
> at
> org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:205)
> at
> org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:127)
> at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:425) at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510) at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:271)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:218)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:231)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:314)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1181)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
> at
> kafka.api.AdminClientIntegrationTest.$anonfun$subscribeAndWaitForAssignment$2(AdminClientIntegrationTest.scala:980)
> at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:781) at
> kafka.api.AdminClientIntegrationTest.subscribeAndWaitForAssignment(AdminClientIntegrationTest.scala:979)
> at
> kafka.api.AdminClientIntegrationTest.testLogStartOffsetCheckpoint(AdminClientIntegrationTest.scala:755)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
