[jira] [Commented] (KAFKA-7156) Deleting topics with long names can bring all brokers to unrecoverable state
[
https://issues.apache.org/jira/browse/KAFKA-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16543348#comment-16543348
]
Vahid Hashemian commented on KAFKA-7156:
Thanks a lot for confirming this. Then I'll mark it as duplicate because the
same fix works for both.
> Deleting topics with long names can bring all brokers to unrecoverable state
>
>
> Key: KAFKA-7156
> URL: https://issues.apache.org/jira/browse/KAFKA-7156
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 1.1.0
>Reporter: Petr Pchelko
>Priority: Major
>
> Kafka limit for the topic name is 249 symbols, so creating a topic with a
> name 248 symbol long is possible. However, when deleting the topic, Kafka
> tries to rename the data directory for the topic to add some hash and
> `-deleted` in the data directory, so that the resulting file name exceeds the
> 255 symbol file name limit in most of the Unix file systems. This provokes a
> java.nio.file.FileSystemException which in turn immediately shuts down all
> the brokers. Further attemts to restart the broker fail with the same
> exception. The only way to resurrect the cluster is to manually delete the
> affected topic from zookeeper and from the disk on all the broker machines.
> Steps to reproduce:
> (Note: delete.topic.enable=true must be set in the config)
> {code:java}
> > kafka-topics.sh --zookeeper localhost:2181 --create --topic
> >
> > --partitions 1 --replication-factor 1
> > kafka-topics.sh --zookeeper localhost:2181 --delete --topic
> > aaa
> {code}
> After these 2 commands executed all the brokers where this topic is
> replicated immediately shut down with the following logs:
> {code:java}
> ERROR Error while renaming dir for
> -0
> in log dir /tmp/kafka-logs (kafka.server.LogDirFailureChannel)
> java.nio.file.FileSystemException:
> /tmp/kafka-logs/-0
> ->
> /tmp/kafka-logs/-0.093fd1e1728f438ea990cbad8a514b9f-delete:
> File name too long
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> ...
> Suppressed: java.nio.file.FileSystemException:
> /tmp/kafka-logs/-0
> ->
> /tmp/kafka-logs/-0.093fd1e1728f438ea990cbad8a514b9f-delete:
> File name too long
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694)
> ... 23 more
> [2018-07-12 13:34:45,847] INFO [ReplicaManager broker=0] Stopping serving
> replicas in dir /tmp/kafka-logs (kafka.server.ReplicaManager
[jira] [Commented] (KAFKA-7156) Deleting topics with long names can bring all brokers to unrecoverable state
[
https://issues.apache.org/jira/browse/KAFKA-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542762#comment-16542762
]
Petr Pchelko commented on KAFKA-7156:
-
I've built Kafka trunk with aforementioned PR and repeated the test, the PR
does fix the issue.
> Deleting topics with long names can bring all brokers to unrecoverable state
>
>
> Key: KAFKA-7156
> URL: https://issues.apache.org/jira/browse/KAFKA-7156
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 1.1.0
>Reporter: Petr Pchelko
>Priority: Major
>
> Kafka limit for the topic name is 249 symbols, so creating a topic with a
> name 248 symbol long is possible. However, when deleting the topic, Kafka
> tries to rename the data directory for the topic to add some hash and
> `-deleted` in the data directory, so that the resulting file name exceeds the
> 255 symbol file name limit in most of the Unix file systems. This provokes a
> java.nio.file.FileSystemException which in turn immediately shuts down all
> the brokers. Further attemts to restart the broker fail with the same
> exception. The only way to resurrect the cluster is to manually delete the
> affected topic from zookeeper and from the disk on all the broker machines.
> Steps to reproduce:
> (Note: delete.topic.enable=true must be set in the config)
> {code:java}
> > kafka-topics.sh --zookeeper localhost:2181 --create --topic
> >
> > --partitions 1 --replication-factor 1
> > kafka-topics.sh --zookeeper localhost:2181 --delete --topic
> > aaa
> {code}
> After these 2 commands executed all the brokers where this topic is
> replicated immediately shut down with the following logs:
> {code:java}
> ERROR Error while renaming dir for
> -0
> in log dir /tmp/kafka-logs (kafka.server.LogDirFailureChannel)
> java.nio.file.FileSystemException:
> /tmp/kafka-logs/-0
> ->
> /tmp/kafka-logs/-0.093fd1e1728f438ea990cbad8a514b9f-delete:
> File name too long
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> ...
> Suppressed: java.nio.file.FileSystemException:
> /tmp/kafka-logs/-0
> ->
> /tmp/kafka-logs/-0.093fd1e1728f438ea990cbad8a514b9f-delete:
> File name too long
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694)
> ... 23 more
> [2018-07-12 13:34:45,847] INFO [ReplicaManager broker=0] Stopping serving
> replicas in dir /tmp/kafka-logs (kafka.server.ReplicaManager)
> [2018-07
[jira] [Commented] (KAFKA-7156) Deleting topics with long names can bring all brokers to unrecoverable state
[
https://issues.apache.org/jira/browse/KAFKA-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541826#comment-16541826
]
Vahid Hashemian commented on KAFKA-7156:
Would it be possible for you to test the PR for KAFKA-4893 to see if that
resolves the issue you reported?
> Deleting topics with long names can bring all brokers to unrecoverable state
>
>
> Key: KAFKA-7156
> URL: https://issues.apache.org/jira/browse/KAFKA-7156
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 1.1.0
>Reporter: Petr Pchelko
>Priority: Major
>
> Kafka limit for the topic name is 249 symbols, so creating a topic with a
> name 248 symbol long is possible. However, when deleting the topic, Kafka
> tries to rename the data directory for the topic to add some hash and
> `-deleted` in the data directory, so that the resulting file name exceeds the
> 255 symbol file name limit in most of the Unix file systems. This provokes a
> java.nio.file.FileSystemException which in turn immediately shuts down all
> the brokers. Further attemts to restart the broker fail with the same
> exception. The only way to resurrect the cluster is to manually delete the
> affected topic from zookeeper and from the disk on all the broker machines.
> Steps to reproduce:
> (Note: delete.topic.enable=true must be set in the config)
> {code:java}
> > kafka-topics.sh --zookeeper localhost:2181 --create --topic
> >
> > --partitions 1 --replication-factor 1
> > kafka-topics.sh --zookeeper localhost:2181 --delete --topic
> > aaa
> {code}
> After these 2 commands executed all the brokers where this topic is
> replicated immediately shut down with the following logs:
> {code:java}
> ERROR Error while renaming dir for
> -0
> in log dir /tmp/kafka-logs (kafka.server.LogDirFailureChannel)
> java.nio.file.FileSystemException:
> /tmp/kafka-logs/-0
> ->
> /tmp/kafka-logs/-0.093fd1e1728f438ea990cbad8a514b9f-delete:
> File name too long
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> ...
> Suppressed: java.nio.file.FileSystemException:
> /tmp/kafka-logs/-0
> ->
> /tmp/kafka-logs/-0.093fd1e1728f438ea990cbad8a514b9f-delete:
> File name too long
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694)
> ... 23 more
> [2018-07-12 13:34:45,847] INFO [ReplicaManager broker=0] Stopping serving
> replicas in dir /tmp/kafka-logs (kafka.server.ReplicaMa
[jira] [Commented] (KAFKA-7156) Deleting topics with long names can bring all brokers to unrecoverable state
[
https://issues.apache.org/jira/browse/KAFKA-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541792#comment-16541792
]
Petr Pchelko commented on KAFKA-7156:
-
Ye, the issue is the same, but now the results are not the same - now all the
brokers in the replica set fall into an unrecoverable failed state, so I guess
the severity is a bit higher.
> Deleting topics with long names can bring all brokers to unrecoverable state
>
>
> Key: KAFKA-7156
> URL: https://issues.apache.org/jira/browse/KAFKA-7156
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 1.1.0
>Reporter: Petr Pchelko
>Priority: Major
>
> Kafka limit for the topic name is 249 symbols, so creating a topic with a
> name 248 symbol long is possible. However, when deleting the topic, Kafka
> tries to rename the data directory for the topic to add some hash and
> `-deleted` in the data directory, so that the resulting file name exceeds the
> 255 symbol file name limit in most of the Unix file systems. This provokes a
> java.nio.file.FileSystemException which in turn immediately shuts down all
> the brokers. Further attemts to restart the broker fail with the same
> exception. The only way to resurrect the cluster is to manually delete the
> affected topic from zookeeper and from the disk on all the broker machines.
> Steps to reproduce:
> (Note: delete.topic.enable=true must be set in the config)
> {code:java}
> > kafka-topics.sh --zookeeper localhost:2181 --create --topic
> >
> > --partitions 1 --replication-factor 1
> > kafka-topics.sh --zookeeper localhost:2181 --delete --topic
> > aaa
> {code}
> After these 2 commands executed all the brokers where this topic is
> replicated immediately shut down with the following logs:
> {code:java}
> ERROR Error while renaming dir for
> -0
> in log dir /tmp/kafka-logs (kafka.server.LogDirFailureChannel)
> java.nio.file.FileSystemException:
> /tmp/kafka-logs/-0
> ->
> /tmp/kafka-logs/-0.093fd1e1728f438ea990cbad8a514b9f-delete:
> File name too long
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> ...
> Suppressed: java.nio.file.FileSystemException:
> /tmp/kafka-logs/-0
> ->
> /tmp/kafka-logs/-0.093fd1e1728f438ea990cbad8a514b9f-delete:
> File name too long
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694)
> ... 23 more
> [2018-07-12 13:34:45,847] INFO [ReplicaManager broker=0]
[jira] [Commented] (KAFKA-7156) Deleting topics with long names can bring all brokers to unrecoverable state
[
https://issues.apache.org/jira/browse/KAFKA-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541776#comment-16541776
]
Vahid Hashemian commented on KAFKA-7156:
I think this is a duplicate of KAFKA-4893.
> Deleting topics with long names can bring all brokers to unrecoverable state
>
>
> Key: KAFKA-7156
> URL: https://issues.apache.org/jira/browse/KAFKA-7156
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 1.1.0
>Reporter: Petr Pchelko
>Priority: Major
>
> Kafka limit for the topic name is 249 symbols, so creating a topic with a
> name 248 symbol long is possible. However, when deleting the topic, Kafka
> tries to rename the data directory for the topic to add some hash and
> `-deleted` in the data directory, so that the resulting file name exceeds the
> 255 symbol file name limit in most of the Unix file systems. This provokes a
> java.nio.file.FileSystemException which in turn immediately shuts down all
> the brokers. Further attemts to restart the broker fail with the same
> exception. The only way to resurrect the cluster is to manually delete the
> affected topic from zookeeper and from the disk on all the broker machines.
> Steps to reproduce:
> (Note: delete.topic.enable=true must be set in the config)
> {code:java}
> > kafka-topics.sh --zookeeper localhost:2181 --create --topic
> >
> > --partitions 1 --replication-factor 1
> > kafka-topics.sh --zookeeper localhost:2181 --delete --topic
> > aaa
> {code}
> After these 2 commands executed all the brokers where this topic is
> replicated immediately shut down with the following logs:
> {code:java}
> ERROR Error while renaming dir for
> -0
> in log dir /tmp/kafka-logs (kafka.server.LogDirFailureChannel)
> java.nio.file.FileSystemException:
> /tmp/kafka-logs/-0
> ->
> /tmp/kafka-logs/-0.093fd1e1728f438ea990cbad8a514b9f-delete:
> File name too long
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> ...
> Suppressed: java.nio.file.FileSystemException:
> /tmp/kafka-logs/-0
> ->
> /tmp/kafka-logs/-0.093fd1e1728f438ea990cbad8a514b9f-delete:
> File name too long
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
> at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> at java.nio.file.Files.move(Files.java:1395)
> at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694)
> ... 23 more
> [2018-07-12 13:34:45,847] INFO [ReplicaManager broker=0] Stopping serving
> replicas in dir /tmp/kafka-logs (kafka.server.ReplicaManager)
> [2018-07-12 13:34:45,848] INFO [ReplicaFetcherManager on
