harmadasg opened a new pull request, #22893: URL: https://github.com/apache/kafka/pull/22893
The issue already came up earlier on the mailing list https://lists.apache.org/thread/ro8mbk70xstywz5yw7mlm4yoswcfqlp5 The bug can be reproduced by pointing log.dirs to a symlinked path, e.g.: /kafka/1/dd -> /navencrypt/1/kafka/kafka/1/dd The storage format step attempts to write the required meta.properties file to the configured log directory and fails with: Error while writing meta.properties file /kafka/3/dd: java.nio.file.FileAlreadyExistsException: /kafka/3/dd The broker later exits with: java.lang.RuntimeException: No readable meta.properties files found. Based on my research the root cause is a bug in JDK's Files.createDirectories which was addressed in https://bugs.openjdk.org/browse/JDK-8294193 and shipped in JDK 20. Specific code location in Kafka https://github.com/apache/kafka/blob/8fc539cb383c34e44bdb31721e0643faf453b8e6/metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java#L452 To conclude, Kafka 4.x is committed to use JDK17 and there is no real restriction on why symlinked paths could not be used for log dirs. Also this bug prevents to use certain tools like NavEncrypt which could handle encryption at rest. I would propose a simple fix to also make symlinked log.dirs possible when on JDK17. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
