Is there any recommendation about header max size?

[Gabriel Giussi]

I have logic in my service to capture exceptions being thrown during
message processing and produce a new message to a different topic with
information about the error. The idea is to leave the message unmodified,
aka produce the exact same bytes to this new topic, therefore I'm planning
on adding the java exception as a header.
By looking at the documentation it is just an array of bytes and it doesn't
say anything about a max size but is there any recommendation about it?
https://kafka.apache.org/documentation/#recordheader

UnwritableMetadataException on Startup

[tobias.bohn]

Hello,

we currently have an "UnwritableMetadataException" error starting Kafka in
our test environment. Unfortunately, I don't know whether it is due to an
incorrect configuration or something similar. Kafka is running in a Docker
Container, but I don't think that's the cause. When I Google this error, I
unfortunately don't find any results. 

We are currently using Kafka version 3.7.0.

Does anyone have any tips or did anyone experience this error before? The
following is a log excerpt:

 

 

kafka  | kafka 06:17:49.01 INFO  ==>

kafka  | kafka 06:17:49.02 INFO  ==> Welcome to the Bitnami kafka container

kafka  | kafka 06:17:49.02 INFO  ==> Subscribe to project updates by
watching https://github.com/bitnami/containers

kafka  | kafka 06:17:49.02 INFO  ==> Submit issues and feature requests at
https://github.com/bitnami/containers/issues

kafka  | kafka 06:17:49.02 INFO  ==> Upgrade to Tanzu Application Catalog
for production environments to access custom-configured and pre-packaged
software components. Gain enhanced features, including Software Bill of
Materials (SBOM), CVE scan result reports, and VEX documents. To learn more,
visit https://bitnami.com/enterprise

kafka  | kafka 06:17:49.02 INFO  ==>

kafka  | kafka 06:17:49.03 INFO  ==> ** Starting Kafka setup **

kafka  | kafka 06:17:50.68 INFO  ==> Initializing Kafka...

kafka  | kafka 06:17:50.69 INFO  ==> No injected configuration files found,
creating default config files

kafka  | kafka 06:17:51.24 INFO  ==> Initializing KRaft storage metadata

kafka  | kafka 06:17:51.25 INFO  ==> Formatting storage directories to add
metadata...

kafka  |
metaPropertiesEnsemble=MetaPropertiesEnsemble(metadataLogDir=Optional.empty,
dirs={/bitnami/kafka/data: MetaProperties(version=1, clusterId=,
nodeId=2, directoryId=85zdFJUTcbfYz0NwP8U3iA)})

kafka  | All of the log directories are already formatted.

kafka  | kafka 06:17:53.21 INFO  ==> ** Kafka setup finished! **

kafka  |

kafka  | kafka 06:17:53.23 INFO  ==> ** Starting Kafka **

kafka  | [2024-04-16 06:17:54,210] INFO Registered
kafka:type=kafka.Log4jController MBean
(kafka.utils.Log4jControllerRegistration$)

kafka  | [2024-04-16 06:17:54,607] INFO Setting -D
jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated
TLS renegotiation (org.apache.zookeeper.common.X509Util)

kafka  | [2024-04-16 06:17:54,815] INFO Registered signal handlers for TERM,
INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)

kafka  | [2024-04-16 06:17:54,819] INFO [ControllerServer id=2] Starting
controller (kafka.server.ControllerServer)

kafka  | [2024-04-16 06:17:55,465] INFO Updated connection-accept-rate max
connection creation rate to 2147483647 (kafka.network.ConnectionQuotas)

kafka  | [2024-04-16 06:17:56,298] INFO [SocketServer
listenerType=CONTROLLER, nodeId=2] Created data-plane acceptor and
processors for endpoint : ListenerName(CONTROLLER)
(kafka.network.SocketServer)

kafka  | [2024-04-16 06:17:56,302] INFO CONTROLLER: resolved wildcard host
to kafka (org.apache.kafka.metadata.ListenerInfo)

kafka  | [2024-04-16 06:17:56,312] INFO authorizerStart completed for
endpoint CONTROLLER. Endpoint is now READY.
(org.apache.kafka.server.network.EndpointReadyFutures)

kafka  | [2024-04-16 06:17:56,312] INFO [SharedServer id=2] Starting
SharedServer (kafka.server.SharedServer)

kafka  | [2024-04-16 06:17:56,405] INFO [LogLoader
partition=__cluster_metadata-0, dir=/bitnami/kafka/data] Recovering
unflushed segment 2639301. 0/3 recovered for __cluster_metadata-0.
(kafka.log.LogLoader)

kafka  | [2024-04-16 06:17:56,408] INFO [LogLoader
partition=__cluster_metadata-0, dir=/bitnami/kafka/data] Loading producer
state till offset 2639301 with message format version 2
(kafka.log.UnifiedLog$)

kafka  | [2024-04-16 06:17:56,409] INFO [LogLoader
partition=__cluster_metadata-0, dir=/bitnami/kafka/data] Reloading from
producer snapshot and rebuilding producer state from offset 2639301
(kafka.log.UnifiedLog$)

kafka  | [2024-04-16 06:17:56,410] INFO Deleted producer state snapshot
/bitnami/kafka/data/__cluster_metadata-0/03954853.snapshot
(org.apache.kafka.storage.internals.log.SnapshotFile)

kafka  | [2024-04-16 06:17:56,410] INFO Deleted producer state snapshot
/bitnami/kafka/data/__cluster_metadata-0/05164322.snapshot
(org.apache.kafka.storage.internals.log.SnapshotFile)

kafka  | [2024-04-16 06:17:56,411] INFO Deleted producer state snapshot
/bitnami/kafka/data/__cluster_metadata-0/06183812.snapshot
(org.apache.kafka.storage.internals.log.SnapshotFile)

kafka  | [2024-04-16 06:17:56,420] INFO [ProducerStateManager
partition=__cluster_metadata-0] Wrote producer snapshot at offset 2639301
with 0 producer ids in 7 ms.
(org.apache.kafka.storage.internals.log.ProducerStateManager)

kafka  | [2024-04-16 06:17:56,420] INFO [LogLoader
partition=__cluster_metadata-0, dir=/bitnami/kafka/data] Producer state
recovery took 1ms for snapshot load and 9ms for segment recovery from