Justin created SAMZA-1639: ----------------------------- Summary: Metrics producer does not produce to all partitions Key: SAMZA-1639 URL: https://issues.apache.org/jira/browse/SAMZA-1639 Project: Samza Issue Type: Bug Components: metrics Affects Versions: 0.9.1 Environment: Samza: 0.9.1
Kafka: 0.8.2.2 Reporter: Justin Hi. I have some tasks that are submitting metrics to a kafka topic, via: {quote}metrics.reporter.snapshot.class=org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory metrics.reporters=snapshot metrics.reporter.snapshot.stream=kafka.samza_metrics serializers.registry.metrics.class=org.apache.samza.serializers.MetricsSnapshotSerdeFactory systems.kafka.streams.samza_metrics.samza.msg.serde=metrics {quote} But we are seeing a weird issue where messages only get sent to a subset of the topic partitions: {quote}./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic samza_metrics --time -1 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/app/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/app/kafka/core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/app/kafka/core/build/dependant-libs-2.10.4/slf4j-simple-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] samza_metrics:0:20311035 samza_metrics:1:0 samza_metrics:2:0 samza_metrics:3:23723469 samza_metrics:4:0 samza_metrics:5:40708920 samza_metrics:6:0 samza_metrics:7:0 samza_metrics:8:10721831 samza_metrics:9:0 samza_metrics:10:50921817 samza_metrics:11:0 samza_metrics:12:0 samza_metrics:13:8637095 samza_metrics:14:11391870 samza_metrics:15:20299587 samza_metrics:16:15689619 samza_metrics:17:0 samza_metrics:18:6834402 samza_metrics:19:0 samza_metrics:20:30792418 samza_metrics:21:0 samza_metrics:22:7186731 samza_metrics:23:5241719 samza_metrics:24:0 samza_metrics:25:29958293 samza_metrics:26:0 samza_metrics:27:8097351 samza_metrics:28:13629794 samza_metrics:29:0 {quote} Even of the ones that are being produced to, it's not very balanced. We don't see this issue with any of of the 'normal' samza input/output topics, or with any of the non-samza related topics we have in this cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)