[
https://issues.apache.org/jira/browse/KAFKA-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16968664#comment-16968664
]
artem edited comment on KAFKA-6947 at 11/6/19 8:16 PM:
-------------------------------------------------------
Face the same error (0.11.0)
{code:java}
[2019-11-06 14:03:03,263] ERROR Error when sending message to topic XXX with
key: 18 bytes, value: 463 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
java.lang.IllegalStateException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:609)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:596)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:188)
at java.lang.Thread.run(Thread.java:748)
[2019-11-06 14:03:03,263] INFO Closing producer due to send failure.
(kafka.tools.MirrorMaker$)
[2019-11-06 14:03:03,263] INFO Closing the Kafka producer with timeoutMillis =
0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2019-11-06 14:03:03,263] INFO Proceeding to force close the producer since
pending requests could not be completed within timeout 0 ms.
(org.apache.kafka.clients.producer.KafkaProducer)
[2019-11-06 14:03:03,263] DEBUG Kafka producer with client id producer-1 has
been closed (org.apache.kafka.clients.producer.KafkaProducer)
[2019-11-06 14:03:03,263] ERROR Error when sending message to topic
mspy_content_service__events with key: 38 bytes, value: 11587 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
java.lang.IllegalStateException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:609)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:596)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:188)
at java.lang.Thread.run(Thread.java:748) {code}
cat consumer.properties
{code:java}
cat consumer.properties
bootstrap.servers=XXX:9092
group.id=mirrormaker-backup
auto.offset.reset=earliest
exclude.internal.topics=true
client.id=mirrormaker-backup {code}
cat producer.properties
{code:java}
bootstrap.servers=YYY:9092
retries=2147483647
acks=all
batch.size=1500
compression.type=none
max.block.ms=9223372036854775807
max.in.flight.requests.per.connection=1
max.request.size=1048576000 {code}
was (Author: artemzhu):
Face the same error (0.11.0)
{code:java}
[2019-11-06 14:03:03,263] ERROR Error when sending message to topic
gwpy_core__document_events with key: 18 bytes, value: 463 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
java.lang.IllegalStateException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:609)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:596)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:188)
at java.lang.Thread.run(Thread.java:748)
[2019-11-06 14:03:03,263] INFO Closing producer due to send failure.
(kafka.tools.MirrorMaker$)
[2019-11-06 14:03:03,263] INFO Closing the Kafka producer with timeoutMillis =
0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2019-11-06 14:03:03,263] INFO Proceeding to force close the producer since
pending requests could not be completed within timeout 0 ms.
(org.apache.kafka.clients.producer.KafkaProducer)
[2019-11-06 14:03:03,263] DEBUG Kafka producer with client id producer-1 has
been closed (org.apache.kafka.clients.producer.KafkaProducer)
[2019-11-06 14:03:03,263] ERROR Error when sending message to topic
mspy_content_service__events with key: 38 bytes, value: 11587 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
java.lang.IllegalStateException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:609)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:596)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:188)
at java.lang.Thread.run(Thread.java:748) {code}
cat consumer.properties
{code:java}
cat consumer.properties
bootstrap.servers=XXX:9092
group.id=mirrormaker-backup
auto.offset.reset=earliest
exclude.internal.topics=true
client.id=mirrormaker-backup {code}
cat producer.properties
{code:java}
bootstrap.servers=YYY:9092
retries=2147483647
acks=all
batch.size=1500
compression.type=none
max.block.ms=9223372036854775807
max.in.flight.requests.per.connection=1
max.request.size=1048576000 {code}
> Mirrormaker Closing producer due to send failure
> ------------------------------------------------
>
> Key: KAFKA-6947
> URL: https://issues.apache.org/jira/browse/KAFKA-6947
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 0.11.0.0
> Reporter: Andrew Holford
> Priority: Major
>
> Hi
> On occasion our mirrormakers fail with the below error....
> [2018-05-25 05:10:31,695] ERROR Error when sending message to topic
> com_snapshot--demo with key: 13 bytes, value: 355 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 38 record(s) for
> com_snapshot--demo-5: 91886 ms has passed since last append
> [2018-05-25 05:10:31,710] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:31,710] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 05:10:31,710] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> and just after this:
> [2018-05-25 05:10:32,041] ERROR Error when sending message to topic
> com_snapshot--demo with key: 13 bytes, value: 323 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for
> com_snapshot--demo-5: 89429 ms has passed since batch creation plus linger
> time
> [2018-05-25 05:10:32,041] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,041] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 05:10:32,041] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> and sometimes we get:
> [2018-05-25 08:24:29,249] ERROR Error when sending message to topic
> com_snapshot--demo with key: 13 bytes, value: 354 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> java.lang.IllegalStateException: Producer is closed forcefully.
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:609)
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:596)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:183)
> at java.lang.Thread.run(Thread.java:745)
> [2018-05-25 08:24:29,249] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 08:24:29,249] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 08:24:29,249] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> then it completely shuts down:
> [2018-05-25 05:10:32,060] INFO Exiting on send failure, skip committing
> offsets. (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,083] INFO [mirrormaker-thread-0] Flushing producer.
> (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,094] INFO [mirrormaker-thread-0] Committing consumer
> offsets. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,094] INFO Exiting on send failure, skip committing
> offsets. (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,105] INFO [mirrormaker-thread-0] Shutting down consumer
> connectors. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,138] INFO Publish thread interrupted!
> (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)
> [2018-05-25 05:10:32,144] INFO Publishing Monitoring Metrics stopped for
> clientID=peach-prod2prod3-mirrormaker-0
> (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)
> [2018-05-25 05:10:32,148] INFO Closing the Kafka producer with timeoutMillis
> = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 05:10:32,183] INFO Closed monitoring interceptor for client
> ID=peach-prod2prod3-mirrormaker-0
> (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)
> [2018-05-25 05:10:32,197] INFO [mirrormaker-thread-0] Mirror maker thread
> stopped (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,202] FATAL [mirrormaker-thread-0] Mirror maker thread
> exited abnormally, stopping the whole mirror maker.
> (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,203] INFO [mirrormaker-thread-0] Mirror maker thread
> shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,245] INFO Start clean shutdown.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,245] INFO Shutting down consumer threads.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,261] INFO [mirrormaker-thread-0] mirrormaker-thread-0
> shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,265] INFO [mirrormaker-thread-0] Mirror maker thread
> shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread)
> On the broker side we see:
> WARN Attempting to send response via channel for which there is no open
> connection, connection id 10.82.6.105:9093-172.27.205.216:32796
> (kafka.network.Processor)
> Does anyone know what could cause this and what a possible solution could be?
> Im a little confused by the "timeoutMillis = 0 ms" as well mentioned, is this
> some setting which needs adjusting somewhere? We have
> request.timeout.ms=60000 on the producer config with most other settings left
> as the defaults.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
