codelipenghui opened a new issue #3056: Client send timeout happens on one broker URL: https://github.com/apache/pulsar/issues/3056 1. 2018.11.25 11:00:00 Restart bookie node in the cluster. some topic's producer send message timeout. After restart all brokers, most problem topic get right, but almost 3 topic still with problem. Then found problem topics owner broker is same, so we restart the broker again. All topics get right. 2. 2018.11.26 18:48:30 Two topic can't write again, after check topic's owner broker, still same broker, after restart the broker, problem topics get right. Some logs at that time: ``` 17:08:31.717 [pulsar-io-21-95] INFO org.apache.pulsar.broker.service.ServerCnx - [/172.30.151.122:44388] Subscribing on topic persistent://platform/passport/user.login-partition-0 / b.userbehavior.user.login 17:08:31.717 [pulsar-io-21-95] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://platform/passport/user.login-partition-0][b.userbehavior.user.login] Created new subscription for 32 17:08:31.717 [pulsar-io-21-95] INFO org.apache.pulsar.broker.service.ServerCnx - [/172.30.151.122:44388] Created subscription on topic persistent://platform/passport/user.login-partition-0 / b.userbehavior.user.login 17:09:31.712 [pulsar-io-21-95] INFO org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers - Removed consumer Consumer{subscription=PersistentSubscription{topic=persistent://platform/passport/user.login-partition-0, name=b.userbehavior.user.login}, consumerId=32, consumerName=e9317, address=/172.30.151.122:44388} with pending 117 acks ``` Error log on log.bk ``` 18:59:05.091 [BookKeeperClientScheduler-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught java.lang.IllegalArgumentException: negative size: -1 at com.google.common.base.Preconditions.badPositionIndex(Preconditions.java:1231) ~[com.google.guava-guava-21.0.jar:?] at com.google.common.base.Preconditions.checkPositionIndex(Preconditions.java:1222) ~[com.google.guava-guava-21.0.jar:?] at com.google.common.base.Preconditions.checkPositionIndex(Preconditions.java:1204) ~[com.google.guava-guava-21.0.jar:?] at com.google.common.collect.LinkedListMultimap$NodeIterator.<init>(LinkedListMultimap.java:327) ~[com.google.guava-guava-21.0.jar:?] at com.google.common.collect.LinkedListMultimap$1ValuesImpl.listIterator(LinkedListMultimap.java:760) ~[com.google.guava-guava-21.0.jar:?] at java.util.AbstractList.listIterator(AbstractList.java:299) ~[?:1.8.0_181] at java.util.AbstractSequentialList.iterator(AbstractSequentialList.java:239) ~[?:1.8.0_181] at org.apache.bookkeeper.proto.PerChannelBookieClient.checkTimeoutOnPendingOperations(PerChannelBookieClient.java:803) ~[org.apache.bookkeeper-bookkeeper-server-4.7.1.jar:4.7.1] at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.checkTimeoutOnPendingOperations(DefaultPerChannelBookieClientPool.java:100) ~[org.apache.bookkeeper-bookkeeper-server-4.7.1.jar:4.7.1] ``` Client send timeout stats: https://files.slack.com/files-pri/T5Z3B324U-FEB7PR341/image.png
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
