zifengmo commented on issue #21557:
URL: https://github.com/apache/pulsar/issues/21557#issuecomment-1820513456
> @zifengmo
>
> > And because of this problem, the broker has been rolled back. But it
cannot be reproduced locally. I will redeploy it later to check.
>
> Thanks. Could you provide the client logs around the issue occurring?
This is the debug log before and after the issue occurred.
2023-11-09T09:02:44,589+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.common.protocol.PulsarDecoder - [/172.22.2.22:44262] Received
cmd SEND
2023-11-09T09:02:44,589+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Received send
message request. producer: 29:1250 mqe-79-11:1250 size: 396, partition key is:
65d1f5964e5191e0422vmi, ordering key is null, uncompressedSize is 332
2023-11-09T09:02:44,589+0000 [pulsar-io-4-4] DEBUG
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29] asyncAddEntry size=396
state=LedgerOpened
2023-11-09T09:02:44,589+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.common.protocol.PulsarDecoder - [/172.22.2.22:44262] Received
cmd SEND
2023-11-09T09:02:44,589+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Received send
message request. producer: 12:780 mqe-79-22:780 size: 971, partition key is:
6556a2caf3c1a01cc0s9cl, ordering key is null, uncompressedSize is 907
2023-11-09T09:02:44,589+0000 [pulsar-io-4-4] DEBUG
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-12] asyncAddEntry size=971
state=LedgerOpened
2023-11-09T09:02:44,589+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.OpAddEntry - Created new OpAddEntry
OpAddEntry{mlName=dlghphniuyhgydjmvkld/out/persistent/event-partition-29,
ledgerId=null, entryId=-1, startTime=7439057180093416, dataLength=396}
2023-11-09T09:02:44,589+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29] Write into current
ledger lh=881992 entries=1332
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.common.protocol.PulsarDecoder - [/172.22.2.22:44262] Received
cmd SEND
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Received send
message request. producer: 9:1005 mqe-79-23:1005 size: 971, partition key is:
658205705425a83918pueo, ordering key is null, uncompressedSize is 907
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-9] asyncAddEntry size=971
state=LedgerOpened
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.common.protocol.PulsarDecoder - [/172.22.2.22:44262] Received
cmd SEND
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Received send
message request. producer: 47:1013 mqe-79-4:1013 size: 414, partition key is:
65d838ee1d11d40104yifc, ordering key is null, uncompressedSize is 351
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-47] asyncAddEntry size=414
state=LedgerOpened
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.common.protocol.PulsarDecoder - [/172.22.2.22:44262] Received
cmd SEND
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Received send
message request. producer: 47:1014 mqe-79-4:1014 size: 414, partition key is:
65ab6d7740f1604bc3ysbx, ordering key is null, uncompressedSize is 351
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-47] asyncAddEntry size=414
state=LedgerOpened
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.common.protocol.PulsarDecoder - [/172.22.2.22:44262] Received
cmd SEND
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Received send
message request. producer: 53:1003 mqe-79-2:1003 size: 478, partition key is:
65c2c89f849a506c26hwpl, ordering key is null, uncompressedSize is 415
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-53] asyncAddEntry size=478
state=LedgerOpened
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.common.protocol.PulsarDecoder - [/172.22.2.22:44262] Received
cmd SEND
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Received send
message request. producer: 0:1199 mqe-79-30:1199 size: 396, partition key is:
65492b7f84a96906b3xgah, ordering key is null, uncompressedSize is 332
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-0] asyncAddEntry size=396
state=LedgerOpened
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.common.protocol.PulsarDecoder - [/172.22.2.22:44262] Received
cmd SEND
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Received send
message request. producer: 34:1268 mqe-79-9:1268 size: 478, partition key is:
65a45c1b6e57b38e08ygvd, ordering key is null, uncompressedSize is 415
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-34] asyncAddEntry size=478
state=LedgerOpened
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.common.protocol.PulsarDecoder - [/172.22.2.22:44262] Received
cmd SEND
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Received send
message request. producer: 24:1353 mqe-79-15:1353 size: 413, partition key is:
576285373c71bf37f221, ordering key is null, uncompressedSize is 351
2023-11-09T09:02:44,591+0000 [pulsar-io-4-4] DEBUG
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-24] asyncAddEntry size=413
state=LedgerOpened
2023-11-09T09:02:44,591+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.OpAddEntry - Created new OpAddEntry
OpAddEntry{mlName=dlghphniuyhgydjmvkld/out/persistent/event-partition-47,
ledgerId=null, entryId=-1, startTime=7439057182221115, dataLength=414}
2023-11-09T09:02:44,591+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-47] Write into current
ledger lh=881971 entries=1158
2023-11-09T09:02:44,591+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.OpAddEntry - Created new OpAddEntry
OpAddEntry{mlName=dlghphniuyhgydjmvkld/out/persistent/event-partition-47,
ledgerId=null, entryId=-1, startTime=7439057182247525, dataLength=414}
2023-11-09T09:02:44,591+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-47] Write into current
ledger lh=881971 entries=1159
2023-11-09T09:02:44,591+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.OpAddEntry - Created new OpAddEntry
OpAddEntry{mlName=dlghphniuyhgydjmvkld/out/persistent/event-partition-53,
ledgerId=null, entryId=-1, startTime=7439057182256545, dataLength=478}
2023-11-09T09:02:44,591+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-53] Write into current
ledger lh=881967 entries=1041
2023-11-09T09:02:44,591+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.OpAddEntry - Created new OpAddEntry
OpAddEntry{mlName=dlghphniuyhgydjmvkld/out/persistent/event-partition-24,
ledgerId=null, entryId=-1, startTime=7439057182268765, dataLength=413}
2023-11-09T09:02:44,591+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-24] Write into current
ledger lh=881985 entries=1572
2023-11-09T09:02:44,591+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.proto.PerChannelBookieClient - Got Add response
from bookie:172.26.3.151:3181 rc:EOK, ledger:881992:entry:1331
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.proto.PerChannelBookieClient - Got Add response
from bookie:172.26.3.149:3181 rc:EOK, ledger:881992:entry:1331
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.client.PendingAddOp - Submit callback (lid:881992,
eid: 1331). rc:0
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.OpAddEntry -
[OpAddEntry{mlName=dlghphniuyhgydjmvkld/out/persistent/event-partition-29,
ledgerId=881992, entryId=1331, startTime=7439057180093416, dataLength=396}]
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29] write-complete:
ledger-id=881992 entry-id=1331 size=396 rc=0
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.cache.RangeEntryCacheImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29] Adding entry to cache:
881992:1331 - size: 396
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.pulsar.broker.service.Producer -
[PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-29}]
[mqe-79-11] [29] triggered send callback. cnx /172.22.2.22:44262, sequenceId
1250
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29]
[dlghphniuyhgydjmvkld-sub] Received ml notification
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29]
[dlghphniuyhgydjmvkld-sub] Received notification of new messages persisted,
reading at 881992:1331 -- last: 881992:1331
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29] Consumer
dlghphniuyhgydjmvkld-sub cursor notification: other counters: consumed 1331
mdPos 881992:1330 rdPos 881992:1331
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29] Reading entries from
ledger 881992 - first=1331 last=1331
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.cache.RangeEntryCacheImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29] Reading entries range
ledger 881992: 1331 to 1331
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.cache.RangeEntryCacheImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29] Ledger 881992 -- Found
in cache entries: 1331-1331
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.OpReadEntry -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29][dlghphniuyhgydjmvkld-sub]
Read entries succeeded batch_size=1 cumulative_size=0 requested_count=100
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29]
[dlghphniuyhgydjmvkld-sub] Filtering entries [881992:1331..881992:1331] -
alreadyDeleted: []
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29]
[dlghphniuyhgydjmvkld-sub] No filtering needed for entries
[881992:1331..881992:1331]
2023-11-09T09:02:44,592+0000 [BookKeeperClientWorker-OrderedExecutor-0-0]
DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://dlghphniuyhgydjmvkld/out/event-partition-29 /
dlghphniuyhgydjmvkld-sub] Distributing 1 messages to 7 consumers
2023-11-09T09:02:44,592+0000 [broker-topic-workers-OrderedExecutor-1-0]
DEBUG org.apache.pulsar.broker.service.Consumer -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-29-PersistentSubscription{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-29,
name=dlghphniuyhgydjmvkld-sub}] Added 881992:1331 ledger entry with batchSize
of 1 to pendingAcks in broker.service.Consumer for consumerId: 29
2023-11-09T09:02:44,592+0000 [broker-topic-workers-OrderedExecutor-1-0]
DEBUG org.apache.pulsar.broker.service.Consumer -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-29-PersistentSubscription{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-29,
name=dlghphniuyhgydjmvkld-sub}] Added 0 minus 1 messages to
MESSAGE_PERMITS_UPDATER in broker.service.Consumer for consumerId: 29;
avgMessagesPerEntry is 1.0
2023-11-09T09:02:44,592+0000 [broker-topic-workers-OrderedExecutor-1-0]
DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://dlghphniuyhgydjmvkld/out/event-partition-29 /
dlghphniuyhgydjmvkld-sub] Added -(1 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
2023-11-09T09:02:44,592+0000 [broker-topic-workers-OrderedExecutor-1-0]
DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://dlghphniuyhgydjmvkld/out/event-partition-29 /
dlghphniuyhgydjmvkld-sub] Schedule read of 100 messages for 7 consumers
2023-11-09T09:02:44,592+0000 [broker-topic-workers-OrderedExecutor-1-0]
DEBUG org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[dlghphniuyhgydjmvkld/out/persistent/event-partition-29]
[dlghphniuyhgydjmvkld-sub] Deferring retry of read at position 881992:1332
2023-11-09T09:02:44,592+0000 [pulsar-io-4-2] DEBUG
org.apache.pulsar.broker.service.PulsarCommandSenderImpl -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-29-PersistentSubscription{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-29,
name=dlghphniuyhgydjmvkld-sub}] Sending message to consumerId 29, msg id
881992-1331 with batchSize 1
2023-11-09T09:02:44,592+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer -
[PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-29}]
[mqe-79-11] [29] Persisted message. cnx
org.apache.pulsar.broker.service.ServerCnx@11792789, sequenceId 1250
2023-11-09T09:02:44,593+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.common.protocol.PulsarDecoder - [/172.22.2.22:44262] Received
cmd SEND
2023-11-09T09:02:44,593+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Received send
message request. producer: 7:696 mqe-79-25:696 size: 1015, partition key is:
65c74c268d2d078505195o, ordering key is null, uncompressedSize is 951
2023-11-09T09:02:44,593+0000 [pulsar-io-4-4] ERROR
org.apache.pulsar.broker.service.Producer -
[PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-7}]
[mqe-79-25] Failed to verify checksum
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] WARN
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Got exception
io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds
5253120: 134623753 - discarded
at
io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:507)
at
io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:493)
at
io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:377)
at
io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:423)
at
io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:333)
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at
org.apache.pulsar.common.protocol.OptionalProxyProtocolDecoder.channelRead(OptionalProxyProtocolDecoder.java:52)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:842)
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] connect state
change to : [Failed]
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] INFO
org.apache.pulsar.broker.service.ServerCnx - Closed connection from
/172.22.2.22:44262
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-0] [mqe-79-30] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=0,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-0},
client=/172.22.2.22:44262, producerName=mqe-79-30, producerId=0}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-0},
client=/172.22.2.22:44262, producerName=mqe-79-30, producerId=0}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-21] [mqe-79-17] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=21,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-21},
client=/172.22.2.22:44262, producerName=mqe-79-17, producerId=21}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-21},
client=/172.22.2.22:44262, producerName=mqe-79-17, producerId=21}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-42] [mqe-79-7] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=42,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-42},
client=/172.22.2.22:44262, producerName=mqe-79-7, producerId=42}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-42},
client=/172.22.2.22:44262, producerName=mqe-79-7, producerId=42}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-4] [mqe-79-27] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=4,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-4},
client=/172.22.2.22:44262, producerName=mqe-79-27, producerId=4}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-4},
client=/172.22.2.22:44262, producerName=mqe-79-27, producerId=4}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-9] [mqe-79-23] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=9,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-9},
client=/172.22.2.22:44262, producerName=mqe-79-23, producerId=9}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-9},
client=/172.22.2.22:44262, producerName=mqe-79-23, producerId=9}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-47] [mqe-79-4] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=47,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-47},
client=/172.22.2.22:44262, producerName=mqe-79-4, producerId=47}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-47},
client=/172.22.2.22:44262, producerName=mqe-79-4, producerId=47}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-13] [mqe-79-21] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=13,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-13},
client=/172.22.2.22:44262, producerName=mqe-79-21, producerId=13}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-13},
client=/172.22.2.22:44262, producerName=mqe-79-21, producerId=13}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-25] [mqe-79-14] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=25,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-25},
client=/172.22.2.22:44262, producerName=mqe-79-14, producerId=25}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-25},
client=/172.22.2.22:44262, producerName=mqe-79-14, producerId=25}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-8] [mqe-79-24] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=8,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-8},
client=/172.22.2.22:44262, producerName=mqe-79-24, producerId=8}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-8},
client=/172.22.2.22:44262, producerName=mqe-79-24, producerId=8}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-53] [mqe-79-2] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=53,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-53},
client=/172.22.2.22:44262, producerName=mqe-79-2, producerId=53}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-53},
client=/172.22.2.22:44262, producerName=mqe-79-2, producerId=53}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-1] [mqe-79-29] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=1,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-1},
client=/172.22.2.22:44262, producerName=mqe-79-29, producerId=1}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-1},
client=/172.22.2.22:44262, producerName=mqe-79-29, producerId=1}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-12] [mqe-79-22] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=12,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-12},
client=/172.22.2.22:44262, producerName=mqe-79-22, producerId=12}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-12},
client=/172.22.2.22:44262, producerName=mqe-79-22, producerId=12}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://kea7r5qxrxmrvwq4vjxe/out/event-partition-6] [mqe-79-34] Removed
producer -- count: 1
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=67,
producer=Producer{topic=PersistentTopic{topic=persistent://kea7r5qxrxmrvwq4vjxe/out/event-partition-6},
client=/172.22.2.22:44262, producerName=mqe-79-34, producerId=67}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://kea7r5qxrxmrvwq4vjxe/out/event-partition-6},
client=/172.22.2.22:44262, producerName=mqe-79-34, producerId=67}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-26] [mqe-79-13] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=26,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-26},
client=/172.22.2.22:44262, producerName=mqe-79-13, producerId=26}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-26},
client=/172.22.2.22:44262, producerName=mqe-79-13, producerId=26}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-5] [mqe-79-26] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=5,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-5},
client=/172.22.2.22:44262, producerName=mqe-79-26, producerId=5}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-5},
client=/172.22.2.22:44262, producerName=mqe-79-26, producerId=5}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-16] [mqe-79-20] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=16,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-16},
client=/172.22.2.22:44262, producerName=mqe-79-20, producerId=16}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-16},
client=/172.22.2.22:44262, producerName=mqe-79-20, producerId=16}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-17] [mqe-79-19] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=17,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-17},
client=/172.22.2.22:44262, producerName=mqe-79-19, producerId=17}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-17},
client=/172.22.2.22:44262, producerName=mqe-79-19, producerId=17}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://kea7r5qxrxmrvwq4vjxe/out/event-partition-9] [mqe-79-35] Removed
producer -- count: 1
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=70,
producer=Producer{topic=PersistentTopic{topic=persistent://kea7r5qxrxmrvwq4vjxe/out/event-partition-9},
client=/172.22.2.22:44262, producerName=mqe-79-35, producerId=70}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://kea7r5qxrxmrvwq4vjxe/out/event-partition-9},
client=/172.22.2.22:44262, producerName=mqe-79-35, producerId=70}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-29] [mqe-79-11] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=29,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-29},
client=/172.22.2.22:44262, producerName=mqe-79-11, producerId=29}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-29},
client=/172.22.2.22:44262, producerName=mqe-79-11, producerId=29}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-52] [mqe-79-3] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=52,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-52},
client=/172.22.2.22:44262, producerName=mqe-79-3, producerId=52}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-52},
client=/172.22.2.22:44262, producerName=mqe-79-3, producerId=52}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://dlghphniuyhgydjmvkld/out/event-partition-43] [mqe-79-6] Removed
producer -- count: 8
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.ServerCnx - [/172.22.2.22:44262] Removed
producer: producerId=43,
producer=Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-43},
client=/172.22.2.22:44262, producerName=mqe-79-6, producerId=43}
2023-11-09T09:02:44,594+0000 [pulsar-io-4-4] DEBUG
org.apache.pulsar.broker.service.Producer - Removed producer:
Producer{topic=PersistentTopic{topic=persistent://dlghphniuyhgydjmvkld/out/event-partition-43},
client=/172.22.2.22:44262, producerName=mqe-79-6, producerId=43}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
