rdhabalia opened a new pull request, #21759:
URL: https://github.com/apache/pulsar/pull/21759
<!--
### Contribution Checklist
- PR title format should be *[type][component] summary*. For details, see
*[Guideline - Pulsar PR Naming
Convention](https://pulsar.apache.org/contribute/develop-semantic-title/)*.
- Fill out the template below to describe the changes contributed by the
pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from
multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and
this checklist, leaving only the filled out template below.
-->
<!-- Either this PR fixes an issue, -->
It fixes https://github.com/apache/pulsar/issues/21751
<!-- If the PR belongs to a PIP, please add the PIP link here -->
PIP: #21752
<!-- Details of when a PIP is required and how the PIP process work, please
see: https://github.com/apache/pulsar/blob/master/pip/README.md -->
### Motivation
We have introduced a configuration called `autoSkipNonRecoverableData`
before open-sourcing Pulsar as we have come across with various situations when
it was not possible to recover ledgers belonging to managed-ledger or
managed-cursors and the broker was not able to load the topics. In such
situations,`autoSkipNonRecoverableData` flag helps to skip non-recoverable
leger-recovery errors such as ledger_not_found and allows the broker to load
topics by skipping such ledgers in disaster recovery.
Brokers can recognize such non-recoverable errors using bookkeeper error
codes but in some cases, it’s very tricky and not possible to conclude
non-recoverable errors. For example, the broker can not differentiate between
all the ensemble bookies of the ledgers that are temporarily unavailable or are
permanently removed from the cluster without graceful recovery, and because of
that broker doesn’t consider all the bookies deleted as a non-recoverable error
though we can not recover ledgers in such situations where all the bookies are
removed due to various reasons such as Dev cluster clean up or system faced
data disaster with multiple bookie loss. In such situations, the system admin
has to manually identify such non-recoverable topics and update those topics’
managed-ledger and managed-cursor’s metadata and reload topics again which
requires a lot of manual effort and sometimes it might not be feasible to
handle such situations with a large number of topics that require
this manual procedure to fix those topics.
```
2023-11-02T00:35:35,582+0000 [BookKeeperClientWorker-OrderedExecutor-5-0]
INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[prop/us-west2/ns1/per
sistent/t1-partition-220] Opened ledger 94267 for consumer
pulsar.repl.dev-eastb. rc=0
2023-11-02T00:35:35,582+0000 [BookKeeperClientWorker-OrderedExecutor-5-0]
INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot
resolve euw1bk--prod-booki
e-0.euw1bk--prod-bookie.bk-.svc.cluster.local:3181, bookie is unknown
org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException:
Bookie handle i
s not available
2023-11-02T00:35:35,582+0000 [BookKeeperClientWorker-OrderedExecutor-5-0]
ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to
euw1bk--prod-bookie-0.
euw1bk--prod-bookie.bk-.svc.cluster.local:3181 as endpoint resolution failed
(probably bookie is down) err
org.apache.bookkeeper.proto.BookieAddressResolver$Bookie
IdNotResolvedException: Cannot resolve bookieId
euw1bk--prod-bookie-0.euw1bk--prod-bookie.bk-.svc.cluster.local:3181, bookie
does not exist or it is not
running
2023-11-02T00:35:35,582+0000 [BookKeeperClientWorker-OrderedExecutor-5-0]
INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not
available while reading L94267
E46 from bookie:
euw1bk--prod-bookie-0.euw1bk--prod-bookie.bk-.svc.cluster.local:3181
2023-11-02T00:35:35,582+0000 [BookKeeperClientWorker-OrderedExecutor-5-0]
INFO org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot
resolve euw1bk--prod-booki
e-2.euw1bk--prod-bookie.bk-.svc.cluster.local:3181, bookie is unknown
org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException:
Bookie handle i
s not available
2023-11-02T00:35:35,582+0000 [BookKeeperClientWorker-OrderedExecutor-5-0]
ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to
euw1bk--prod-bookie-2.
euw1bk--prod-bookie.bk-.svc.cluster.local:3181 as endpoint resolution failed
(probably bookie is down) err
org.apache.bookkeeper.proto.BookieAddressResolver$Bookie
IdNotResolvedException: Cannot resolve bookieId
euw1bk--prod-bookie-2.euw1bk--prod-bookie.bk-.svc.cluster.local:3181, bookie
does not exist or it is not
running
2023-11-02T00:35:35,582+0000 [BookKeeperClientWorker-OrderedExecutor-5-0]
INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not
available while reading L94267
E46 from bookie:
euw1bk--prod-bookie-2.euw1bk--prod-bookie.bk-.svc.cluster.local:3181
2023-11-02T00:35:35,582+0000 [BookKeeperClientWorker-OrderedExecutor-5-0]
ERROR org.apache.bookkeeper.client.PendingReadOp - Read of ledger entry failed:
L94267 E46-E46, Sent to [euw1
bk--prod-bookie-2.euw1bk--prod-bookie.bk-.svc.cluster.local:3181,
euw1bk--prod-bookie-0.euw1bk--prod-bookie.bk-.svc.cluster
.local:3181], Heard from [] : bitset = {}, Error = 'Bookie handle is not
available'. First unread entry is (-1, rc = null)
2023-11-02T00:35:35,582+0000 [BookKeeperClientWorker-OrderedExecutor-5-0]
WARN org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[prop/us-west2/ns1/per
sistent/t1-partition-220] Error reading from metadata ledger 94267 for
consumer pulsar.repl.dev-eastb: Bookie handle is not available
2023-11-02T00:35:35,582+0000 [BookKeeperClientWorker-OrderedExecutor-5-0]
WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[prop/us-west2/ns1/per
sistent/t1-partition-220] Recovery for cursor pulsar.repl.dev-eastb failed
org.apache.bookkeeper.mledger.ManagedLedgerException: Bookie handle is not
available
2023-11-02T00:35:35,582+0000 [BookKeeperClientWorker-OrderedExecutor-5-0]
INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[prop/us-west2/ns1/per
sistent/t1-partition-220] Closing managed ledger
2023-11-02T00:35:35,582+0000 [BookKeeperClientWorker-OrderedExecutor-5-0]
WARN org.apache.pulsar.broker.service.BrokerService - Failed to create topic
persistent://prop/us-west2/ns1/per
sistent/t1-partition-220
org.apache.bookkeeper.mledger.ManagedLedgerException: Bookie handle is not
available
2023-11-02T00:35:35,582+0000 [BookKeeperClientWorker-OrderedExecutor-5-0]
WARN org.apache.pulsar.broker.service.ServerCnx -
[/1.1.1.1:59962][persistent://prop/us-west2/ns1/t1-partition-220][my-local]
Failed to create consumer: consumerId=5369828,
org.apache.bookkeeper.mledger.ManagedLedgerException: Bookie handle is not
available
```
### Modifications
Therefore, the system admin should have a dynamic configuration called
`managedLedgerForceRecovery` to use in such situations to allow brokers to
forcefully load topics by skipping ledger failures to avoid topic
unavailability and perform auto repairs of the topics. This will allow the
admin to handle disaster recovery situations in a controlled and automated
manner and maintain the topic availability by mitigating such failures.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe
tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads
(10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update
later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: <!-- ENTER URL HERE -->
<!--
After opening this PR, the build in apache/pulsar will fail and instructions
will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since
the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed
in
a forked repository.
The tests will be run in the forked repository until all PR review comments
have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
