If it is a permanent error, - check if the log file (indicated in the error message) exists or not. - use `bookkeeper shell readledger` to dump the index of the given ledger. - see if the index points to the right entry log file or not
- Sijie On Fri, Nov 29, 2019 at 12:46 AM Sharda, Ravi <ravi.sha...@dell.com> wrote: > The latest instance we have seen is a permanent error. The bookies haven't > recovered in the environment (last 2 days). In some previous instances, > developers had reported that the bookies had recovered, but it is also > possible that the error was slightly different from what we are seeing now. > > Thanks & best regards, > Ravi > ------------------------------ > *From:* Sijie Guo <guosi...@gmail.com> > *Sent:* Friday, November 29, 2019 2:10 PM > *To:* user <user@bookkeeper.apache.org> > *Cc:* Flavio Junqueira <f...@apache.org>; Sharda, Ravi < > ravi.sha...@dell.com> > *Subject:* Re: Bookeeper exception on pods restart > > > [EXTERNAL EMAIL] > Sorry for jumping into the discussion. But the error message indicates > that the entry log file 1 is not found. > It seems to me that entry log file was removed but the entry index still > points to the old location. Is this error transient error or a permanent > error? > > - Sijie > > > > On Fri, Nov 29, 2019 at 12:11 AM <prajakta.belgu...@dell.com> wrote: > > + Ravi, who will be looking into this …. > > > > *From:* Enrico Olivelli - Diennea <enrico.olive...@diennea.com> > *Sent:* Thursday, November 28, 2019 7:00 PM > *To:* user@bookkeeper.apache.org > *Cc:* f...@apache.org > *Subject:* Re: Bookeeper exception on pods restart > > > > [EXTERNAL EMAIL] > > From the error it looks like one client is trying to read an entry from > the Bookie but the entry is not there. > > I see two reasons: > > 1) The write never reached the bookie > > 2) The bookie is missing some file > > > > For 1) > > Do you have logs on the writer ? something that could tell us that a write > did not succeed ? > > How old is supposed to be the entry ? > > Do you have logs on the reader that is trying to read the entry ? > > > > > > For 2) > > Do you have other errors in the logs about failed writes or whatever ? > > If you were on 4.9 we could use the ‘localconsistency checker’ and check > for inconsistency on the bookie, it scans the bookie looking for every > entry that should reside on the bookie itsself. > > If you were writing your ledgers with writequorum >= 2 maybe you can > recover your data. > > > > > > In order to debug the problem we should compare the logs of: > > - The bookie > - The writer > - The reader > > > > > > Enrico > > > > > > Il giorno 28/11/19, 13:59 "prajakta.belgu...@dell.com" < > prajakta.belgu...@dell.com> ha scritto: > > > > I understand that auto recovery would replicate data for under replicated > ledgers. > > But it is scheduled to run only once in a while and may not have run > before a reader tries to read this data from a certain bookie. > > > > Generally what does below exception indicate about the state of BK? > > Does it indicate that the entry is missing on the specific bookie and so > we don’t find it? > > Or that something in the ledger metadata or ledgers could have been > corrupted?? > > > > Found the same issue with another product where you seem to have provided > a custom fix: > > https://github.com/diennea/herddb/issues/194 > > > > All in all want to understand if this can be the result of BK > misconfiguration or is just a temporary unavailability problem that will > resolve itself when auto-replication runs?? > > > > -Thanks, > > Prajakta > > > > *From:* Enrico Olivelli - Diennea <enrico.olive...@diennea.com> > *Sent:* Thursday, November 28, 2019 6:11 PM > *To:* user@bookkeeper.apache.org > *Cc:* f...@apache.org > *Subject:* Re: Bookeeper exception on pods restart > > > > [EXTERNAL EMAIL] > > I don’t think there is a good value. > > You can use WriteQuorumSize = AckQuorumSize, this way you will see an > error on the writing client in case of write failure to any of the bookies > > > > Usually you are enabling the Autorecovery feature to fill in the gaps of > underreplicated ledgers: > > http://bookkeeper.apache.org/docs/4.10.0/admin/autorecovery/ > > > > > > Hope that helps > > Enrico > > > > Il giorno 28/11/19, 13:30 "prajakta.belgu...@dell.com" < > prajakta.belgu...@dell.com> ha scritto: > > > > What EnsembleSize, WriteQuorumSize and AckQuorumSize would you recommend, > so we never see this? > > What other ledger creation parameters do you need information about? > > > > -Thanks, > > Prajakta > > *From:* Enrico Olivelli - Diennea <enrico.olive...@diennea.com> > *Sent:* Thursday, November 28, 2019 5:19 PM > *To:* user@bookkeeper.apache.org > *Cc:* f...@apache.org > *Subject:* Re: Bookeeper exception on pods restart > > > > [EXTERNAL EMAIL] > > Hi Prajakta, > > What ledger creation parameters are you using ? Ensamble size, Write > quorum size, Ack quorum size ? > > If ackQuorumSize < WriteQuorumSize it is possible that a write to the > bookie failed and even if the entry is supposed to be on the bookie it > never reached it but the overall single write succeeded because a > writequorum of bookies acknowledged the write. > > > > Enrico > > > > Il giorno 28/11/19, 12:44 "prajakta.belgu...@dell.com" < > prajakta.belgu...@dell.com> ha scritto: > > > > Hello Team, > > > > We have a question about an issue we are running into with Bookeeper. > > We use bookkeeper version 4.7.3. > > > > This issue occurs occasionally when Bookkeeper servers are restarted. > > We see the following error in the logs for some time, which blocks > Pravega's operations for the same duration. Not knowing the internals of > Bookeeper, but just based on the exception alone, it seems like Bookeeper > might not be locate the files temporarily. What could be causing this? > > > > 2019-11-28 03:52:26,491 - ERROR - > [BookieReadThreadPool-OrderedExecutor-0-0:ReadEntryProcessorV3@235] - > IOException while reading entry: 25 from ledger 56 > java.io > <https://slack-redir.net/link?url=http%3A%2F%2Fjava.io>.FileNotFoundException: > No file for log 1 for 56 with location 4744138143 > at > org.apache.bookkeeper.bookie.EntryLogger.findFile(EntryLogger.java:1165) > at > org.apache.bookkeeper.bookie.EntryLogger.getChannelForLogId(EntryLogger.java:1100) > at > org.apache.bookkeeper.bookie.EntryLogger.internalReadEntry(EntryLogger.java:1002) > at > org.apache.bookkeeper.bookie.EntryLogger.readEntry(EntryLogger.java:1051) > at > org.apache.bookkeeper.bookie.InterleavedLedgerStorage.getEntry(InterleavedLedgerStorage.java:305) > at > org.apache.bookkeeper.bookie.SortedLedgerStorage.getEntry(SortedLedgerStorage.java:153) > at > org.apache.bookkeeper.bookie.LedgerDescriptorImpl.readEntry(LedgerDescriptorImpl.java:153) > at org.apache.bookkeeper.bookie.Bookie.readEntry(Bookie.java:1305) > at > org.apache.bookkeeper.proto.ReadEntryProcessorV3.readEntry(ReadEntryProcessorV3.java:175) > at > org.apache.bookkeeper.proto.ReadEntryProcessorV3.readEntry(ReadEntryProcessorV3.java:155) > at > org.apache.bookkeeper.proto.ReadEntryProcessorV3.getReadResponse(ReadEntryProcessorV3.java:218) > at > org.apache.bookkeeper.proto.ReadEntryProcessorV3.executeOp(ReadEntryProcessorV3.java:264) > at > org.apache.bookkeeper.proto.ReadEntryProcessorV3.safeRun(ReadEntryProcessorV3.java:260) > at > org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > -Thanks, > > Prajakta > > > > > ------------------------------ > > > CONFIDENTIALITY & PRIVACY NOTICE > This e-mail (including any attachments) is strictly confidential and may > also contain privileged information. If you are not the intended recipient > you are not authorised to read, print, save, process or disclose this > message. If you have received this message by mistake, please inform the > sender immediately and destroy this e-mail, its attachments and any copies. > Any use, distribution, reproduction or disclosure by any person other than > the intended recipient is strictly prohibited and the person responsible > may incur in penalties. > The use of this e-mail is only for professional purposes; there is no > guarantee that the correspondence towards this e-mail will be read only by > the recipient, because, under certain circumstances, there may be a need to > access this email by third subjects belonging to the Company. > > > ------------------------------ > > > CONFIDENTIALITY & PRIVACY NOTICE > This e-mail (including any attachments) is strictly confidential and may > also contain privileged information. If you are not the intended recipient > you are not authorised to read, print, save, process or disclose this > message. If you have received this message by mistake, please inform the > sender immediately and destroy this e-mail, its attachments and any copies. > Any use, distribution, reproduction or disclosure by any person other than > the intended recipient is strictly prohibited and the person responsible > may incur in penalties. > The use of this e-mail is only for professional purposes; there is no > guarantee that the correspondence towards this e-mail will be read only by > the recipient, because, under certain circumstances, there may be a need to > access this email by third subjects belonging to the Company. > > > ------------------------------ > > > CONFIDENTIALITY & PRIVACY NOTICE > This e-mail (including any attachments) is strictly confidential and may > also contain privileged information. If you are not the intended recipient > you are not authorised to read, print, save, process or disclose this > message. If you have received this message by mistake, please inform the > sender immediately and destroy this e-mail, its attachments and any copies. > Any use, distribution, reproduction or disclosure by any person other than > the intended recipient is strictly prohibited and the person responsible > may incur in penalties. > The use of this e-mail is only for professional purposes; there is no > guarantee that the correspondence towards this e-mail will be read only by > the recipient, because, under certain circumstances, there may be a need to > access this email by third subjects belonging to the Company. > >
