I mean the error "ERROR: invalid ledger id 56" is raised due to using a wrong ledger id formatter. I was suggesting you rerunning the command to collect more information so that we can debug.
- Sijie On Tue, Dec 3, 2019 at 12:17 AM Sharda, Ravi <ravi.sha...@dell.com> wrote: > Did you mean we should run this on a running environment to recover from > the failure? > > “bin/bookkeeper shell ledger -ledgeridformat long -m [ledger-id]" > ------------------------------ > *From:* Sijie Guo <guosi...@gmail.com> > *Sent:* Tuesday, December 3, 2019 1:10 PM > *To:* Sharda, Ravi <ravi.sha...@dell.com> > *Cc:* Enrico Olivelli <eolive...@gmail.com>; user < > user@bookkeeper.apache.org>; Flavio Junqueira <f...@apache.org> > *Subject:* Re: Bookeeper exception on pods restart > > > [EXTERNAL EMAIL] > I think 4.7.2 is using UUID as the ledger id formatter by default (it was > a mistake, and reverted in its subsequent releases). > > So you might have to run “bin/bookkeeper shell ledger -ledgeridformat long > -m [ledger-id]". > > Can you rerun this command again? > > --- > > > I saw this occurring for several ledgers in this environment. > > The IOException might be related to disk issues. Although I don't have > enough information to tell. > > Thanks, > Sijie > > On Mon, Dec 2, 2019 at 7:24 AM Sharda, Ravi <ravi.sha...@dell.com> wrote: > > > Enrico, > > I saw this occurring for several ledgers in this environment. > > ----------- > > ERROR - [BookieReadThreadPool-OrderedExecutor-3-0:ReadEntryProcessorV3@235] > - IOException while reading entry: 5 from ledger 1243 > > [BookieReadThreadPool-OrderedExecutor-7-0:ReadEntryProcessorV3@235] - > IOException while reading entry: 15 from ledger 1239 > > ERROR - [BookieReadThreadPool-OrderedExecutor-0-0:ReadEntryProcessorV3@235] > - IOException while reading entry: 102 from ledger 64 > > ERROR - [BookieReadThreadPool-OrderedExecutor-0-0:ReadEntryProcessorV3@235] > - IOException while reading entry: 7 from ledger 728 > ------------------------------ > *From:* Enrico Olivelli <eolive...@gmail.com> > *Sent:* Monday, December 2, 2019 7:01 PM > *To:* user <user@bookkeeper.apache.org> > *Cc:* Sijie Guo <guosi...@gmail.com>; Flavio Junqueira <f...@apache.org> > *Subject:* Re: Bookeeper exception on pods restart > > > [EXTERNAL EMAIL] > > > sh-4.2# ./bookkeeper shell ledger -m 56 > ERROR: invalid ledger id 56 > ledger: Dump ledger index entries into readable format. > usage: ledger [-m] <ledger_id> > -m,--meta Print meta information > > Does it work for other ledgers ? > > Enrico > > > > Il giorno lun 2 dic 2019 alle ore 10:06 Sharda, Ravi <ravi.sha...@dell.com> > ha scritto: > > Hello Sijie, > > Any luck with this? Please let us know what could be going wrong. > > Thanks & best regards, > Ravi > ------------------------------ > *From:* Sharda, Ravi <ravi.sha...@dell.com> > *Sent:* Friday, November 29, 2019 3:28 PM > *To:* Sijie Guo <guosi...@gmail.com> > *Cc:* user <user@bookkeeper.apache.org>; Flavio Junqueira <f...@apache.org> > *Subject:* Re: Bookeeper exception on pods restart > > Thanks. Here's the output of the command: > > sh-4.2# ./bookkeeper shell ledger -m 56 > ERROR: invalid ledger id 56 > ledger: Dump ledger index entries into readable format. > usage: ledger [-m] <ledger_id> > -m,--meta Print meta information > ------------------------------ > *From:* Sijie Guo <guosi...@gmail.com> > *Sent:* Friday, November 29, 2019 3:15 PM > *To:* Sharda, Ravi <ravi.sha...@dell.com> > *Cc:* user <user@bookkeeper.apache.org>; Flavio Junqueira <f...@apache.org> > *Subject:* Re: Bookeeper exception on pods restart > > > [EXTERNAL EMAIL] > Sorry, my bad. The command for reading ledger index should be "bookkeeper > shell ledger". > > From the `ls` output, I didn't find entry 1.log under ledgers directory. > So I guess the log file doesn't exist. If you can provide the output of > `bookkeeper shell ledger`, we can take a look at the index file to > understand more. > > On Fri, Nov 29, 2019 at 1:20 AM Sharda, Ravi <ravi.sha...@dell.com> wrote: > > For the following error, > > OrderedExecutor-0-0:ReadEntryProcessorV3@235] - IOException while reading > entry: 25 from ledger 56 > > java.io.FileNotFoundException: No file for log 1 for 56 with location > 4744138143 > > at org.apache.bookkeeper.bookie.EntryLogger.findFile(EntryLogger.java:1165) > > at > org.apache.bookkeeper.bookie.EntryLogger.getChannelForLogId(EntryLogger.java:1100) > > at > org.apache.bookkeeper.bookie.EntryLogger.internalReadEntry(EntryLogger.java:1002) > > at > org.apache.bookkeeper.bookie.EntryLogger.readEntry(EntryLogger.java:1051) > > at > org.apache.bookkeeper.bookie.InterleavedLedgerStorage.getEntry(InterleavedLedgerStorage.java:305) > > at > org.apache.bookkeeper.bookie.SortedLedgerStorage.getEntry(SortedLedgerStorage.java:153) > > at org.apache.bookkeeper.bookie.L > ----------- > sh-4.2# ./bookkeeper shell readledger --ledgerid 56 > > ERROR: invalid value for option ledgerid : 56 > Must specify a ledger id > > ----------- > I didn't know how to check that the log file exists. Attaching the output > of "ls -R -L", instead. > > ------------------------------ > *From:* Sijie Guo <guosi...@gmail.com> > *Sent:* Friday, November 29, 2019 2:31 PM > *To:* Sharda, Ravi <ravi.sha...@dell.com> > *Cc:* user <user@bookkeeper.apache.org>; Flavio Junqueira <f...@apache.org> > *Subject:* Re: Bookeeper exception on pods restart > > > [EXTERNAL EMAIL] > If it is a permanent error, > > - check if the log file (indicated in the error message) exists or not. > - use `bookkeeper shell readledger` to dump the index of the given ledger. > - see if the index points to the right entry log file or not > > - Sijie > > On Fri, Nov 29, 2019 at 12:46 AM Sharda, Ravi <ravi.sha...@dell.com> > wrote: > > The latest instance we have seen is a permanent error. The bookies haven't > recovered in the environment (last 2 days). In some previous instances, > developers had reported that the bookies had recovered, but it is also > possible that the error was slightly different from what we are seeing now. > > Thanks & best regards, > Ravi > ------------------------------ > *From:* Sijie Guo <guosi...@gmail.com> > *Sent:* Friday, November 29, 2019 2:10 PM > *To:* user <user@bookkeeper.apache.org> > *Cc:* Flavio Junqueira <f...@apache.org>; Sharda, Ravi < > ravi.sha...@dell.com> > *Subject:* Re: Bookeeper exception on pods restart > > > [EXTERNAL EMAIL] > Sorry for jumping into the discussion. But the error message indicates > that the entry log file 1 is not found. > It seems to me that entry log file was removed but the entry index still > points to the old location. Is this error transient error or a permanent > error? > > - Sijie > > > > On Fri, Nov 29, 2019 at 12:11 AM <prajakta.belgu...@dell.com> wrote: > > + Ravi, who will be looking into this …. > > > > *From:* Enrico Olivelli - Diennea <enrico.olive...@diennea.com> > *Sent:* Thursday, November 28, 2019 7:00 PM > *To:* user@bookkeeper.apache.org > *Cc:* f...@apache.org > *Subject:* Re: Bookeeper exception on pods restart > > > > [EXTERNAL EMAIL] > > From the error it looks like one client is trying to read an entry from > the Bookie but the entry is not there. > > I see two reasons: > > 1) The write never reached the bookie > > 2) The bookie is missing some file > > > > For 1) > > Do you have logs on the writer ? something that could tell us that a write > did not succeed ? > > How old is supposed to be the entry ? > > Do you have logs on the reader that is trying to read the entry ? > > > > > > For 2) > > Do you have other errors in the logs about failed writes or whatever ? > > If you were on 4.9 we could use the ‘localconsistency checker’ and check > for inconsistency on the bookie, it scans the bookie looking for every > entry that should reside on the bookie itsself. > > If you were writing your ledgers with writequorum >= 2 maybe you can > recover your data. > > > > > > In order to debug the problem we should compare the logs of: > > - The bookie > - The writer > - The reader > > > > > > Enrico > > > > > > Il giorno 28/11/19, 13:59 "prajakta.belgu...@dell.com" < > prajakta.belgu...@dell.com> ha scritto: > > > > I understand that auto recovery would replicate data for under replicated > ledgers. > > But it is scheduled to run only once in a while and may not have run > before a reader tries to read this data from a certain bookie. > > > > Generally what does below exception indicate about the state of BK? > > Does it indicate that the entry is missing on the specific bookie and so > we don’t find it? > > Or that something in the ledger metadata or ledgers could have been > corrupted?? > > > > Found the same issue with another product where you seem to have provided > a custom fix: > > https://github.com/diennea/herddb/issues/194 > > > > All in all want to understand if this can be the result of BK > misconfiguration or is just a temporary unavailability problem that will > resolve itself when auto-replication runs?? > > > > -Thanks, > > Prajakta > > > > *From:* Enrico Olivelli - Diennea <enrico.olive...@diennea.com> > *Sent:* Thursday, November 28, 2019 6:11 PM > *To:* user@bookkeeper.apache.org > *Cc:* f...@apache.org > *Subject:* Re: Bookeeper exception on pods restart > > > > [EXTERNAL EMAIL] > > I don’t think there is a good value. > > You can use WriteQuorumSize = AckQuorumSize, this way you will see an > error on the writing client in case of write failure to any of the bookies > > > > Usually you are enabling the Autorecovery feature to fill in the gaps of > underreplicated ledgers: > > http://bookkeeper.apache.org/docs/4.10.0/admin/autorecovery/ > > > > > > Hope that helps > > Enrico > > > > Il giorno 28/11/19, 13:30 "prajakta.belgu...@dell.com" < > prajakta.belgu...@dell.com> ha scritto: > > > > What EnsembleSize, WriteQuorumSize and AckQuorumSize would you recommend, > so we never see this? > > What other ledger creation parameters do you need information about? > > > > -Thanks, > > Prajakta > > *From:* Enrico Olivelli - Diennea <enrico.olive...@diennea.com> > *Sent:* Thursday, November 28, 2019 5:19 PM > *To:* user@bookkeeper.apache.org > *Cc:* f...@apache.org > *Subject:* Re: Bookeeper exception on pods restart > > > > [EXTERNAL EMAIL] > > Hi Prajakta, > > What ledger creation parameters are you using ? Ensamble size, Write > quorum size, Ack quorum size ? > > If ackQuorumSize < WriteQuorumSize it is possible that a write to the > bookie failed and even if the entry is supposed to be on the bookie it > never reached it but the overall single write succeeded because a > writequorum of bookies acknowledged the write. > > > > Enrico > > > > Il giorno 28/11/19, 12:44 "prajakta.belgu...@dell.com" < > prajakta.belgu...@dell.com> ha scritto: > > > > Hello Team, > > > > We have a question about an issue we are running into with Bookeeper. > > We use bookkeeper version 4.7.3. > > > > This issue occurs occasionally when Bookkeeper servers are restarted. > > We see the following error in the logs for some time, which blocks > Pravega's operations for the same duration. Not knowing the internals of > Bookeeper, but just based on the exception alone, it seems like Bookeeper > might not be locate the files temporarily. What could be causing this? > > > > 2019-11-28 03:52:26,491 - ERROR - > [BookieReadThreadPool-OrderedExecutor-0-0:ReadEntryProcessorV3@235] - > IOException while reading entry: 25 from ledger 56 > java.io > <https://slack-redir.net/link?url=http%3A%2F%2Fjava.io>.FileNotFoundException: > No file for log 1 for 56 with location 4744138143 > at > org.apache.bookkeeper.bookie.EntryLogger.findFile(EntryLogger.java:1165) > at > org.apache.bookkeeper.bookie.EntryLogger.getChannelForLogId(EntryLogger.java:1100) > at > org.apache.bookkeeper.bookie.EntryLogger.internalReadEntry(EntryLogger.java:1002) > at > org.apache.bookkeeper.bookie.EntryLogger.readEntry(EntryLogger.java:1051) > at > org.apache.bookkeeper.bookie.InterleavedLedgerStorage.getEntry(InterleavedLedgerStorage.java:305) > at > org.apache.bookkeeper.bookie.SortedLedgerStorage.getEntry(SortedLedgerStorage.java:153) > at > org.apache.bookkeeper.bookie.LedgerDescriptorImpl.readEntry(LedgerDescriptorImpl.java:153) > at org.apache.bookkeeper.bookie.Bookie.readEntry(Bookie.java:1305) > at > org.apache.bookkeeper.proto.ReadEntryProcessorV3.readEntry(ReadEntryProcessorV3.java:175) > at > org.apache.bookkeeper.proto.ReadEntryProcessorV3.readEntry(ReadEntryProcessorV3.java:155) > at > org.apache.bookkeeper.proto.ReadEntryProcessorV3.getReadResponse(ReadEntryProcessorV3.java:218) > at > org.apache.bookkeeper.proto.ReadEntryProcessorV3.executeOp(ReadEntryProcessorV3.java:264) > at > org.apache.bookkeeper.proto.ReadEntryProcessorV3.safeRun(ReadEntryProcessorV3.java:260) > at > org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > -Thanks, > > Prajakta > > > > > ------------------------------ > > > CONFIDENTIALITY & PRIVACY NOTICE > This e-mail (including any attachments) is strictly confidential and may > also contain privileged information. If you are not the intended recipient > you are not authorised to read, print, save, process or disclose this > message. If you have received this message by mistake, please inform the > sender immediately and destroy this e-mail, its attachments and any copies. > Any use, distribution, reproduction or disclosure by any person other than > the intended recipient is strictly prohibited and the person responsible > may incur in penalties. > The use of this e-mail is only for professional purposes; there is no > guarantee that the correspondence towards this e-mail will be read only by > the recipient, because, under certain circumstances, there may be a need to > access this email by third subjects belonging to the Company. > > > ------------------------------ > > > CONFIDENTIALITY & PRIVACY NOTICE > This e-mail (including any attachments) is strictly confidential and may > also contain privileged information. If you are not the intended recipient > you are not authorised to read, print, save, process or disclose this > message. If you have received this message by mistake, please inform the > sender immediately and destroy this e-mail, its attachments and any copies. > Any use, distribution, reproduction or disclosure by any person other than > the intended recipient is strictly prohibited and the person responsible > may incur in penalties. > The use of this e-mail is only for professional purposes; there is no > guarantee that the correspondence towards this e-mail will be read only by > the recipient, because, under certain circumstances, there may be a need to > access this email by third subjects belonging to the Company. > > > ------------------------------ > > > CONFIDENTIALITY & PRIVACY NOTICE > This e-mail (including any attachments) is strictly confidential and may > also contain privileged information. If you are not the intended recipient > you are not authorised to read, print, save, process or disclose this > message. If you have received this message by mistake, please inform the > sender immediately and destroy this e-mail, its attachments and any copies. > Any use, distribution, reproduction or disclosure by any person other than > the intended recipient is strictly prohibited and the person responsible > may incur in penalties. > The use of this e-mail is only for professional purposes; there is no > guarantee that the correspondence towards this e-mail will be read only by > the recipient, because, under certain circumstances, there may be a need to > access this email by third subjects belonging to the Company. > >
