Thank you Jia, The error is clear. The bookie is in fact able to work but it cannot reclaim space I am not sure if a good patch would to simply drop the broken file...
Enrico On mer 20 set 2017, 10:45 Jia Zhai <zhaiji...@gmail.com> wrote: > Seems the failing place was scan the old EntryLog, not doing write, maybe > it not broken the EntryLog file. > > The error indicates that we read a negative entrySize from entryLog file. > > /** > * Scan entry log. > * > * @param entryLogId Entry Log Id > * @param scanner Entry Log Scanner > * @throws IOException > */ > protected void scanEntryLog(long entryLogId, EntryLogScanner scanner) throws > IOException { > ByteBuffer sizeBuff = ByteBuffer.allocate(4); > ByteBuffer lidBuff = ByteBuffer.allocate(8); > BufferedReadChannel bc; > // Get the BufferedChannel for the current entry log file > try { > bc = getChannelForLogId(entryLogId); > } catch (IOException e) { > LOG.warn("Failed to get channel to scan entry log: " + entryLogId + > ".log"); > throw e; > } > // Start the read position in the current entry log file to be after > // the header where all of the ledger entries are. > long pos = LOGFILE_HEADER_SIZE; > > // Read through the entry log file and extract the ledger ID's. > while (true) { > // Check if we've finished reading the entry log file. > if (pos >= bc.size()) { > break; > } > if (readFromLogChannel(entryLogId, bc, sizeBuff, pos) != > sizeBuff.capacity()) { > LOG.warn("Short read for entry size from entrylog {}", > entryLogId); > return; > } > long offset = pos; > pos += 4; > sizeBuff.flip(); > int entrySize = sizeBuff.getInt(); < === 2, here we get entrySize > from buffer. > > sizeBuff.clear(); > // try to read ledger id first > if (readFromLogChannel(entryLogId, bc, lidBuff, pos) != > lidBuff.capacity()) { > LOG.warn("Short read for ledger id from entrylog {}", entryLogId); > return; > } > lidBuff.flip(); > long lid = lidBuff.getLong(); > lidBuff.clear(); > if (lid == INVALID_LID || !scanner.accept(lid)) { > // skip this entry > pos += entrySize; > continue; > } > // read the entry > byte data[] = new byte[entrySize]; < === 1, here is line 1006, > seems it get a negative entrySize. > ByteBuffer buff = ByteBuffer.wrap(data); > int rc = readFromLogChannel(entryLogId, bc, buff, pos); > if (rc != data.length) { > LOG.warn("Short read for ledger entry from entryLog {}@{} ({} != > {})", new Object[] { entryLogId, pos, > rc, data.length }); > return; > } > buff.flip(); > // process the entry > scanner.process(lid, offset, buff); > // Advance position to the next entry > pos += entrySize; > } > } > > > > > > On Tue, Sep 19, 2017 at 8:42 PM, Enrico Olivelli <eolive...@gmail.com> > wrote: > >> Hi >> in one of my staging sites I got this error and the bookie cannot reclaim >> disk space >> >> I am running a pure 4.5 BookKeeper bookie >> below is the error >> >> Surely it is a broken EntryLog file. >> Do you think the bookie could be recoverable ? >> I did a backup, it does not contain sensitive data, I could share it >> >> -- Enrico >> >> 17-09-19-14-37-22 Unexpected throwable caught >> 17-09-19-14-37-22 java.lang.NegativeArraySizeException >> java.lang.NegativeArraySizeException >> at >> org.apache.bookkeeper.bookie.EntryLogger.scanEntryLog(EntryLogger.java:1006) >> at >> org.apache.bookkeeper.bookie.EntryLogger.extractEntryLogMetadataByScanning(EntryLogger.java:1118) >> at >> org.apache.bookkeeper.bookie.EntryLogger.getEntryLogMetadata(EntryLogger.java:1031) >> at >> org.apache.bookkeeper.bookie.GarbageCollectorThread.extractMetaFromEntryLogs(GarbageCollectorThread.java:569) >> at >> org.apache.bookkeeper.bookie.GarbageCollectorThread.safeRun(GarbageCollectorThread.java:342) >> at org.apache.bookkeeper.util.SafeRunnable.run(SafeRunnable.java:33) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> > > -- -- Enrico Olivelli