[
https://issues.apache.org/jira/browse/BOOKKEEPER-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075433#comment-14075433
]
Sijie Guo commented on BOOKKEEPER-774:
--------------------------------------
The sequence that triggered the flakiness is described as below:
- in the directory, a directory is added as filled directory.
- EntryLogger detected current dir is full. It set shouldCreateNewEntryLog to
true.
{code}
public void diskFull(File disk) {
// If the current entry log disk is full, then create new entry
// log.
if (currentDir != null && currentDir.equals(disk)) {
shouldCreateNewEntryLog.set(true);
}
}
{code}
- addEntry came in and it found shouldCreateNewEntryLog is true. it means we
need to stop writing to current entry log, and try to create a new entry log.
the logic here is to add current log channel to a ToFlush list and then create
a new log. but since all directories were marked as filled, so createNewLog
will throw exception, which bring the entry logger into a state that *the log
channel was added in ToFlush list and current log channel was still referencing
to it*.
{code}
2014-07-26 07:03:14,559 - INFO - [BookieWriteThread-15006-0:EntryLogger@373] -
Flushing entry logger 0 back to filesystem, pending for syncing entry loggers :
[org.apache.bookkeeper.bookie.EntryLogger$BufferedLogChannel@49f10a67].
if (null != logChannel) {
if (null == logChannelsToFlush) {
logChannelsToFlush = new LinkedList<BufferedLogChannel>();
}
// flush the internal buffer back to filesystem but not sync disk
// so the readers could access the data from filesystem.
logChannel.flush(false);
logChannelsToFlush.add(logChannel);
LOG.info("Flushing entry logger {} back to filesystem, pending for
syncing entry loggers : {}.",
logChannel.getLogId(), logChannelsToFlush);
for (EntryLogListener listener : listeners) {
listener.onRotateEntryLog();
}
}
logChannel = entryLoggerAllocator.createNewLog();
{code}
- at some point, the ToFlush list was synced to disks and closed by SyncThread.
{code}
2014-07-26 07:03:14,576 - INFO - [SyncThread-15006-0:EntryLogger@596] - Synced
entry logger 0 to disk.
{code}
- the directory was added back to writable directories. so the bookie was
transitioned from readonly to writable. when it tried to write entry to current
*logChannel* and it found that the channel was closed. it was closed by the
SyncThread.
> Flaky test
> org.apache.bookkeeper.test.ReadOnlyBookieTest.testBookieShouldTurnWritableFromReadOnly
> -------------------------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-774
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-774
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Labels: test
> Fix For: 4.3.0
>
>
> it failed at https://builds.apache.org/job/bookkeeper-trunk/716 .
> It seems that an addEntry hit a ClosedChannelException on file io after the
> bookie turned to writable from readonly.
> {code}
> 2014-07-26 07:03:14,584 - ERROR -
> [BookieWriteThread-15006-0:WriteEntryProcessorV3@103] - Error writing entry:0
> to ledger:4
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:184)
> at
> org.apache.bookkeeper.bookie.BufferedChannel.flushInternal(BufferedChannel.java:126)
> at
> org.apache.bookkeeper.bookie.BufferedChannel.flush(BufferedChannel.java:111)
> at
> org.apache.bookkeeper.bookie.EntryLogger.createNewLog(EntryLogger.java:371)
> at
> org.apache.bookkeeper.bookie.EntryLogger.addEntry(EntryLogger.java:623)
> at
> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.processEntry(InterleavedLedgerStorage.java:296)
> at
> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.processEntry(InterleavedLedgerStorage.java:283)
> at
> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:196)
> at
> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:80)
> at
> org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:1046)
> at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:1085)
> at
> org.apache.bookkeeper.proto.WriteEntryProcessorV3.getAddResponse(WriteEntryProcessorV3.java:99)
> at
> org.apache.bookkeeper.proto.WriteEntryProcessorV3.run(WriteEntryProcessorV3.java:132)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
