[GitHub] [bookkeeper] zhaohaidao opened a new issue, #3588: In DirectIO mode, GarbageCollectorThread failed to read metadata from log header as meta is not flushed into the log file.

[GitBox]

zhaohaidao opened a new issue, #3588:
URL: https://github.com/apache/bookkeeper/issues/3588

   **BUG REPORT**
   
   ***Describe the bug***
   <img width="1008" alt="image" 
src="https://user-images.githubusercontent.com/6216286/198606479-71993726-107e-48f8-936f-5b5c156c9db2.png";>
   When I was testing DirectIO mode, I observed a lot of unidentified read 
traffic from the metrics (I turned off consumption) after test finished. 
Through analysis, it can be seen that this traffic originated from the failure 
of GCThread to read metadta from logheader, and had to be downgraded to scan 
the entire file(The READ_METADATA_FALLBACK event from following logs reflect 
this). The root cause of metadata not being written to the log file is that 
metadata writing happens after fsync. If I understand correctly, in DirectIO 
mode the metaa write should happen before the fsync
   
   > Oct 28 18:58:19 test_host pulsar[108543]: 2022-10-28T18:58:19,481+0800 
[GarbageCollectorThread-44-1] INFO  
org.apache.bookkeeper.slogger.slf4j.Slf4jSlogger - 
{directory=/cache2/storage/8/current, entryLogId=86, 
event=READ_METADATA_FALLBACK, reason=Error reading 
index(logId=86,reason=ledgerMap never 
written(ledgerMapOffset=0,ledgerCountOffset=0))} READ_METADATA_FALLBACK
   
   
   ***To Reproduce***
   
   Steps to reproduce the behavior:
   1. Setup a test env based on bookkeeper-4.16
   2. Start a 2GB/s publish workload and no consumption workload(the publish 
rate doesn't necessarily need to be 2GB/s), This workload should last 5min.
   
   
   ***Expected behavior***
   
   After ten minutes(GC Wait Time). You should see a lot of unidentified read 
traffic from bookie host and the READ_METADATA_FALLBACK log
   
   ***Screenshots***
   <img width="1494" alt="image" 
src="https://user-images.githubusercontent.com/6216286/198615026-fec69df8-577b-4240-98d3-19bad3b6ce09.png";>
   
   
   ***Additional context***
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@bookkeeper.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org