On Wed, 3 Mar 2010 03:26:04 -0800 Joel Becker <joel.bec...@oracle.com> wrote:
> On Wed, Mar 03, 2010 at 09:47:08PM +1100, Brad Plant wrote: > > Turns out I'd hit a free space fragmentation problem. While df reported I > > had heaps of free space (>50% from memory!), I couldn't write (echo >>) to > > the log files on the problem web server. Note that you'll find you can > > still create small files and append to small files, but not the larger > > apache log files. > > > > The fact that it happens late at night was very confusing, but eventually > > made sense. As the day goes on, the log files get bigger and bigger pieces > > of contiguous free space are required to extend the file. Eventually, a > > contiguous piece of free space cannot be found and your writes will start > > to fail. > > This is my assumption, which is why I asked for the debugfs > output. This will give us layout information. > There are actually two fragmentation issues here. First, free > space is fragmented. This prevents us from allocating metadata blocks. > These metadata blocks are used in the housekeeping of files with many > data extents. This fits your description of lots of free space, yet > files are unable to grow. > The second issue is fragmentation of the actual file. Log files > are written in small hunks. As each node takes turns extending the log > file, the hunks come from disparate parts of the disk. This means the > log files have lots of small extents instead of a few large ones. This > is why we recommend separate log files for each node. Separate log > files would grow in a much more contiguous fashion, allowing fewer, > larger extents. With fewer extents, they would need fewer metadata > blocks to grow. I forgot to mention that I was actually using a separate log file per web server. Cheers, Brad
signature.asc
Description: PGP signature
_______________________________________________ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-users
