Hi Todd, thanks for the answer! > Any chance you've done something like copy the files away and back that might cause them to lose their sparseness?
No, I don't think so. Recently we experienced some problems with stability with Kudu, and ran rebalance a couple of times, if this related. But we never used fs commands like cp/mv against Kudu dirs. I ran du on all-WALs dir: # du -sh /mnt/data01/kudu-tserver-wal/ 12G /mnt/data01/kudu-tserver-wal/ # du -sh --apparent-size /mnt/data01/kudu-tserver-wal/ 25G /mnt/data01/kudu-tserver-wal/ And on WAL with a many indexes: # du -sh --apparent-size /mnt/data01/kudu-tserver-wal/wals/779a382ea4e6464aa80ea398070a391f 306M /mnt/data01/kudu-tserver-wal/wals/779a382ea4e6464aa80ea398070a391f # du -sh /mnt/data01/kudu-tserver-wal/wals/779a382ea4e6464aa80ea398070a391f 296M /mnt/data01/kudu-tserver-wal/wals/779a382ea4e6464aa80ea398070a391f > Also, any chance you're using XFS here? Yes, exactly XFS. We use CentOS 7.6. What is interesting, there are no many holes in index files in /mnt/data01/kudu-tserver-wal/wals/779a382ea4e6464aa80ea398070a391f (WAL dir that I mention before). Only single hole in single index file (of 13 files): # xfs_bmap -v index.000000120 index.000000120: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..4231]: 1176541248..1176545479 2 (4429888..4434119) 4232 1: [4232..9815]: 1176546592..1176552175 2 (4435232..4440815) 5584 2: [9816..11583]: 1176552832..1176554599 2 (4441472..4443239) 1768 3: [11584..13319]: 1176558672..1176560407 2 (4447312..4449047) 1736 4: [13320..15239]: 1176565336..1176567255 2 (4453976..4455895) 1920 5: [15240..17183]: 1176570776..1176572719 2 (4459416..4461359) 1944 6: [17184..18999]: 1176575856..1176577671 2 (4464496..4466311) 1816 7: [19000..20927]: 1176593552..1176595479 2 (4482192..4484119) 1928 8: [20928..22703]: 1176599128..1176600903 2 (4487768..4489543) 1776 9: [22704..24575]: 1176602704..1176604575 2 (4491344..4493215) 1872 10: [24576..26495]: 1176611936..1176613855 2 (4500576..4502495) 1920 11: [26496..26655]: 1176615040..1176615199 2 (4503680..4503839) 160 12: [26656..46879]: hole 20224 But in some other WAL I see like this: # xfs_bmap -v /mnt/data01/kudu-tserver-wal/wals/508ecdfa8904bdb97a02078a91822af/index.000000000 /mnt/data01/kudu-tserver-wal/wals/508ecdfa89054bdb97a02078a91822af/index.000000000: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..7]: 1758753776..1758753783 3 (586736..586743) 8 1: [8..46879]: hole 46872 Looks like there actually used only 8 blocks and all other blocks are the hole. So looks like I can use formulas with confidence. Normal case: 8 MB/segment * 80 max segments * 2000 tablets = 1,280,000 MB = ~1.3 TB (+ some minor index overhead) Worse case: 8 MB/segment * 1 segment * 2000 tablets = 1,280,000 MB = ~16 GB (+ some minor index overhead) Right? ср, 19 июн. 2019 г. в 09:35, Todd Lipcon <[email protected]>: > Hi Pavel, > > That's not quite expected. For example, on one of our test clusters here, > we have about 65GB of WALs and about 1GB of index files. If I recall > correctly, the index files store 8 bytes per WAL entry, so typically a > couple orders of magnitude smaller than the WALs themselves. > > One thing is that the index files are sparse. Any chance you've done > something like copy the files away and back that might cause them to lose > their sparseness? If I use du --apparent-size on mine, it's total of about > 180GB vs the 1GB of actual size. > > Also, any chance you're using XFS here? XFS sometimes likes to preallocate > large amounts of data into files while they're open, and only frees it up > if disk space is contended. I think you can use 'xfs_bmap' on an index file > to see the allocation status, which might be interesting. > > -Todd > > On Tue, Jun 18, 2019 at 11:12 PM Pavel Martynov <[email protected]> > wrote: > >> Hi guys! >> >> We want to buy SSDs for TServers WALs for our cluster. I'm working on >> capacity estimation for this SSDs using "Getting Started with Kudu" book, >> Chapter 4, Write-Ahead Log ( >> https://www.oreilly.com/library/view/getting-started-with/9781491980248/ch04.html >> <https://www.oreilly.com/library/view/getting-started-with/9781491980248/ch04.html#idm139738927926240> >> ). >> >> NB: we use default Kudu WAL configuration settings. >> >> There is a formula for worse-case: >> 8 MB/segment * 80 max segments * 2000 tablets = 1,280,000 MB = ~1.3 TB >> >> So, this formula takes into account only segment files. But in our >> cluster, I see that every segment file has >= 1 corresponding index files. >> And every index file actually larger than segment file. >> >> Numbers from one of our nodes. >> WALs count: >> $ ls /mnt/data01/kudu-tserver-wal/wals/ | wc -l >> 711 >> >> Overall WAL size: >> $ du -d 0 -h /mnt/data01/kudu-tserver-wal/ >> 13G /mnt/data01/kudu-tserver-wal/ >> >> Size of all segment files: >> $ find /mnt/data01/kudu-tserver-wal/ -type f -name 'wal-*' -exec du -ch >> {} + | grep total$ >> 6.1G total >> >> Size of all index files: >> $ find /mnt/data01/kudu-tserver-wal/ -type f -name 'index*' -exec du -ch >> {} + | grep total$ >> 6.5G total >> >> So I have questions. >> >> 1. How can I estimate the size of index files? >> Looks like in our cluster size of index files approximately equal to size >> segment files. >> >> 2. There is some WALs with more than one index files. For example: >> $ ls -lh >> /mnt/data01/kudu-tserver-wal/wals/779a382ea4e6464aa80ea398070a391f/ >> total 296M >> -rw-r--r-- 1 root root 23M Jun 18 21:31 index.000000108 >> -rw-r--r-- 1 root root 23M Jun 18 21:41 index.000000109 >> -rw-r--r-- 1 root root 23M Jun 18 21:52 index.000000110 >> -rw-r--r-- 1 root root 23M Jun 18 22:10 index.000000111 >> -rw-r--r-- 1 root root 23M Jun 18 22:22 index.000000112 >> -rw-r--r-- 1 root root 23M Jun 18 22:35 index.000000113 >> -rw-r--r-- 1 root root 23M Jun 18 22:48 index.000000114 >> -rw-r--r-- 1 root root 23M Jun 18 23:01 index.000000115 >> -rw-r--r-- 1 root root 23M Jun 18 23:14 index.000000116 >> -rw-r--r-- 1 root root 23M Jun 18 23:27 index.000000117 >> -rw-r--r-- 1 root root 23M Jun 18 23:40 index.000000118 >> -rw-r--r-- 1 root root 23M Jun 18 23:52 index.000000119 >> -rw-r--r-- 1 root root 23M Jun 19 01:13 index.000000120 >> -rw-r--r-- 1 root root 8.0M Jun 19 01:13 wal-000007799 >> >> Is this a normal situation? >> >> 3. Not a question. Please, consider adding documentation about the >> estimation of WAL storage. Also, I can't found any mentions about index >> files, except here >> https://kudu.apache.org/docs/scaling_guide.html#file_descriptors. >> >> Thanks! >> >> -- >> with best regards, Pavel Martynov >> > > > -- > Todd Lipcon > Software Engineer, Cloudera > -- with best regards, Pavel Martynov
