On 2022/3/16 16:43, Juhyung Park wrote:
Hi Jaegeuk,
I'll perform the trace debugging later.
Meanwhile, we encountered another issue.
For now, I added a workaround that performs mlockall() on all files
(vmtouch -L $val.mount -d -w -P $val.vmtouch) under the f2fs partition
so that the cp command works.
After confirming that extraction works, I've parallelized the script
again (on our production server again, so v5.15 without f2fs-stable
backports) and while userspace processes don't complain, there's
something going on on dmesg:
[224319.321668] loop0: detected capacity change from 0 to 11041080
[224319.361530] F2FS-fs (loop0): Found nat_bits in checkpoint
[224319.365029] F2FS-fs (loop0): write access unavailable, skipping recovery
[224319.365033] F2FS-fs (loop0): Mounted with checkpoint version = 4321
[224340.047471] loop1: detected capacity change from 0 to 11769352
[224340.101509] F2FS-fs (loop1): Found nat_bits in checkpoint
[224340.103799] F2FS-fs (loop1): write access unavailable, skipping recovery
[224340.103802] F2FS-fs (loop1): Mounted with checkpoint version = 4321
[224361.763528] loop2: detected capacity change from 0 to 11386480
[224361.812891] F2FS-fs (loop2): Found nat_bits in checkpoint
[224361.814881] F2FS-fs (loop2): write access unavailable, skipping recovery
[224361.814886] F2FS-fs (loop2): Mounted with checkpoint version = 4321
[224386.701011] F2FS-fs (loop2): lz4 invalid rlen:16384, expected:16384
[224386.701248] F2FS-fs (loop2): lz4 invalid rlen:16384, expected:16384
[224386.701779] F2FS-fs (loop2): lz4 invalid rlen:16384, expected:16384
[224386.701806] F2FS-fs (loop2): lz4 invalid rlen:16384, expected:16384
[224386.701826] F2FS-fs (loop2): lz4 invalid rlen:16384, expected:16384
[224391.823216] loop3: detected capacity change from 0 to 11045240
[224391.884383] F2FS-fs (loop3): Found nat_bits in checkpoint
[224392.194608] F2FS-fs (loop3): write access unavailable, skipping recovery
[224392.194612] F2FS-fs (loop3): Mounted with checkpoint version = 4321
[224401.386330] loop4: detected capacity change from 0 to 11428112
[224401.424575] F2FS-fs (loop4): Found nat_bits in checkpoint
[224401.428872] F2FS-fs (loop4): write access unavailable, skipping recovery
[224401.428875] F2FS-fs (loop4): Mounted with checkpoint version = 4321
[224576.100839] loop0: detected capacity change from 0 to 11041080
[224576.140862] F2FS-fs (loop0): Found nat_bits in checkpoint
[224576.142694] F2FS-fs (loop0): write access unavailable, skipping recovery
[224576.142699] F2FS-fs (loop0): Mounted with checkpoint version = 4321
[224599.330003] loop1: detected capacity change from 0 to 2913344
[224599.360425] F2FS-fs (loop1): Found nat_bits in checkpoint
[224599.362257] F2FS-fs (loop1): write access unavailable, skipping recovery
[224599.362260] F2FS-fs (loop1): Mounted with checkpoint version = 4321
[224602.241868] loop2: detected capacity change from 0 to 2064344
[224602.276146] F2FS-fs (loop2): Found nat_bits in checkpoint
[224602.277955] F2FS-fs (loop2): write access unavailable, skipping recovery
[224602.277971] F2FS-fs (loop2): Mounted with checkpoint version = 4321
[224629.346056] loop3: detected capacity change from 0 to 41776
[224629.375324] F2FS-fs (loop3): Found nat_bits in checkpoint
[224629.377167] F2FS-fs (loop3): write access unavailable, skipping recovery
[224629.377170] F2FS-fs (loop3): Mounted with checkpoint version = 4321
[225029.224078] loop0: detected capacity change from 0 to 11411472
[225029.270396] F2FS-fs (loop0): Found nat_bits in checkpoint
[225029.271345] F2FS-fs (loop0): write access unavailable, skipping recovery
[225029.271348] F2FS-fs (loop0): Mounted with checkpoint version = 4321
[225625.045254] loop0: detected capacity change from 0 to 11045240
[225625.078435] F2FS-fs (loop0): Found nat_bits in checkpoint
[225625.079453] F2FS-fs (loop0): write access unavailable, skipping recovery
[225625.079455] F2FS-fs (loop0): Mounted with checkpoint version = 4321
[225651.466438] loop1: detected capacity change from 0 to 11428112
[225651.532489] F2FS-fs (loop1): Found nat_bits in checkpoint
[225651.582589] F2FS-fs (loop1): write access unavailable, skipping recovery
[225651.582594] F2FS-fs (loop1): Mounted with checkpoint version = 4321
[225653.729775] loop2: detected capacity change from 0 to 2896696
[225653.817115] F2FS-fs (loop2): Found nat_bits in checkpoint
[225653.842649] F2FS-fs (loop2): write access unavailable, skipping recovery
[225653.842654] F2FS-fs (loop2): Mounted with checkpoint version = 4321
[225665.480652] loop3: detected capacity change from 0 to 2064344
[225665.548952] F2FS-fs (loop3): Found nat_bits in checkpoint
[225665.551460] F2FS-fs (loop3): write access unavailable, skipping recovery
[225665.551465] F2FS-fs (loop3): Mounted with checkpoint version = 4321
[225687.173551] loop4: detected capacity change from 0 to 3753984
[225687.297132] F2FS-fs (loop4): Found nat_bits in checkpoint
[225687.299702] F2FS-fs (loop4): write access unavailable, skipping recovery
[225687.299705] F2FS-fs (loop4): Mounted with checkpoint version = 4321
[225694.934235] loop5: detected capacity change from 0 to 1868760
[225695.047462] F2FS-fs (loop5): Found nat_bits in checkpoint
[225695.049562] F2FS-fs (loop5): write access unavailable, skipping recovery
[225695.049567] F2FS-fs (loop5): Mounted with checkpoint version = 4321
[225704.254531] loop6: detected capacity change from 0 to 41776
[225704.282972] F2FS-fs (loop6): Found nat_bits in checkpoint
[225704.286725] F2FS-fs (loop6): write access unavailable, skipping recovery
[225704.286730] F2FS-fs (loop6): Mounted with checkpoint version = 4321
[225723.689515] loop0: detected capacity change from 0 to 11411472
[225723.723250] F2FS-fs (loop0): Found nat_bits in checkpoint
[225723.725740] F2FS-fs (loop0): write access unavailable, skipping recovery
[225723.725744] F2FS-fs (loop0): Mounted with checkpoint version = 4321
[225727.574004] loop2: detected capacity change from 0 to 11386480
[225727.621016] F2FS-fs (loop2): Found nat_bits in checkpoint
[225727.623296] F2FS-fs (loop2): write access unavailable, skipping recovery
[225727.623300] F2FS-fs (loop2): Mounted with checkpoint version = 4321
[225728.404524] loop3: detected capacity change from 0 to 41776
[225728.440403] F2FS-fs (loop3): Found nat_bits in checkpoint
[225728.442743] F2FS-fs (loop3): write access unavailable, skipping recovery
[225728.442748] F2FS-fs (loop3): Mounted with checkpoint version = 4321
[225731.328330] F2FS-fs (loop2): lz4 invalid rlen:16384, expected:16384
[225731.328400] F2FS-fs (loop2): lz4 invalid rlen:16384, expected:16384
[225731.328424] F2FS-fs (loop2): lz4 invalid rlen:16384, expected:16384
[225743.064116] loop1: detected capacity change from 0 to 11769352
[225743.110306] F2FS-fs (loop1): Found nat_bits in checkpoint
[225743.113408] F2FS-fs (loop1): write access unavailable, skipping recovery
[225743.113414] F2FS-fs (loop1): Mounted with checkpoint version = 4321
[225761.148410] loop3: detected capacity change from 0 to 3758152
[225761.195014] F2FS-fs (loop3): Found nat_bits in checkpoint
[225761.196431] F2FS-fs (loop3): write access unavailable, skipping recovery
[225761.196432] F2FS-fs (loop3): Mounted with checkpoint version = 4321
[225766.885693] loop4: detected capacity change from 0 to 1868760
[225767.010849] F2FS-fs (loop4): Found nat_bits in checkpoint
[225767.013024] F2FS-fs (loop4): write access unavailable, skipping recovery
[225767.013027] F2FS-fs (loop4): Mounted with checkpoint version = 4321
[225781.475142] loop5: detected capacity change from 0 to 3766472
[225781.500625] F2FS-fs (loop5): Found nat_bits in checkpoint
[225781.503148] F2FS-fs (loop5): write access unavailable, skipping recovery
[225781.503150] F2FS-fs (loop5): Mounted with checkpoint version = 4321
[225785.367635] loop6: detected capacity change from 0 to 1868760
[225785.474029] F2FS-fs (loop6): Found nat_bits in checkpoint
[225785.475673] F2FS-fs (loop6): write access unavailable, skipping recovery
[225785.475678] F2FS-fs (loop6): Mounted with checkpoint version = 4321
[225802.664721] loop7: detected capacity change from 0 to 41776
[225802.706846] F2FS-fs (loop7): Found nat_bits in checkpoint
[225802.709092] F2FS-fs (loop7): write access unavailable, skipping recovery
[225802.709095] F2FS-fs (loop7): Mounted with checkpoint version = 4321
[225812.972186] loop0: detected capacity change from 0 to 41776
[225813.000175] F2FS-fs (loop0): Found nat_bits in checkpoint
[225813.003617] F2FS-fs (loop0): write access unavailable, skipping recovery
[225813.003619] F2FS-fs (loop0): Mounted with checkpoint version = 4321
Those "lz4 invalid rlen:16384, expected:16384", is this normal?
Juhyung, good catch.
I guess this is the place in where it returns EIO in your scenario.
Actually, it looks we printed the wrong variable in the message... could
you please change the code as below, using "ret" instead of "dic->rlen",
and try another test to check the log.
static int lz4_decompress_pages()
{
...
if (ret != PAGE_SIZE << dic->log_cluster_size) {
printk_ratelimited("%sF2FS-fs (%s): lz4 invalid rlen:%zu, "
"expected:%lu\n", KERN_ERR,
F2FS_I_SB(dic->inode)->sb->s_id,
ret,
PAGE_SIZE << dic->log_cluster_size);
return -EIO;
}
return 0;
}
Not sure, is it possible that you can share this f2fs image in somewhere?
then I can check details of this issue directly w/ the image.
Thanks,
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
