Re: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Good Morning Everyone, A little update regarding my initial request. Meanwhile we know a little bit more about one or the other thing. #1 statfs reporting wrong sizes: The current nuttx code doesn't follow the fat32 white paper [1] recommendation of recomputing the number of free clusters at every mount time. Instead, nuttx takes the information from the FSInfo.sector as granted. A pull request is pending [2]. As it seems does Windows stop updating the FSInfo sector once it recognises any sort of corruption (e.g. indicated number of free cluster conflicts with the FAT table, corrupted dir entries ...). If then are files moved around outside of nuttx, nuttx won't recognise it since that value didn't get updated. It might look like as the card is full while the card is empty. In the case the fs_fsifreecountis is low enough and a file / dir is allocated before a statfs call, it will re-trigger due to the extension of the chain and therefore reduction of that count. [1] https://www.win.tue.nl/~aeb/linux/fs/fat/fatgen103.pdf [2] https://github.com/apache/incubator-nuttx/pull/3760 #2 corrupted directory entries and file content. No update here, this is still the big question. It might help if others have recognised it at some point. A bit more information of the directory entries corruption, this might start in the middle of a file entry, which is 32 Bytes long, but seems to end with the end of a logical block. Therefore, you can have files which at once have a totally wrong file size, create time and so on. A fsck.msdos of the SD-card once in a while might reveal such behaviour. @Jukka, your fix might have an impact here, not sure. But hard to check. While looking into the issue we discovered one more thing. We might make a follow up request once we have time to look into it. If one iterates over a directory with readdir, then the last entry in a directory cluster won't be returned if that one is allocated on the last slot (1024) of the last cluster! If there is another partially filled cluster following for that dir all will be shown. Anyone aware what other projects do use FAT32 partitions which will fill up 4GB regularly and create lots of files (>1024) in a folder? Anyone aware of a tool like fsck.msdos for nuttx or ported to c and a quickformat routine? Best Regards, Reto On 2021/05/18 13:10:45, Reto Gähwiler wrote: > Dear All, > > First of all, in case a similar thread pops-up authored by myself, please > ignore. > > Recently we discovered some issues with the FAT32 partition on the SD-Card > used in our device running on nuttx. There are actually a bunch of issues > we are strugle to understand related to the filesystem. > The issue discovered recently is, that a statfs call won't return the true > number of free clusters. It rather returns what ever is in the FS INFO > section of the FAT32 partition. Inserting such a "corrupted" card into an > SD-Card reader and mounting it to Windows shows following: > > #1 Windows file explorer reports the correct free space. > #2 Comparing the free space in file explorer and looging at FS INFO section > with "Active - Disk Editor" doesn't line up. Windows wouldn't write any > longer the FS INFO section on that card. Even if all the files/dires are > wiped from the card. > #3 Whenever files are added / removed under nuttx the FS INFO section would > be updated by nuttx, but to the wrong number since the base is wrong. > #4 Mounting the card to linux and properly unmount it might actually fix > the issue. But not always! > #5 Quick format the card resolved the issue too. Windows would once again > write the FS INFO section and nuttx would shows the correct free space. > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes the > issue too. Reporting a broken file in a root folder on that card. > > We indeed had issues with corrupted files in the past. Usually in the > logging directory, which is accessed most. How they corrupt we do not know. > The corruption is recognised in the following ways: > > #A cryptic and too long file names, we only make use of short file names > (8.3) > #B files whcih became folders > #C sizes which are not possible > #D combinations of the above > > Our application supervises the SD-Card with statfs and removes files once > we hit a quota of 0.85. Therefore, the card is not filling up. However, the > most recent recognition of this behvaiour was due to an application bug > which allowed the card to fill up under some circumstances. > Playing around with a quickformated SD-Card, filled up with data and trying > to retrigger the issue revealed that I never get the ENOSPC but only EIO > once the card is full! But never managed to get the statfs issue. > For the logging we use the fstream family (fopen, fwrite, fread, fflush, > fsync) commands. For config files and other things also the low level > read/write are used. > > One more thing, among all the logging files we once in a
Re: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Good Morning Alan, Can't really tell you why, decisions made by the hardware designer. Bet bet I have. Unfortunately the layout is hardwired like this, so no qspi or SDIO available without a change. Reto On 2021/05/19 15:27:38, Alan Carvalho de Assis wrote: > Hi Reto, > > Just a curiosity: why are you using SDCard support over SPI? > Isn't your board prepared to use native SDCard/SDIO peripheral? > > BR, > > Alan > > On 5/19/21, Reto Gähwiler wrote: > > Hey again, > > > > About the requested configuration of the SD-Card, here we go: > > CONFIG_MMCSD=y > > CONFIG_MMCSD_SPI=y > > CONFIG_MMCSD_SPICLOCK=2500 > > > > About the multiblock, I don't think we are using that. Let me know how I > > could recognise it if not in the config. > > > > Thanks, Reto > > > > On 2021/05/19 12:13:35, David Sidrane wrote: > >> Hi Reto, > >> > >> What is the clock rate to the card and is multiblock enabled? > >> > >> David > >> > >> -----Original Message- > >> From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > >> Sent: Wednesday, May 19, 2021 5:08 AM > >> To: dev@nuttx.apache.org > >> Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated > >> free > >> clusters by statfs > >> > >> Hi David, > >> > >> It is running on an STM32h743zi version V or Y. The SD-card in use is a > >> SwissBit S-45u. > >> > >> Reto > >> > >> On 2021/05/19 11:11:22, David Sidrane wrote: > >> > hi Reto, > >> > > >> > What SoC is this on? What type of card are you using? > >> > > >> > David > >> > > >> > -Original Message- > >> > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > >> > Sent: Wednesday, May 19, 2021 3:22 AM > >> > To: dev@nuttx.apache.org > >> > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free > >> > clusters by statfs > >> > > >> > Hello Jukka, > >> > > >> > Thanks for your quick reaction. I applied your PR to our "outdated" > >> > nuttx. > >> > Unfortunately, it did not show any impact on the statfs behaviour if > >> > the > >> > card is corrupted. Though, meanwhile I do have hands on two images. > >> > > >> > #1 the mentioned one. Although, the files/dirs are all deleted. But > >> > still > >> > same behvaiour and a chkdsk error. statfs reports 703 MB free while the > >> > sd-card was deleted! > >> > > >> > #2 Having the file structure in place and contains plenty of logfiles. > >> > Many > >> > of them with the mentioned corruptions. The card is filled to 100%. But > >> > here > >> > the behaviour is slightly different. Checking the FSINFO block and then > >> > removing some files under Windows doesn't impact the FSINFO section. > >> > But > >> > once inserted to the device statfs reports the correct size and updates > >> > the > >> > FSINFO section. However, the card also shows corruptions if chkdsk is > >> > applied to it. > >> > This behaviour is identical before and after applying your PR. > >> > > >> > Brgds, Reto > >> > > >> > On 2021/05/18 19:55:06, Jukka Laitinen wrote: > >> > > Hi, > >> > > > >> > > There seems to be a bug in sector calculation which may trigger with > >> > > e.g. corrupted fs. I just made a PR for this, > >> > > > >> > > https://github.com/apache/incubator-nuttx/pull/3740 > >> > > > >> > > But I really don't know if this is related to your issues, this is > >> > > just > >> > > something that suddenly popped up elsewhere. > >> > > > >> > > -Jukka > >> > > > >> > > > >> > > On 18.5.2021 16.10, Reto Gähwiler wrote: > >> > > > Dear All, > >> > > > > >> > > > First of all, in case a similar thread pops-up authored by myself, > >> > > > please > >> > > > ignore. > >> > > > > >> > > > Recently we discovered some issues with the FAT32 partition on the > >> > > > SD-Card > >> > > > used in our device running on nuttx. There are actually a b
RE: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Hi Reto, The multiblock is a disable so if CONFIG_MMCSD_MULTIBLOCK_DISABLE=y is not in the config it is using multiblock However, that is only for the SDMMC driver. What I was thinking was [1],[2] for the SDMMC driver not the SPI one. [1] https://github.com/apache/incubator-nuttx/pull/3669 [2] https://github.com/PX4/PX4-Autopilot/issues/17442#issuecomment-835303021 David -Original Message- From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] Sent: Wednesday, May 19, 2021 8:21 AM To: dev@nuttx.apache.org Subject: Re: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs Hey again, About the requested configuration of the SD-Card, here we go: CONFIG_MMCSD=y CONFIG_MMCSD_SPI=y CONFIG_MMCSD_SPICLOCK=2500 About the multiblock, I don't think we are using that. Let me know how I could recognise it if not in the config. Thanks, Reto On 2021/05/19 12:13:35, David Sidrane wrote: > Hi Reto, > > What is the clock rate to the card and is multiblock enabled? > > David > > -Original Message- > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > Sent: Wednesday, May 19, 2021 5:08 AM > To: dev@nuttx.apache.org > Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free > clusters by statfs > > Hi David, > > It is running on an STM32h743zi version V or Y. The SD-card in use is a > SwissBit S-45u. > > Reto > > On 2021/05/19 11:11:22, David Sidrane wrote: > > hi Reto, > > > > What SoC is this on? What type of card are you using? > > > > David > > > > -Original Message- > > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > > Sent: Wednesday, May 19, 2021 3:22 AM > > To: dev@nuttx.apache.org > > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free > > clusters by statfs > > > > Hello Jukka, > > > > Thanks for your quick reaction. I applied your PR to our "outdated" > > nuttx. > > Unfortunately, it did not show any impact on the statfs behaviour if the > > card is corrupted. Though, meanwhile I do have hands on two images. > > > > #1 the mentioned one. Although, the files/dirs are all deleted. But > > still > > same behvaiour and a chkdsk error. statfs reports 703 MB free while the > > sd-card was deleted! > > > > #2 Having the file structure in place and contains plenty of logfiles. > > Many > > of them with the mentioned corruptions. The card is filled to 100%. But > > here > > the behaviour is slightly different. Checking the FSINFO block and then > > removing some files under Windows doesn't impact the FSINFO section. But > > once inserted to the device statfs reports the correct size and updates > > the > > FSINFO section. However, the card also shows corruptions if chkdsk is > > applied to it. > > This behaviour is identical before and after applying your PR. > > > > Brgds, Reto > > > > On 2021/05/18 19:55:06, Jukka Laitinen wrote: > > > Hi, > > > > > > There seems to be a bug in sector calculation which may trigger with > > > e.g. corrupted fs. I just made a PR for this, > > > > > > https://github.com/apache/incubator-nuttx/pull/3740 > > > > > > But I really don't know if this is related to your issues, this is > > > just > > > something that suddenly popped up elsewhere. > > > > > > -Jukka > > > > > > > > > On 18.5.2021 16.10, Reto Gähwiler wrote: > > > > Dear All, > > > > > > > > First of all, in case a similar thread pops-up authored by myself, > > > > please > > > > ignore. > > > > > > > > Recently we discovered some issues with the FAT32 partition on the > > > > SD-Card > > > > used in our device running on nuttx. There are actually a bunch of > > > > issues > > > > we are strugle to understand related to the filesystem. > > > > The issue discovered recently is, that a statfs call won't return > > > > the > > > > true > > > > number of free clusters. It rather returns what ever is in the FS > > > > INFO > > > > section of the FAT32 partition. Inserting such a "corrupted" card > > > > into > > > > an > > > > SD-Card reader and mounting it to Windows shows following: > > > > > > > > #1 Windows file explorer reports the correct free space. > > > > #2 Comparing the free space in file explorer and looging at FS INFO > > > > section > > > > wit
Re: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Hi Reto, Just a curiosity: why are you using SDCard support over SPI? Isn't your board prepared to use native SDCard/SDIO peripheral? BR, Alan On 5/19/21, Reto Gähwiler wrote: > Hey again, > > About the requested configuration of the SD-Card, here we go: > CONFIG_MMCSD=y > CONFIG_MMCSD_SPI=y > CONFIG_MMCSD_SPICLOCK=2500 > > About the multiblock, I don't think we are using that. Let me know how I > could recognise it if not in the config. > > Thanks, Reto > > On 2021/05/19 12:13:35, David Sidrane wrote: >> Hi Reto, >> >> What is the clock rate to the card and is multiblock enabled? >> >> David >> >> -Original Message- >> From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] >> Sent: Wednesday, May 19, 2021 5:08 AM >> To: dev@nuttx.apache.org >> Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated >> free >> clusters by statfs >> >> Hi David, >> >> It is running on an STM32h743zi version V or Y. The SD-card in use is a >> SwissBit S-45u. >> >> Reto >> >> On 2021/05/19 11:11:22, David Sidrane wrote: >> > hi Reto, >> > >> > What SoC is this on? What type of card are you using? >> > >> > David >> > >> > -Original Message----- >> > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] >> > Sent: Wednesday, May 19, 2021 3:22 AM >> > To: dev@nuttx.apache.org >> > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free >> > clusters by statfs >> > >> > Hello Jukka, >> > >> > Thanks for your quick reaction. I applied your PR to our "outdated" >> > nuttx. >> > Unfortunately, it did not show any impact on the statfs behaviour if >> > the >> > card is corrupted. Though, meanwhile I do have hands on two images. >> > >> > #1 the mentioned one. Although, the files/dirs are all deleted. But >> > still >> > same behvaiour and a chkdsk error. statfs reports 703 MB free while the >> > sd-card was deleted! >> > >> > #2 Having the file structure in place and contains plenty of logfiles. >> > Many >> > of them with the mentioned corruptions. The card is filled to 100%. But >> > here >> > the behaviour is slightly different. Checking the FSINFO block and then >> > removing some files under Windows doesn't impact the FSINFO section. >> > But >> > once inserted to the device statfs reports the correct size and updates >> > the >> > FSINFO section. However, the card also shows corruptions if chkdsk is >> > applied to it. >> > This behaviour is identical before and after applying your PR. >> > >> > Brgds, Reto >> > >> > On 2021/05/18 19:55:06, Jukka Laitinen wrote: >> > > Hi, >> > > >> > > There seems to be a bug in sector calculation which may trigger with >> > > e.g. corrupted fs. I just made a PR for this, >> > > >> > > https://github.com/apache/incubator-nuttx/pull/3740 >> > > >> > > But I really don't know if this is related to your issues, this is >> > > just >> > > something that suddenly popped up elsewhere. >> > > >> > > -Jukka >> > > >> > > >> > > On 18.5.2021 16.10, Reto Gähwiler wrote: >> > > > Dear All, >> > > > >> > > > First of all, in case a similar thread pops-up authored by myself, >> > > > please >> > > > ignore. >> > > > >> > > > Recently we discovered some issues with the FAT32 partition on the >> > > > SD-Card >> > > > used in our device running on nuttx. There are actually a bunch of >> > > > issues >> > > > we are strugle to understand related to the filesystem. >> > > > The issue discovered recently is, that a statfs call won't return >> > > > the >> > > > true >> > > > number of free clusters. It rather returns what ever is in the FS >> > > > INFO >> > > > section of the FAT32 partition. Inserting such a "corrupted" card >> > > > into >> > > > an >> > > > SD-Card reader and mounting it to Windows shows following: >> > > > >> > > > #1 Windows file explorer reports the correct free space. >> > > > #2 Comparing the free space in file explorer and looging
Re: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Hey again, About the requested configuration of the SD-Card, here we go: CONFIG_MMCSD=y CONFIG_MMCSD_SPI=y CONFIG_MMCSD_SPICLOCK=2500 About the multiblock, I don't think we are using that. Let me know how I could recognise it if not in the config. Thanks, Reto On 2021/05/19 12:13:35, David Sidrane wrote: > Hi Reto, > > What is the clock rate to the card and is multiblock enabled? > > David > > -Original Message- > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > Sent: Wednesday, May 19, 2021 5:08 AM > To: dev@nuttx.apache.org > Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free > clusters by statfs > > Hi David, > > It is running on an STM32h743zi version V or Y. The SD-card in use is a > SwissBit S-45u. > > Reto > > On 2021/05/19 11:11:22, David Sidrane wrote: > > hi Reto, > > > > What SoC is this on? What type of card are you using? > > > > David > > > > -Original Message- > > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > > Sent: Wednesday, May 19, 2021 3:22 AM > > To: dev@nuttx.apache.org > > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free > > clusters by statfs > > > > Hello Jukka, > > > > Thanks for your quick reaction. I applied your PR to our "outdated" nuttx. > > Unfortunately, it did not show any impact on the statfs behaviour if the > > card is corrupted. Though, meanwhile I do have hands on two images. > > > > #1 the mentioned one. Although, the files/dirs are all deleted. But still > > same behvaiour and a chkdsk error. statfs reports 703 MB free while the > > sd-card was deleted! > > > > #2 Having the file structure in place and contains plenty of logfiles. > > Many > > of them with the mentioned corruptions. The card is filled to 100%. But > > here > > the behaviour is slightly different. Checking the FSINFO block and then > > removing some files under Windows doesn't impact the FSINFO section. But > > once inserted to the device statfs reports the correct size and updates > > the > > FSINFO section. However, the card also shows corruptions if chkdsk is > > applied to it. > > This behaviour is identical before and after applying your PR. > > > > Brgds, Reto > > > > On 2021/05/18 19:55:06, Jukka Laitinen wrote: > > > Hi, > > > > > > There seems to be a bug in sector calculation which may trigger with > > > e.g. corrupted fs. I just made a PR for this, > > > > > > https://github.com/apache/incubator-nuttx/pull/3740 > > > > > > But I really don't know if this is related to your issues, this is just > > > something that suddenly popped up elsewhere. > > > > > > -Jukka > > > > > > > > > On 18.5.2021 16.10, Reto Gähwiler wrote: > > > > Dear All, > > > > > > > > First of all, in case a similar thread pops-up authored by myself, > > > > please > > > > ignore. > > > > > > > > Recently we discovered some issues with the FAT32 partition on the > > > > SD-Card > > > > used in our device running on nuttx. There are actually a bunch of > > > > issues > > > > we are strugle to understand related to the filesystem. > > > > The issue discovered recently is, that a statfs call won't return the > > > > true > > > > number of free clusters. It rather returns what ever is in the FS INFO > > > > section of the FAT32 partition. Inserting such a "corrupted" card into > > > > an > > > > SD-Card reader and mounting it to Windows shows following: > > > > > > > > #1 Windows file explorer reports the correct free space. > > > > #2 Comparing the free space in file explorer and looging at FS INFO > > > > section > > > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write > > > > any > > > > longer the FS INFO section on that card. Even if all the files/dires > > > > are > > > > wiped from the card. > > > > #3 Whenever files are added / removed under nuttx the FS INFO section > > > > would > > > > be updated by nuttx, but to the wrong number since the base is wrong. > > > > #4 Mounting the card to linux and properly unmount it might actually > > > > fix > > > > the issue. But not always! > > > > #5 Quick format the card resolved the issue too. Windows would once > > >
RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Hi Reto, What is the clock rate to the card and is multiblock enabled? David -Original Message- From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] Sent: Wednesday, May 19, 2021 5:08 AM To: dev@nuttx.apache.org Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs Hi David, It is running on an STM32h743zi version V or Y. The SD-card in use is a SwissBit S-45u. Reto On 2021/05/19 11:11:22, David Sidrane wrote: > hi Reto, > > What SoC is this on? What type of card are you using? > > David > > -Original Message- > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > Sent: Wednesday, May 19, 2021 3:22 AM > To: dev@nuttx.apache.org > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free > clusters by statfs > > Hello Jukka, > > Thanks for your quick reaction. I applied your PR to our "outdated" nuttx. > Unfortunately, it did not show any impact on the statfs behaviour if the > card is corrupted. Though, meanwhile I do have hands on two images. > > #1 the mentioned one. Although, the files/dirs are all deleted. But still > same behvaiour and a chkdsk error. statfs reports 703 MB free while the > sd-card was deleted! > > #2 Having the file structure in place and contains plenty of logfiles. > Many > of them with the mentioned corruptions. The card is filled to 100%. But > here > the behaviour is slightly different. Checking the FSINFO block and then > removing some files under Windows doesn't impact the FSINFO section. But > once inserted to the device statfs reports the correct size and updates > the > FSINFO section. However, the card also shows corruptions if chkdsk is > applied to it. > This behaviour is identical before and after applying your PR. > > Brgds, Reto > > On 2021/05/18 19:55:06, Jukka Laitinen wrote: > > Hi, > > > > There seems to be a bug in sector calculation which may trigger with > > e.g. corrupted fs. I just made a PR for this, > > > > https://github.com/apache/incubator-nuttx/pull/3740 > > > > But I really don't know if this is related to your issues, this is just > > something that suddenly popped up elsewhere. > > > > -Jukka > > > > > > On 18.5.2021 16.10, Reto Gähwiler wrote: > > > Dear All, > > > > > > First of all, in case a similar thread pops-up authored by myself, > > > please > > > ignore. > > > > > > Recently we discovered some issues with the FAT32 partition on the > > > SD-Card > > > used in our device running on nuttx. There are actually a bunch of > > > issues > > > we are strugle to understand related to the filesystem. > > > The issue discovered recently is, that a statfs call won't return the > > > true > > > number of free clusters. It rather returns what ever is in the FS INFO > > > section of the FAT32 partition. Inserting such a "corrupted" card into > > > an > > > SD-Card reader and mounting it to Windows shows following: > > > > > > #1 Windows file explorer reports the correct free space. > > > #2 Comparing the free space in file explorer and looging at FS INFO > > > section > > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write > > > any > > > longer the FS INFO section on that card. Even if all the files/dires > > > are > > > wiped from the card. > > > #3 Whenever files are added / removed under nuttx the FS INFO section > > > would > > > be updated by nuttx, but to the wrong number since the base is wrong. > > > #4 Mounting the card to linux and properly unmount it might actually > > > fix > > > the issue. But not always! > > > #5 Quick format the card resolved the issue too. Windows would once > > > again > > > write the FS INFO section and nuttx would shows the correct free > > > space. > > > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes > > > the > > > issue too. Reporting a broken file in a root folder on that card. > > > > > > We indeed had issues with corrupted files in the past. Usually in the > > > logging directory, which is accessed most. How they corrupt we do not > > > know. > > > The corruption is recognised in the following ways: > > > > > > #A cryptic and too long file names, we only make use of short file > > > names > > > (8.3) > > > #B files whcih became folders > > > #C sizes which are not possible > > > #D combinations of the above &g
Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Hi David, It is running on an STM32h743zi version V or Y. The SD-card in use is a SwissBit S-45u. Reto On 2021/05/19 11:11:22, David Sidrane wrote: > hi Reto, > > What SoC is this on? What type of card are you using? > > David > > -Original Message- > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > Sent: Wednesday, May 19, 2021 3:22 AM > To: dev@nuttx.apache.org > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free > clusters by statfs > > Hello Jukka, > > Thanks for your quick reaction. I applied your PR to our "outdated" nuttx. > Unfortunately, it did not show any impact on the statfs behaviour if the > card is corrupted. Though, meanwhile I do have hands on two images. > > #1 the mentioned one. Although, the files/dirs are all deleted. But still > same behvaiour and a chkdsk error. statfs reports 703 MB free while the > sd-card was deleted! > > #2 Having the file structure in place and contains plenty of logfiles. Many > of them with the mentioned corruptions. The card is filled to 100%. But here > the behaviour is slightly different. Checking the FSINFO block and then > removing some files under Windows doesn't impact the FSINFO section. But > once inserted to the device statfs reports the correct size and updates the > FSINFO section. However, the card also shows corruptions if chkdsk is > applied to it. > This behaviour is identical before and after applying your PR. > > Brgds, Reto > > On 2021/05/18 19:55:06, Jukka Laitinen wrote: > > Hi, > > > > There seems to be a bug in sector calculation which may trigger with > > e.g. corrupted fs. I just made a PR for this, > > > > https://github.com/apache/incubator-nuttx/pull/3740 > > > > But I really don't know if this is related to your issues, this is just > > something that suddenly popped up elsewhere. > > > > -Jukka > > > > > > On 18.5.2021 16.10, Reto Gähwiler wrote: > > > Dear All, > > > > > > First of all, in case a similar thread pops-up authored by myself, > > > please > > > ignore. > > > > > > Recently we discovered some issues with the FAT32 partition on the > > > SD-Card > > > used in our device running on nuttx. There are actually a bunch of > > > issues > > > we are strugle to understand related to the filesystem. > > > The issue discovered recently is, that a statfs call won't return the > > > true > > > number of free clusters. It rather returns what ever is in the FS INFO > > > section of the FAT32 partition. Inserting such a "corrupted" card into > > > an > > > SD-Card reader and mounting it to Windows shows following: > > > > > > #1 Windows file explorer reports the correct free space. > > > #2 Comparing the free space in file explorer and looging at FS INFO > > > section > > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write any > > > longer the FS INFO section on that card. Even if all the files/dires are > > > wiped from the card. > > > #3 Whenever files are added / removed under nuttx the FS INFO section > > > would > > > be updated by nuttx, but to the wrong number since the base is wrong. > > > #4 Mounting the card to linux and properly unmount it might actually fix > > > the issue. But not always! > > > #5 Quick format the card resolved the issue too. Windows would once > > > again > > > write the FS INFO section and nuttx would shows the correct free space. > > > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes the > > > issue too. Reporting a broken file in a root folder on that card. > > > > > > We indeed had issues with corrupted files in the past. Usually in the > > > logging directory, which is accessed most. How they corrupt we do not > > > know. > > > The corruption is recognised in the following ways: > > > > > > #A cryptic and too long file names, we only make use of short file names > > > (8.3) > > > #B files whcih became folders > > > #C sizes which are not possible > > > #D combinations of the above > > > > > > Our application supervises the SD-Card with statfs and removes files > > > once > > > we hit a quota of 0.85. Therefore, the card is not filling up. However, > > > the > > > most recent recognition of this behvaiour was due to an application bug > > > which allowed the card to fill up under some circumstances. > > &g
RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
hi Reto, What SoC is this on? What type of card are you using? David -Original Message- From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] Sent: Wednesday, May 19, 2021 3:22 AM To: dev@nuttx.apache.org Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs Hello Jukka, Thanks for your quick reaction. I applied your PR to our "outdated" nuttx. Unfortunately, it did not show any impact on the statfs behaviour if the card is corrupted. Though, meanwhile I do have hands on two images. #1 the mentioned one. Although, the files/dirs are all deleted. But still same behvaiour and a chkdsk error. statfs reports 703 MB free while the sd-card was deleted! #2 Having the file structure in place and contains plenty of logfiles. Many of them with the mentioned corruptions. The card is filled to 100%. But here the behaviour is slightly different. Checking the FSINFO block and then removing some files under Windows doesn't impact the FSINFO section. But once inserted to the device statfs reports the correct size and updates the FSINFO section. However, the card also shows corruptions if chkdsk is applied to it. This behaviour is identical before and after applying your PR. Brgds, Reto On 2021/05/18 19:55:06, Jukka Laitinen wrote: > Hi, > > There seems to be a bug in sector calculation which may trigger with > e.g. corrupted fs. I just made a PR for this, > > https://github.com/apache/incubator-nuttx/pull/3740 > > But I really don't know if this is related to your issues, this is just > something that suddenly popped up elsewhere. > > -Jukka > > > On 18.5.2021 16.10, Reto Gähwiler wrote: > > Dear All, > > > > First of all, in case a similar thread pops-up authored by myself, > > please > > ignore. > > > > Recently we discovered some issues with the FAT32 partition on the > > SD-Card > > used in our device running on nuttx. There are actually a bunch of > > issues > > we are strugle to understand related to the filesystem. > > The issue discovered recently is, that a statfs call won't return the > > true > > number of free clusters. It rather returns what ever is in the FS INFO > > section of the FAT32 partition. Inserting such a "corrupted" card into > > an > > SD-Card reader and mounting it to Windows shows following: > > > > #1 Windows file explorer reports the correct free space. > > #2 Comparing the free space in file explorer and looging at FS INFO > > section > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write any > > longer the FS INFO section on that card. Even if all the files/dires are > > wiped from the card. > > #3 Whenever files are added / removed under nuttx the FS INFO section > > would > > be updated by nuttx, but to the wrong number since the base is wrong. > > #4 Mounting the card to linux and properly unmount it might actually fix > > the issue. But not always! > > #5 Quick format the card resolved the issue too. Windows would once > > again > > write the FS INFO section and nuttx would shows the correct free space. > > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes the > > issue too. Reporting a broken file in a root folder on that card. > > > > We indeed had issues with corrupted files in the past. Usually in the > > logging directory, which is accessed most. How they corrupt we do not > > know. > > The corruption is recognised in the following ways: > > > > #A cryptic and too long file names, we only make use of short file names > > (8.3) > > #B files whcih became folders > > #C sizes which are not possible > > #D combinations of the above > > > > Our application supervises the SD-Card with statfs and removes files > > once > > we hit a quota of 0.85. Therefore, the card is not filling up. However, > > the > > most recent recognition of this behvaiour was due to an application bug > > which allowed the card to fill up under some circumstances. > > Playing around with a quickformated SD-Card, filled up with data and > > trying > > to retrigger the issue revealed that I never get the ENOSPC but only EIO > > once the card is full! But never managed to get the statfs issue. > > For the logging we use the fstream family (fopen, fwrite, fread, fflush, > > fsync) commands. For config files and other things also the low level > > read/write are used. > > > > One more thing, among all the logging files we once in a while have > > corrupted file content. Typically that is a skipped byte or doubled byte > > (we use protobuf and can see that encoding the binary). T
Re: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Hey Alan, Thanks for your input. See reply to Jukka. About the dd command, I am aware of and make use of it. Also the reason I could test different things with the same image :) Greetings, Reto On 2021/05/18 20:53:39, Alan Carvalho de Assis wrote: > Hi Reto, > > Mr. Jukka just opened a PR that could help the issue related to the > cluster issue: > > https://github.com/apache/incubator-nuttx/pull/3740 > > When you face an issue with the SDCard a good idea is to do a raw copy > of it on Linux using the dd command, then you could duplicated the > issue later. > > If you find an easy way to reproduce the issue, it will make it easier > for other people duplicate your issue and help to debug it. > > BR, > > Alan > > On 5/18/21, Reto Gähwiler wrote: > > Dear All, > > > > First of all, in case a similar thread pops-up authored by myself, please > > ignore. > > > > Recently we discovered some issues with the FAT32 partition on the SD-Card > > used in our device running on nuttx. There are actually a bunch of issues > > we are strugle to understand related to the filesystem. > > The issue discovered recently is, that a statfs call won't return the true > > number of free clusters. It rather returns what ever is in the FS INFO > > section of the FAT32 partition. Inserting such a "corrupted" card into an > > SD-Card reader and mounting it to Windows shows following: > > > > #1 Windows file explorer reports the correct free space. > > #2 Comparing the free space in file explorer and looging at FS INFO section > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write any > > longer the FS INFO section on that card. Even if all the files/dires are > > wiped from the card. > > #3 Whenever files are added / removed under nuttx the FS INFO section would > > be updated by nuttx, but to the wrong number since the base is wrong. > > #4 Mounting the card to linux and properly unmount it might actually fix > > the issue. But not always! > > #5 Quick format the card resolved the issue too. Windows would once again > > write the FS INFO section and nuttx would shows the correct free space. > > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes the > > issue too. Reporting a broken file in a root folder on that card. > > > > We indeed had issues with corrupted files in the past. Usually in the > > logging directory, which is accessed most. How they corrupt we do not know. > > The corruption is recognised in the following ways: > > > > #A cryptic and too long file names, we only make use of short file names > > (8.3) > > #B files whcih became folders > > #C sizes which are not possible > > #D combinations of the above > > > > Our application supervises the SD-Card with statfs and removes files once > > we hit a quota of 0.85. Therefore, the card is not filling up. However, the > > most recent recognition of this behvaiour was due to an application bug > > which allowed the card to fill up under some circumstances. > > Playing around with a quickformated SD-Card, filled up with data and trying > > to retrigger the issue revealed that I never get the ENOSPC but only EIO > > once the card is full! But never managed to get the statfs issue. > > For the logging we use the fstream family (fopen, fwrite, fread, fflush, > > fsync) commands. For config files and other things also the low level > > read/write are used. > > > > One more thing, among all the logging files we once in a while have > > corrupted file content. Typically that is a skipped byte or doubled byte > > (we use protobuf and can see that encoding the binary). This seems to > > happen during reading but also writing the file. > > > > At the moment we didn't investigate the fat driver of nuttx but rather > > tried to analyse the problem. And now we have a bunch of question marks. > > > > #1 Has anyone recongised similar issues? > > #2 Does anyone know a tool which can be used in nuttx equivalent to chkdsk > > under windows? Or do quick format? > > #3 Why is there no ENOSPC but EIO once the card is full? > > #4 What triggers the corrupted files? How robust is the nuttx FAT32 > > considering stackdumps and powerlosses? > > > > For the purpose of investigation, I have an image of the SD-Card when it > > was in the state of reporting wrong sizes in nuttx. > > > > Looking forward to your answers and ideas. Appreciate your help, > > > > Reto > > >
Re: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Hello Jukka, Thanks for your quick reaction. I applied your PR to our "outdated" nuttx. Unfortunately, it did not show any impact on the statfs behaviour if the card is corrupted. Though, meanwhile I do have hands on two images. #1 the mentioned one. Although, the files/dirs are all deleted. But still same behvaiour and a chkdsk error. statfs reports 703 MB free while the sd-card was deleted! #2 Having the file structure in place and contains plenty of logfiles. Many of them with the mentioned corruptions. The card is filled to 100%. But here the behaviour is slightly different. Checking the FSINFO block and then removing some files under Windows doesn't impact the FSINFO section. But once inserted to the device statfs reports the correct size and updates the FSINFO section. However, the card also shows corruptions if chkdsk is applied to it. This behaviour is identical before and after applying your PR. Brgds, Reto On 2021/05/18 19:55:06, Jukka Laitinen wrote: > Hi, > > There seems to be a bug in sector calculation which may trigger with > e.g. corrupted fs. I just made a PR for this, > > https://github.com/apache/incubator-nuttx/pull/3740 > > But I really don't know if this is related to your issues, this is just > something that suddenly popped up elsewhere. > > -Jukka > > > On 18.5.2021 16.10, Reto Gähwiler wrote: > > Dear All, > > > > First of all, in case a similar thread pops-up authored by myself, please > > ignore. > > > > Recently we discovered some issues with the FAT32 partition on the SD-Card > > used in our device running on nuttx. There are actually a bunch of issues > > we are strugle to understand related to the filesystem. > > The issue discovered recently is, that a statfs call won't return the true > > number of free clusters. It rather returns what ever is in the FS INFO > > section of the FAT32 partition. Inserting such a "corrupted" card into an > > SD-Card reader and mounting it to Windows shows following: > > > > #1 Windows file explorer reports the correct free space. > > #2 Comparing the free space in file explorer and looging at FS INFO section > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write any > > longer the FS INFO section on that card. Even if all the files/dires are > > wiped from the card. > > #3 Whenever files are added / removed under nuttx the FS INFO section would > > be updated by nuttx, but to the wrong number since the base is wrong. > > #4 Mounting the card to linux and properly unmount it might actually fix > > the issue. But not always! > > #5 Quick format the card resolved the issue too. Windows would once again > > write the FS INFO section and nuttx would shows the correct free space. > > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes the > > issue too. Reporting a broken file in a root folder on that card. > > > > We indeed had issues with corrupted files in the past. Usually in the > > logging directory, which is accessed most. How they corrupt we do not know. > > The corruption is recognised in the following ways: > > > > #A cryptic and too long file names, we only make use of short file names > > (8.3) > > #B files whcih became folders > > #C sizes which are not possible > > #D combinations of the above > > > > Our application supervises the SD-Card with statfs and removes files once > > we hit a quota of 0.85. Therefore, the card is not filling up. However, the > > most recent recognition of this behvaiour was due to an application bug > > which allowed the card to fill up under some circumstances. > > Playing around with a quickformated SD-Card, filled up with data and trying > > to retrigger the issue revealed that I never get the ENOSPC but only EIO > > once the card is full! But never managed to get the statfs issue. > > For the logging we use the fstream family (fopen, fwrite, fread, fflush, > > fsync) commands. For config files and other things also the low level > > read/write are used. > > > > One more thing, among all the logging files we once in a while have > > corrupted file content. Typically that is a skipped byte or doubled byte > > (we use protobuf and can see that encoding the binary). This seems to > > happen during reading but also writing the file. > > > > At the moment we didn't investigate the fat driver of nuttx but rather > > tried to analyse the problem. And now we have a bunch of question marks. > > > > #1 Has anyone recongised similar issues? > > #2 Does anyone know a tool which can be used in nuttx equivalent to chkdsk > > under windows? Or do quick format? > > #3 Why is there no ENOSPC but EIO once the card is full? > > #4 What triggers the corrupted files? How robust is the nuttx FAT32 > > considering stackdumps and powerlosses? > > > > For the purpose of investigation, I have an image of the SD-Card when it > > was in the state of reporting wrong sizes in nuttx. > > > > Looking forward to your answers and ideas. Appreciate your help, > > > > Reto > >
Re: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Hi Reto, Mr. Jukka just opened a PR that could help the issue related to the cluster issue: https://github.com/apache/incubator-nuttx/pull/3740 When you face an issue with the SDCard a good idea is to do a raw copy of it on Linux using the dd command, then you could duplicated the issue later. If you find an easy way to reproduce the issue, it will make it easier for other people duplicate your issue and help to debug it. BR, Alan On 5/18/21, Reto Gähwiler wrote: > Dear All, > > First of all, in case a similar thread pops-up authored by myself, please > ignore. > > Recently we discovered some issues with the FAT32 partition on the SD-Card > used in our device running on nuttx. There are actually a bunch of issues > we are strugle to understand related to the filesystem. > The issue discovered recently is, that a statfs call won't return the true > number of free clusters. It rather returns what ever is in the FS INFO > section of the FAT32 partition. Inserting such a "corrupted" card into an > SD-Card reader and mounting it to Windows shows following: > > #1 Windows file explorer reports the correct free space. > #2 Comparing the free space in file explorer and looging at FS INFO section > with "Active - Disk Editor" doesn't line up. Windows wouldn't write any > longer the FS INFO section on that card. Even if all the files/dires are > wiped from the card. > #3 Whenever files are added / removed under nuttx the FS INFO section would > be updated by nuttx, but to the wrong number since the base is wrong. > #4 Mounting the card to linux and properly unmount it might actually fix > the issue. But not always! > #5 Quick format the card resolved the issue too. Windows would once again > write the FS INFO section and nuttx would shows the correct free space. > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes the > issue too. Reporting a broken file in a root folder on that card. > > We indeed had issues with corrupted files in the past. Usually in the > logging directory, which is accessed most. How they corrupt we do not know. > The corruption is recognised in the following ways: > > #A cryptic and too long file names, we only make use of short file names > (8.3) > #B files whcih became folders > #C sizes which are not possible > #D combinations of the above > > Our application supervises the SD-Card with statfs and removes files once > we hit a quota of 0.85. Therefore, the card is not filling up. However, the > most recent recognition of this behvaiour was due to an application bug > which allowed the card to fill up under some circumstances. > Playing around with a quickformated SD-Card, filled up with data and trying > to retrigger the issue revealed that I never get the ENOSPC but only EIO > once the card is full! But never managed to get the statfs issue. > For the logging we use the fstream family (fopen, fwrite, fread, fflush, > fsync) commands. For config files and other things also the low level > read/write are used. > > One more thing, among all the logging files we once in a while have > corrupted file content. Typically that is a skipped byte or doubled byte > (we use protobuf and can see that encoding the binary). This seems to > happen during reading but also writing the file. > > At the moment we didn't investigate the fat driver of nuttx but rather > tried to analyse the problem. And now we have a bunch of question marks. > > #1 Has anyone recongised similar issues? > #2 Does anyone know a tool which can be used in nuttx equivalent to chkdsk > under windows? Or do quick format? > #3 Why is there no ENOSPC but EIO once the card is full? > #4 What triggers the corrupted files? How robust is the nuttx FAT32 > considering stackdumps and powerlosses? > > For the purpose of investigation, I have an image of the SD-Card when it > was in the state of reporting wrong sizes in nuttx. > > Looking forward to your answers and ideas. Appreciate your help, > > Reto >
