Re: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Good Morning Alan, Can't really tell you why, decisions made by the hardware designer. Bet bet I have. Unfortunately the layout is hardwired like this, so no qspi or SDIO available without a change. Reto On 2021/05/19 15:27:38, Alan Carvalho de Assis wrote: > Hi Reto, > > Just a curiosity: why are you using SDCard support over SPI? > Isn't your board prepared to use native SDCard/SDIO peripheral? > > BR, > > Alan > > On 5/19/21, Reto Gähwiler wrote: > > Hey again, > > > > About the requested configuration of the SD-Card, here we go: > > CONFIG_MMCSD=y > > CONFIG_MMCSD_SPI=y > > CONFIG_MMCSD_SPICLOCK=2500 > > > > About the multiblock, I don't think we are using that. Let me know how I > > could recognise it if not in the config. > > > > Thanks, Reto > > > > On 2021/05/19 12:13:35, David Sidrane wrote: > >> Hi Reto, > >> > >> What is the clock rate to the card and is multiblock enabled? > >> > >> David > >> > >> -----Original Message- > >> From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > >> Sent: Wednesday, May 19, 2021 5:08 AM > >> To: dev@nuttx.apache.org > >> Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated > >> free > >> clusters by statfs > >> > >> Hi David, > >> > >> It is running on an STM32h743zi version V or Y. The SD-card in use is a > >> SwissBit S-45u. > >> > >> Reto > >> > >> On 2021/05/19 11:11:22, David Sidrane wrote: > >> > hi Reto, > >> > > >> > What SoC is this on? What type of card are you using? > >> > > >> > David > >> > > >> > -Original Message- > >> > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > >> > Sent: Wednesday, May 19, 2021 3:22 AM > >> > To: dev@nuttx.apache.org > >> > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free > >> > clusters by statfs > >> > > >> > Hello Jukka, > >> > > >> > Thanks for your quick reaction. I applied your PR to our "outdated" > >> > nuttx. > >> > Unfortunately, it did not show any impact on the statfs behaviour if > >> > the > >> > card is corrupted. Though, meanwhile I do have hands on two images. > >> > > >> > #1 the mentioned one. Although, the files/dirs are all deleted. But > >> > still > >> > same behvaiour and a chkdsk error. statfs reports 703 MB free while the > >> > sd-card was deleted! > >> > > >> > #2 Having the file structure in place and contains plenty of logfiles. > >> > Many > >> > of them with the mentioned corruptions. The card is filled to 100%. But > >> > here > >> > the behaviour is slightly different. Checking the FSINFO block and then > >> > removing some files under Windows doesn't impact the FSINFO section. > >> > But > >> > once inserted to the device statfs reports the correct size and updates > >> > the > >> > FSINFO section. However, the card also shows corruptions if chkdsk is > >> > applied to it. > >> > This behaviour is identical before and after applying your PR. > >> > > >> > Brgds, Reto > >> > > >> > On 2021/05/18 19:55:06, Jukka Laitinen wrote: > >> > > Hi, > >> > > > >> > > There seems to be a bug in sector calculation which may trigger with > >> > > e.g. corrupted fs. I just made a PR for this, > >> > > > >> > > https://github.com/apache/incubator-nuttx/pull/3740 > >> > > > >> > > But I really don't know if this is related to your issues, this is > >> > > just > >> > > something that suddenly popped up elsewhere. > >> > > > >> > > -Jukka > >> > > > >> > > > >> > > On 18.5.2021 16.10, Reto Gähwiler wrote: > >> > > > Dear All, > >> > > > > >> > > > First of all, in case a similar thread pops-up authored by myself, > >> > > > please > >> > > > ignore. > >> > > > > >> > > > Recently we discovered some issues with the FAT32 partition on the > >> > > > SD-Card > >> > > > used in our device running on nuttx. There are actually a b
RE: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Hi Reto, The multiblock is a disable so if CONFIG_MMCSD_MULTIBLOCK_DISABLE=y is not in the config it is using multiblock However, that is only for the SDMMC driver. What I was thinking was [1],[2] for the SDMMC driver not the SPI one. [1] https://github.com/apache/incubator-nuttx/pull/3669 [2] https://github.com/PX4/PX4-Autopilot/issues/17442#issuecomment-835303021 David -Original Message- From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] Sent: Wednesday, May 19, 2021 8:21 AM To: dev@nuttx.apache.org Subject: Re: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs Hey again, About the requested configuration of the SD-Card, here we go: CONFIG_MMCSD=y CONFIG_MMCSD_SPI=y CONFIG_MMCSD_SPICLOCK=2500 About the multiblock, I don't think we are using that. Let me know how I could recognise it if not in the config. Thanks, Reto On 2021/05/19 12:13:35, David Sidrane wrote: > Hi Reto, > > What is the clock rate to the card and is multiblock enabled? > > David > > -Original Message- > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > Sent: Wednesday, May 19, 2021 5:08 AM > To: dev@nuttx.apache.org > Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free > clusters by statfs > > Hi David, > > It is running on an STM32h743zi version V or Y. The SD-card in use is a > SwissBit S-45u. > > Reto > > On 2021/05/19 11:11:22, David Sidrane wrote: > > hi Reto, > > > > What SoC is this on? What type of card are you using? > > > > David > > > > -Original Message- > > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > > Sent: Wednesday, May 19, 2021 3:22 AM > > To: dev@nuttx.apache.org > > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free > > clusters by statfs > > > > Hello Jukka, > > > > Thanks for your quick reaction. I applied your PR to our "outdated" > > nuttx. > > Unfortunately, it did not show any impact on the statfs behaviour if the > > card is corrupted. Though, meanwhile I do have hands on two images. > > > > #1 the mentioned one. Although, the files/dirs are all deleted. But > > still > > same behvaiour and a chkdsk error. statfs reports 703 MB free while the > > sd-card was deleted! > > > > #2 Having the file structure in place and contains plenty of logfiles. > > Many > > of them with the mentioned corruptions. The card is filled to 100%. But > > here > > the behaviour is slightly different. Checking the FSINFO block and then > > removing some files under Windows doesn't impact the FSINFO section. But > > once inserted to the device statfs reports the correct size and updates > > the > > FSINFO section. However, the card also shows corruptions if chkdsk is > > applied to it. > > This behaviour is identical before and after applying your PR. > > > > Brgds, Reto > > > > On 2021/05/18 19:55:06, Jukka Laitinen wrote: > > > Hi, > > > > > > There seems to be a bug in sector calculation which may trigger with > > > e.g. corrupted fs. I just made a PR for this, > > > > > > https://github.com/apache/incubator-nuttx/pull/3740 > > > > > > But I really don't know if this is related to your issues, this is > > > just > > > something that suddenly popped up elsewhere. > > > > > > -Jukka > > > > > > > > > On 18.5.2021 16.10, Reto Gähwiler wrote: > > > > Dear All, > > > > > > > > First of all, in case a similar thread pops-up authored by myself, > > > > please > > > > ignore. > > > > > > > > Recently we discovered some issues with the FAT32 partition on the > > > > SD-Card > > > > used in our device running on nuttx. There are actually a bunch of > > > > issues > > > > we are strugle to understand related to the filesystem. > > > > The issue discovered recently is, that a statfs call won't return > > > > the > > > > true > > > > number of free clusters. It rather returns what ever is in the FS > > > > INFO > > > > section of the FAT32 partition. Inserting such a "corrupted" card > > > > into > > > > an > > > > SD-Card reader and mounting it to Windows shows following: > > > > > > > > #1 Windows file explorer reports the correct free space. > > > > #2 Comparing the free space in file explorer and looging at FS INFO > > > > section > > > > wit
Re: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Hi Reto, Just a curiosity: why are you using SDCard support over SPI? Isn't your board prepared to use native SDCard/SDIO peripheral? BR, Alan On 5/19/21, Reto Gähwiler wrote: > Hey again, > > About the requested configuration of the SD-Card, here we go: > CONFIG_MMCSD=y > CONFIG_MMCSD_SPI=y > CONFIG_MMCSD_SPICLOCK=2500 > > About the multiblock, I don't think we are using that. Let me know how I > could recognise it if not in the config. > > Thanks, Reto > > On 2021/05/19 12:13:35, David Sidrane wrote: >> Hi Reto, >> >> What is the clock rate to the card and is multiblock enabled? >> >> David >> >> -Original Message- >> From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] >> Sent: Wednesday, May 19, 2021 5:08 AM >> To: dev@nuttx.apache.org >> Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated >> free >> clusters by statfs >> >> Hi David, >> >> It is running on an STM32h743zi version V or Y. The SD-card in use is a >> SwissBit S-45u. >> >> Reto >> >> On 2021/05/19 11:11:22, David Sidrane wrote: >> > hi Reto, >> > >> > What SoC is this on? What type of card are you using? >> > >> > David >> > >> > -Original Message----- >> > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] >> > Sent: Wednesday, May 19, 2021 3:22 AM >> > To: dev@nuttx.apache.org >> > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free >> > clusters by statfs >> > >> > Hello Jukka, >> > >> > Thanks for your quick reaction. I applied your PR to our "outdated" >> > nuttx. >> > Unfortunately, it did not show any impact on the statfs behaviour if >> > the >> > card is corrupted. Though, meanwhile I do have hands on two images. >> > >> > #1 the mentioned one. Although, the files/dirs are all deleted. But >> > still >> > same behvaiour and a chkdsk error. statfs reports 703 MB free while the >> > sd-card was deleted! >> > >> > #2 Having the file structure in place and contains plenty of logfiles. >> > Many >> > of them with the mentioned corruptions. The card is filled to 100%. But >> > here >> > the behaviour is slightly different. Checking the FSINFO block and then >> > removing some files under Windows doesn't impact the FSINFO section. >> > But >> > once inserted to the device statfs reports the correct size and updates >> > the >> > FSINFO section. However, the card also shows corruptions if chkdsk is >> > applied to it. >> > This behaviour is identical before and after applying your PR. >> > >> > Brgds, Reto >> > >> > On 2021/05/18 19:55:06, Jukka Laitinen wrote: >> > > Hi, >> > > >> > > There seems to be a bug in sector calculation which may trigger with >> > > e.g. corrupted fs. I just made a PR for this, >> > > >> > > https://github.com/apache/incubator-nuttx/pull/3740 >> > > >> > > But I really don't know if this is related to your issues, this is >> > > just >> > > something that suddenly popped up elsewhere. >> > > >> > > -Jukka >> > > >> > > >> > > On 18.5.2021 16.10, Reto Gähwiler wrote: >> > > > Dear All, >> > > > >> > > > First of all, in case a similar thread pops-up authored by myself, >> > > > please >> > > > ignore. >> > > > >> > > > Recently we discovered some issues with the FAT32 partition on the >> > > > SD-Card >> > > > used in our device running on nuttx. There are actually a bunch of >> > > > issues >> > > > we are strugle to understand related to the filesystem. >> > > > The issue discovered recently is, that a statfs call won't return >> > > > the >> > > > true >> > > > number of free clusters. It rather returns what ever is in the FS >> > > > INFO >> > > > section of the FAT32 partition. Inserting such a "corrupted" card >> > > > into >> > > > an >> > > > SD-Card reader and mounting it to Windows shows following: >> > > > >> > > > #1 Windows file explorer reports the correct free space. >> > > > #2 Comparing the free space in file explorer and looging
Re: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Hey again, About the requested configuration of the SD-Card, here we go: CONFIG_MMCSD=y CONFIG_MMCSD_SPI=y CONFIG_MMCSD_SPICLOCK=2500 About the multiblock, I don't think we are using that. Let me know how I could recognise it if not in the config. Thanks, Reto On 2021/05/19 12:13:35, David Sidrane wrote: > Hi Reto, > > What is the clock rate to the card and is multiblock enabled? > > David > > -Original Message- > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > Sent: Wednesday, May 19, 2021 5:08 AM > To: dev@nuttx.apache.org > Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free > clusters by statfs > > Hi David, > > It is running on an STM32h743zi version V or Y. The SD-card in use is a > SwissBit S-45u. > > Reto > > On 2021/05/19 11:11:22, David Sidrane wrote: > > hi Reto, > > > > What SoC is this on? What type of card are you using? > > > > David > > > > -Original Message- > > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > > Sent: Wednesday, May 19, 2021 3:22 AM > > To: dev@nuttx.apache.org > > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free > > clusters by statfs > > > > Hello Jukka, > > > > Thanks for your quick reaction. I applied your PR to our "outdated" nuttx. > > Unfortunately, it did not show any impact on the statfs behaviour if the > > card is corrupted. Though, meanwhile I do have hands on two images. > > > > #1 the mentioned one. Although, the files/dirs are all deleted. But still > > same behvaiour and a chkdsk error. statfs reports 703 MB free while the > > sd-card was deleted! > > > > #2 Having the file structure in place and contains plenty of logfiles. > > Many > > of them with the mentioned corruptions. The card is filled to 100%. But > > here > > the behaviour is slightly different. Checking the FSINFO block and then > > removing some files under Windows doesn't impact the FSINFO section. But > > once inserted to the device statfs reports the correct size and updates > > the > > FSINFO section. However, the card also shows corruptions if chkdsk is > > applied to it. > > This behaviour is identical before and after applying your PR. > > > > Brgds, Reto > > > > On 2021/05/18 19:55:06, Jukka Laitinen wrote: > > > Hi, > > > > > > There seems to be a bug in sector calculation which may trigger with > > > e.g. corrupted fs. I just made a PR for this, > > > > > > https://github.com/apache/incubator-nuttx/pull/3740 > > > > > > But I really don't know if this is related to your issues, this is just > > > something that suddenly popped up elsewhere. > > > > > > -Jukka > > > > > > > > > On 18.5.2021 16.10, Reto Gähwiler wrote: > > > > Dear All, > > > > > > > > First of all, in case a similar thread pops-up authored by myself, > > > > please > > > > ignore. > > > > > > > > Recently we discovered some issues with the FAT32 partition on the > > > > SD-Card > > > > used in our device running on nuttx. There are actually a bunch of > > > > issues > > > > we are strugle to understand related to the filesystem. > > > > The issue discovered recently is, that a statfs call won't return the > > > > true > > > > number of free clusters. It rather returns what ever is in the FS INFO > > > > section of the FAT32 partition. Inserting such a "corrupted" card into > > > > an > > > > SD-Card reader and mounting it to Windows shows following: > > > > > > > > #1 Windows file explorer reports the correct free space. > > > > #2 Comparing the free space in file explorer and looging at FS INFO > > > > section > > > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write > > > > any > > > > longer the FS INFO section on that card. Even if all the files/dires > > > > are > > > > wiped from the card. > > > > #3 Whenever files are added / removed under nuttx the FS INFO section > > > > would > > > > be updated by nuttx, but to the wrong number since the base is wrong. > > > > #4 Mounting the card to linux and properly unmount it might actually > > > > fix > > > > the issue. But not always! > > > > #5 Quick format the card resolved the issue too. Windows would once > > >
RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Hi Reto, What is the clock rate to the card and is multiblock enabled? David -Original Message- From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] Sent: Wednesday, May 19, 2021 5:08 AM To: dev@nuttx.apache.org Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs Hi David, It is running on an STM32h743zi version V or Y. The SD-card in use is a SwissBit S-45u. Reto On 2021/05/19 11:11:22, David Sidrane wrote: > hi Reto, > > What SoC is this on? What type of card are you using? > > David > > -Original Message- > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > Sent: Wednesday, May 19, 2021 3:22 AM > To: dev@nuttx.apache.org > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free > clusters by statfs > > Hello Jukka, > > Thanks for your quick reaction. I applied your PR to our "outdated" nuttx. > Unfortunately, it did not show any impact on the statfs behaviour if the > card is corrupted. Though, meanwhile I do have hands on two images. > > #1 the mentioned one. Although, the files/dirs are all deleted. But still > same behvaiour and a chkdsk error. statfs reports 703 MB free while the > sd-card was deleted! > > #2 Having the file structure in place and contains plenty of logfiles. > Many > of them with the mentioned corruptions. The card is filled to 100%. But > here > the behaviour is slightly different. Checking the FSINFO block and then > removing some files under Windows doesn't impact the FSINFO section. But > once inserted to the device statfs reports the correct size and updates > the > FSINFO section. However, the card also shows corruptions if chkdsk is > applied to it. > This behaviour is identical before and after applying your PR. > > Brgds, Reto > > On 2021/05/18 19:55:06, Jukka Laitinen wrote: > > Hi, > > > > There seems to be a bug in sector calculation which may trigger with > > e.g. corrupted fs. I just made a PR for this, > > > > https://github.com/apache/incubator-nuttx/pull/3740 > > > > But I really don't know if this is related to your issues, this is just > > something that suddenly popped up elsewhere. > > > > -Jukka > > > > > > On 18.5.2021 16.10, Reto Gähwiler wrote: > > > Dear All, > > > > > > First of all, in case a similar thread pops-up authored by myself, > > > please > > > ignore. > > > > > > Recently we discovered some issues with the FAT32 partition on the > > > SD-Card > > > used in our device running on nuttx. There are actually a bunch of > > > issues > > > we are strugle to understand related to the filesystem. > > > The issue discovered recently is, that a statfs call won't return the > > > true > > > number of free clusters. It rather returns what ever is in the FS INFO > > > section of the FAT32 partition. Inserting such a "corrupted" card into > > > an > > > SD-Card reader and mounting it to Windows shows following: > > > > > > #1 Windows file explorer reports the correct free space. > > > #2 Comparing the free space in file explorer and looging at FS INFO > > > section > > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write > > > any > > > longer the FS INFO section on that card. Even if all the files/dires > > > are > > > wiped from the card. > > > #3 Whenever files are added / removed under nuttx the FS INFO section > > > would > > > be updated by nuttx, but to the wrong number since the base is wrong. > > > #4 Mounting the card to linux and properly unmount it might actually > > > fix > > > the issue. But not always! > > > #5 Quick format the card resolved the issue too. Windows would once > > > again > > > write the FS INFO section and nuttx would shows the correct free > > > space. > > > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes > > > the > > > issue too. Reporting a broken file in a root folder on that card. > > > > > > We indeed had issues with corrupted files in the past. Usually in the > > > logging directory, which is accessed most. How they corrupt we do not > > > know. > > > The corruption is recognised in the following ways: > > > > > > #A cryptic and too long file names, we only make use of short file > > > names > > > (8.3) > > > #B files whcih became folders > > > #C sizes which are not possible > > > #D combinations of the above &g
Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs
Hi David, It is running on an STM32h743zi version V or Y. The SD-card in use is a SwissBit S-45u. Reto On 2021/05/19 11:11:22, David Sidrane wrote: > hi Reto, > > What SoC is this on? What type of card are you using? > > David > > -Original Message- > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com] > Sent: Wednesday, May 19, 2021 3:22 AM > To: dev@nuttx.apache.org > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free > clusters by statfs > > Hello Jukka, > > Thanks for your quick reaction. I applied your PR to our "outdated" nuttx. > Unfortunately, it did not show any impact on the statfs behaviour if the > card is corrupted. Though, meanwhile I do have hands on two images. > > #1 the mentioned one. Although, the files/dirs are all deleted. But still > same behvaiour and a chkdsk error. statfs reports 703 MB free while the > sd-card was deleted! > > #2 Having the file structure in place and contains plenty of logfiles. Many > of them with the mentioned corruptions. The card is filled to 100%. But here > the behaviour is slightly different. Checking the FSINFO block and then > removing some files under Windows doesn't impact the FSINFO section. But > once inserted to the device statfs reports the correct size and updates the > FSINFO section. However, the card also shows corruptions if chkdsk is > applied to it. > This behaviour is identical before and after applying your PR. > > Brgds, Reto > > On 2021/05/18 19:55:06, Jukka Laitinen wrote: > > Hi, > > > > There seems to be a bug in sector calculation which may trigger with > > e.g. corrupted fs. I just made a PR for this, > > > > https://github.com/apache/incubator-nuttx/pull/3740 > > > > But I really don't know if this is related to your issues, this is just > > something that suddenly popped up elsewhere. > > > > -Jukka > > > > > > On 18.5.2021 16.10, Reto Gähwiler wrote: > > > Dear All, > > > > > > First of all, in case a similar thread pops-up authored by myself, > > > please > > > ignore. > > > > > > Recently we discovered some issues with the FAT32 partition on the > > > SD-Card > > > used in our device running on nuttx. There are actually a bunch of > > > issues > > > we are strugle to understand related to the filesystem. > > > The issue discovered recently is, that a statfs call won't return the > > > true > > > number of free clusters. It rather returns what ever is in the FS INFO > > > section of the FAT32 partition. Inserting such a "corrupted" card into > > > an > > > SD-Card reader and mounting it to Windows shows following: > > > > > > #1 Windows file explorer reports the correct free space. > > > #2 Comparing the free space in file explorer and looging at FS INFO > > > section > > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write any > > > longer the FS INFO section on that card. Even if all the files/dires are > > > wiped from the card. > > > #3 Whenever files are added / removed under nuttx the FS INFO section > > > would > > > be updated by nuttx, but to the wrong number since the base is wrong. > > > #4 Mounting the card to linux and properly unmount it might actually fix > > > the issue. But not always! > > > #5 Quick format the card resolved the issue too. Windows would once > > > again > > > write the FS INFO section and nuttx would shows the correct free space. > > > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes the > > > issue too. Reporting a broken file in a root folder on that card. > > > > > > We indeed had issues with corrupted files in the past. Usually in the > > > logging directory, which is accessed most. How they corrupt we do not > > > know. > > > The corruption is recognised in the following ways: > > > > > > #A cryptic and too long file names, we only make use of short file names > > > (8.3) > > > #B files whcih became folders > > > #C sizes which are not possible > > > #D combinations of the above > > > > > > Our application supervises the SD-Card with statfs and removes files > > > once > > > we hit a quota of 0.85. Therefore, the card is not filling up. However, > > > the > > > most recent recognition of this behvaiour was due to an application bug > > > which allowed the card to fill up under some circumstances. > > &g
