Re: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs

2021-05-19 Thread Reto Gähwiler
Good Morning Alan,

Can't really tell you why, decisions made by the hardware designer. Bet bet  I 
have. Unfortunately the layout is hardwired like this, so no qspi or SDIO 
available without a change.

Reto

On 2021/05/19 15:27:38, Alan Carvalho de Assis  wrote: 
> Hi Reto,
> 
> Just a curiosity: why are you using SDCard support over SPI?
> Isn't your board prepared to use native SDCard/SDIO peripheral?
> 
> BR,
> 
> Alan
> 
> On 5/19/21, Reto Gähwiler  wrote:
> > Hey again,
> >
> > About the requested configuration of the SD-Card, here we go:
> >   CONFIG_MMCSD=y
> >   CONFIG_MMCSD_SPI=y
> >   CONFIG_MMCSD_SPICLOCK=2500
> >
> > About the multiblock, I don't think we are using that. Let me know how I
> > could recognise it if not in the config.
> >
> > Thanks, Reto
> >
> > On 2021/05/19 12:13:35, David Sidrane  wrote:
> >> Hi Reto,
> >>
> >> What is the clock rate to the card and is multiblock enabled?
> >>
> >> David
> >>
> >> -Original Message-
> >> From: Reto Gähwiler [mailto:gret.hexa...@gmail.com]
> >> Sent: Wednesday, May 19, 2021 5:08 AM
> >> To: dev@nuttx.apache.org
> >> Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated
> >> free
> >> clusters by statfs
> >>
> >> Hi David,
> >>
> >> It is running on an STM32h743zi version V or Y. The SD-card in use is a
> >> SwissBit S-45u.
> >>
> >> Reto
> >>
> >> On 2021/05/19 11:11:22, David Sidrane  wrote:
> >> > hi Reto,
> >> >
> >> > What SoC is this on? What type of card are you using?
> >> >
> >> > David
> >> >
> >> > -Original Message-
> >> > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com]
> >> > Sent: Wednesday, May 19, 2021 3:22 AM
> >> > To: dev@nuttx.apache.org
> >> > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free
> >> > clusters by statfs
> >> >
> >> > Hello Jukka,
> >> >
> >> > Thanks for your quick reaction. I applied your PR to our "outdated"
> >> > nuttx.
> >> > Unfortunately, it did not show any impact on the statfs behaviour if
> >> > the
> >> > card is corrupted. Though, meanwhile I do have hands on two images.
> >> >
> >> > #1 the mentioned one. Although, the files/dirs are all deleted. But
> >> > still
> >> > same behvaiour and a chkdsk error. statfs reports 703 MB free while the
> >> > sd-card was deleted!
> >> >
> >> > #2 Having the file structure in place and contains plenty of logfiles.
> >> > Many
> >> > of them with the mentioned corruptions. The card is filled to 100%. But
> >> > here
> >> > the behaviour is slightly different. Checking the FSINFO block and then
> >> > removing some files under Windows doesn't impact the FSINFO section.
> >> > But
> >> > once inserted to the device statfs reports the correct size and updates
> >> > the
> >> > FSINFO section. However, the card also shows corruptions if chkdsk is
> >> > applied to it.
> >> > This behaviour is identical before and after applying your PR.
> >> >
> >> > Brgds, Reto
> >> >
> >> > On 2021/05/18 19:55:06, Jukka Laitinen  wrote:
> >> > > Hi,
> >> > >
> >> > > There seems to be a bug in sector calculation which may trigger with
> >> > > e.g. corrupted fs. I just made a PR for this,
> >> > >
> >> > > https://github.com/apache/incubator-nuttx/pull/3740
> >> > >
> >> > > But I really don't know if this is related to your issues, this is
> >> > > just
> >> > > something that suddenly popped up elsewhere.
> >> > >
> >> > > -Jukka
> >> > >
> >> > >
> >> > > On 18.5.2021 16.10, Reto Gähwiler wrote:
> >> > > > Dear All,
> >> > > >
> >> > > > First of all, in case a similar thread pops-up authored by myself,
> >> > > > please
> >> > > > ignore.
> >> > > >
> >> > > > Recently we discovered some issues with the FAT32 partition on the
> >> > > > SD-Card
> >> > > > used in our device running on nuttx. There are actually a bunch of
> >> > > > issues
> >> > > > we are strugle to understand related to the filesystem.
> >> > > > The issue discovered recently is, that a statfs call won't return
> >> > > > the
> >> > > > true
> >> > > > number of free clusters. It rather returns what ever is in the FS
> >> > > > INFO
> >> > > > section of the FAT32 partition. Inserting such a "corrupted" card
> >> > > > into
> >> > > > an
> >> > > > SD-Card reader and mounting it to Windows shows following:
> >> > > >
> >> > > > #1 Windows file explorer reports the correct free space.
> >> > > > #2 Comparing the free space in file explorer and looging at FS INFO
> >> > > > section
> >> > > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write
> >> > > > any
> >> > > > longer the FS INFO section on that card. Even if all the
> >> > > > files/dires
> >> > > > are
> >> > > > wiped from the card.
> >> > > > #3 Whenever files are added / removed under nuttx the FS INFO
> >> > > > section
> >> > > > would
> >> > > > be updated by nuttx, but to the wrong number since the base is
> >> > > > wrong.
> >> > > > #4 Mounting the card to linux and properly unmount it might
> >> > > > actually
> >> > > > fix
> >> > > > the issue. But not 

Re: Ethernet cable (network interface availability) detection

2021-05-19 Thread Flavio Castro Alves Filho
Hello David,

I will try this option.

Thank you very much.

Best regards,

Flavio


Em qua., 19 de mai. de 2021 19:28, David Sidrane 
escreveu:

> This https://github.com/apache/incubator-nuttx/pull/1924 was merged. It
> adds
> the polling.
>
> This is the app https://github.com/PX4/NuttX-apps/pull/8  does the rest.
>
> It was not upstreamed because it was not everything others wanted and I
> could not test what they wanted.
>
> David
>
> -Original Message-
> From: Gregory Nutt [mailto:spudan...@gmail.com]
> Sent: Wednesday, May 19, 2021 2:56 PM
> To: dev@nuttx.apache.org
> Subject: Re: Ethernet cable (network interface availability) detection
>
>
> > Considering this scenario, is there any alternative approach that I
> > could use to have this detection? Polling the phy, or something
> > similar?
>
> As I recall, David Sidrane submitted a PR to do just this but it was not
> incoporated.  I don't recall why.   I recall having some concerns that
> polling the PHY in maintenance mode would interfere with normal
> operational mode, but I think David demonstrated that there was not an
> issue with that.
>
> I think these:
>
> https://github.com/apache/incubator-nuttx-apps/pull/402
>
> https://github.com/apache/incubator-nuttx-apps/pull/415
>


RE: Ethernet cable (network interface availability) detection

2021-05-19 Thread David Sidrane
This https://github.com/apache/incubator-nuttx/pull/1924 was merged. It adds
the polling.

This is the app https://github.com/PX4/NuttX-apps/pull/8  does the rest.

It was not upstreamed because it was not everything others wanted and I
could not test what they wanted.

David

-Original Message-
From: Gregory Nutt [mailto:spudan...@gmail.com]
Sent: Wednesday, May 19, 2021 2:56 PM
To: dev@nuttx.apache.org
Subject: Re: Ethernet cable (network interface availability) detection


> Considering this scenario, is there any alternative approach that I
> could use to have this detection? Polling the phy, or something
> similar?

As I recall, David Sidrane submitted a PR to do just this but it was not
incoporated.  I don't recall why.   I recall having some concerns that
polling the PHY in maintenance mode would interfere with normal
operational mode, but I think David demonstrated that there was not an
issue with that.

I think these:

https://github.com/apache/incubator-nuttx-apps/pull/402

https://github.com/apache/incubator-nuttx-apps/pull/415


Re: Ethernet cable (network interface availability) detection

2021-05-19 Thread Flavio Castro Alves Filho
Hello Gregory,

Thank you for your quick response.

Looking at the PRs, they seemed a bit complex for me :-| And I'm not
sure if it is that what I want or need.

What I was thinking was to use this call:

stm32_phyread(CONFIG_STM32F4_PHYADDR,MII_MSR,);

Then make the check:

if ((phyval & MII_MSR_LINKSTATUS) != 0) {

==> change the net driver flag that indicates if the network interface
is up or not;

}

Then in the application, I continue to check regularly the netlib_getifstatus().

Is there any way, or is it feasible, to have something like that? How
should I proceed?

Best regards,

Flavio



Em qua., 19 de mai. de 2021 às 18:55, Gregory Nutt
 escreveu:
>
>
> > Considering this scenario, is there any alternative approach that I
> > could use to have this detection? Polling the phy, or something
> > similar?
>
> As I recall, David Sidrane submitted a PR to do just this but it was not
> incoporated.  I don't recall why.   I recall having some concerns that
> polling the PHY in maintenance mode would interfere with normal
> operational mode, but I think David demonstrated that there was not an
> issue with that.
>
> I think these:
>
> https://github.com/apache/incubator-nuttx-apps/pull/402
>
> https://github.com/apache/incubator-nuttx-apps/pull/415
>
>


-- 
Flavio de Castro Alves Filho

flavio.al...@gmail.com
Twitter: http://twitter.com/#!/fraviofii
LinkedIn profile: www.linkedin.com/in/flaviocastroalves


Re: Ethernet cable (network interface availability) detection

2021-05-19 Thread Gregory Nutt




Considering this scenario, is there any alternative approach that I
could use to have this detection? Polling the phy, or something
similar?


As I recall, David Sidrane submitted a PR to do just this but it was not 
incoporated.  I don't recall why.   I recall having some concerns that 
polling the PHY in maintenance mode would interfere with normal 
operational mode, but I think David demonstrated that there was not an 
issue with that.


I think these:

https://github.com/apache/incubator-nuttx-apps/pull/402

https://github.com/apache/incubator-nuttx-apps/pull/415




RE: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs

2021-05-19 Thread David Sidrane
Hi Reto,

The multiblock is a disable so if CONFIG_MMCSD_MULTIBLOCK_DISABLE=y is not
in the config it is using multiblock

However, that is only for the SDMMC driver. What I was thinking was [1],[2]
for the SDMMC driver not the SPI one.

[1] https://github.com/apache/incubator-nuttx/pull/3669
[2] https://github.com/PX4/PX4-Autopilot/issues/17442#issuecomment-835303021


David

-Original Message-
From: Reto Gähwiler [mailto:gret.hexa...@gmail.com]
Sent: Wednesday, May 19, 2021 8:21 AM
To: dev@nuttx.apache.org
Subject: Re: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated
free clusters by statfs

Hey again,

About the requested configuration of the SD-Card, here we go:
  CONFIG_MMCSD=y
  CONFIG_MMCSD_SPI=y
  CONFIG_MMCSD_SPICLOCK=2500

About the multiblock, I don't think we are using that. Let me know how I
could recognise it if not in the config.

Thanks, Reto

On 2021/05/19 12:13:35, David Sidrane  wrote:
> Hi Reto,
>
> What is the clock rate to the card and is multiblock enabled?
>
> David
>
> -Original Message-
> From: Reto Gähwiler [mailto:gret.hexa...@gmail.com]
> Sent: Wednesday, May 19, 2021 5:08 AM
> To: dev@nuttx.apache.org
> Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free
> clusters by statfs
>
> Hi David,
>
> It is running on an STM32h743zi version V or Y. The SD-card in use is a
> SwissBit S-45u.
>
> Reto
>
> On 2021/05/19 11:11:22, David Sidrane  wrote:
> > hi Reto,
> >
> > What SoC is this on? What type of card are you using?
> >
> > David
> >
> > -Original Message-
> > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com]
> > Sent: Wednesday, May 19, 2021 3:22 AM
> > To: dev@nuttx.apache.org
> > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free
> > clusters by statfs
> >
> > Hello Jukka,
> >
> > Thanks for your quick reaction. I applied your PR to our "outdated"
> > nuttx.
> > Unfortunately, it did not show any impact on the statfs behaviour if the
> > card is corrupted. Though, meanwhile I do have hands on two images.
> >
> > #1 the mentioned one. Although, the files/dirs are all deleted. But
> > still
> > same behvaiour and a chkdsk error. statfs reports 703 MB free while the
> > sd-card was deleted!
> >
> > #2 Having the file structure in place and contains plenty of logfiles.
> > Many
> > of them with the mentioned corruptions. The card is filled to 100%. But
> > here
> > the behaviour is slightly different. Checking the FSINFO block and then
> > removing some files under Windows doesn't impact the FSINFO section. But
> > once inserted to the device statfs reports the correct size and updates
> > the
> > FSINFO section. However, the card also shows corruptions if chkdsk is
> > applied to it.
> > This behaviour is identical before and after applying your PR.
> >
> > Brgds, Reto
> >
> > On 2021/05/18 19:55:06, Jukka Laitinen  wrote:
> > > Hi,
> > >
> > > There seems to be a bug in sector calculation which may trigger with
> > > e.g. corrupted fs. I just made a PR for this,
> > >
> > > https://github.com/apache/incubator-nuttx/pull/3740
> > >
> > > But I really don't know if this is related to your issues, this is
> > > just
> > > something that suddenly popped up elsewhere.
> > >
> > > -Jukka
> > >
> > >
> > > On 18.5.2021 16.10, Reto Gähwiler wrote:
> > > > Dear All,
> > > >
> > > > First of all, in case a similar thread pops-up authored by myself,
> > > > please
> > > > ignore.
> > > >
> > > > Recently we discovered some issues with the FAT32 partition on the
> > > > SD-Card
> > > > used in our device running on nuttx. There are actually a bunch of
> > > > issues
> > > > we are strugle to understand related to the filesystem.
> > > > The issue discovered recently is, that a statfs call won't return
> > > > the
> > > > true
> > > > number of free clusters. It rather returns what ever is in the FS
> > > > INFO
> > > > section of the FAT32 partition. Inserting such a "corrupted" card
> > > > into
> > > > an
> > > > SD-Card reader and mounting it to Windows shows following:
> > > >
> > > > #1 Windows file explorer reports the correct free space.
> > > > #2 Comparing the free space in file explorer and looging at FS INFO
> > > > section
> > > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write
> > > > any
> > > > longer the FS INFO section on that card. Even if all the files/dires
> > > > are
> > > > wiped from the card.
> > > > #3 Whenever files are added / removed under nuttx the FS INFO
> > > > section
> > > > would
> > > > be updated by nuttx, but to the wrong number since the base is
> > > > wrong.
> > > > #4 Mounting the card to linux and properly unmount it might actually
> > > > fix
> > > > the issue. But not always!
> > > > #5 Quick format the card resolved the issue too. Windows would once
> > > > again
> > > > write the FS INFO section and nuttx would shows the correct free
> > > > space.
> > > > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes
> > > > 

Re: Ethernet cable (network interface availability) detection

2021-05-19 Thread Flavio Castro Alves Filho
Hello,

Coming back to this issue.

Looking at STM32F4DISCOVERY code, I could see that the NETINIT_THREAD
is disabled because the IRQ pin from the PHY is not available.

It happens when the RMII configuration is set and the IRQ pin is also
used as REFCLK0.

This is also the situation with my hardware.

Considering this scenario, is there any alternative approach that I
could use to have this detection? Polling the phy, or something
similar?

Best regards,

Flavio

Em qua., 14 de abr. de 2021 às 11:45, Flavio Castro Alves Filho
 escreveu:
>
> Gregory,
>
> Thank you for your support. I will work on this.
>
> Best regards.
>
> Flavio
>
> Em qua., 14 de abr. de 2021 11:09, Gregory Nutt  
> escreveu:
>>
>> You board must provide CONFIG_ARCH_PHY_INTERRUPT:
>>
>> $ find boards/ -name Kconfig | xargs grep ARCH_PHY_INTERRUPT
>> boards/Kconfig: select ARCH_PHY_INTERRUPT
>> boards/Kconfig: select ARCH_PHY_INTERRUPT if SAMA5_EMACA ||
>> SAMA5_EMAC0 || SAMA5_EMAC1 || SAMA5_GMAC
>> boards/Kconfig: select ARCH_PHY_INTERRUPT
>> boards/Kconfig: select ARCH_PHY_INTERRUPT
>> boards/Kconfig: select ARCH_PHY_INTERRUPT
>> boards/Kconfig: select ARCH_PHY_INTERRUPT
>> boards/Kconfig: select ARCH_PHY_INTERRUPT
>>
>> and you must implement PHY interrupt handling as documented in
>> include/nuttx/arch.h:
>>
>> 
>> /
>>   * Name: arch_phy_irq
>>   *
>>   * Description:
>>   *   This function may be called to register an interrupt handler
>> that will
>>   *   be called when a PHY interrupt occurs.  This function both
>> attaches
>>   *   the interrupt handler and enables the interrupt if 'handler'
>> is non-
>>   *   NULL.  If handler is NULL, then the interrupt is detached and
>> disabled
>>   *   instead.
>>   *
>>   *   The PHY interrupt is always disabled upon return.  The caller must
>>   *   call back through the enable function point to control the
>> state of
>>   *   the interrupt.
>>   *
>>   *   This interrupt may or may not be available on a given platform
>> depending
>>   *   on how the network hardware architecture is implemented. In a
>> typical
>>   *   case, the PHY interrupt is provided to board-level logic as a GPIO
>>   *   interrupt (in which case this is a board-specific interface
>> and really
>>   *   should be called board_phy_irq()); In other cases, the PHY
>> interrupt
>>   *   may be cause by the chip's MAC logic (in which case
>> arch_phy_irq()) is
>>   *   an appropriate name.  Other other boards, there may be no PHY
>> interrupts
>>   *   available at all.  If client attachable PHY interrupts are
>> available
>>   *   from the board or from the chip, then
>> CONFIG_ARCH_PHY_INTERRUPT should
>>   *   be defined to indicate that fact.
>>   *
>>   *   Typical usage:
>>   *   a. OS service logic (not application logic*) attaches to the PHY
>>   *  PHY interrupt and enables the PHY interrupt.
>>   *   b. When the PHY interrupt occurs:  (1) the interrupt should be
>>   *  disabled and () work should be scheduled on the worker
>> thread (or
>>   *  perhaps a dedicated application thread).
>>   *   c. That worker thread should use the SIOCGMIIPHY, SIOCGMIIREG,
>>   *  and SIOCSMIIREG ioctl calls** to communicate with the PHY,
>>   *  determine what network event took place (Link Up/Down?), and
>>   *  take the appropriate actions.
>>   *   d. It should then interact the PHY to clear any pending
>>   *  interrupts, then re-enable the PHY interrupt.
>>   *
>>   ** This is an OS internal interface and should not be used from
>>   *  application space.  Rather applications should use the
>> SIOCMIISIG
>>   *  ioctl to receive a signal when a PHY event occurs.
>>   *   ** This interrupt is really of no use if the Ethernet MAC driver
>>   *  does not support these ioctl calls.
>>   *
>>   * Input Parameters:
>>   *   intf- Identifies the network interface.  For example
>> "eth0".  Only
>>   * useful on platforms that support multiple Ethernet
>> interfaces
>>   * and, hence, multiple PHYs and PHY interrupts.
>>   *   handler - The client interrupt handler to be invoked when the PHY
>>   * asserts an interrupt.  Must reside in OS space, but can
>>   * signal tasks in user space.  A value of NULL can be
>> passed
>>   * in order to detach and disable the PHY interrupt.
>>   *   arg - The argument that will accompany the interrupt
>>   *   enable  - A function pointer that be unused to enable or
>> disable the
>>   * PHY interrupt.
>>   *
>>   * Returned Value:
>>   *   Zero (OK) 

Re: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs

2021-05-19 Thread Alan Carvalho de Assis
Hi Reto,

Just a curiosity: why are you using SDCard support over SPI?
Isn't your board prepared to use native SDCard/SDIO peripheral?

BR,

Alan

On 5/19/21, Reto Gähwiler  wrote:
> Hey again,
>
> About the requested configuration of the SD-Card, here we go:
>   CONFIG_MMCSD=y
>   CONFIG_MMCSD_SPI=y
>   CONFIG_MMCSD_SPICLOCK=2500
>
> About the multiblock, I don't think we are using that. Let me know how I
> could recognise it if not in the config.
>
> Thanks, Reto
>
> On 2021/05/19 12:13:35, David Sidrane  wrote:
>> Hi Reto,
>>
>> What is the clock rate to the card and is multiblock enabled?
>>
>> David
>>
>> -Original Message-
>> From: Reto Gähwiler [mailto:gret.hexa...@gmail.com]
>> Sent: Wednesday, May 19, 2021 5:08 AM
>> To: dev@nuttx.apache.org
>> Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated
>> free
>> clusters by statfs
>>
>> Hi David,
>>
>> It is running on an STM32h743zi version V or Y. The SD-card in use is a
>> SwissBit S-45u.
>>
>> Reto
>>
>> On 2021/05/19 11:11:22, David Sidrane  wrote:
>> > hi Reto,
>> >
>> > What SoC is this on? What type of card are you using?
>> >
>> > David
>> >
>> > -Original Message-
>> > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com]
>> > Sent: Wednesday, May 19, 2021 3:22 AM
>> > To: dev@nuttx.apache.org
>> > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free
>> > clusters by statfs
>> >
>> > Hello Jukka,
>> >
>> > Thanks for your quick reaction. I applied your PR to our "outdated"
>> > nuttx.
>> > Unfortunately, it did not show any impact on the statfs behaviour if
>> > the
>> > card is corrupted. Though, meanwhile I do have hands on two images.
>> >
>> > #1 the mentioned one. Although, the files/dirs are all deleted. But
>> > still
>> > same behvaiour and a chkdsk error. statfs reports 703 MB free while the
>> > sd-card was deleted!
>> >
>> > #2 Having the file structure in place and contains plenty of logfiles.
>> > Many
>> > of them with the mentioned corruptions. The card is filled to 100%. But
>> > here
>> > the behaviour is slightly different. Checking the FSINFO block and then
>> > removing some files under Windows doesn't impact the FSINFO section.
>> > But
>> > once inserted to the device statfs reports the correct size and updates
>> > the
>> > FSINFO section. However, the card also shows corruptions if chkdsk is
>> > applied to it.
>> > This behaviour is identical before and after applying your PR.
>> >
>> > Brgds, Reto
>> >
>> > On 2021/05/18 19:55:06, Jukka Laitinen  wrote:
>> > > Hi,
>> > >
>> > > There seems to be a bug in sector calculation which may trigger with
>> > > e.g. corrupted fs. I just made a PR for this,
>> > >
>> > > https://github.com/apache/incubator-nuttx/pull/3740
>> > >
>> > > But I really don't know if this is related to your issues, this is
>> > > just
>> > > something that suddenly popped up elsewhere.
>> > >
>> > > -Jukka
>> > >
>> > >
>> > > On 18.5.2021 16.10, Reto Gähwiler wrote:
>> > > > Dear All,
>> > > >
>> > > > First of all, in case a similar thread pops-up authored by myself,
>> > > > please
>> > > > ignore.
>> > > >
>> > > > Recently we discovered some issues with the FAT32 partition on the
>> > > > SD-Card
>> > > > used in our device running on nuttx. There are actually a bunch of
>> > > > issues
>> > > > we are strugle to understand related to the filesystem.
>> > > > The issue discovered recently is, that a statfs call won't return
>> > > > the
>> > > > true
>> > > > number of free clusters. It rather returns what ever is in the FS
>> > > > INFO
>> > > > section of the FAT32 partition. Inserting such a "corrupted" card
>> > > > into
>> > > > an
>> > > > SD-Card reader and mounting it to Windows shows following:
>> > > >
>> > > > #1 Windows file explorer reports the correct free space.
>> > > > #2 Comparing the free space in file explorer and looging at FS INFO
>> > > > section
>> > > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write
>> > > > any
>> > > > longer the FS INFO section on that card. Even if all the
>> > > > files/dires
>> > > > are
>> > > > wiped from the card.
>> > > > #3 Whenever files are added / removed under nuttx the FS INFO
>> > > > section
>> > > > would
>> > > > be updated by nuttx, but to the wrong number since the base is
>> > > > wrong.
>> > > > #4 Mounting the card to linux and properly unmount it might
>> > > > actually
>> > > > fix
>> > > > the issue. But not always!
>> > > > #5 Quick format the card resolved the issue too. Windows would once
>> > > > again
>> > > > write the FS INFO section and nuttx would shows the correct free
>> > > > space.
>> > > > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes
>> > > > the
>> > > > issue too. Reporting a broken file in a root folder on that card.
>> > > >
>> > > > We indeed had issues with corrupted files in the past. Usually in
>> > > > the
>> > > > logging directory, which is accessed most. How they corrupt we do
>> > > > not
>> > > > 

Re: RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs

2021-05-19 Thread Reto Gähwiler
Hey again, 

About the requested configuration of the SD-Card, here we go: 
  CONFIG_MMCSD=y
  CONFIG_MMCSD_SPI=y
  CONFIG_MMCSD_SPICLOCK=2500

About the multiblock, I don't think we are using that. Let me know how I could 
recognise it if not in the config. 

Thanks, Reto

On 2021/05/19 12:13:35, David Sidrane  wrote: 
> Hi Reto,
> 
> What is the clock rate to the card and is multiblock enabled?
> 
> David
> 
> -Original Message-
> From: Reto Gähwiler [mailto:gret.hexa...@gmail.com]
> Sent: Wednesday, May 19, 2021 5:08 AM
> To: dev@nuttx.apache.org
> Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free
> clusters by statfs
> 
> Hi David,
> 
> It is running on an STM32h743zi version V or Y. The SD-card in use is a
> SwissBit S-45u.
> 
> Reto
> 
> On 2021/05/19 11:11:22, David Sidrane  wrote:
> > hi Reto,
> >
> > What SoC is this on? What type of card are you using?
> >
> > David
> >
> > -Original Message-
> > From: Reto Gähwiler [mailto:gret.hexa...@gmail.com]
> > Sent: Wednesday, May 19, 2021 3:22 AM
> > To: dev@nuttx.apache.org
> > Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free
> > clusters by statfs
> >
> > Hello Jukka,
> >
> > Thanks for your quick reaction. I applied your PR to our "outdated" nuttx.
> > Unfortunately, it did not show any impact on the statfs behaviour if the
> > card is corrupted. Though, meanwhile I do have hands on two images.
> >
> > #1 the mentioned one. Although, the files/dirs are all deleted. But still
> > same behvaiour and a chkdsk error. statfs reports 703 MB free while the
> > sd-card was deleted!
> >
> > #2 Having the file structure in place and contains plenty of logfiles.
> > Many
> > of them with the mentioned corruptions. The card is filled to 100%. But
> > here
> > the behaviour is slightly different. Checking the FSINFO block and then
> > removing some files under Windows doesn't impact the FSINFO section. But
> > once inserted to the device statfs reports the correct size and updates
> > the
> > FSINFO section. However, the card also shows corruptions if chkdsk is
> > applied to it.
> > This behaviour is identical before and after applying your PR.
> >
> > Brgds, Reto
> >
> > On 2021/05/18 19:55:06, Jukka Laitinen  wrote:
> > > Hi,
> > >
> > > There seems to be a bug in sector calculation which may trigger with
> > > e.g. corrupted fs. I just made a PR for this,
> > >
> > > https://github.com/apache/incubator-nuttx/pull/3740
> > >
> > > But I really don't know if this is related to your issues, this is just
> > > something that suddenly popped up elsewhere.
> > >
> > > -Jukka
> > >
> > >
> > > On 18.5.2021 16.10, Reto Gähwiler wrote:
> > > > Dear All,
> > > >
> > > > First of all, in case a similar thread pops-up authored by myself,
> > > > please
> > > > ignore.
> > > >
> > > > Recently we discovered some issues with the FAT32 partition on the
> > > > SD-Card
> > > > used in our device running on nuttx. There are actually a bunch of
> > > > issues
> > > > we are strugle to understand related to the filesystem.
> > > > The issue discovered recently is, that a statfs call won't return the
> > > > true
> > > > number of free clusters. It rather returns what ever is in the FS INFO
> > > > section of the FAT32 partition. Inserting such a "corrupted" card into
> > > > an
> > > > SD-Card reader and mounting it to Windows shows following:
> > > >
> > > > #1 Windows file explorer reports the correct free space.
> > > > #2 Comparing the free space in file explorer and looging at FS INFO
> > > > section
> > > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write
> > > > any
> > > > longer the FS INFO section on that card. Even if all the files/dires
> > > > are
> > > > wiped from the card.
> > > > #3 Whenever files are added / removed under nuttx the FS INFO section
> > > > would
> > > > be updated by nuttx, but to the wrong number since the base is wrong.
> > > > #4 Mounting the card to linux and properly unmount it might actually
> > > > fix
> > > > the issue. But not always!
> > > > #5 Quick format the card resolved the issue too. Windows would once
> > > > again
> > > > write the FS INFO section and nuttx would shows the correct free
> > > > space.
> > > > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes
> > > > the
> > > > issue too. Reporting a broken file in a root folder on that card.
> > > >
> > > > We indeed had issues with corrupted files in the past. Usually in the
> > > > logging directory, which is accessed most. How they corrupt we do not
> > > > know.
> > > > The corruption is recognised in the following ways:
> > > >
> > > > #A cryptic and too long file names, we only make use of short file
> > > > names
> > > > (8.3)
> > > > #B files whcih became folders
> > > > #C sizes which are not possible
> > > > #D combinations of the above
> > > >
> > > > Our application supervises the SD-Card with statfs and removes files
> > > > once
> > > > we hit a quota of 

RE: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs

2021-05-19 Thread David Sidrane
Hi Reto,

What is the clock rate to the card and is multiblock enabled?

David

-Original Message-
From: Reto Gähwiler [mailto:gret.hexa...@gmail.com]
Sent: Wednesday, May 19, 2021 5:08 AM
To: dev@nuttx.apache.org
Subject: Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free
clusters by statfs

Hi David,

It is running on an STM32h743zi version V or Y. The SD-card in use is a
SwissBit S-45u.

Reto

On 2021/05/19 11:11:22, David Sidrane  wrote:
> hi Reto,
>
> What SoC is this on? What type of card are you using?
>
> David
>
> -Original Message-
> From: Reto Gähwiler [mailto:gret.hexa...@gmail.com]
> Sent: Wednesday, May 19, 2021 3:22 AM
> To: dev@nuttx.apache.org
> Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free
> clusters by statfs
>
> Hello Jukka,
>
> Thanks for your quick reaction. I applied your PR to our "outdated" nuttx.
> Unfortunately, it did not show any impact on the statfs behaviour if the
> card is corrupted. Though, meanwhile I do have hands on two images.
>
> #1 the mentioned one. Although, the files/dirs are all deleted. But still
> same behvaiour and a chkdsk error. statfs reports 703 MB free while the
> sd-card was deleted!
>
> #2 Having the file structure in place and contains plenty of logfiles.
> Many
> of them with the mentioned corruptions. The card is filled to 100%. But
> here
> the behaviour is slightly different. Checking the FSINFO block and then
> removing some files under Windows doesn't impact the FSINFO section. But
> once inserted to the device statfs reports the correct size and updates
> the
> FSINFO section. However, the card also shows corruptions if chkdsk is
> applied to it.
> This behaviour is identical before and after applying your PR.
>
> Brgds, Reto
>
> On 2021/05/18 19:55:06, Jukka Laitinen  wrote:
> > Hi,
> >
> > There seems to be a bug in sector calculation which may trigger with
> > e.g. corrupted fs. I just made a PR for this,
> >
> > https://github.com/apache/incubator-nuttx/pull/3740
> >
> > But I really don't know if this is related to your issues, this is just
> > something that suddenly popped up elsewhere.
> >
> > -Jukka
> >
> >
> > On 18.5.2021 16.10, Reto Gähwiler wrote:
> > > Dear All,
> > >
> > > First of all, in case a similar thread pops-up authored by myself,
> > > please
> > > ignore.
> > >
> > > Recently we discovered some issues with the FAT32 partition on the
> > > SD-Card
> > > used in our device running on nuttx. There are actually a bunch of
> > > issues
> > > we are strugle to understand related to the filesystem.
> > > The issue discovered recently is, that a statfs call won't return the
> > > true
> > > number of free clusters. It rather returns what ever is in the FS INFO
> > > section of the FAT32 partition. Inserting such a "corrupted" card into
> > > an
> > > SD-Card reader and mounting it to Windows shows following:
> > >
> > > #1 Windows file explorer reports the correct free space.
> > > #2 Comparing the free space in file explorer and looging at FS INFO
> > > section
> > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write
> > > any
> > > longer the FS INFO section on that card. Even if all the files/dires
> > > are
> > > wiped from the card.
> > > #3 Whenever files are added / removed under nuttx the FS INFO section
> > > would
> > > be updated by nuttx, but to the wrong number since the base is wrong.
> > > #4 Mounting the card to linux and properly unmount it might actually
> > > fix
> > > the issue. But not always!
> > > #5 Quick format the card resolved the issue too. Windows would once
> > > again
> > > write the FS INFO section and nuttx would shows the correct free
> > > space.
> > > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes
> > > the
> > > issue too. Reporting a broken file in a root folder on that card.
> > >
> > > We indeed had issues with corrupted files in the past. Usually in the
> > > logging directory, which is accessed most. How they corrupt we do not
> > > know.
> > > The corruption is recognised in the following ways:
> > >
> > > #A cryptic and too long file names, we only make use of short file
> > > names
> > > (8.3)
> > > #B files whcih became folders
> > > #C sizes which are not possible
> > > #D combinations of the above
> > >
> > > Our application supervises the SD-Card with statfs and removes files
> > > once
> > > we hit a quota of 0.85. Therefore, the card is not filling up.
> > > However,
> > > the
> > > most recent recognition of this behvaiour was due to an application
> > > bug
> > > which allowed the card to fill up under some circumstances.
> > > Playing around with a quickformated SD-Card, filled up with data and
> > > trying
> > > to retrigger the issue revealed that I never get the ENOSPC but only
> > > EIO
> > > once the card is full! But never managed to get the statfs issue.
> > > For the logging we use the fstream family (fopen, fwrite, fread,
> > > fflush,
> > > fsync) commands. For config 

Re: RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs

2021-05-19 Thread Reto Gähwiler
Hi David, 

It is running on an STM32h743zi version V or Y. The SD-card in use is a 
SwissBit S-45u.

Reto

On 2021/05/19 11:11:22, David Sidrane  wrote: 
> hi Reto,
> 
> What SoC is this on? What type of card are you using?
> 
> David
> 
> -Original Message-
> From: Reto Gähwiler [mailto:gret.hexa...@gmail.com]
> Sent: Wednesday, May 19, 2021 3:22 AM
> To: dev@nuttx.apache.org
> Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free
> clusters by statfs
> 
> Hello Jukka,
> 
> Thanks for your quick reaction. I applied your PR to our "outdated" nuttx.
> Unfortunately, it did not show any impact on the statfs behaviour if the
> card is corrupted. Though, meanwhile I do have hands on two images.
> 
> #1 the mentioned one. Although, the files/dirs are all deleted. But still
> same behvaiour and a chkdsk error. statfs reports 703 MB free while the
> sd-card was deleted!
> 
> #2 Having the file structure in place and contains plenty of logfiles. Many
> of them with the mentioned corruptions. The card is filled to 100%. But here
> the behaviour is slightly different. Checking the FSINFO block and then
> removing some files under Windows doesn't impact the FSINFO section. But
> once inserted to the device statfs reports the correct size and updates the
> FSINFO section. However, the card also shows corruptions if chkdsk is
> applied to it.
> This behaviour is identical before and after applying your PR.
> 
> Brgds, Reto
> 
> On 2021/05/18 19:55:06, Jukka Laitinen  wrote:
> > Hi,
> >
> > There seems to be a bug in sector calculation which may trigger with
> > e.g. corrupted fs. I just made a PR for this,
> >
> > https://github.com/apache/incubator-nuttx/pull/3740
> >
> > But I really don't know if this is related to your issues, this is just
> > something that suddenly popped up elsewhere.
> >
> > -Jukka
> >
> >
> > On 18.5.2021 16.10, Reto Gähwiler wrote:
> > > Dear All,
> > >
> > > First of all, in case a similar thread pops-up authored by myself,
> > > please
> > > ignore.
> > >
> > > Recently we discovered some issues with the FAT32 partition on the
> > > SD-Card
> > > used in our device running on nuttx. There are actually a bunch of
> > > issues
> > > we are strugle to understand related to the filesystem.
> > > The issue discovered recently is, that a statfs call won't return the
> > > true
> > > number of free clusters. It rather returns what ever is in the FS INFO
> > > section of the FAT32 partition. Inserting such a "corrupted" card into
> > > an
> > > SD-Card reader and mounting it to Windows shows following:
> > >
> > > #1 Windows file explorer reports the correct free space.
> > > #2 Comparing the free space in file explorer and looging at FS INFO
> > > section
> > > with "Active - Disk Editor" doesn't line up. Windows wouldn't write any
> > > longer the FS INFO section on that card. Even if all the files/dires are
> > > wiped from the card.
> > > #3 Whenever files are added / removed under nuttx the FS INFO section
> > > would
> > > be updated by nuttx, but to the wrong number since the base is wrong.
> > > #4 Mounting the card to linux and properly unmount it might actually fix
> > > the issue. But not always!
> > > #5 Quick format the card resolved the issue too. Windows would once
> > > again
> > > write the FS INFO section and nuttx would shows the correct free space.
> > > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes the
> > > issue too. Reporting a broken file in a root folder on that card.
> > >
> > > We indeed had issues with corrupted files in the past. Usually in the
> > > logging directory, which is accessed most. How they corrupt we do not
> > > know.
> > > The corruption is recognised in the following ways:
> > >
> > > #A cryptic and too long file names, we only make use of short file names
> > > (8.3)
> > > #B files whcih became folders
> > > #C sizes which are not possible
> > > #D combinations of the above
> > >
> > > Our application supervises the SD-Card with statfs and removes files
> > > once
> > > we hit a quota of 0.85. Therefore, the card is not filling up. However,
> > > the
> > > most recent recognition of this behvaiour was due to an application bug
> > > which allowed the card to fill up under some circumstances.
> > > Playing around with a quickformated SD-Card, filled up with data and
> > > trying
> > > to retrigger the issue revealed that I never get the ENOSPC but only EIO
> > > once the card is full! But never managed to get the statfs issue.
> > > For the logging we use the fstream family (fopen, fwrite, fread, fflush,
> > > fsync) commands. For config files and other things also the low level
> > > read/write are used.
> > >
> > > One more thing, among all the logging files we once in a while have
> > > corrupted file content. Typically that is a skipped byte or doubled byte
> > > (we use protobuf and can see that encoding the binary). This seems to
> > > happen during reading but also writing the file.
> > >
> > > At 

RE: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs

2021-05-19 Thread David Sidrane
hi Reto,

What SoC is this on? What type of card are you using?

David

-Original Message-
From: Reto Gähwiler [mailto:gret.hexa...@gmail.com]
Sent: Wednesday, May 19, 2021 3:22 AM
To: dev@nuttx.apache.org
Subject: Re: Nuttx FAT32 issues - corrupted files, wrongly stated free
clusters by statfs

Hello Jukka,

Thanks for your quick reaction. I applied your PR to our "outdated" nuttx.
Unfortunately, it did not show any impact on the statfs behaviour if the
card is corrupted. Though, meanwhile I do have hands on two images.

#1 the mentioned one. Although, the files/dirs are all deleted. But still
same behvaiour and a chkdsk error. statfs reports 703 MB free while the
sd-card was deleted!

#2 Having the file structure in place and contains plenty of logfiles. Many
of them with the mentioned corruptions. The card is filled to 100%. But here
the behaviour is slightly different. Checking the FSINFO block and then
removing some files under Windows doesn't impact the FSINFO section. But
once inserted to the device statfs reports the correct size and updates the
FSINFO section. However, the card also shows corruptions if chkdsk is
applied to it.
This behaviour is identical before and after applying your PR.

Brgds, Reto

On 2021/05/18 19:55:06, Jukka Laitinen  wrote:
> Hi,
>
> There seems to be a bug in sector calculation which may trigger with
> e.g. corrupted fs. I just made a PR for this,
>
> https://github.com/apache/incubator-nuttx/pull/3740
>
> But I really don't know if this is related to your issues, this is just
> something that suddenly popped up elsewhere.
>
> -Jukka
>
>
> On 18.5.2021 16.10, Reto Gähwiler wrote:
> > Dear All,
> >
> > First of all, in case a similar thread pops-up authored by myself,
> > please
> > ignore.
> >
> > Recently we discovered some issues with the FAT32 partition on the
> > SD-Card
> > used in our device running on nuttx. There are actually a bunch of
> > issues
> > we are strugle to understand related to the filesystem.
> > The issue discovered recently is, that a statfs call won't return the
> > true
> > number of free clusters. It rather returns what ever is in the FS INFO
> > section of the FAT32 partition. Inserting such a "corrupted" card into
> > an
> > SD-Card reader and mounting it to Windows shows following:
> >
> > #1 Windows file explorer reports the correct free space.
> > #2 Comparing the free space in file explorer and looging at FS INFO
> > section
> > with "Active - Disk Editor" doesn't line up. Windows wouldn't write any
> > longer the FS INFO section on that card. Even if all the files/dires are
> > wiped from the card.
> > #3 Whenever files are added / removed under nuttx the FS INFO section
> > would
> > be updated by nuttx, but to the wrong number since the base is wrong.
> > #4 Mounting the card to linux and properly unmount it might actually fix
> > the issue. But not always!
> > #5 Quick format the card resolved the issue too. Windows would once
> > again
> > write the FS INFO section and nuttx would shows the correct free space.
> > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes the
> > issue too. Reporting a broken file in a root folder on that card.
> >
> > We indeed had issues with corrupted files in the past. Usually in the
> > logging directory, which is accessed most. How they corrupt we do not
> > know.
> > The corruption is recognised in the following ways:
> >
> > #A cryptic and too long file names, we only make use of short file names
> > (8.3)
> > #B files whcih became folders
> > #C sizes which are not possible
> > #D combinations of the above
> >
> > Our application supervises the SD-Card with statfs and removes files
> > once
> > we hit a quota of 0.85. Therefore, the card is not filling up. However,
> > the
> > most recent recognition of this behvaiour was due to an application bug
> > which allowed the card to fill up under some circumstances.
> > Playing around with a quickformated SD-Card, filled up with data and
> > trying
> > to retrigger the issue revealed that I never get the ENOSPC but only EIO
> > once the card is full! But never managed to get the statfs issue.
> > For the logging we use the fstream family (fopen, fwrite, fread, fflush,
> > fsync) commands. For config files and other things also the low level
> > read/write are used.
> >
> > One more thing, among all the logging files we once in a while have
> > corrupted file content. Typically that is a skipped byte or doubled byte
> > (we use protobuf and can see that encoding the binary). This seems to
> > happen during reading but also writing the file.
> >
> > At the moment we didn't investigate the fat driver of nuttx but rather
> > tried to analyse the problem. And now we have a bunch of question marks.
> >
> > #1 Has anyone recongised similar issues?
> > #2 Does anyone know a tool which can be used in nuttx equivalent to
> > chkdsk
> > under windows? Or do quick format?
> > #3 Why is there no ENOSPC but EIO once the card is full?
> > #4 

Re: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs

2021-05-19 Thread Reto Gähwiler
Hey Alan, 

Thanks for your input. See reply to Jukka. About the dd command, I am aware of 
and make use of it. Also the reason I could test different things with the same 
image :) 

Greetings, Reto

On 2021/05/18 20:53:39, Alan Carvalho de Assis  wrote: 
> Hi Reto,
> 
> Mr. Jukka just opened a PR that could help the issue related to the
> cluster issue:
> 
> https://github.com/apache/incubator-nuttx/pull/3740
> 
> When you face an issue with the SDCard a good idea is to do a raw copy
> of it on Linux using the dd command, then you could duplicated the
> issue later.
> 
> If you find an easy way to reproduce the issue, it will make it easier
> for other people duplicate your issue and help to debug it.
> 
> BR,
> 
> Alan
> 
> On 5/18/21, Reto Gähwiler  wrote:
> > Dear All,
> >
> > First of all, in case a similar thread pops-up authored by myself, please
> > ignore.
> >
> > Recently we discovered some issues with the FAT32 partition on the SD-Card
> > used in our device running on nuttx. There are actually a bunch of issues
> > we are strugle to understand related to the filesystem.
> > The issue discovered recently is, that a statfs call won't return the true
> > number of free clusters. It rather returns what ever is in the FS INFO
> > section of the FAT32 partition. Inserting such a "corrupted" card into an
> > SD-Card reader and mounting it to Windows shows following:
> >
> > #1 Windows file explorer reports the correct free space.
> > #2 Comparing the free space in file explorer and looging at FS INFO section
> > with "Active - Disk Editor" doesn't line up. Windows wouldn't write any
> > longer the FS INFO section on that card. Even if all the files/dires are
> > wiped from the card.
> > #3 Whenever files are added / removed under nuttx the FS INFO section would
> > be updated by nuttx, but to the wrong number since the base is wrong.
> > #4 Mounting the card to linux and properly unmount it might actually fix
> > the issue. But not always!
> > #5 Quick format the card resolved the issue too. Windows would once again
> > write the FS INFO section and nuttx would shows the correct free space.
> > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes the
> > issue too. Reporting a broken file in a root folder on that card.
> >
> > We indeed had issues with corrupted files in the past. Usually in the
> > logging directory, which is accessed most. How they corrupt we do not know.
> > The corruption is recognised in the following ways:
> >
> > #A cryptic and too long file names, we only make use of short file names
> > (8.3)
> > #B files whcih became folders
> > #C sizes which are not possible
> > #D combinations of the above
> >
> > Our application supervises the SD-Card with statfs and removes files once
> > we hit a quota of 0.85. Therefore, the card is not filling up. However, the
> > most recent recognition of this behvaiour was due to an application bug
> > which allowed the card to fill up under some circumstances.
> > Playing around with a quickformated SD-Card, filled up with data and trying
> > to retrigger the issue revealed that I never get the ENOSPC but only EIO
> > once the card is full! But never managed to get the statfs issue.
> > For the logging we use the fstream family (fopen, fwrite, fread, fflush,
> > fsync) commands. For config files and other things also the low level
> > read/write are used.
> >
> > One more thing, among all the logging files we once in a while have
> > corrupted file content. Typically that is a skipped byte or doubled byte
> > (we use protobuf and can see that encoding the binary). This seems to
> > happen during reading but also writing the file.
> >
> > At the moment we didn't investigate the fat driver of nuttx but rather
> > tried to analyse the problem. And now we have a bunch of question marks.
> >
> > #1 Has anyone recongised similar issues?
> > #2 Does anyone know a tool which can be used in nuttx equivalent to chkdsk
> > under windows? Or do quick format?
> > #3 Why is there no ENOSPC but EIO once the card is full?
> > #4 What triggers the corrupted files? How robust is the nuttx FAT32
> > considering stackdumps and powerlosses?
> >
> > For the purpose of investigation, I have an image of the SD-Card when it
> > was in the state of reporting wrong sizes in nuttx.
> >
> > Looking forward to your answers and ideas. Appreciate your help,
> >
> > Reto
> >
> 


Re: Nuttx FAT32 issues - corrupted files, wrongly stated free clusters by statfs

2021-05-19 Thread Reto Gähwiler
Hello Jukka, 

Thanks for your quick reaction. I applied your PR to our "outdated" nuttx. 
Unfortunately, it did not show any impact on the statfs behaviour if the card 
is corrupted. Though, meanwhile I do have hands on two images. 

#1 the mentioned one. Although, the files/dirs are all deleted. But still same 
behvaiour and a chkdsk error. statfs reports 703 MB free while the sd-card was 
deleted! 

#2 Having the file structure in place and contains plenty of logfiles. Many of 
them with the mentioned corruptions. The card is filled to 100%. But here the 
behaviour is slightly different. Checking the FSINFO block and then removing 
some files under Windows doesn't impact the FSINFO section. But once inserted 
to the device statfs reports the correct size and updates the FSINFO section. 
However, the card also shows corruptions if chkdsk is applied to it. 
This behaviour is identical before and after applying your PR. 

Brgds, Reto

On 2021/05/18 19:55:06, Jukka Laitinen  wrote: 
> Hi,
> 
> There seems to be a bug in sector calculation which may trigger with 
> e.g. corrupted fs. I just made a PR for this,
> 
> https://github.com/apache/incubator-nuttx/pull/3740
> 
> But I really don't know if this is related to your issues, this is just 
> something that suddenly popped up elsewhere.
> 
> -Jukka
> 
> 
> On 18.5.2021 16.10, Reto Gähwiler wrote:
> > Dear All,
> >
> > First of all, in case a similar thread pops-up authored by myself, please
> > ignore.
> >
> > Recently we discovered some issues with the FAT32 partition on the SD-Card
> > used in our device running on nuttx. There are actually a bunch of issues
> > we are strugle to understand related to the filesystem.
> > The issue discovered recently is, that a statfs call won't return the true
> > number of free clusters. It rather returns what ever is in the FS INFO
> > section of the FAT32 partition. Inserting such a "corrupted" card into an
> > SD-Card reader and mounting it to Windows shows following:
> >
> > #1 Windows file explorer reports the correct free space.
> > #2 Comparing the free space in file explorer and looging at FS INFO section
> > with "Active - Disk Editor" doesn't line up. Windows wouldn't write any
> > longer the FS INFO section on that card. Even if all the files/dires are
> > wiped from the card.
> > #3 Whenever files are added / removed under nuttx the FS INFO section would
> > be updated by nuttx, but to the wrong number since the base is wrong.
> > #4 Mounting the card to linux and properly unmount it might actually fix
> > the issue. But not always!
> > #5 Quick format the card resolved the issue too. Windows would once again
> > write the FS INFO section and nuttx would shows the correct free space.
> > #5 Running a "chkdsk /f" under windows on that broken SD-Card fixes the
> > issue too. Reporting a broken file in a root folder on that card.
> >
> > We indeed had issues with corrupted files in the past. Usually in the
> > logging directory, which is accessed most. How they corrupt we do not know.
> > The corruption is recognised in the following ways:
> >
> > #A cryptic and too long file names, we only make use of short file names
> > (8.3)
> > #B files whcih became folders
> > #C sizes which are not possible
> > #D combinations of the above
> >
> > Our application supervises the SD-Card with statfs and removes files once
> > we hit a quota of 0.85. Therefore, the card is not filling up. However, the
> > most recent recognition of this behvaiour was due to an application bug
> > which allowed the card to fill up under some circumstances.
> > Playing around with a quickformated SD-Card, filled up with data and trying
> > to retrigger the issue revealed that I never get the ENOSPC but only EIO
> > once the card is full! But never managed to get the statfs issue.
> > For the logging we use the fstream family (fopen, fwrite, fread, fflush,
> > fsync) commands. For config files and other things also the low level
> > read/write are used.
> >
> > One more thing, among all the logging files we once in a while have
> > corrupted file content. Typically that is a skipped byte or doubled byte
> > (we use protobuf and can see that encoding the binary). This seems to
> > happen during reading but also writing the file.
> >
> > At the moment we didn't investigate the fat driver of nuttx but rather
> > tried to analyse the problem. And now we have a bunch of question marks.
> >
> > #1 Has anyone recongised similar issues?
> > #2 Does anyone know a tool which can be used in nuttx equivalent to chkdsk
> > under windows? Or do quick format?
> > #3 Why is there no ENOSPC but EIO once the card is full?
> > #4 What triggers the corrupted files? How robust is the nuttx FAT32
> > considering stackdumps and powerlosses?
> >
> > For the purpose of investigation, I have an image of the SD-Card when it
> > was in the state of reporting wrong sizes in nuttx.
> >
> > Looking forward to your answers and ideas. Appreciate your help,
> >
> > Reto
> >