subject:"Re\: \[Gluster\-devel\] could you help to check about a glusterfs issue seems to be related to ctime"

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

2020-03-17 Thread Zhou, Cynthia (NSB - CN/Hangzhou)

I have not tested with option attribute-timeout=0 -o entry-timeout=0,
But I think to set that two options to 0 will have impact on performance.
Also set consistent-metadata on will also have impact for performance, so I’d 
rather keep consistent-metadata to be off and use ctime feature to solve the 
tar issue.

cynthia

From: Ravishankar N 
Sent: 2020年3月17日 17:08
To: Zhou, Cynthia (NSB - CN/Hangzhou) ; Kotresh 
Hiremath Ravishankar 
Cc: Gluster Devel 
Subject: Re: [Gluster-devel] could you help to check about a glusterfs issue 
seems to be related to ctime



On 17/03/20 12:56 pm, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:
Hi,
I know that this may not be your presumption of this ctime feature, however, 
our delima is that

  1.  product need to support date change scenario.
  2.  if we disable ctime feature, we will meet tar issue.
  3.  if we set consistent-metadata to on to work around this tar issue, there 
is another even worse problem: 
https://bugzilla.redhat.com/show_bug.cgi?id=1773476, app just cranshed!
So I have to make this patch.

For what it is worth, the 'file changed as we read it' warning from tar can be 
suppressed with the --warning=no-file-changed flag.

For 3, do you see any visible usability/performance issues with the work-around 
mentioned in the bug (comment 5 and 6)? Just curious.
Regards,
Ravi

Maybe I will keep this as internal use just for our local use scenarios.
Cynthia.

From: Kotresh Hiremath Ravishankar 

Sent: 2020年3月17日 15:17
To: Zhou, Cynthia (NSB - CN/Hangzhou) 

Cc: Amar Tumballi ; Gluster Devel 

Subject: Re: [Gluster-devel] could you help to check about a glusterfs issue 
seems to be related to ctime

The whole ctime features relies on time provided by clients which are time 
synchronized. This patch brings in the time from server to compare against the 
time sent from client.
As Amar mentioned, this doesn't fit well into the scheme of how ctime is 
designed. Definitely keeping it optional and disabling it by default is one 
way. But is that what your intention
here?

On Tue, Mar 17, 2020 at 10:56 AM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
Ok, thanks for your feedback!
I will do local test to verify this patch first.

cynthia

From: Amar Tumballi mailto:a...@kadalu.io>>
Sent: 2020年3月17日 13:18
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>>
Cc: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>; Gluster Devel 
mailto:gluster-devel@gluster.org>>
Subject: Re: [Gluster-devel] could you help to check about a glusterfs issue 
seems to be related to ctime


On Tue, Mar 17, 2020 at 10:18 AM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
Hi glusterfs expert,
Our product need to tolerate change date to future and then change back.
How about change like this ?
https://review.gluster.org/#/c/glusterfs/+/24229/1/xlators/storage/posix/src/posix-metadata.c

when time change to future and change back , should still be able to update 
mdata, so the following changes to file can be populated to other clients.


We do like to have people integrating with GlusterFS. But this change is not 
inline with the 'assumptions' we had about the feature.

If you have verified this change works for you, please add it as an 'option' in 
posix, which can be changed through volume set, and keep this option 
disable/off by default. That should be an easier way to get the patch reviewed 
and take it further. Please make sure to provide the 'description' for the 
option with details.

Regards,
Amar


cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 17:31
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Cc: 'Gluster Devel' 
mailto:gluster-devel@gluster.org>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,
One more question, I find each client has the same future time stamp where are 
those time stamps from, since Since it is different from any brick stored time 
stamp. And after I modify files  from clients, it remains the same.
[root@mn-0:/home/robot]
# stat /mnt/export/testfile
  File: /mnt/export/testfile
  Size: 193 Blocks: 1  IO Block: 131072 regular file
Device: 28h/40d Inode: 10383279039841136109  Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-04-11 12:20:22.114365172 +0300
Modify: 2020-04-11 12:20:22.121552573 +0300
Change: 2020-04-11 12:20:22.121552573 +0300

[root@mn-0:/home/robot]
# date
Thu Mar 12 11:27:33 EET 2020
[root@mn-0:/home/robot]

[root@mn-0:/home/robot]
# stat /mnt/bricks/export/brick/testfile
  File: /mnt/bricks/export/brick/testfile
  Size: 193 Blocks: 16 IO Block: 4096   regular file
Device: fc02h/64514dInode: 512015  Links: 2
Access: (0644/-rw-r--r--)  Uid: (0/

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

2020-03-17 Thread Zhou, Cynthia (NSB - CN/Hangzhou)

Hi,
I know that this may not be your presumption of this ctime feature, however, 
our delima is that

  1.  product need to support date change scenario.
  2.  if we disable ctime feature, we will meet tar issue.
  3.  if we set consistent-metadata to on to work around this tar issue, there 
is another even worse problem: 
https://bugzilla.redhat.com/show_bug.cgi?id=1773476, app just cranshed!
So I have to make this patch.
Maybe I will keep this as internal use just for our local use scenarios.
Cynthia.

From: Kotresh Hiremath Ravishankar 
Sent: 2020年3月17日 15:17
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
Cc: Amar Tumballi ; Gluster Devel 
Subject: Re: [Gluster-devel] could you help to check about a glusterfs issue 
seems to be related to ctime

The whole ctime features relies on time provided by clients which are time 
synchronized. This patch brings in the time from server to compare against the 
time sent from client.
As Amar mentioned, this doesn't fit well into the scheme of how ctime is 
designed. Definitely keeping it optional and disabling it by default is one 
way. But is that what your intention
here?

On Tue, Mar 17, 2020 at 10:56 AM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
Ok, thanks for your feedback!
I will do local test to verify this patch first.

cynthia

From: Amar Tumballi mailto:a...@kadalu.io>>
Sent: 2020年3月17日 13:18
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>>
Cc: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>; Gluster Devel 
mailto:gluster-devel@gluster.org>>
Subject: Re: [Gluster-devel] could you help to check about a glusterfs issue 
seems to be related to ctime


On Tue, Mar 17, 2020 at 10:18 AM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
Hi glusterfs expert,
Our product need to tolerate change date to future and then change back.
How about change like this ?
https://review.gluster.org/#/c/glusterfs/+/24229/1/xlators/storage/posix/src/posix-metadata.c

when time change to future and change back , should still be able to update 
mdata, so the following changes to file can be populated to other clients.


We do like to have people integrating with GlusterFS. But this change is not 
inline with the 'assumptions' we had about the feature.

If you have verified this change works for you, please add it as an 'option' in 
posix, which can be changed through volume set, and keep this option 
disable/off by default. That should be an easier way to get the patch reviewed 
and take it further. Please make sure to provide the 'description' for the 
option with details.

Regards,
Amar


cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 17:31
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Cc: 'Gluster Devel' 
mailto:gluster-devel@gluster.org>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,
One more question, I find each client has the same future time stamp where are 
those time stamps from, since Since it is different from any brick stored time 
stamp. And after I modify files  from clients, it remains the same.
[root@mn-0:/home/robot]
# stat /mnt/export/testfile
  File: /mnt/export/testfile
  Size: 193 Blocks: 1  IO Block: 131072 regular file
Device: 28h/40d Inode: 10383279039841136109  Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-04-11 12:20:22.114365172 +0300
Modify: 2020-04-11 12:20:22.121552573 +0300
Change: 2020-04-11 12:20:22.121552573 +0300

[root@mn-0:/home/robot]
# date
Thu Mar 12 11:27:33 EET 2020
[root@mn-0:/home/robot]

[root@mn-0:/home/robot]
# stat /mnt/bricks/export/brick/testfile
  File: /mnt/bricks/export/brick/testfile
  Size: 193 Blocks: 16 IO Block: 4096   regular file
Device: fc02h/64514dInode: 512015  Links: 2
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-04-11 12:20:22.100395536 +0300
Modify: 2020-03-12 11:25:04.095981276 +0200
Change: 2020-03-12 11:25:04.095981276 +0200
Birth: 2020-04-11 08:53:26.805163816 +0300


[root@mn-1:/root]
# stat /mnt/bricks/export/brick/testfile
  File: /mnt/bricks/export/brick/testfile
  Size: 193 Blocks: 16 IO Block: 4096   regular file
Device: fc02h/64514dInode: 512015  Links: 2
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-04-11 12:20:22.100395536 +0300
Modify: 2020-03-12 11:25:04.094913452 +0200
Change: 2020-03-12 11:25:04.095913453 +0200
Birth: 2020-03-12 07:53:26.803783053 +0200



From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 16:09
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,
This is abnormal test case, however, when this happened

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

2020-03-17 Thread Kotresh Hiremath Ravishankar

The whole ctime features relies on time provided by clients which are time
synchronized. This patch brings in the time from server to compare against
the time sent from client.
As Amar mentioned, this doesn't fit well into the scheme of how ctime is
designed. Definitely keeping it optional and disabling it by default is one
way. But is that what your intention
here?

On Tue, Mar 17, 2020 at 10:56 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.z...@nokia-sbell.com> wrote:

> Ok, thanks for your feedback!
>
> I will do local test to verify this patch first.
>
>
>
> cynthia
>
>
>
> *From:* Amar Tumballi 
> *Sent:* 2020年3月17日 13:18
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) 
> *Cc:* Kotresh Hiremath Ravishankar ; Gluster Devel <
> gluster-devel@gluster.org>
> *Subject:* Re: [Gluster-devel] could you help to check about a glusterfs
> issue seems to be related to ctime
>
>
>
>
>
> On Tue, Mar 17, 2020 at 10:18 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.z...@nokia-sbell.com> wrote:
>
> Hi glusterfs expert,
>
> Our product need to tolerate change date to future and then change back.
>
> How about change like this ?
>
>
> https://review.gluster.org/#/c/glusterfs/+/24229/1/xlators/storage/posix/src/posix-metadata.c
>
>
>
> when time change to future and change back , should still be able to
> update mdata, so the following changes to file can be populated to other
> clients.
>
>
>
>
>
> We do like to have people integrating with GlusterFS. But this change is
> not inline with the 'assumptions' we had about the feature.
>
>
>
> If you have verified this change works for you, please add it as an
> 'option' in posix, which can be changed through volume set, and keep this
> option disable/off by default. That should be an easier way to get the
> patch reviewed and take it further. Please make sure to provide the
> 'description' for the option with details.
>
>
>
> Regards,
>
> Amar
>
>
>
>
>
> cynthia
>
>
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* 2020年3月12日 17:31
> *To:* 'Kotresh Hiremath Ravishankar' 
> *Cc:* 'Gluster Devel' 
> *Subject:* RE: could you help to check about a glusterfs issue seems to
> be related to ctime
>
>
>
> Hi,
>
> One more question, I find each client has the same future time stamp where
> are those time stamps from, since Since it is different from any brick
> stored time stamp. And after I modify files  from clients, it remains the
> same.
>
> [root@mn-0:/home/robot]
>
> # stat /mnt/export/testfile
>
>   File: /mnt/export/testfile
>
>   Size: 193 Blocks: 1  IO Block: 131072 regular file
>
> Device: 28h/40d Inode: 10383279039841136109  Links: 1
>
> Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (
> 615/_nokfsuifileshare)
>
> Access: 2020-04-11 12:20:22.114365172 +0300
>
> Modify: 2020-04-11 12:20:22.121552573 +0300
>
> Change: 2020-04-11 12:20:22.121552573 +0300
>
>
>
> [root@mn-0:/home/robot]
>
> # date
>
> Thu Mar 12 11:27:33 EET 2020
>
> [root@mn-0:/home/robot]
>
>
>
> [root@mn-0:/home/robot]
>
> # stat /mnt/bricks/export/brick/testfile
>
>   File: /mnt/bricks/export/brick/testfile
>
>   Size: 193 Blocks: 16 IO Block: 4096   regular file
>
> Device: fc02h/64514dInode: 512015  Links: 2
>
> Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (
> 615/_nokfsuifileshare)
>
> Access: 2020-04-11 12:20:22.100395536 +0300
>
> Modify: 2020-03-12 11:25:04.095981276 +0200
>
> Change: 2020-03-12 11:25:04.095981276 +0200
>
> Birth: 2020-04-11 08:53:26.805163816 +0300
>
>
>
>
>
> [root@mn-1:/root]
>
> # stat /mnt/bricks/export/brick/testfile
>
>   File: /mnt/bricks/export/brick/testfile
>
>   Size: 193 Blocks: 16 IO Block: 4096   regular file
>
> Device: fc02h/64514dInode: 512015  Links: 2
>
> Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (
> 615/_nokfsuifileshare)
>
> Access: 2020-04-11 12:20:22.100395536 +0300
>
> Modify: 2020-03-12 11:25:04.094913452 +0200
>
> Change: 2020-03-12 11:25:04.095913453 +0200
>
> Birth: 2020-03-12 07:53:26.803783053 +0200
>
>
>
>
>
>
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* 2020年3月12日 16:09
> *To:* 'Kotresh Hiremath Ravishankar' 
> *Cc:* Gluster Devel 
> *Subject:* RE: could you help to check about a glusterfs issue seems to
> be related to ctime
>
>
>
> Hi,
>
> This is abnormal test case, however, when this happened it will have big
> impact on the apps using those files. And this can not be restored
> automatically unless disable some xlator, I think it is unacceptable for
> the user apps.
>
>
>
>
>
> cynthia
>
>
>
> *From:* Kotresh Hiremath Ravishankar 
> *Sent:* 2020年3月12日 14:37
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) 
> *Cc:* Gluster Devel 
> *Subject:* Re: could you help to check about a glusterfs issue seems to
> be related to ctime
>
>
>
> All the perf xlators depend on time (mostly mtime I guess). In my setup,
> only quick read was enabled and hence disabling it worked for me.
> All perf xlators needs to be disabled to make it work

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

2020-03-16 Thread Zhou, Cynthia (NSB - CN/Hangzhou)

Ok, thanks for your feedback!
I will do local test to verify this patch first.

cynthia

From: Amar Tumballi 
Sent: 2020年3月17日 13:18
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
Cc: Kotresh Hiremath Ravishankar ; Gluster Devel 

Subject: Re: [Gluster-devel] could you help to check about a glusterfs issue 
seems to be related to ctime

On Tue, Mar 17, 2020 at 10:18 AM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
Hi glusterfs expert,
Our product need to tolerate change date to future and then change back.
How about change like this ?
https://review.gluster.org/#/c/glusterfs/+/24229/1/xlators/storage/posix/src/posix-metadata.c

when time change to future and change back , should still be able to update 
mdata, so the following changes to file can be populated to other clients.

We do like to have people integrating with GlusterFS. But this change is not 
inline with the 'assumptions' we had about the feature.

If you have verified this change works for you, please add it as an 'option' in 
posix, which can be changed through volume set, and keep this option 
disable/off by default. That should be an easier way to get the patch reviewed 
and take it further. Please make sure to provide the 'description' for the 
option with details.

Regards,
Amar

cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 17:31
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Cc: 'Gluster Devel' 
mailto:gluster-devel@gluster.org>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,
One more question, I find each client has the same future time stamp where are 
those time stamps from, since Since it is different from any brick stored time 
stamp. And after I modify files  from clients, it remains the same.
[root@mn-0:/home/robot]
# stat /mnt/export/testfile
  File: /mnt/export/testfile
  Size: 193 Blocks: 1  IO Block: 131072 regular file
Device: 28h/40d Inode: 10383279039841136109  Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-04-11 12:20:22.114365172 +0300
Modify: 2020-04-11 12:20:22.121552573 +0300
Change: 2020-04-11 12:20:22.121552573 +0300

[root@mn-0:/home/robot]
# date
Thu Mar 12 11:27:33 EET 2020
[root@mn-0:/home/robot]

[root@mn-0:/home/robot]
# stat /mnt/bricks/export/brick/testfile
  File: /mnt/bricks/export/brick/testfile
  Size: 193 Blocks: 16 IO Block: 4096   regular file
Device: fc02h/64514dInode: 512015  Links: 2
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-04-11 12:20:22.100395536 +0300
Modify: 2020-03-12 11:25:04.095981276 +0200
Change: 2020-03-12 11:25:04.095981276 +0200
Birth: 2020-04-11 08:53:26.805163816 +0300

[root@mn-1:/root]
# stat /mnt/bricks/export/brick/testfile
  File: /mnt/bricks/export/brick/testfile
  Size: 193 Blocks: 16 IO Block: 4096   regular file
Device: fc02h/64514dInode: 512015  Links: 2
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-04-11 12:20:22.100395536 +0300
Modify: 2020-03-12 11:25:04.094913452 +0200
Change: 2020-03-12 11:25:04.095913453 +0200
Birth: 2020-03-12 07:53:26.803783053 +0200

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 16:09
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,
This is abnormal test case, however, when this happened it will have big impact 
on the apps using those files. And this can not be restored automatically 
unless disable some xlator, I think it is unacceptable for the user apps.

cynthia

From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: 2020年3月12日 14:37
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: Re: could you help to check about a glusterfs issue seems to be 
related to ctime

All the perf xlators depend on time (mostly mtime I guess). In my setup, only 
quick read was enabled and hence disabling it worked for me.
All perf xlators needs to be disabled to make it work correctly. But I still 
failed to understand how normal this kind of workload ?

Thanks,
Kotresh

On Thu, Mar 12, 2020 at 11:20 AM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
When disable both quick-read and performance.io-cache off everything is back to 
normal
I attached the log when only enable quick-read and performance.io-cache is 
still on glusterfs trace log
When execute command “cat /mnt/export/testfile”
Can you help to find why this still to fail to show correct content?
The file size showed is 141, but actually in brick it is longer than that.

cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 12:53
To:

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

2020-03-16 Thread Amar Tumballi

On Tue, Mar 17, 2020 at 10:18 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.z...@nokia-sbell.com> wrote:

> Hi glusterfs expert,
>
> Our product need to tolerate change date to future and then change back.
>
> How about change like this ?
>
>
> https://review.gluster.org/#/c/glusterfs/+/24229/1/xlators/storage/posix/src/posix-metadata.c
>
>
>
> when time change to future and change back , should still be able to
> update mdata, so the following changes to file can be populated to other
> clients.
>
>
>

We do like to have people integrating with GlusterFS. But this change is
not inline with the 'assumptions' we had about the feature.

If you have verified this change works for you, please add it as an
'option' in posix, which can be changed through volume set, and keep this
option disable/off by default. That should be an easier way to get the
patch reviewed and take it further. Please make sure to provide the
'description' for the option with details.

Regards,
Amar



> cynthia
>
>
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* 2020年3月12日 17:31
> *To:* 'Kotresh Hiremath Ravishankar' 
> *Cc:* 'Gluster Devel' 
> *Subject:* RE: could you help to check about a glusterfs issue seems to
> be related to ctime
>
>
>
> Hi,
>
> One more question, I find each client has the same future time stamp where
> are those time stamps from, since Since it is different from any brick
> stored time stamp. And after I modify files  from clients, it remains the
> same.
>
> [root@mn-0:/home/robot]
>
> # stat /mnt/export/testfile
>
>   File: /mnt/export/testfile
>
>   Size: 193 Blocks: 1  IO Block: 131072 regular file
>
> Device: 28h/40d Inode: 10383279039841136109  Links: 1
>
> Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (
> 615/_nokfsuifileshare)
>
> Access: 2020-04-11 12:20:22.114365172 +0300
>
> Modify: 2020-04-11 12:20:22.121552573 +0300
>
> Change: 2020-04-11 12:20:22.121552573 +0300
>
>
>
> [root@mn-0:/home/robot]
>
> # date
>
> Thu Mar 12 11:27:33 EET 2020
>
> [root@mn-0:/home/robot]
>
>
>
> [root@mn-0:/home/robot]
>
> # stat /mnt/bricks/export/brick/testfile
>
>   File: /mnt/bricks/export/brick/testfile
>
>   Size: 193 Blocks: 16 IO Block: 4096   regular file
>
> Device: fc02h/64514dInode: 512015  Links: 2
>
> Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (
> 615/_nokfsuifileshare)
>
> Access: 2020-04-11 12:20:22.100395536 +0300
>
> Modify: 2020-03-12 11:25:04.095981276 +0200
>
> Change: 2020-03-12 11:25:04.095981276 +0200
>
> Birth: 2020-04-11 08:53:26.805163816 +0300
>
>
>
>
>
> [root@mn-1:/root]
>
> # stat /mnt/bricks/export/brick/testfile
>
>   File: /mnt/bricks/export/brick/testfile
>
>   Size: 193 Blocks: 16 IO Block: 4096   regular file
>
> Device: fc02h/64514dInode: 512015  Links: 2
>
> Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (
> 615/_nokfsuifileshare)
>
> Access: 2020-04-11 12:20:22.100395536 +0300
>
> Modify: 2020-03-12 11:25:04.094913452 +0200
>
> Change: 2020-03-12 11:25:04.095913453 +0200
>
> Birth: 2020-03-12 07:53:26.803783053 +0200
>
>
>
>
>
>
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* 2020年3月12日 16:09
> *To:* 'Kotresh Hiremath Ravishankar' 
> *Cc:* Gluster Devel 
> *Subject:* RE: could you help to check about a glusterfs issue seems to
> be related to ctime
>
>
>
> Hi,
>
> This is abnormal test case, however, when this happened it will have big
> impact on the apps using those files. And this can not be restored
> automatically unless disable some xlator, I think it is unacceptable for
> the user apps.
>
>
>
>
>
> cynthia
>
>
>
> *From:* Kotresh Hiremath Ravishankar 
> *Sent:* 2020年3月12日 14:37
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) 
> *Cc:* Gluster Devel 
> *Subject:* Re: could you help to check about a glusterfs issue seems to
> be related to ctime
>
>
>
> All the perf xlators depend on time (mostly mtime I guess). In my setup,
> only quick read was enabled and hence disabling it worked for me.
> All perf xlators needs to be disabled to make it work correctly. But I
> still failed to understand how normal this kind of workload ?
>
>
>
> Thanks,
> Kotresh
>
>
>
> On Thu, Mar 12, 2020 at 11:20 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.z...@nokia-sbell.com> wrote:
>
> When disable both quick-read and performance.io-cache off everything is
> back to normal
>
> I attached the log when only enable quick-read and performance.io-cache is
> still on glusterfs trace log
>
> When execute command “cat /mnt/export/testfile”
>
> Can you help to find why this still to fail to show correct content?
>
> The file size showed is 141, but actually in brick it is longer than that.
>
>
>
>
>
> cynthia
>
>
>
>
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* 2020年3月12日 12:53
> *To:* 'Kotresh Hiremath Ravishankar' 
> *Cc:* 'Gluster Devel' 
> *Subject:* RE: could you help to check about a glusterfs issue seems to
> be related to ctime
>
>
>
> From my local test only when

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

2020-03-16 Thread Zhou, Cynthia (NSB - CN/Hangzhou)

Hi glusterfs expert,
Our product need to tolerate change date to future and then change back.
How about change like this ?
https://review.gluster.org/#/c/glusterfs/+/24229/1/xlators/storage/posix/src/posix-metadata.c

when time change to future and change back , should still be able to update 
mdata, so the following changes to file can be populated to other clients.

cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 17:31
To: 'Kotresh Hiremath Ravishankar' 
Cc: 'Gluster Devel' 
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,
One more question, I find each client has the same future time stamp where are 
those time stamps from, since Since it is different from any brick stored time 
stamp. And after I modify files  from clients, it remains the same.
[root@mn-0:/home/robot]
# stat /mnt/export/testfile
  File: /mnt/export/testfile
  Size: 193 Blocks: 1  IO Block: 131072 regular file
Device: 28h/40d Inode: 10383279039841136109  Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-04-11 12:20:22.114365172 +0300
Modify: 2020-04-11 12:20:22.121552573 +0300
Change: 2020-04-11 12:20:22.121552573 +0300

[root@mn-0:/home/robot]
# date
Thu Mar 12 11:27:33 EET 2020
[root@mn-0:/home/robot]

[root@mn-0:/home/robot]
# stat /mnt/bricks/export/brick/testfile
  File: /mnt/bricks/export/brick/testfile
  Size: 193 Blocks: 16 IO Block: 4096   regular file
Device: fc02h/64514dInode: 512015  Links: 2
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-04-11 12:20:22.100395536 +0300
Modify: 2020-03-12 11:25:04.095981276 +0200
Change: 2020-03-12 11:25:04.095981276 +0200
Birth: 2020-04-11 08:53:26.805163816 +0300


[root@mn-1:/root]
# stat /mnt/bricks/export/brick/testfile
  File: /mnt/bricks/export/brick/testfile
  Size: 193 Blocks: 16 IO Block: 4096   regular file
Device: fc02h/64514dInode: 512015  Links: 2
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-04-11 12:20:22.100395536 +0300
Modify: 2020-03-12 11:25:04.094913452 +0200
Change: 2020-03-12 11:25:04.095913453 +0200
Birth: 2020-03-12 07:53:26.803783053 +0200



From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 16:09
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,
This is abnormal test case, however, when this happened it will have big impact 
on the apps using those files. And this can not be restored automatically 
unless disable some xlator, I think it is unacceptable for the user apps.


cynthia

From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: 2020年3月12日 14:37
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: Re: could you help to check about a glusterfs issue seems to be 
related to ctime

All the perf xlators depend on time (mostly mtime I guess). In my setup, only 
quick read was enabled and hence disabling it worked for me.
All perf xlators needs to be disabled to make it work correctly. But I still 
failed to understand how normal this kind of workload ?

Thanks,
Kotresh

On Thu, Mar 12, 2020 at 11:20 AM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
When disable both quick-read and performance.io-cache off everything is back to 
normal
I attached the log when only enable quick-read and performance.io-cache is 
still on glusterfs trace log
When execute command “cat /mnt/export/testfile”
Can you help to find why this still to fail to show correct content?
The file size showed is 141, but actually in brick it is longer than that.


cynthia


From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 12:53
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Cc: 'Gluster Devel' 
mailto:gluster-devel@gluster.org>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

From my local test only when disable both features.ctime and ctime.noatime this 
issue is gone.
Or
Do echo 3 >/proc/sys/vm/drop_caches after each time when some client change the 
file , can cat command show correct data(same as brick )

cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 9:53
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,
Thanks for your responding!
I’ve tried to disable quick-read:
[root@mn-0:/home/robot]
# gluster v get export all| grep quick
performance.quick-read  off
performance.nfs.quick-read  off

however, this issue still exists.
Two

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

2020-03-12 Thread Zhou, Cynthia (NSB - CN/Hangzhou)

Hi,
One more question, I find each client has the same future time stamp where are 
those time stamps from, since Since it is different from any brick stored time 
stamp. And after I modify files  from clients, it remains the same.
[root@mn-0:/home/robot]
# stat /mnt/export/testfile
  File: /mnt/export/testfile
  Size: 193 Blocks: 1  IO Block: 131072 regular file
Device: 28h/40d Inode: 10383279039841136109  Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-04-11 12:20:22.114365172 +0300
Modify: 2020-04-11 12:20:22.121552573 +0300
Change: 2020-04-11 12:20:22.121552573 +0300

[root@mn-0:/home/robot]
# date
Thu Mar 12 11:27:33 EET 2020
[root@mn-0:/home/robot]

[root@mn-0:/home/robot]
# stat /mnt/bricks/export/brick/testfile
  File: /mnt/bricks/export/brick/testfile
  Size: 193 Blocks: 16 IO Block: 4096   regular file
Device: fc02h/64514dInode: 512015  Links: 2
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-04-11 12:20:22.100395536 +0300
Modify: 2020-03-12 11:25:04.095981276 +0200
Change: 2020-03-12 11:25:04.095981276 +0200
Birth: 2020-04-11 08:53:26.805163816 +0300


[root@mn-1:/root]
# stat /mnt/bricks/export/brick/testfile
  File: /mnt/bricks/export/brick/testfile
  Size: 193 Blocks: 16 IO Block: 4096   regular file
Device: fc02h/64514dInode: 512015  Links: 2
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-04-11 12:20:22.100395536 +0300
Modify: 2020-03-12 11:25:04.094913452 +0200
Change: 2020-03-12 11:25:04.095913453 +0200
Birth: 2020-03-12 07:53:26.803783053 +0200



From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 16:09
To: 'Kotresh Hiremath Ravishankar' 
Cc: Gluster Devel 
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,
This is abnormal test case, however, when this happened it will have big impact 
on the apps using those files. And this can not be restored automatically 
unless disable some xlator, I think it is unacceptable for the user apps.


cynthia

From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: 2020年3月12日 14:37
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: Re: could you help to check about a glusterfs issue seems to be 
related to ctime

All the perf xlators depend on time (mostly mtime I guess). In my setup, only 
quick read was enabled and hence disabling it worked for me.
All perf xlators needs to be disabled to make it work correctly. But I still 
failed to understand how normal this kind of workload ?

Thanks,
Kotresh

On Thu, Mar 12, 2020 at 11:20 AM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
When disable both quick-read and performance.io-cache off everything is back to 
normal
I attached the log when only enable quick-read and performance.io-cache is 
still on glusterfs trace log
When execute command “cat /mnt/export/testfile”
Can you help to find why this still to fail to show correct content?
The file size showed is 141, but actually in brick it is longer than that.


cynthia


From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 12:53
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Cc: 'Gluster Devel' 
mailto:gluster-devel@gluster.org>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

From my local test only when disable both features.ctime and ctime.noatime this 
issue is gone.
Or
Do echo 3 >/proc/sys/vm/drop_caches after each time when some client change the 
file , can cat command show correct data(same as brick )

cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 9:53
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,
Thanks for your responding!
I’ve tried to disable quick-read:
[root@mn-0:/home/robot]
# gluster v get export all| grep quick
performance.quick-read  off
performance.nfs.quick-read  off

however, this issue still exists.
Two clients see different contents.

it seems only after I disable utime this issue is completely gone.
features.ctime  off
ctime.noatime   off


Do you know why is this?


Cynthia
Nokia storage team
From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: 2020年3月11日 22:05
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: Re: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,

I figured out what's happening. The issue is that the file has 'c|a|m' time set 
to future (The file

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

2020-03-12 Thread Zhou, Cynthia (NSB - CN/Hangzhou)

Hi,
This is abnormal test case, however, when this happened it will have big impact 
on the apps using those files. And this can not be restored automatically 
unless disable some xlator, I think it is unacceptable for the user apps.


cynthia

From: Kotresh Hiremath Ravishankar 
Sent: 2020年3月12日 14:37
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
Cc: Gluster Devel 
Subject: Re: could you help to check about a glusterfs issue seems to be 
related to ctime

All the perf xlators depend on time (mostly mtime I guess). In my setup, only 
quick read was enabled and hence disabling it worked for me.
All perf xlators needs to be disabled to make it work correctly. But I still 
failed to understand how normal this kind of workload ?

Thanks,
Kotresh

On Thu, Mar 12, 2020 at 11:20 AM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
When disable both quick-read and performance.io-cache off everything is back to 
normal
I attached the log when only enable quick-read and performance.io-cache is 
still on glusterfs trace log
When execute command “cat /mnt/export/testfile”
Can you help to find why this still to fail to show correct content?
The file size showed is 141, but actually in brick it is longer than that.


cynthia


From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 12:53
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Cc: 'Gluster Devel' 
mailto:gluster-devel@gluster.org>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

From my local test only when disable both features.ctime and ctime.noatime this 
issue is gone.
Or
Do echo 3 >/proc/sys/vm/drop_caches after each time when some client change the 
file , can cat command show correct data(same as brick )

cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 9:53
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,
Thanks for your responding!
I’ve tried to disable quick-read:
[root@mn-0:/home/robot]
# gluster v get export all| grep quick
performance.quick-read  off
performance.nfs.quick-read  off

however, this issue still exists.
Two clients see different contents.

it seems only after I disable utime this issue is completely gone.
features.ctime  off
ctime.noatime   off


Do you know why is this?


Cynthia
Nokia storage team
From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: 2020年3月11日 22:05
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: Re: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,

I figured out what's happening. The issue is that the file has 'c|a|m' time set 
to future (The file is created after the date is set to +30 days). This
is done from client-1. On client-2 with correct date, when data is appended, it 
doesn't update the mtime and ctime because of both mtime and ctime is less than
already set time on the file. This protection is required to keep the latest 
time when two clients are writing to the same file. We update c|m|a time only 
if it's greater than
existing time. As a result, the perf xlators on client1 which relies on mtime 
doesn't send read to server as it thinks nothing is changed as in this case the 
times haven't
changed.

Workarounds:
1. Disabling quick-read solved the issue for me.
I don't know how real this kind of workload is? Is this a normal scenario ?
The other thing to do is to remove that protection of updating time only if 
it's greater but that would open up the race when two clients are updating the 
same file.
This would result in keeping the older time than the latest. This requires code 
change and I don't think that should be done.

Thanks,
Kotresh

On Wed, Mar 11, 2020 at 3:02 PM Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>> wrote:
Exactly, I am also curious about this. I will debug and update about what's 
exactly happening.

Thanks,
Kotresh

On Wed, Mar 11, 2020 at 1:56 PM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
I used to think the file is cached in some client side buffer, because I’ve 
checked from different sn brick, the file content are all correct. But when I 
open client side trace level log, and cat the file, I only find 
lookup/open/flush fop from fuse-bridge side, I am just wondering how is file 
content served to client side? Should not there be readv fop seen from trace 
log?

cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月11日 15:54
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Does that require, that for all the time client should be time

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

2020-03-12 Thread Kotresh Hiremath Ravishankar

All the perf xlators depend on time (mostly mtime I guess). In my setup,
only quick read was enabled and hence disabling it worked for me.
All perf xlators needs to be disabled to make it work correctly. But I
still failed to understand how normal this kind of workload ?

Thanks,
Kotresh

On Thu, Mar 12, 2020 at 11:20 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.z...@nokia-sbell.com> wrote:

> When disable both quick-read and performance.io-cache off everything is
> back to normal
>
> I attached the log when only enable quick-read and performance.io-cache is
> still on glusterfs trace log
>
> When execute command “cat /mnt/export/testfile”
>
> Can you help to find why this still to fail to show correct content?
>
> The file size showed is 141, but actually in brick it is longer than that.
>
>
>
>
>
> cynthia
>
>
>
>
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* 2020年3月12日 12:53
> *To:* 'Kotresh Hiremath Ravishankar' 
> *Cc:* 'Gluster Devel' 
> *Subject:* RE: could you help to check about a glusterfs issue seems to
> be related to ctime
>
>
>
> From my local test only when disable both features.ctime and ctime.noatime
> this issue is gone.
>
> Or
>
> Do echo 3 >/proc/sys/vm/drop_caches after each time when some client
> change the file , can cat command show correct data(same as brick )
>
>
>
> cynthia
>
>
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* 2020年3月12日 9:53
> *To:* 'Kotresh Hiremath Ravishankar' 
> *Cc:* Gluster Devel 
> *Subject:* RE: could you help to check about a glusterfs issue seems to
> be related to ctime
>
>
>
> Hi,
>
> Thanks for your responding!
>
> I’ve tried to disable quick-read:
>
> [root@mn-0:/home/robot]
>
> # gluster v get export all| grep quick
>
> performance.quick-read  off
>
> performance.nfs.quick-read  off
>
>
>
> however, this issue still exists.
>
> Two clients see different contents.
>
>
>
> it seems only after I disable utime this issue is completely gone.
>
> features.ctime  off
>
> ctime.noatime   off
>
>
>
>
>
> Do you know why is this?
>
>
>
>
>
> Cynthia
>
> Nokia storage team
>
> *From:* Kotresh Hiremath Ravishankar 
> *Sent:* 2020年3月11日 22:05
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) 
> *Cc:* Gluster Devel 
> *Subject:* Re: could you help to check about a glusterfs issue seems to
> be related to ctime
>
>
>
> Hi,
>
> I figured out what's happening. The issue is that the file has 'c|a|m'
> time set to future (The file is created after the date is set to +30 days).
> This
> is done from client-1. On client-2 with correct date, when data is
> appended, it doesn't update the mtime and ctime because of both mtime and
> ctime is less than
> already set time on the file. This protection is required to keep the
> latest time when two clients are writing to the same file. We update c|m|a
> time only if it's greater than
> existing time. As a result, the perf xlators on client1 which relies on
> mtime doesn't send read to server as it thinks nothing is changed as in
> this case the times haven't
> changed.
>
>
>
> Workarounds:
> 1. Disabling quick-read solved the issue for me.
>
> I don't know how real this kind of workload is? Is this a normal scenario ?
> The other thing to do is to remove that protection of updating time only
> if it's greater but that would open up the race when two clients are
> updating the same file.
>
> This would result in keeping the older time than the latest. This requires
> code change and I don't think that should be done.
>
>
>
> Thanks,
> Kotresh
>
>
>
> On Wed, Mar 11, 2020 at 3:02 PM Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
> Exactly, I am also curious about this. I will debug and update about
> what's exactly happening.
>
>
>
> Thanks,
> Kotresh
>
>
>
> On Wed, Mar 11, 2020 at 1:56 PM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.z...@nokia-sbell.com> wrote:
>
> I used to think the file is cached in some client side buffer, because
> I’ve checked from different sn brick, the file content are all correct. But
> when I open client side trace level log, and cat the file, I only find
> lookup/open/flush fop from fuse-bridge side, I am just wondering how is
> file content served to client side? Should not there be readv fop seen from
> trace log?
>
>
>
> cynthia
>
>
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* 2020年3月11日 15:54
> *To:* 'Kotresh Hiremath Ravishankar' 
> *Subject:* RE: could you help to check about a glusterfs issue seems to
> be related to ctime
>
>
>
> Does that require, that for all the time client should be time synched?
> What if the client time is not synched for a while? And then restored?
>
> I make a test when time has been restored and then client change the file,
> the file’s modify time, access times remains to be wrong, is that correct?
>
>
>
> root@mn-0:/home/robot]
>
> # echo "fromm mn-0">>/mnt/export/testfile
>
> [root@mn-0:/home/robot]
>
> # stat /mnt/export/testfile
>
>   File:

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

2020-03-11 Thread Zhou, Cynthia (NSB - CN/Hangzhou)

When disable both quick-read and performance.io-cache off everything is back to 
normal
I attached the log when only enable quick-read and performance.io-cache is 
still on glusterfs trace log
When execute command “cat /mnt/export/testfile”
Can you help to find why this still to fail to show correct content?
The file size showed is 141, but actually in brick it is longer than that.


cynthia


From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 12:53
To: 'Kotresh Hiremath Ravishankar' 
Cc: 'Gluster Devel' 
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

From my local test only when disable both features.ctime and ctime.noatime this 
issue is gone.
Or
Do echo 3 >/proc/sys/vm/drop_caches after each time when some client change the 
file , can cat command show correct data(same as brick )

cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 9:53
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,
Thanks for your responding!
I’ve tried to disable quick-read:
[root@mn-0:/home/robot]
# gluster v get export all| grep quick
performance.quick-read  off
performance.nfs.quick-read  off

however, this issue still exists.
Two clients see different contents.

it seems only after I disable utime this issue is completely gone.
features.ctime  off
ctime.noatime   off


Do you know why is this?


Cynthia
Nokia storage team
From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: 2020年3月11日 22:05
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: Re: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,

I figured out what's happening. The issue is that the file has 'c|a|m' time set 
to future (The file is created after the date is set to +30 days). This
is done from client-1. On client-2 with correct date, when data is appended, it 
doesn't update the mtime and ctime because of both mtime and ctime is less than
already set time on the file. This protection is required to keep the latest 
time when two clients are writing to the same file. We update c|m|a time only 
if it's greater than
existing time. As a result, the perf xlators on client1 which relies on mtime 
doesn't send read to server as it thinks nothing is changed as in this case the 
times haven't
changed.

Workarounds:
1. Disabling quick-read solved the issue for me.
I don't know how real this kind of workload is? Is this a normal scenario ?
The other thing to do is to remove that protection of updating time only if 
it's greater but that would open up the race when two clients are updating the 
same file.
This would result in keeping the older time than the latest. This requires code 
change and I don't think that should be done.

Thanks,
Kotresh

On Wed, Mar 11, 2020 at 3:02 PM Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>> wrote:
Exactly, I am also curious about this. I will debug and update about what's 
exactly happening.

Thanks,
Kotresh

On Wed, Mar 11, 2020 at 1:56 PM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
I used to think the file is cached in some client side buffer, because I’ve 
checked from different sn brick, the file content are all correct. But when I 
open client side trace level log, and cat the file, I only find 
lookup/open/flush fop from fuse-bridge side, I am just wondering how is file 
content served to client side? Should not there be readv fop seen from trace 
log?

cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月11日 15:54
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Does that require, that for all the time client should be time synched? What if 
the client time is not synched for a while? And then restored?
I make a test when time has been restored and then client change the file, the 
file’s modify time, access times remains to be wrong, is that correct?

root@mn-0:/home/robot]
# echo "fromm mn-0">>/mnt/export/testfile
[root@mn-0:/home/robot]
# stat /mnt/export/testfile
  File: /mnt/export/testfile
  Size: 30  Blocks: 1  IO Block: 131072 regular file
Device: 28h/40d Inode: 9855109080001305442  Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-05-10 09:33:59.713840197 +0300
Modify: 2020-05-10 09:33:59.713840197 +0300
Change: 2020-05-10 09:33:59.714413772 +0300  //remains to be future time
Birth: -
[root@mn-0:/home/robot]
# cat /mnt/export/testfil
cat: /mnt/export/testfil: No such file or directory
[root@mn-0:/home/robot]
# cat /mnt/export/testfile
from mn0
from mn-1

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

2020-03-11 Thread Zhou, Cynthia (NSB - CN/Hangzhou)

From my local test only when disable both features.ctime and ctime.noatime this 
issue is gone.
Or
Do echo 3 >/proc/sys/vm/drop_caches after each time when some client change the 
file , can cat command show correct data(same as brick )

cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月12日 9:53
To: 'Kotresh Hiremath Ravishankar' 
Cc: Gluster Devel 
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,
Thanks for your responding!
I’ve tried to disable quick-read:
[root@mn-0:/home/robot]
# gluster v get export all| grep quick
performance.quick-read  off
performance.nfs.quick-read  off

however, this issue still exists.
Two clients see different contents.

it seems only after I disable utime this issue is completely gone.
features.ctime  off
ctime.noatime   off


Do you know why is this?


Cynthia
Nokia storage team
From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: 2020年3月11日 22:05
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>>
Cc: Gluster Devel mailto:gluster-devel@gluster.org>>
Subject: Re: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,

I figured out what's happening. The issue is that the file has 'c|a|m' time set 
to future (The file is created after the date is set to +30 days). This
is done from client-1. On client-2 with correct date, when data is appended, it 
doesn't update the mtime and ctime because of both mtime and ctime is less than
already set time on the file. This protection is required to keep the latest 
time when two clients are writing to the same file. We update c|m|a time only 
if it's greater than
existing time. As a result, the perf xlators on client1 which relies on mtime 
doesn't send read to server as it thinks nothing is changed as in this case the 
times haven't
changed.

Workarounds:
1. Disabling quick-read solved the issue for me.
I don't know how real this kind of workload is? Is this a normal scenario ?
The other thing to do is to remove that protection of updating time only if 
it's greater but that would open up the race when two clients are updating the 
same file.
This would result in keeping the older time than the latest. This requires code 
change and I don't think that should be done.

Thanks,
Kotresh

On Wed, Mar 11, 2020 at 3:02 PM Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>> wrote:
Exactly, I am also curious about this. I will debug and update about what's 
exactly happening.

Thanks,
Kotresh

On Wed, Mar 11, 2020 at 1:56 PM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
I used to think the file is cached in some client side buffer, because I’ve 
checked from different sn brick, the file content are all correct. But when I 
open client side trace level log, and cat the file, I only find 
lookup/open/flush fop from fuse-bridge side, I am just wondering how is file 
content served to client side? Should not there be readv fop seen from trace 
log?

cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月11日 15:54
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Does that require, that for all the time client should be time synched? What if 
the client time is not synched for a while? And then restored?
I make a test when time has been restored and then client change the file, the 
file’s modify time, access times remains to be wrong, is that correct?

root@mn-0:/home/robot]
# echo "fromm mn-0">>/mnt/export/testfile
[root@mn-0:/home/robot]
# stat /mnt/export/testfile
  File: /mnt/export/testfile
  Size: 30  Blocks: 1  IO Block: 131072 regular file
Device: 28h/40d Inode: 9855109080001305442  Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-05-10 09:33:59.713840197 +0300
Modify: 2020-05-10 09:33:59.713840197 +0300
Change: 2020-05-10 09:33:59.714413772 +0300  //remains to be future time
Birth: -
[root@mn-0:/home/robot]
# cat /mnt/export/testfil
cat: /mnt/export/testfil: No such file or directory
[root@mn-0:/home/robot]
# cat /mnt/export/testfile
from mn0
from mn-1
fromm mn-0
[root@mn-0:/home/robot]
# date
Wed 11 Mar 2020 09:05:58 AM EET
[root@mn-0:/home/robot]

cynthia

From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: 2020年3月11日 15:41
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>>
Subject: Re: could you help to check about a glusterfs issue seems to be 
related to ctime



On Wed, Mar 11, 2020 at 12:46 PM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
But there are times that ntp service went wrong, and time on two storage nodes 
may be not synced.
Or do you mean when can not guarantee that the time on two clients is synched, 
we should

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

2020-03-11 Thread Zhou, Cynthia (NSB - CN/Hangzhou)

Hi,
Thanks for your responding!
I’ve tried to disable quick-read:
[root@mn-0:/home/robot]
# gluster v get export all| grep quick
performance.quick-read  off
performance.nfs.quick-read  off

however, this issue still exists.
Two clients see different contents.

it seems only after I disable utime this issue is completely gone.
features.ctime  off
ctime.noatime   off


Do you know why is this?


Cynthia
Nokia storage team
From: Kotresh Hiremath Ravishankar 
Sent: 2020年3月11日 22:05
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
Cc: Gluster Devel 
Subject: Re: could you help to check about a glusterfs issue seems to be 
related to ctime

Hi,

I figured out what's happening. The issue is that the file has 'c|a|m' time set 
to future (The file is created after the date is set to +30 days). This
is done from client-1. On client-2 with correct date, when data is appended, it 
doesn't update the mtime and ctime because of both mtime and ctime is less than
already set time on the file. This protection is required to keep the latest 
time when two clients are writing to the same file. We update c|m|a time only 
if it's greater than
existing time. As a result, the perf xlators on client1 which relies on mtime 
doesn't send read to server as it thinks nothing is changed as in this case the 
times haven't
changed.

Workarounds:
1. Disabling quick-read solved the issue for me.
I don't know how real this kind of workload is? Is this a normal scenario ?
The other thing to do is to remove that protection of updating time only if 
it's greater but that would open up the race when two clients are updating the 
same file.
This would result in keeping the older time than the latest. This requires code 
change and I don't think that should be done.

Thanks,
Kotresh

On Wed, Mar 11, 2020 at 3:02 PM Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>> wrote:
Exactly, I am also curious about this. I will debug and update about what's 
exactly happening.

Thanks,
Kotresh

On Wed, Mar 11, 2020 at 1:56 PM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
I used to think the file is cached in some client side buffer, because I’ve 
checked from different sn brick, the file content are all correct. But when I 
open client side trace level log, and cat the file, I only find 
lookup/open/flush fop from fuse-bridge side, I am just wondering how is file 
content served to client side? Should not there be readv fop seen from trace 
log?

cynthia

From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: 2020年3月11日 15:54
To: 'Kotresh Hiremath Ravishankar' 
mailto:khire...@redhat.com>>
Subject: RE: could you help to check about a glusterfs issue seems to be 
related to ctime

Does that require, that for all the time client should be time synched? What if 
the client time is not synched for a while? And then restored?
I make a test when time has been restored and then client change the file, the 
file’s modify time, access times remains to be wrong, is that correct?

root@mn-0:/home/robot]
# echo "fromm mn-0">>/mnt/export/testfile
[root@mn-0:/home/robot]
# stat /mnt/export/testfile
  File: /mnt/export/testfile
  Size: 30  Blocks: 1  IO Block: 131072 regular file
Device: 28h/40d Inode: 9855109080001305442  Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (  
615/_nokfsuifileshare)
Access: 2020-05-10 09:33:59.713840197 +0300
Modify: 2020-05-10 09:33:59.713840197 +0300
Change: 2020-05-10 09:33:59.714413772 +0300  //remains to be future time
Birth: -
[root@mn-0:/home/robot]
# cat /mnt/export/testfil
cat: /mnt/export/testfil: No such file or directory
[root@mn-0:/home/robot]
# cat /mnt/export/testfile
from mn0
from mn-1
fromm mn-0
[root@mn-0:/home/robot]
# date
Wed 11 Mar 2020 09:05:58 AM EET
[root@mn-0:/home/robot]

cynthia

From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: 2020年3月11日 15:41
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>>
Subject: Re: could you help to check about a glusterfs issue seems to be 
related to ctime



On Wed, Mar 11, 2020 at 12:46 PM Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>> wrote:
But there are times that ntp service went wrong, and time on two storage nodes 
may be not synced.
Or do you mean when can not guarantee that the time on two clients is synched, 
we should not enable this ctime feature?
Yes, that's correct. The ctime feature relies on the time generated at the 
client (that's the utime xlator loaded in client) and hence
expects all clients to be ntp synced.

Without ctime feature, is there some way to avoid this “file changed as we read 
it” issue?
Unfortunately no. That's the only way as of now.
cynthia

From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: 2020年3月11日 15:12
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
mailto:cynthia.z...@nokia-sbell.com>>
Subject: Re: could you help to check

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

2020-03-11 Thread Kotresh Hiremath Ravishankar

Hi,

I figured out what's happening. The issue is that the file has 'c|a|m' time
set to future (The file is created after the date is set to +30 days). This
is done from client-1. On client-2 with correct date, when data is
appended, it doesn't update the mtime and ctime because of both mtime and
ctime is less than
already set time on the file. This protection is required to keep the
latest time when two clients are writing to the same file. We update c|m|a
time only if it's greater than
existing time. As a result, the perf xlators on client1 which relies on
mtime doesn't send read to server as it thinks nothing is changed as in
this case the times haven't
changed.

Workarounds:
1. Disabling quick-read solved the issue for me.

I don't know how real this kind of workload is? Is this a normal scenario ?
The other thing to do is to remove that protection of updating time only if
it's greater but that would open up the race when two clients are updating
the same file.
This would result in keeping the older time than the latest. This requires
code change and I don't think that should be done.

Thanks,
Kotresh

On Wed, Mar 11, 2020 at 3:02 PM Kotresh Hiremath Ravishankar <
khire...@redhat.com> wrote:

> Exactly, I am also curious about this. I will debug and update about
> what's exactly happening.
>
> Thanks,
> Kotresh
>
> On Wed, Mar 11, 2020 at 1:56 PM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.z...@nokia-sbell.com> wrote:
>
>> I used to think the file is cached in some client side buffer, because
>> I’ve checked from different sn brick, the file content are all correct. But
>> when I open client side trace level log, and cat the file, I only find
>> lookup/open/flush fop from fuse-bridge side, I am just wondering how is
>> file content served to client side? Should not there be readv fop seen from
>> trace log?
>>
>>
>>
>> cynthia
>>
>>
>>
>> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
>> *Sent:* 2020年3月11日 15:54
>> *To:* 'Kotresh Hiremath Ravishankar' 
>> *Subject:* RE: could you help to check about a glusterfs issue seems to
>> be related to ctime
>>
>>
>>
>> Does that require, that for all the time client should be time synched?
>> What if the client time is not synched for a while? And then restored?
>>
>> I make a test when time has been restored and then client change the
>> file, the file’s modify time, access times remains to be wrong, is that
>> correct?
>>
>>
>>
>> root@mn-0:/home/robot]
>>
>> # echo "fromm mn-0">>/mnt/export/testfile
>>
>> [root@mn-0:/home/robot]
>>
>> # stat /mnt/export/testfile
>>
>>   File: /mnt/export/testfile
>>
>>   Size: 30  Blocks: 1  IO Block: 131072 regular file
>>
>> Device: 28h/40d Inode: 9855109080001305442  Links: 1
>>
>> Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (
>> 615/_nokfsuifileshare)
>>
>> Access: 2020-05-10 09:33:59.713840197 +0300
>>
>> Modify: 2020-05-10 09:33:59.713840197 +0300
>>
>> Change: 2020-05-10 09:33:59.714413772 +0300  //remains to be future time
>>
>> Birth: -
>>
>> [root@mn-0:/home/robot]
>>
>> # cat /mnt/export/testfil
>>
>> cat: /mnt/export/testfil: No such file or directory
>>
>> [root@mn-0:/home/robot]
>>
>> # cat /mnt/export/testfile
>>
>> from mn0
>>
>> from mn-1
>>
>> fromm mn-0
>>
>> [root@mn-0:/home/robot]
>>
>> # date
>>
>> Wed 11 Mar 2020 09:05:58 AM EET
>>
>> [root@mn-0:/home/robot]
>>
>>
>>
>> cynthia
>>
>>
>>
>> *From:* Kotresh Hiremath Ravishankar 
>> *Sent:* 2020年3月11日 15:41
>> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) 
>> *Subject:* Re: could you help to check about a glusterfs issue seems to
>> be related to ctime
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Mar 11, 2020 at 12:46 PM Zhou, Cynthia (NSB - CN/Hangzhou) <
>> cynthia.z...@nokia-sbell.com> wrote:
>>
>> But there are times that ntp service went wrong, and time on two storage
>> nodes may be not synced.
>>
>> Or do you mean when can not guarantee that the time on two clients is
>> synched, we should not enable this ctime feature?
>>
>> Yes, that's correct. The ctime feature relies on the time generated at
>> the client (that's the utime xlator loaded in client) and hence
>> expects all clients to be ntp synced.
>>
>>
>>
>> Without ctime feature, is there some way to avoid this “file changed as
>> we read it” issue?
>>
>> Unfortunately no. That's the only way as of now.
>>
>> cynthia
>>
>>
>>
>> *From:* Kotresh Hiremath Ravishankar 
>> *Sent:* 2020年3月11日 15:12
>> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) 
>> *Subject:* Re: could you help to check about a glusterfs issue seems to
>> be related to ctime
>>
>>
>>
>> Hi,
>>
>> I have not looked at it. I will take a look and update you. But one of
>> the pre-requisite for ctime feature is that the clients should be time
>> synced (ntp or other means).
>> Could you try your reproducer by syncing the time of all clients and
>> update me back ?
>>
>> Thanks,
>>
>> Kotresh HR
>>
>>
>>
>> On Wed, Mar 11, 2020 at 12:37 PM Zhou, Cynthia (NSB - CN/Hangzhou) <
>> cynthia.z...@nokia-sbell.com> wrote:
>>
>> I make

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

Re: [Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

13 matches

Site Navigation

Mail list logo

Footer information