Hi Artem, As this is a replicated volume , can you try the following: 0. Check for pending heals and wait to complete before proceeding gluster volume heal <Volume Name> 1. On first brick - kill all processes via the script in /usr/share/gluster/scripts/ 2. Run a start via : gluster volume start <Volume Name > force 3. Verify that brick is started and running gluster volune status <Volume Name> 4. Check the logs for entries containing 'data-client-0' . If there is none -> wait for the heal to end and repeat with the rest of the bricks.
Best Regards, Strahil NikolovOn Dec 24, 2019 22:35, Artem Russakovskii <archon...@gmail.com> wrote: > > Hi everyone, > > I'm preparing the upgrade for our 5.10 gluster cluster running 1x4 replicate > volumes to 7.X and decided to upgrade our test cluster first. > > As soon as I upgraded to 7.0 (and now 7.1), I started seeing the following > messages every 10 minutes in the log for one of the volumes: > >> [2019-12-19 21:27:55.041949] W [MSGID: 114031] >> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-0: >> remote operation failed [Permission denied] >> [2019-12-19 21:27:55.042002] W [MSGID: 114031] >> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-2: >> remote operation failed [Permission denied] >> [2019-12-19 21:27:55.042013] W [MSGID: 114031] >> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-1: >> remote operation failed [Permission denied] >> [2019-12-19 21:27:55.042634] E [MSGID: 148002] >> [utime.c:146:gf_utime_set_mdata_setxattr_cbk] 0-dev_SNIP_data-utime: dict >> set of key for set-ctime-mdata failed [Permission denied] >> The message "W [MSGID: 114031] >> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-0: >> remote operation failed [Permission denied]" repeated 2 times between >> [2019-12-19 21:27:55.041949] and [2019-12-19 21:27:55.047300] >> The message "W [MSGID: 114031] >> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-2: >> remote operation failed [Permission denied]" repeated 2 times between >> [2019-12-19 21:27:55.042002] and [2019-12-19 21:27:55.047312] >> The message "W [MSGID: 114031] >> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-1: >> remote operation failed [Permission denied]" repeated 2 times between >> [2019-12-19 21:27:55.042013] and [2019-12-19 21:27:55.047524] >> The message "E [MSGID: 148002] [utime.c:146:gf_utime_set_mdata_setxattr_cbk] >> 0-dev_SNIP_data-utime: dict set of key for set-ctime-mdata failed >> [Permission denied]" repeated 2 times between [2019-12-19 21:27:55.042634] >> and [2019-12-19 21:27:55.047887] > > >> >> [2019-12-19 21:37:55.541329] W [MSGID: 114031] >> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-2: >> remote operation failed [Permission denied] >> [2019-12-19 21:37:55.541644] W [MSGID: 114031] >> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-1: >> remote operation failed [Permission denied] >> [2019-12-19 21:37:55.541681] W [MSGID: 114031] >> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-0: >> remote operation failed [Permission denied] >> [2019-12-19 21:37:55.542067] E [MSGID: 148002] >> [utime.c:146:gf_utime_set_mdata_setxattr_cbk] 0-dev_SNIP_data-utime: dict >> set of key for set-ctime-mdata failed [Permission denied] >> The message "W [MSGID: 114031] >> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-2: >> remote operation failed [Permission denied]" repeated 3 times between >> [2019-12-19 21:37:55.541329] and [2019-12-19 21:37:55.546695] >> The message "W [MSGID: 114031] >> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-1: >> remote operation failed [Permission denied]" repeated 3 times between >> [2019-12-19 21:37:55.541644] and [2019-12-19 21:37:55.546711] >> The message "W [MSGID: 114031] >> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-0: >> remote operation failed [Permission denied]" repeated 3 times between >> [2019-12-19 21:37:55.541681] and [2019-12-19 21:37:55.546761] >> The message "E [MSGID: 148002] [utime.c:146:gf_utime_set_mdata_setxattr_cbk] >> 0-dev_SNIP_data-utime: dict set of key for set-ctime-mdata failed >> [Permission denied]" repeated 3 times between [2019-12-19 21:37:55.542067] >> and [2019-12-19 21:37:55.547042] > > > etc. > > The questions are: > Is it a cause for concern? They weren't there before the upgrade. > How can I determine what's causing the errors? > How can I fix them and prevent them from spamming the logs? > Thank you and happy holidays! > > Sincerely, > Artem > > -- > Founder, Android Police, APK Mirror, Illogical Robot LLC > beerpla.net | @ArtemR
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
