Hi all, We have a problem with tasks (written in PHP) doing a lot of file renaming/moving (even several renames per second).
The pattern is always the same: the task rename files with variable filename to the SAME final file (that is: the destination filename is ALWAYS the same). Problem: according to glusterfs and task logs, some rename are successful but other don't and we don't understand why or what is causing the error in the failing renames because the error is "File exists", but the rename supossedly must address these situations correctly and "overwrite" the destination file if it exists (it uses the PHP rename() function) More info: - We use a distributed-replicated volume in a 3 node cluster gluster volume info moodle-cv Volume Name: moodle-cv Type: Distributed-Replicate Volume ID: 1eef9714-3943-4d77-b42f-6b1144389c56 Status: Started Snapshot Count: 0 Number of Bricks: 5 x (2 + 1) = 15 Transport-type: tcp Bricks: Brick1: moodle2017-n2:/glusterfs/moodle-cv/brick1/datos Brick2: moodle2017-n3:/glusterfs/moodle-cv/brick1/replica Brick3: moodle2017-n1:/glusterfs/moodle-cv/brick1/arbiter (arbiter) Brick4: moodle2017-n3:/glusterfs/moodle-cv/brick2/datos Brick5: moodle2017-n2:/glusterfs/moodle-cv/brick2/replica Brick6: moodle2017-n1:/glusterfs/moodle-cv/brick2/arbiter (arbiter) Brick7: moodle2017-n2:/glusterfs/moodle-cv/brick3/datos Brick8: moodle2017-n3:/glusterfs/moodle-cv/brick3/replica Brick9: moodle2017-n1:/glusterfs/moodle-cv/brick3/arbiter (arbiter) Brick10: moodle2017-n3:/glusterfs/moodle-cv/brick4/datos Brick11: moodle2017-n2:/glusterfs/moodle-cv/brick4/replica Brick12: moodle2017-n1:/glusterfs/moodle-cv/brick4/arbiter (arbiter) Brick13: moodle2017-n2:/glusterfs/moodle-cv/brick5/datos Brick14: moodle2017-n3:/glusterfs/moodle-cv/brick5/replica Brick15: moodle2017-n1:/glusterfs/moodle-cv/brick5/arbiter (arbiter) Options Reconfigured: nfs.disable: on storage.fips-mode-rchecksum: on performance.open-behind: off performance.lazy-open: no cluster.self-heal-daemon: enable - The nodes use Oracle Linux 7.9 (RedHat clone) and GlusterFS 8.5 - The tasks use the PHP rename() function - The tasks renaming files are executed IN ONE NODE OF THE CLUSTER that mounts the volume using the FUSE client - At the end of the message I paste a (very) little fragment of the volume log Any ideas of the possible cause of the problem and/or suggestions to avoid it? Regards. This is the fragment of the log: * I've "separated" the lines by "rename operation" but all the lines appear together in the log --->SUCCESSFUL RENAME: [2021-09-03 09:04:02.478321] I [MSGID: 109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e50272f326.07618090.temp (4a875eac-5389-400c-9359-458e151054f7) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) => /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache (11fdf019-16f8-48ae-bd1f-05b670b29ec1) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-2) --->UNSUCCESSFUL RENAME: [2021-09-03 09:04:02.497845] I [MSGID: 109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp (350698df-35d9-4489-90cc-faa004bfc861) (hash=moodle-cv-replicate-0/cache=moodle-cv-replicate-0) => /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache (4a875eac-5389-400c-9359-458e151054f7) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) [2021-09-03 09:04:02.503271] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-0: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}] [2021-09-03 09:04:02.503339] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-1: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}] [2021-09-03 09:04:02.503393] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-2: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}] [2021-09-03 09:04:02.504997] W [fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse: 781438412: /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp -> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache => -1 (File exists) --->UNSUCCESSFUL RENAME: [2021-09-03 09:04:02.729314] I [MSGID: 109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp (5bff6629-80b8-4d9c-8e37-abfa7691124e) (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) => /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache (4a875eac-5389-400c-9359-458e151054f7) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) [2021-09-03 09:04:02.735631] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-5: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}] [2021-09-03 09:04:02.735709] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-4: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}] [2021-09-03 09:04:02.735747] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-3: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}] [2021-09-03 09:04:02.737356] W [fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse: 781438861: /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp -> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache => -1 (File exists) --->SUCCESSFUL RENAME: [2021-09-03 09:04:02.799776] I [MSGID: 109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502c175a4.68869416.temp (82725bab-b888-4d24-b239-31b86f50425f) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) => /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache (4a875eac-5389-400c-9359-458e151054f7) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) --->UNSUCCESFULL RENAME: [2021-09-03 09:04:02.963545] I [MSGID: 109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp (b3c5435c-f79b-402f-ba00-0e78dcdb6cd2) (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) => /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache (82725bab-b888-4d24-b239-31b86f50425f) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) [2021-09-03 09:04:02.968665] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-5: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}] [2021-09-03 09:04:02.968801] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-4: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}] [2021-09-03 09:04:02.968847] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-3: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}] [2021-09-03 09:04:02.970438] W [fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse: 781439391: /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp -> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache => -1 (File exists) -- <http://www.uva.es> <http://www.uva.es>Escudo UVa <http://www.uva.es> Jose Manuel Blanco Analista Sistemas · josemanuel.bla...@uva.es <mailto:josemanuel.bla...@uva.es> Tfno.: (+34) 983 18 6411 Universidad de Valladolid Servicio de las TIC Edificio Alfonso VIII · C/ Real de Burgos s/n 47011 · Valladolid · España (/Spain/) | Facebook <https://www.facebook.com/Universidad-de-Valladolid-187763507920209/> | Twitter <https://twitter.com/uva_es> | UVa <http://www.uva.es> Este mensaje puede contener información confidencial, sometida al secreto profesional, cuya divulgación no está permitida por la ley. Si usted no es su destinatario, por favor, notifíquelo al remitente y borre este correo de su sistema. A los efectos de la protección de datos y el RGPD, consulte: protección de datos en la UVa <http://www.uva.es/protecciondedatos/>. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros. Piensa en verde: lee en la pantalla. This message may contain confidential information covered by the obligation of professional secrecy, the disclosure of which would be contrary to the law. If you are not the intended recipient, please advise the sender and delete this e-mail from your system. For the purposes of data protection and GDPR, read: protección de datos en la UVa <http://www.uva.es/protecciondedatos/>. The sender does not guarantee the integrity, the accuracy, the swift delivery or the security of this email transmission, and assumes no responsibility for any possible damage incurred through data capture, virus incorporation or any manipulation carried out by third parties. Think green: read on the screen.
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users