Nithyya, Raghavendra, Davide Thank you for your help.
*>how the permissions are displayed for the file on the servers?* Trying to answer this question is what fixed the problem. It looked just fine on all 3 servers. And it looks like running "ls" on the servers fixed it on the clients. I had to repeat it on all 3 servers. It fixed file permissions on 2 clients and made the file show up on the 3rd client. Even though it fixed the file permissions and I could now view contents of the file, the software was still having issues with renaming ".download_suspensions.memo.writing" to ".download_suspensions.memo" When I tried to replace the file manually, I got $ mv .download_suspensions.memo.writing .download_suspensions.memo mv: ‘.download_suspensions.memo.writing’ and ‘.download_suspensions.memo’ are the same file I ended up removing both files and having the software rebuild them. > *Wondering whether its a case of split brain.*Very possible. All 3 servers were rebooted. It brought down linux cluster running on the same 3 servers as well. > On Thu, Jan 10, 2019 at 10:00 AM Raghavendra Gowdappa <rgowd...@redhat.com> > wrote: > >> >> >> On Wed, Jan 9, 2019 at 7:48 PM Dmitry Isakbayev <isak...@gmail.com> >> wrote: >> >>> I am seeing a broken file that exists on 2 out of 3 nodes. >>> >> >> Wondering whether its a case of split brain. >> >> >>> The application trying to use the file throws file permissions error. >>> ls, rm, mv, touch all throw "Input/output error" >>> >>> $ ls -la >>> ls: cannot access .download_suspensions.memo: Input/output error >>> drwxrwxr-x. 2 ossadmin ossadmin 4096 Jan 9 08:06 . >>> drwxrwxr-x. 5 ossadmin ossadmin 4096 Jan 3 11:36 .. >>> -?????????? ? ? ? ? ? >>> .download_suspensions.memo >>> >>> $ rm ".download_suspensions.memo" >>> rm: cannot remove ‘.download_suspensions.memo’: Input/output error >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users@gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users@gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > > > -- > Davide Obbi > Senior System Administrator > > Booking.com B.V. > Vijzelstraat 66-80 Amsterdam 1017HL Netherlands > Direct +31207031558 > [image: Booking.com] <https://www.booking.com/> > Empowering people to experience the world since 1996 > 43 languages, 214+ offices worldwide, 141,000+ global destinations, 29 > million reported listings > Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG) >
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users