Re: [Gluster-users] [External] Re: A broken file that can not be deleted

[Dmitry Isakbayev]

Nithyya, Raghavendra, Davide

Thank you for your help.


*>how the permissions are displayed for the file on the servers?*
Trying to answer this question is what fixed the problem.  It looked just
fine on all 3 servers.  And it looks like running "ls" on the servers fixed
it on the clients.  I had to repeat it on all 3 servers.
It fixed file permissions on 2 clients and made the file show up on the 3rd
client.

Even though it fixed the file permissions and I could now view contents of
the file, the software was still having issues with renaming
".download_suspensions.memo.writing"  to  ".download_suspensions.memo"
When I tried to replace the file manually, I got
$ mv .download_suspensions.memo.writing .download_suspensions.memo
mv: ‘.download_suspensions.memo.writing’ and ‘.download_suspensions.memo’
are the same file

I ended up removing both files and having the software rebuild them.

>
*Wondering whether its a case of split brain.*Very possible.  All 3 servers
were rebooted.  It brought down linux cluster running on the same 3 servers
as well.



> On Thu, Jan 10, 2019 at 10:00 AM Raghavendra Gowdappa 
> wrote:
>
>>
>>
>> On Wed, Jan 9, 2019 at 7:48 PM Dmitry Isakbayev 
>> wrote:
>>
>>> I am seeing a broken file that exists on 2 out of 3 nodes.
>>>
>>
>> Wondering whether its a case of split brain.
>>
>>
>>> The application trying to use the file throws file permissions error.
>>> ls, rm, mv, touch all throw "Input/output error"
>>>
>>> $ ls -la
>>> ls: cannot access .download_suspensions.memo: Input/output error
>>> drwxrwxr-x. 2 ossadmin ossadmin  4096 Jan  9 08:06 .
>>> drwxrwxr-x. 5 ossadmin ossadmin  4096 Jan  3 11:36 ..
>>> -?? ? ????
>>> .download_suspensions.memo
>>>
>>> $ rm ".download_suspensions.memo"
>>> rm: cannot remove ‘.download_suspensions.memo’: Input/output error
>>>
>>>
>>>
>>>
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> --
> Davide Obbi
> Senior System Administrator
>
> Booking.com B.V.
> Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
> Direct +31207031558
> [image: Booking.com] 
> Empowering people to experience the world since 1996
> 43 languages, 214+ offices worldwide, 141,000+ global destinations, 29
> million reported listings
> Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [External] Re: A broken file that can not be deleted

[Davide Obbi]

does selfheal reports anything?
did you re-mount on the client side?
how the permissions are displayed for the file on the servers?

On Thu, Jan 10, 2019 at 10:00 AM Raghavendra Gowdappa 
wrote:

>
>
> On Wed, Jan 9, 2019 at 7:48 PM Dmitry Isakbayev  wrote:
>
>> I am seeing a broken file that exists on 2 out of 3 nodes.
>>
>
> Wondering whether its a case of split brain.
>
>
>> The application trying to use the file throws file permissions error.
>> ls, rm, mv, touch all throw "Input/output error"
>>
>> $ ls -la
>> ls: cannot access .download_suspensions.memo: Input/output error
>> drwxrwxr-x. 2 ossadmin ossadmin  4096 Jan  9 08:06 .
>> drwxrwxr-x. 5 ossadmin ossadmin  4096 Jan  3 11:36 ..
>> -?? ? ????
>> .download_suspensions.memo
>>
>> $ rm ".download_suspensions.memo"
>> rm: cannot remove ‘.download_suspensions.memo’: Input/output error
>>
>>
>>
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users



-- 
Davide Obbi
Senior System Administrator

Booking.com B.V.
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
Direct +31207031558
[image: Booking.com] 
Empowering people to experience the world since 1996
43 languages, 214+ offices worldwide, 141,000+ global destinations, 29
million reported listings
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users