[Gluster-devel] Patch backport request

[Serkan Çoban]

Hi,

I see that this (1) patch not back ported to 3.10. Am I correct?
Is there any plans to back port it?

1- https://review.gluster.org/16468
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] glfsheal-v0.log Too many open files

[Serkan Çoban]

Hi,

When I run gluster v heal v0 info, it gives "v0: Not able to fetch
volfile from glusterd" error message.
I see too many open files errors in glfsheal-v0.log file. How can I
increase open file limit for glfsheal?
I already increased nfile limit in /etc/init.d/glusterd and
/etc/init.d/gluserfsd but it did not help.

Any suggestions?
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Brick count limit in a volume

[Serkan Çoban]

Hi, I think this is the line limiting brick count:
https://github.com/gluster/glusterfs/blob/c136024613c697fec87aaff3a070862b92c57977/cli/src/cli-cmd-parser.c#L84

Can gluster-devs increase this limit? Should I open a github issue?

On Mon, Aug 21, 2017 at 7:01 PM, Serkan Çoban <cobanser...@gmail.com> wrote:
> Hi,
> Gluster version is 3.10.5. I am trying to create a 5500 brick volume,
> but getting an error stating that  bricks is the limit. Is this a
> known limit? Can I change this with an option?
>
> Thanks,
> Serkan
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Brick count limit in a volume

[Serkan Çoban]

This is the command line output:
Total brick list is larger than a request. Can take (brick_count )
Usage: volume create  [stripe ] [replica ] 

I am testing if a big single volume will work for us. Now I am
continuing testing with three volumes each 13PB...
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Announcing release 3.11 : Scope, schedule and feature tracking

[Serkan Çoban]

How this affect CPU usage? Does it read whole file and calculates a
hash after it is being written?
Will this patch land in 3.10.x?

On Tue, Apr 25, 2017 at 10:32 AM, Kotresh Hiremath Ravishankar
 wrote:
> Hi
>
> https://github.com/gluster/glusterfs/issues/188 is merged in master
> and needs to go in 3.11
>
> Thanks and Regards,
> Kotresh H R
>
> - Original Message -
>> From: "Kaushal M" 
>> To: "Shyam" 
>> Cc: gluster-us...@gluster.org, "Gluster Devel" 
>> Sent: Thursday, April 20, 2017 12:16:39 PM
>> Subject: Re: [Gluster-devel] Announcing release 3.11 : Scope, schedule and 
>> feature tracking
>>
>> On Thu, Apr 13, 2017 at 8:17 PM, Shyam  wrote:
>> > On 02/28/2017 10:17 AM, Shyam wrote:
>> >>
>> >> Hi,
>> >>
>> >> With release 3.10 shipped [1], it is time to set the dates for release
>> >> 3.11 (and subsequently 4.0).
>> >>
>> >> This mail has the following sections, so please read or revisit as needed,
>> >>   - Release 3.11 dates (the schedule)
>> >>   - 3.11 focus areas
>> >
>> >
>> > Pinging the list on the above 2 items.
>> >
>> >> *Release 3.11 dates:*
>> >> Based on our release schedule [2], 3.11 would be 3 months from the 3.10
>> >> release and would be a Short Term Maintenance (STM) release.
>> >>
>> >> This puts 3.11 schedule as (working from the release date backwards):
>> >> - Release: May 30th, 2017
>> >> - Branching: April 27th, 2017
>> >
>> >
>> > Branching is about 2 weeks away, other than the initial set of overflow
>> > features from 3.10 nothing else has been raised on the lists and in github
>> > as requests for 3.11.
>> >
>> > So, a reminder to folks who are working on features, to raise the relevant
>> > github issue for the same, and post it to devel list for consideration in
>> > 3.11 (also this helps tracking and ensuring we are waiting for the right
>> > things at the time of branching).
>> >
>> >>
>> >> *3.11 focus areas:*
>> >> As maintainers of gluster, we want to harden testing around the various
>> >> gluster features in this release. Towards this the focus area for this
>> >> release are,
>> >>
>> >> 1) Testing improvements in Gluster
>> >>   - Primary focus would be to get automated test cases to determine
>> >> release health, rather than repeating a manual exercise every 3 months
>> >>   - Further, we would also attempt to focus on maturing Glusto[7] for
>> >> this, and other needs (as much as possible)
>> >>
>> >> 2) Merge all (or as much as possible) Facebook patches into master, and
>> >> hence into release 3.11
>> >>   - Facebook has (as announced earlier [3]) started posting their
>> >> patches mainline, and this needs some attention to make it into master
>> >>
>> >
>> > Further to the above, we are also considering the following features for
>> > this release, request feature owners to let us know if these are actively
>> > being worked on and if these will make the branching dates. (calling out
>> > folks that I think are the current feature owners for the same)
>> >
>> > 1) Halo - Initial Cut (@pranith)
>> > 2) IPv6 support (@kaushal)
>>
>> This is under review at https://review.gluster.org/16228 . The patch
>> mostly looks fine.
>>
>> The only issue is that it currently depends and links with an internal
>> FB fork of tirpc (mainly for some helper functions and utilities).
>> This makes it hard for the community to make actual use of  and test,
>> the IPv6 features/fixes introduced by the change.
>>
>> If the change were refactored the use publicly available versions of
>> tirpc or ntirpc, I'm OK for it to be merged. I did try it out myself.
>> While I was able to build it against available versions of tirpc, I
>> wasn't able to get it working correctly.
>>
>> > 3) Negative lookup (@poornima)
>> > 4) Parallel Readdirp - More changes to default settings. (@poornima, @du)
>> >
>> >
>> >> [1] 3.10 release announcement:
>> >> http://lists.gluster.org/pipermail/gluster-devel/2017-February/052188.html
>> >>
>> >> [2] Gluster release schedule:
>> >> https://www.gluster.org/community/release-schedule/
>> >>
>> >> [3] Mail regarding facebook patches:
>> >> http://lists.gluster.org/pipermail/gluster-devel/2016-December/051784.html
>> >>
>> >> [4] Release scope: https://github.com/gluster/glusterfs/projects/1
>> >>
>> >> [5] glusterfs github issues: https://github.com/gluster/glusterfs/issues
>> >>
>> >> [6] github issues for features and major fixes:
>> >> https://hackmd.io/s/BkgH8sdtg#
>> >>
>> >> [7] Glusto tests: https://github.com/gluster/glusto-tests
>> >> ___
>> >> Gluster-devel mailing list
>> >> Gluster-devel@gluster.org
>> >> http://lists.gluster.org/mailman/listinfo/gluster-devel
>> >
>> > ___
>> > Gluster-devel mailing list
>> > Gluster-devel@gluster.org
>> > http://lists.gluster.org/mailman/listinfo/gluster-devel
>> 

Re: [Gluster-devel] [Gluster-users] Notice: https://download.gluster.org:/pub/gluster/glusterfs/LATEST has changed

[Serkan Çoban]

Hi,

I want to try multi threaded disperse heal in 3.9 but I have questions:
Currently whatever I do (find /bricks -exec stat... from multiple
clients) I cannot get more than 10MB/sec heal speed for one brick.
Will multi-thread heal improve this? I will do a test but I also ask you...
Secondly, if I upgrade to 3.9 and it gives problems is it safe to
downgrade to 3.7.11? Any suggestions?

On Sat, Nov 19, 2016 at 9:12 PM, Serkan Çoban <cobanser...@gmail.com> wrote:
> Hi,
>
> Sorry for late reply. I think I will wait for 3.10 LTS release to try
> it. I am on 3.7.11 and it is very stable for us.
>
> On Thu, Nov 17, 2016 at 1:05 PM, Pranith Kumar Karampuri
> <pkara...@redhat.com> wrote:
>>
>>
>> On Wed, Nov 16, 2016 at 11:47 PM, Serkan Çoban <cobanser...@gmail.com>
>> wrote:
>>>
>>> Hi,
>>> Will disperse related new futures be ported to 3.7? or we should
>>> upgrade for those features?
>>
>>
>> hi Serkan,
>>   Unfortunately, no they won't be backported to 3.7. We are adding
>> new features to latest releases to prevent accidental bugs slipping in
>> stable releases. While the features are working well, we did see a
>> performance problem very late in the cycle in the I/O path just with EC for
>> small files. You should wait before you upgrade IMO.
>>
>> You were trying to test how long it takes to heal data with multi-threaded
>> heal in EC right? Do you want to give us feedback by trying this feature
>> out?
>>
>>>
>>> On Wed, Nov 16, 2016 at 8:51 PM, Kaleb S. KEITHLEY <kkeit...@redhat.com>
>>> wrote:
>>> > Hi,
>>> >
>>> > As some of you may have noticed, GlusterFS-3.9.0 was released. Watch
>>> > this space for the official announcement soon.
>>> >
>>> > If you are using Community GlusterFS packages from download.gluster.org
>>> > you should check your package metadata to be sure that an update doesn't
>>> > inadvertently update your system to 3.9.
>>> >
>>> > There is a new symlink:
>>> > https://download.gluster.org:/pub/gluster/glusterfs/LTM-3.8 which will
>>> > remain pointed at the GlusterFS-3.8 packages. Use this instead of
>>> > .../LATEST to keep getting 3.8 updates without risk of accidentally
>>> > getting 3.9. There is also a new LTM-3.7 symlink that you can use for
>>> > 3.7 updates.
>>> >
>>> > Also note that there is a new package signing key for the 3.9 packages
>>> > that are on download.gluster.org. The old key remains the same for 3.8
>>> > and earlier packages. New releases of 3.8 and 3.7 packages will continue
>>> > to use the old key.
>>> >
>>> > GlusterFS-3.9 is the first "short term" release; it will be supported
>>> > for approximately six months. 3.7 and 3.8 are Long Term Maintenance
>>> > (LTM) releases. 3.9 will be followed by 3.10; 3.10 will be a LTM release
>>> > and 3.9 and 3.7 will be End-of-Life (EOL) at that time.
>>> >
>>> >
>>> > --
>>> >
>>> > Kaleb
>>> > ___
>>> > Gluster-users mailing list
>>> > gluster-us...@gluster.org
>>> > http://www.gluster.org/mailman/listinfo/gluster-users
>>> ___
>>> Gluster-users mailing list
>>> gluster-us...@gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>>
>> --
>> Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Notice: https://download.gluster.org:/pub/gluster/glusterfs/LATEST has changed

[Serkan Çoban]

Hi,

Sorry for late reply. I think I will wait for 3.10 LTS release to try
it. I am on 3.7.11 and it is very stable for us.

On Thu, Nov 17, 2016 at 1:05 PM, Pranith Kumar Karampuri
<pkara...@redhat.com> wrote:
>
>
> On Wed, Nov 16, 2016 at 11:47 PM, Serkan Çoban <cobanser...@gmail.com>
> wrote:
>>
>> Hi,
>> Will disperse related new futures be ported to 3.7? or we should
>> upgrade for those features?
>
>
> hi Serkan,
>   Unfortunately, no they won't be backported to 3.7. We are adding
> new features to latest releases to prevent accidental bugs slipping in
> stable releases. While the features are working well, we did see a
> performance problem very late in the cycle in the I/O path just with EC for
> small files. You should wait before you upgrade IMO.
>
> You were trying to test how long it takes to heal data with multi-threaded
> heal in EC right? Do you want to give us feedback by trying this feature
> out?
>
>>
>> On Wed, Nov 16, 2016 at 8:51 PM, Kaleb S. KEITHLEY <kkeit...@redhat.com>
>> wrote:
>> > Hi,
>> >
>> > As some of you may have noticed, GlusterFS-3.9.0 was released. Watch
>> > this space for the official announcement soon.
>> >
>> > If you are using Community GlusterFS packages from download.gluster.org
>> > you should check your package metadata to be sure that an update doesn't
>> > inadvertently update your system to 3.9.
>> >
>> > There is a new symlink:
>> > https://download.gluster.org:/pub/gluster/glusterfs/LTM-3.8 which will
>> > remain pointed at the GlusterFS-3.8 packages. Use this instead of
>> > .../LATEST to keep getting 3.8 updates without risk of accidentally
>> > getting 3.9. There is also a new LTM-3.7 symlink that you can use for
>> > 3.7 updates.
>> >
>> > Also note that there is a new package signing key for the 3.9 packages
>> > that are on download.gluster.org. The old key remains the same for 3.8
>> > and earlier packages. New releases of 3.8 and 3.7 packages will continue
>> > to use the old key.
>> >
>> > GlusterFS-3.9 is the first "short term" release; it will be supported
>> > for approximately six months. 3.7 and 3.8 are Long Term Maintenance
>> > (LTM) releases. 3.9 will be followed by 3.10; 3.10 will be a LTM release
>> > and 3.9 and 3.7 will be End-of-Life (EOL) at that time.
>> >
>> >
>> > --
>> >
>> > Kaleb
>> > ___
>> > Gluster-users mailing list
>> > gluster-us...@gluster.org
>> > http://www.gluster.org/mailman/listinfo/gluster-users
>> ___
>> Gluster-users mailing list
>> gluster-us...@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> --
> Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] A question of GlusterFS dentries!

[Serkan Çoban]

+1 for "no-rewinddir-support" option in DHT.
We are seeing very slow directory listing specially with 1500+ brick
volume, 'ls' takes 20+ second with 1000+ files.

On Wed, Nov 2, 2016 at 7:08 AM, Raghavendra Gowdappa
 wrote:
>
>
> - Original Message -
>> From: "Keiviw" 
>> To: gluster-devel@gluster.org
>> Sent: Tuesday, November 1, 2016 12:41:02 PM
>> Subject: [Gluster-devel] A question of GlusterFS dentries!
>>
>> Hi,
>> In GlusterFS distributed volumes, listing a non-empty directory was slow.
>> Then I read the dht codes and found the reasons. But I was confused that
>> GlusterFS dht travesed all the bricks(in the volume) sequentially,why not
>> use multi-thread to read dentries from multiple bricks simultaneously.
>> That's a question that's always puzzled me, Couly you please tell me
>> something about this???
>
> readdir across subvols is sequential mostly because we have to support 
> rewinddir(3). We need to maintain the mapping of offset and dentry across 
> multiple invocations of readdir. In other words if someone did a rewinddir to 
> an offset corresponding to earlier dentry, subsequent readdirs should return 
> same set of dentries what the earlier invocation of readdir returned. For 
> example, in an hypothetical scenario, readdir returned following dentries:
>
> 1. a, off=10
> 2. b, off=2
> 3. c, off=5
> 4. d, off=15
> 5. e, off=17
> 6. f, off=13
>
> Now if we did rewinddir to off 5 and issue readdir again we should get 
> following dentries:
> (c, off=5), (d, off=15), (e, off=17), (f, off=13)
>
> Within a subvol backend filesystem provides rewinddir guarantee for the 
> dentries present on that subvol. However, across subvols it is the 
> responsibility of DHT to provide the above guarantee. Which means we 
> should've some well defined order in which we send readdir calls (Note that 
> order is not well defined if we do a parallel readdir across all subvols). 
> So, DHT has sequential readdir which is a well defined order of reading 
> dentries.
>
> To give an example if we have another subvol - subvol2 - (in addiction to the 
> subvol above - say subvol1) with following listing:
> 1. g, off=16
> 2. h, off=20
> 3. i, off=3
> 4. j, off=19
>
> With parallel readdir we can have many ordering like - (a, b, g, h, i, c, d, 
> e, f, j), (g, h, a, b, c, i, j, d, e, f) etc. Now if we do (with readdir done 
> parallely):
>
> 1. A complete listing of the directory (which can be any one of 10P1 = 10 
> ways - I hope math is correct here).
> 2. Do rewinddir (20)
>
> We cannot predict what are the set of dentries that come _after_ offset 20. 
> However, if we do a readdir sequentially across subvols there is only one 
> directory listing i.e, (a, b, c, d, e, f, g, h, i, j). So, its easier to 
> support rewinddir.
>
> If there is no POSIX requirement for rewinddir support, I think a parallel 
> readdir can easily be implemented (which improves performance too). But 
> unfortunately rewinddir is still a POSIX requirement. This also opens up 
> another possibility of a "no-rewinddir-support" option in DHT, which if 
> enabled results in parallel readdirs across subvols. What I am not sure is 
> how many users still use rewinddir? If there is a critical mass which wants 
> performance with a tradeoff of no rewinddir support this can be a good 
> feature.
>
> +gluster-users to get an opinion on this.
>
> regards,
> Raghavendra
>
>>
>>
>>
>>
>>
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-devel
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] GlusterFS-3.7.14 released

[Serkan Çoban]

I can wait for the patch to complete, please inform me when you ready.
If it will take too much time to solve the crawl issue I can test
without it too...

Serkan

On Thu, Aug 11, 2016 at 5:52 AM, Pranith Kumar Karampuri
<pkara...@redhat.com> wrote:
>
>
> On Wed, Aug 10, 2016 at 1:58 PM, Serkan Çoban <cobanser...@gmail.com> wrote:
>>
>> Hi,
>>
>> Any progress about the patch?
>
>
> hi Serkan,
>While testing the patch by myself, I am seeing that it is taking more
> than one crawl to complete heals even when there are no  directory
> hierarchies. It is faster than before but it shouldn't take more than 1
> crawl to complete the heal because all the files exist already. I am
> investigating why that is the case now. If you want to test things out
> without this patch I will give you rpms today. Otherwise we need to find
> until we find RCA for this crawl problem. Let me know your decision. If you
> are okay with testing progressive versions of this feature, that would be
> great. We can compare how each patch improved the performance.
>
> Pranith
>
>>
>>
>> On Thu, Aug 4, 2016 at 10:16 AM, Pranith Kumar Karampuri
>> <pkara...@redhat.com> wrote:
>> >
>> >
>> > On Thu, Aug 4, 2016 at 11:30 AM, Serkan Çoban <cobanser...@gmail.com>
>> > wrote:
>> >>
>> >> Thanks Pranith,
>> >> I am waiting for RPMs to show, I will do the tests as soon as possible
>> >> and inform you.
>> >
>> >
>> > I guess on 3.7.x the RPMs are not automatically built. Let me find how
>> > it
>> > can be done. I will inform you after finding that out. Give me a day.
>> >
>> >>
>> >>
>> >> On Wed, Aug 3, 2016 at 11:19 PM, Pranith Kumar Karampuri
>> >> <pkara...@redhat.com> wrote:
>> >> >
>> >> >
>> >> > On Thu, Aug 4, 2016 at 1:47 AM, Pranith Kumar Karampuri
>> >> > <pkara...@redhat.com> wrote:
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Thu, Aug 4, 2016 at 12:51 AM, Serkan Çoban
>> >> >> <cobanser...@gmail.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> I use rpms for installation. Redhat/Centos 6.8.
>> >> >>
>> >> >>
>> >> >> http://review.gluster.org/#/c/15084 is the patch. In some time the
>> >> >> rpms
>> >> >> will be built actually.
>> >> >
>> >> >
>> >> > In the same URL above it will actually post the rpms for
>> >> > fedora/el6/el7
>> >> > at
>> >> > the end of the page.
>> >> >
>> >> >>
>> >> >>
>> >> >> Use gluster volume set  disperse.shd-max-threads
>> >> >> > >> >> (range: 1-64)>
>> >> >>
>> >> >> While testing this I thought of ways to decrease the number of
>> >> >> crawls
>> >> >> as
>> >> >> well. But they are a bit involved. Try to create same set of data
>> >> >> and
>> >> >> see
>> >> >> what is the time it takes to complete heals using number of threads
>> >> >> as
>> >> >> you
>> >> >> increase the number of parallel heals from 1 to 64.
>> >> >>
>> >> >>>
>> >> >>> On Wed, Aug 3, 2016 at 10:16 PM, Pranith Kumar Karampuri
>> >> >>> <pkara...@redhat.com> wrote:
>> >> >>> >
>> >> >>> >
>> >> >>> > On Thu, Aug 4, 2016 at 12:45 AM, Serkan Çoban
>> >> >>> > <cobanser...@gmail.com>
>> >> >>> > wrote:
>> >> >>> >>
>> >> >>> >> I prefer 3.7 if it is ok for you. Can you also provide build
>> >> >>> >> instructions?
>> >> >>> >
>> >> >>> >
>> >> >>> > 3.7 should be fine. Do you use rpms/debs/anything-else?
>> >> >>> >
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> On Wed, Aug 3, 2016 at 10:12 PM, Pranith Kumar Karampuri
>> >> >>> >> <pkara...@redhat.com> wrote:
>> >> >>> >> >
>> >> >>>

Re: [Gluster-devel] [Gluster-users] GlusterFS-3.7.14 released

[Serkan Çoban]

Thanks Pranith,
I am waiting for RPMs to show, I will do the tests as soon as possible
and inform you.

On Wed, Aug 3, 2016 at 11:19 PM, Pranith Kumar Karampuri
<pkara...@redhat.com> wrote:
>
>
> On Thu, Aug 4, 2016 at 1:47 AM, Pranith Kumar Karampuri
> <pkara...@redhat.com> wrote:
>>
>>
>>
>> On Thu, Aug 4, 2016 at 12:51 AM, Serkan Çoban <cobanser...@gmail.com>
>> wrote:
>>>
>>> I use rpms for installation. Redhat/Centos 6.8.
>>
>>
>> http://review.gluster.org/#/c/15084 is the patch. In some time the rpms
>> will be built actually.
>
>
> In the same URL above it will actually post the rpms for fedora/el6/el7 at
> the end of the page.
>
>>
>>
>> Use gluster volume set  disperse.shd-max-threads > (range: 1-64)>
>>
>> While testing this I thought of ways to decrease the number of crawls as
>> well. But they are a bit involved. Try to create same set of data and see
>> what is the time it takes to complete heals using number of threads as you
>> increase the number of parallel heals from 1 to 64.
>>
>>>
>>> On Wed, Aug 3, 2016 at 10:16 PM, Pranith Kumar Karampuri
>>> <pkara...@redhat.com> wrote:
>>> >
>>> >
>>> > On Thu, Aug 4, 2016 at 12:45 AM, Serkan Çoban <cobanser...@gmail.com>
>>> > wrote:
>>> >>
>>> >> I prefer 3.7 if it is ok for you. Can you also provide build
>>> >> instructions?
>>> >
>>> >
>>> > 3.7 should be fine. Do you use rpms/debs/anything-else?
>>> >
>>> >>
>>> >>
>>> >> On Wed, Aug 3, 2016 at 10:12 PM, Pranith Kumar Karampuri
>>> >> <pkara...@redhat.com> wrote:
>>> >> >
>>> >> >
>>> >> > On Thu, Aug 4, 2016 at 12:37 AM, Serkan Çoban
>>> >> > <cobanser...@gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> Yes, but I can create 2+1(or 8+2) ec using two servers right? I
>>> >> >> have
>>> >> >> 26 disks on each server.
>>> >> >
>>> >> >
>>> >> > On which release-branch do you want the patch? I am testing it on
>>> >> > master-branch now.
>>> >> >
>>> >> >>
>>> >> >>
>>> >> >> On Wed, Aug 3, 2016 at 9:59 PM, Pranith Kumar Karampuri
>>> >> >> <pkara...@redhat.com> wrote:
>>> >> >> >
>>> >> >> >
>>> >> >> > On Thu, Aug 4, 2016 at 12:23 AM, Serkan Çoban
>>> >> >> > <cobanser...@gmail.com>
>>> >> >> > wrote:
>>> >> >> >>
>>> >> >> >> I have two of my storage servers free, I think I can use them
>>> >> >> >> for
>>> >> >> >> testing. Is two server testing environment ok for you?
>>> >> >> >
>>> >> >> >
>>> >> >> > I think it would be better if you have at least 3. You can test
>>> >> >> > it
>>> >> >> > with
>>> >> >> > 2+1
>>> >> >> > ec configuration.
>>> >> >> >
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> On Wed, Aug 3, 2016 at 9:44 PM, Pranith Kumar Karampuri
>>> >> >> >> <pkara...@redhat.com> wrote:
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > On Wed, Aug 3, 2016 at 6:01 PM, Serkan Çoban
>>> >> >> >> > <cobanser...@gmail.com>
>>> >> >> >> > wrote:
>>> >> >> >> >>
>>> >> >> >> >> Hi,
>>> >> >> >> >>
>>> >> >> >> >> May I ask if multi-threaded self heal for distributed
>>> >> >> >> >> disperse
>>> >> >> >> >> volumes
>>> >> >> >> >> implemented in this release?
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > Serkan,
>>> >> >> >> > At the moment I am a bit busy with different work, Is
>>> >

Re: [Gluster-devel] [Gluster-users] GlusterFS-3.7.14 released

[Serkan Çoban]

I prefer 3.7 if it is ok for you. Can you also provide build instructions?

On Wed, Aug 3, 2016 at 10:12 PM, Pranith Kumar Karampuri
<pkara...@redhat.com> wrote:
>
>
> On Thu, Aug 4, 2016 at 12:37 AM, Serkan Çoban <cobanser...@gmail.com> wrote:
>>
>> Yes, but I can create 2+1(or 8+2) ec using two servers right? I have
>> 26 disks on each server.
>
>
> On which release-branch do you want the patch? I am testing it on
> master-branch now.
>
>>
>>
>> On Wed, Aug 3, 2016 at 9:59 PM, Pranith Kumar Karampuri
>> <pkara...@redhat.com> wrote:
>> >
>> >
>> > On Thu, Aug 4, 2016 at 12:23 AM, Serkan Çoban <cobanser...@gmail.com>
>> > wrote:
>> >>
>> >> I have two of my storage servers free, I think I can use them for
>> >> testing. Is two server testing environment ok for you?
>> >
>> >
>> > I think it would be better if you have at least 3. You can test it with
>> > 2+1
>> > ec configuration.
>> >
>> >>
>> >>
>> >> On Wed, Aug 3, 2016 at 9:44 PM, Pranith Kumar Karampuri
>> >> <pkara...@redhat.com> wrote:
>> >> >
>> >> >
>> >> > On Wed, Aug 3, 2016 at 6:01 PM, Serkan Çoban <cobanser...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> May I ask if multi-threaded self heal for distributed disperse
>> >> >> volumes
>> >> >> implemented in this release?
>> >> >
>> >> >
>> >> > Serkan,
>> >> > At the moment I am a bit busy with different work, Is it
>> >> > possible
>> >> > for you to help test the feature if I provide a patch? Actually the
>> >> > patch
>> >> > should be small. Testing is where lot of time will be spent on.
>> >> >
>> >> >>
>> >> >>
>> >> >> Thanks,
>> >> >> Serkan
>> >> >>
>> >> >> On Tue, Aug 2, 2016 at 5:30 PM, David Gossage
>> >> >> <dgoss...@carouselchecks.com> wrote:
>> >> >> > On Tue, Aug 2, 2016 at 6:01 AM, Lindsay Mathieson
>> >> >> > <lindsay.mathie...@gmail.com> wrote:
>> >> >> >>
>> >> >> >> On 2/08/2016 5:07 PM, Kaushal M wrote:
>> >> >> >>>
>> >> >> >>> GlusterFS-3.7.14 has been released. This is a regular minor
>> >> >> >>> release.
>> >> >> >>> The release-notes are available at
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> https://github.com/gluster/glusterfs/blob/release-3.7/doc/release-notes/3.7.14.md
>> >> >> >>
>> >> >> >>
>> >> >> >> Thanks Kaushal, I'll check it out
>> >> >> >>
>> >> >> >
>> >> >> > So far on my test box its working as expected.  At least the
>> >> >> > issues
>> >> >> > that
>> >> >> > prevented it from running as before have disappeared.  Will need
>> >> >> > to
>> >> >> > see
>> >> >> > how
>> >> >> > my test VM behaves after a few days.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >> --
>> >> >> >> Lindsay Mathieson
>> >> >> >>
>> >> >> >> ___
>> >> >> >> Gluster-users mailing list
>> >> >> >> gluster-us...@gluster.org
>> >> >> >> http://www.gluster.org/mailman/listinfo/gluster-users
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > ___
>> >> >> > Gluster-users mailing list
>> >> >> > gluster-us...@gluster.org
>> >> >> > http://www.gluster.org/mailman/listinfo/gluster-users
>> >> >> ___
>> >> >> Gluster-users mailing list
>> >> >> gluster-us...@gluster.org
>> >> >> http://www.gluster.org/mailman/listinfo/gluster-users
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Pranith
>> >
>> >
>> >
>> >
>> > --
>> > Pranith
>
>
>
>
> --
> Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] GlusterFS-3.7.14 released

[Serkan Çoban]

I use rpms for installation. Redhat/Centos 6.8.

On Wed, Aug 3, 2016 at 10:16 PM, Pranith Kumar Karampuri
<pkara...@redhat.com> wrote:
>
>
> On Thu, Aug 4, 2016 at 12:45 AM, Serkan Çoban <cobanser...@gmail.com> wrote:
>>
>> I prefer 3.7 if it is ok for you. Can you also provide build instructions?
>
>
> 3.7 should be fine. Do you use rpms/debs/anything-else?
>
>>
>>
>> On Wed, Aug 3, 2016 at 10:12 PM, Pranith Kumar Karampuri
>> <pkara...@redhat.com> wrote:
>> >
>> >
>> > On Thu, Aug 4, 2016 at 12:37 AM, Serkan Çoban <cobanser...@gmail.com>
>> > wrote:
>> >>
>> >> Yes, but I can create 2+1(or 8+2) ec using two servers right? I have
>> >> 26 disks on each server.
>> >
>> >
>> > On which release-branch do you want the patch? I am testing it on
>> > master-branch now.
>> >
>> >>
>> >>
>> >> On Wed, Aug 3, 2016 at 9:59 PM, Pranith Kumar Karampuri
>> >> <pkara...@redhat.com> wrote:
>> >> >
>> >> >
>> >> > On Thu, Aug 4, 2016 at 12:23 AM, Serkan Çoban <cobanser...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> I have two of my storage servers free, I think I can use them for
>> >> >> testing. Is two server testing environment ok for you?
>> >> >
>> >> >
>> >> > I think it would be better if you have at least 3. You can test it
>> >> > with
>> >> > 2+1
>> >> > ec configuration.
>> >> >
>> >> >>
>> >> >>
>> >> >> On Wed, Aug 3, 2016 at 9:44 PM, Pranith Kumar Karampuri
>> >> >> <pkara...@redhat.com> wrote:
>> >> >> >
>> >> >> >
>> >> >> > On Wed, Aug 3, 2016 at 6:01 PM, Serkan Çoban
>> >> >> > <cobanser...@gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Hi,
>> >> >> >>
>> >> >> >> May I ask if multi-threaded self heal for distributed disperse
>> >> >> >> volumes
>> >> >> >> implemented in this release?
>> >> >> >
>> >> >> >
>> >> >> > Serkan,
>> >> >> > At the moment I am a bit busy with different work, Is it
>> >> >> > possible
>> >> >> > for you to help test the feature if I provide a patch? Actually
>> >> >> > the
>> >> >> > patch
>> >> >> > should be small. Testing is where lot of time will be spent on.
>> >> >> >
>> >> >> >>
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> Serkan
>> >> >> >>
>> >> >> >> On Tue, Aug 2, 2016 at 5:30 PM, David Gossage
>> >> >> >> <dgoss...@carouselchecks.com> wrote:
>> >> >> >> > On Tue, Aug 2, 2016 at 6:01 AM, Lindsay Mathieson
>> >> >> >> > <lindsay.mathie...@gmail.com> wrote:
>> >> >> >> >>
>> >> >> >> >> On 2/08/2016 5:07 PM, Kaushal M wrote:
>> >> >> >> >>>
>> >> >> >> >>> GlusterFS-3.7.14 has been released. This is a regular minor
>> >> >> >> >>> release.
>> >> >> >> >>> The release-notes are available at
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >> >>> https://github.com/gluster/glusterfs/blob/release-3.7/doc/release-notes/3.7.14.md
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> Thanks Kaushal, I'll check it out
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > So far on my test box its working as expected.  At least the
>> >> >> >> > issues
>> >> >> >> > that
>> >> >> >> > prevented it from running as before have disappeared.  Will
>> >> >> >> > need
>> >> >> >> > to
>> >> >> >> > see
>> >> >> >> > how
>> >> >> >> > my test VM behaves after a few days.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >> --
>> >> >> >> >> Lindsay Mathieson
>> >> >> >> >>
>> >> >> >> >> ___
>> >> >> >> >> Gluster-users mailing list
>> >> >> >> >> gluster-us...@gluster.org
>> >> >> >> >> http://www.gluster.org/mailman/listinfo/gluster-users
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > ___
>> >> >> >> > Gluster-users mailing list
>> >> >> >> > gluster-us...@gluster.org
>> >> >> >> > http://www.gluster.org/mailman/listinfo/gluster-users
>> >> >> >> ___
>> >> >> >> Gluster-users mailing list
>> >> >> >> gluster-us...@gluster.org
>> >> >> >> http://www.gluster.org/mailman/listinfo/gluster-users
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Pranith
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Pranith
>> >
>> >
>> >
>> >
>> > --
>> > Pranith
>
>
>
>
> --
> Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] GlusterFS-3.7.14 released

[Serkan Çoban]

I have two of my storage servers free, I think I can use them for
testing. Is two server testing environment ok for you?

On Wed, Aug 3, 2016 at 9:44 PM, Pranith Kumar Karampuri
<pkara...@redhat.com> wrote:
>
>
> On Wed, Aug 3, 2016 at 6:01 PM, Serkan Çoban <cobanser...@gmail.com> wrote:
>>
>> Hi,
>>
>> May I ask if multi-threaded self heal for distributed disperse volumes
>> implemented in this release?
>
>
> Serkan,
> At the moment I am a bit busy with different work, Is it possible
> for you to help test the feature if I provide a patch? Actually the patch
> should be small. Testing is where lot of time will be spent on.
>
>>
>>
>> Thanks,
>> Serkan
>>
>> On Tue, Aug 2, 2016 at 5:30 PM, David Gossage
>> <dgoss...@carouselchecks.com> wrote:
>> > On Tue, Aug 2, 2016 at 6:01 AM, Lindsay Mathieson
>> > <lindsay.mathie...@gmail.com> wrote:
>> >>
>> >> On 2/08/2016 5:07 PM, Kaushal M wrote:
>> >>>
>> >>> GlusterFS-3.7.14 has been released. This is a regular minor release.
>> >>> The release-notes are available at
>> >>>
>> >>>
>> >>> https://github.com/gluster/glusterfs/blob/release-3.7/doc/release-notes/3.7.14.md
>> >>
>> >>
>> >> Thanks Kaushal, I'll check it out
>> >>
>> >
>> > So far on my test box its working as expected.  At least the issues that
>> > prevented it from running as before have disappeared.  Will need to see
>> > how
>> > my test VM behaves after a few days.
>> >
>> >
>> >
>> >> --
>> >> Lindsay Mathieson
>> >>
>> >> ___
>> >> Gluster-users mailing list
>> >> gluster-us...@gluster.org
>> >> http://www.gluster.org/mailman/listinfo/gluster-users
>> >
>> >
>> >
>> > ___
>> > Gluster-users mailing list
>> > gluster-us...@gluster.org
>> > http://www.gluster.org/mailman/listinfo/gluster-users
>> ___
>> Gluster-users mailing list
>> gluster-us...@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> --
> Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] GlusterFS-3.7.14 released

[Serkan Çoban]

Yes, but I can create 2+1(or 8+2) ec using two servers right? I have
26 disks on each server.

On Wed, Aug 3, 2016 at 9:59 PM, Pranith Kumar Karampuri
<pkara...@redhat.com> wrote:
>
>
> On Thu, Aug 4, 2016 at 12:23 AM, Serkan Çoban <cobanser...@gmail.com> wrote:
>>
>> I have two of my storage servers free, I think I can use them for
>> testing. Is two server testing environment ok for you?
>
>
> I think it would be better if you have at least 3. You can test it with 2+1
> ec configuration.
>
>>
>>
>> On Wed, Aug 3, 2016 at 9:44 PM, Pranith Kumar Karampuri
>> <pkara...@redhat.com> wrote:
>> >
>> >
>> > On Wed, Aug 3, 2016 at 6:01 PM, Serkan Çoban <cobanser...@gmail.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> May I ask if multi-threaded self heal for distributed disperse volumes
>> >> implemented in this release?
>> >
>> >
>> > Serkan,
>> > At the moment I am a bit busy with different work, Is it
>> > possible
>> > for you to help test the feature if I provide a patch? Actually the
>> > patch
>> > should be small. Testing is where lot of time will be spent on.
>> >
>> >>
>> >>
>> >> Thanks,
>> >> Serkan
>> >>
>> >> On Tue, Aug 2, 2016 at 5:30 PM, David Gossage
>> >> <dgoss...@carouselchecks.com> wrote:
>> >> > On Tue, Aug 2, 2016 at 6:01 AM, Lindsay Mathieson
>> >> > <lindsay.mathie...@gmail.com> wrote:
>> >> >>
>> >> >> On 2/08/2016 5:07 PM, Kaushal M wrote:
>> >> >>>
>> >> >>> GlusterFS-3.7.14 has been released. This is a regular minor
>> >> >>> release.
>> >> >>> The release-notes are available at
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> https://github.com/gluster/glusterfs/blob/release-3.7/doc/release-notes/3.7.14.md
>> >> >>
>> >> >>
>> >> >> Thanks Kaushal, I'll check it out
>> >> >>
>> >> >
>> >> > So far on my test box its working as expected.  At least the issues
>> >> > that
>> >> > prevented it from running as before have disappeared.  Will need to
>> >> > see
>> >> > how
>> >> > my test VM behaves after a few days.
>> >> >
>> >> >
>> >> >
>> >> >> --
>> >> >> Lindsay Mathieson
>> >> >>
>> >> >> ___
>> >> >> Gluster-users mailing list
>> >> >> gluster-us...@gluster.org
>> >> >> http://www.gluster.org/mailman/listinfo/gluster-users
>> >> >
>> >> >
>> >> >
>> >> > ___
>> >> > Gluster-users mailing list
>> >> > gluster-us...@gluster.org
>> >> > http://www.gluster.org/mailman/listinfo/gluster-users
>> >> ___
>> >> Gluster-users mailing list
>> >> gluster-us...@gluster.org
>> >> http://www.gluster.org/mailman/listinfo/gluster-users
>> >
>> >
>> >
>> >
>> > --
>> > Pranith
>
>
>
>
> --
> Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] GlusterFS-3.7.14 released

[Serkan Çoban]

Hi,

May I ask if multi-threaded self heal for distributed disperse volumes
implemented in this release?

Thanks,
Serkan

On Tue, Aug 2, 2016 at 5:30 PM, David Gossage
 wrote:
> On Tue, Aug 2, 2016 at 6:01 AM, Lindsay Mathieson
>  wrote:
>>
>> On 2/08/2016 5:07 PM, Kaushal M wrote:
>>>
>>> GlusterFS-3.7.14 has been released. This is a regular minor release.
>>> The release-notes are available at
>>>
>>> https://github.com/gluster/glusterfs/blob/release-3.7/doc/release-notes/3.7.14.md
>>
>>
>> Thanks Kaushal, I'll check it out
>>
>
> So far on my test box its working as expected.  At least the issues that
> prevented it from running as before have disappeared.  Will need to see how
> my test VM behaves after a few days.
>
>
>
>> --
>> Lindsay Mathieson
>>
>> ___
>> Gluster-users mailing list
>> gluster-us...@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Fwd: dht_is_subvol_filled messages on client

[Serkan Çoban]

Hi,

You can find the output below link:
https://www.dropbox.com/s/wzrh5yp494ogksc/status_detail.txt?dl=0

Thanks,
Serkan

On Thu, May 5, 2016 at 9:33 AM, Xavier Hernandez <xhernan...@datalab.es> wrote:
> Can you post the result of 'gluster volume status v0 detail' ?
>
>
> On 05/05/16 06:49, Serkan Çoban wrote:
>>
>> Hi, Can anyone suggest something for this issue? df, du has no issue
>> for the bricks yet one subvolume not being used by gluster..
>>
>> On Wed, May 4, 2016 at 4:40 PM, Serkan Çoban <cobanser...@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I changed cluster.min-free-inodes to "0". Remount the volume on
>>> clients. inode full messages not coming to syslog anymore but I see
>>> disperse-56 subvolume still not being used.
>>> Anything I can do to resolve this issue? Maybe I can destroy and
>>> recreate the volume but I am not sure It will fix this issue...
>>> Maybe the disperse size 16+4 is too big should I change it to 8+2?
>>>
>>> On Tue, May 3, 2016 at 2:36 PM, Serkan Çoban <cobanser...@gmail.com>
>>> wrote:
>>>>
>>>> I also checked the df output all 20 bricks are same like below:
>>>> /dev/sdu1 7.3T 34M 7.3T 1% /bricks/20
>>>>
>>>> On Tue, May 3, 2016 at 1:40 PM, Raghavendra G <raghaven...@gluster.com>
>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Mon, May 2, 2016 at 11:41 AM, Serkan Çoban <cobanser...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>> 1. What is the out put of du -hs ? Please get this
>>>>>>> information for each of the brick that are part of disperse.
>>>>>
>>>>>
>>>>>
>>>>> Sorry. I needed df output of the filesystem containing brick. Not du.
>>>>> Sorry
>>>>> about that.
>>>>>
>>>>>>
>>>>>> There are 20 bricks in disperse-56 and the du -hs output is like:
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 1.8M /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>> 80K /bricks/20
>>>>>>
>>>>>> I see that gluster is not writing to this disperse set. All other
>>>>>> disperse sets are filled 13GB but this one is empty. I see directory
>>>>>> structure created but no files in directories.
>>>>>> How can I fix the issue? I will try to rebalance but I don't think it
>>>>>> will write to this disperse set...
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Apr 30, 2016 at 9:22 AM, Raghavendra G
>>>>>> <raghaven...@gluster.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Apr 29, 2016 at 12:32 AM, Serkan Çoban
>>>>>>> <cobanser...@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi, I cannot get an answer from user list, so asking to devel list.
>>>>>>>>
>>>>>>>> I am getting [dht-diskusage.c:277:dht_is_subvol_filled] 0-v0-dht:
>>>>>>>> inodes on subvolume 'v0-disperse-56' are at (100.00 %), consider
>>>>>>>> adding more bricks.
>>>>>>>>
>>>>>>>> message on client logs.My cluster is empty there are only a couple
>>>>>>>> of
>>>>>>>> GB files for testing. Why this message appear in syslog?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> dht uses disk usage information from backend export.
>>>>>>>
>>>>>>> 1. What is the out put of du -hs ? Please get this
>>&

Re: [Gluster-devel] [Gluster-users] Fwd: dht_is_subvol_filled messages on client

[Serkan Çoban]

Hi, Can anyone suggest something for this issue? df, du has no issue
for the bricks yet one subvolume not being used by gluster..

On Wed, May 4, 2016 at 4:40 PM, Serkan Çoban <cobanser...@gmail.com> wrote:
> Hi,
>
> I changed cluster.min-free-inodes to "0". Remount the volume on
> clients. inode full messages not coming to syslog anymore but I see
> disperse-56 subvolume still not being used.
> Anything I can do to resolve this issue? Maybe I can destroy and
> recreate the volume but I am not sure It will fix this issue...
> Maybe the disperse size 16+4 is too big should I change it to 8+2?
>
> On Tue, May 3, 2016 at 2:36 PM, Serkan Çoban <cobanser...@gmail.com> wrote:
>> I also checked the df output all 20 bricks are same like below:
>> /dev/sdu1 7.3T 34M 7.3T 1% /bricks/20
>>
>> On Tue, May 3, 2016 at 1:40 PM, Raghavendra G <raghaven...@gluster.com> 
>> wrote:
>>>
>>>
>>> On Mon, May 2, 2016 at 11:41 AM, Serkan Çoban <cobanser...@gmail.com> wrote:
>>>>
>>>> >1. What is the out put of du -hs ? Please get this
>>>> > information for each of the brick that are part of disperse.
>>>
>>>
>>> Sorry. I needed df output of the filesystem containing brick. Not du. Sorry
>>> about that.
>>>
>>>>
>>>> There are 20 bricks in disperse-56 and the du -hs output is like:
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 1.8M /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>> 80K /bricks/20
>>>>
>>>> I see that gluster is not writing to this disperse set. All other
>>>> disperse sets are filled 13GB but this one is empty. I see directory
>>>> structure created but no files in directories.
>>>> How can I fix the issue? I will try to rebalance but I don't think it
>>>> will write to this disperse set...
>>>>
>>>>
>>>>
>>>> On Sat, Apr 30, 2016 at 9:22 AM, Raghavendra G <raghaven...@gluster.com>
>>>> wrote:
>>>> >
>>>> >
>>>> > On Fri, Apr 29, 2016 at 12:32 AM, Serkan Çoban <cobanser...@gmail.com>
>>>> > wrote:
>>>> >>
>>>> >> Hi, I cannot get an answer from user list, so asking to devel list.
>>>> >>
>>>> >> I am getting [dht-diskusage.c:277:dht_is_subvol_filled] 0-v0-dht:
>>>> >> inodes on subvolume 'v0-disperse-56' are at (100.00 %), consider
>>>> >> adding more bricks.
>>>> >>
>>>> >> message on client logs.My cluster is empty there are only a couple of
>>>> >> GB files for testing. Why this message appear in syslog?
>>>> >
>>>> >
>>>> > dht uses disk usage information from backend export.
>>>> >
>>>> > 1. What is the out put of du -hs ? Please get this
>>>> > information for each of the brick that are part of disperse.
>>>> > 2. Once you get du information from each brick, the value seen by dht
>>>> > will
>>>> > be based on how cluster/disperse aggregates du info (basically statfs
>>>> > fop).
>>>> >
>>>> > The reason for 100% disk usage may be,
>>>> > In case of 1, backend fs might be shared by data other than brick.
>>>> > In case of 2, some issues with aggregation.
>>>> >
>>>> >> Is is safe to
>>>> >> ignore it?
>>>> >
>>>> >
>>>> > dht will try not to have data files on the subvol in question
>>>> > (v0-disperse-56). Hence lookup cost will be two hops for files hashing
>>>> > to
>>>> > disperse-56 (note that other fops like read/write/open still have the
>>>> > cost
>>>> > of single hop and dont suffer from this penalty). Other than that there
>>>> > is
>>>> > no significant harm unless disperse-56 is really running out of space.
>>>> >
>>>> > regards,
>>>> > Raghavendra
>>>> >
>>>> >> ___
>>>> >> Gluster-devel mailing list
>>>> >> Gluster-devel@gluster.org
>>>> >> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Raghavendra G
>>>> ___
>>>> Gluster-devel mailing list
>>>> Gluster-devel@gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>
>>>
>>>
>>>
>>> --
>>> Raghavendra G
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Fwd: dht_is_subvol_filled messages on client

[Serkan Çoban]

Ah I see how can I overlook this, my bad sorry everyone for taking
time to help me...
>BTW, how large is the volume you have?
9PB usable :)

Serkan

On Thu, May 5, 2016 at 2:07 PM, Xavier Hernandez <xhernan...@datalab.es> wrote:
> On 05/05/16 11:31, Kaushal M wrote:
>>
>> On Thu, May 5, 2016 at 2:36 PM, David Gossage
>> <dgoss...@carouselchecks.com> wrote:
>>>
>>>
>>>
>>>
>>> On Thu, May 5, 2016 at 3:28 AM, Serkan Çoban <cobanser...@gmail.com>
>>> wrote:
>>>>
>>>>
>>>> Hi,
>>>>
>>>> You can find the output below link:
>>>> https://www.dropbox.com/s/wzrh5yp494ogksc/status_detail.txt?dl=0
>>>>
>>>> Thanks,
>>>> Serkan
>>>
>>>
>>>
>>> Maybe not issue, but playing one of these things is not like the other I
>>> notice of all the bricks only one seems to be different at a quick glance
>>>
>>> Brick: Brick 1.1.1.235:/bricks/20
>>> TCP Port : 49170
>>> RDMA Port: 0
>>> Online   : Y
>>> Pid  : 26736
>>> File System  : ext4
>>> Device   : /dev/mapper/vol0-vol_root
>>> Mount Options: rw,relatime,data=ordered
>>> Inode Size   : 256
>>> Disk Space Free  : 86.1GB
>>> Total Disk Space : 96.0GB
>>> Inode Count  : 6406144
>>> Free Inodes  : 6381374
>>>
>>> Every other brick seems to be 7TB and xfs but this one.
>>
>>
>> Looks like the brick fs isn't mounted, and the root-fs is being used
>> instead. But that still leaves enough inodes free.
>>
>> What I suspect is that one of the cluster translators is mixing up
>> stats when aggregating from multiple bricks.
>> From the log snippet you gave in the first mail, it seems like the
>> disperse translator is possibly involved.
>
>
> Currently ec takes the number of potential files in the subvolume (f_files)
> as the maximum of all its subvolumes, but it takes the available count
> (f_ffree) as the minumum of all its volumes.
>
> This causes max to be ~781.000.000, but free will be ~6.300.000. This gives
> a ~0.8% available, i.e. almost 100% full.
>
> Given the circumstances I think it's the correct thing to do.
>
> Xavi
>
>
>>
>> BTW, how large is the volume you have? Those are a lot of bricks!
>>
>> ~kaushal
>>
>>
>>>
>>>
>>>
>>>>
>>>>
>>>> On Thu, May 5, 2016 at 9:33 AM, Xavier Hernandez <xhernan...@datalab.es>
>>>> wrote:
>>>>>
>>>>> Can you post the result of 'gluster volume status v0 detail' ?
>>>>>
>>>>>
>>>>> On 05/05/16 06:49, Serkan Çoban wrote:
>>>>>>
>>>>>>
>>>>>> Hi, Can anyone suggest something for this issue? df, du has no issue
>>>>>> for the bricks yet one subvolume not being used by gluster..
>>>>>>
>>>>>> On Wed, May 4, 2016 at 4:40 PM, Serkan Çoban <cobanser...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I changed cluster.min-free-inodes to "0". Remount the volume on
>>>>>>> clients. inode full messages not coming to syslog anymore but I see
>>>>>>> disperse-56 subvolume still not being used.
>>>>>>> Anything I can do to resolve this issue? Maybe I can destroy and
>>>>>>> recreate the volume but I am not sure It will fix this issue...
>>>>>>> Maybe the disperse size 16+4 is too big should I change it to 8+2?
>>>>>>>
>>>>>>> On Tue, May 3, 2016 at 2:36 PM, Serkan Çoban <cobanser...@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> I also checked the df output all 20 bricks are same like below:
>>>>>>>> /dev/sdu1 7.3T 34M 7.3T 1% /bricks/20
>>>>>>>>
>>>>>>>> On Tue, May 3, 2016 at 1:40 PM, Raghavendra G
>>>>>>>> <raghaven...@gluster.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>&

Re: [Gluster-devel] [Gluster-users] Fwd: dht_is_subvol_filled messages on client

[Serkan Çoban]

Hi,

I changed cluster.min-free-inodes to "0". Remount the volume on
clients. inode full messages not coming to syslog anymore but I see
disperse-56 subvolume still not being used.
Anything I can do to resolve this issue? Maybe I can destroy and
recreate the volume but I am not sure It will fix this issue...
Maybe the disperse size 16+4 is too big should I change it to 8+2?

On Tue, May 3, 2016 at 2:36 PM, Serkan Çoban <cobanser...@gmail.com> wrote:
> I also checked the df output all 20 bricks are same like below:
> /dev/sdu1 7.3T 34M 7.3T 1% /bricks/20
>
> On Tue, May 3, 2016 at 1:40 PM, Raghavendra G <raghaven...@gluster.com> wrote:
>>
>>
>> On Mon, May 2, 2016 at 11:41 AM, Serkan Çoban <cobanser...@gmail.com> wrote:
>>>
>>> >1. What is the out put of du -hs ? Please get this
>>> > information for each of the brick that are part of disperse.
>>
>>
>> Sorry. I needed df output of the filesystem containing brick. Not du. Sorry
>> about that.
>>
>>>
>>> There are 20 bricks in disperse-56 and the du -hs output is like:
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 1.8M /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>> 80K /bricks/20
>>>
>>> I see that gluster is not writing to this disperse set. All other
>>> disperse sets are filled 13GB but this one is empty. I see directory
>>> structure created but no files in directories.
>>> How can I fix the issue? I will try to rebalance but I don't think it
>>> will write to this disperse set...
>>>
>>>
>>>
>>> On Sat, Apr 30, 2016 at 9:22 AM, Raghavendra G <raghaven...@gluster.com>
>>> wrote:
>>> >
>>> >
>>> > On Fri, Apr 29, 2016 at 12:32 AM, Serkan Çoban <cobanser...@gmail.com>
>>> > wrote:
>>> >>
>>> >> Hi, I cannot get an answer from user list, so asking to devel list.
>>> >>
>>> >> I am getting [dht-diskusage.c:277:dht_is_subvol_filled] 0-v0-dht:
>>> >> inodes on subvolume 'v0-disperse-56' are at (100.00 %), consider
>>> >> adding more bricks.
>>> >>
>>> >> message on client logs.My cluster is empty there are only a couple of
>>> >> GB files for testing. Why this message appear in syslog?
>>> >
>>> >
>>> > dht uses disk usage information from backend export.
>>> >
>>> > 1. What is the out put of du -hs ? Please get this
>>> > information for each of the brick that are part of disperse.
>>> > 2. Once you get du information from each brick, the value seen by dht
>>> > will
>>> > be based on how cluster/disperse aggregates du info (basically statfs
>>> > fop).
>>> >
>>> > The reason for 100% disk usage may be,
>>> > In case of 1, backend fs might be shared by data other than brick.
>>> > In case of 2, some issues with aggregation.
>>> >
>>> >> Is is safe to
>>> >> ignore it?
>>> >
>>> >
>>> > dht will try not to have data files on the subvol in question
>>> > (v0-disperse-56). Hence lookup cost will be two hops for files hashing
>>> > to
>>> > disperse-56 (note that other fops like read/write/open still have the
>>> > cost
>>> > of single hop and dont suffer from this penalty). Other than that there
>>> > is
>>> > no significant harm unless disperse-56 is really running out of space.
>>> >
>>> > regards,
>>> > Raghavendra
>>> >
>>> >> ___
>>> >> Gluster-devel mailing list
>>> >> Gluster-devel@gluster.org
>>> >> http://www.gluster.org/mailman/listinfo/gluster-devel
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Raghavendra G
>>> ___
>>> Gluster-devel mailing list
>>> Gluster-devel@gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>
>>
>>
>>
>> --
>> Raghavendra G
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Fwd: dht_is_subvol_filled messages on client

[Serkan Çoban]

I also checked the df output all 20 bricks are same like below:
/dev/sdu1 7.3T 34M 7.3T 1% /bricks/20

On Tue, May 3, 2016 at 1:40 PM, Raghavendra G <raghaven...@gluster.com> wrote:
>
>
> On Mon, May 2, 2016 at 11:41 AM, Serkan Çoban <cobanser...@gmail.com> wrote:
>>
>> >1. What is the out put of du -hs ? Please get this
>> > information for each of the brick that are part of disperse.
>
>
> Sorry. I needed df output of the filesystem containing brick. Not du. Sorry
> about that.
>
>>
>> There are 20 bricks in disperse-56 and the du -hs output is like:
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 1.8M /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>>
>> I see that gluster is not writing to this disperse set. All other
>> disperse sets are filled 13GB but this one is empty. I see directory
>> structure created but no files in directories.
>> How can I fix the issue? I will try to rebalance but I don't think it
>> will write to this disperse set...
>>
>>
>>
>> On Sat, Apr 30, 2016 at 9:22 AM, Raghavendra G <raghaven...@gluster.com>
>> wrote:
>> >
>> >
>> > On Fri, Apr 29, 2016 at 12:32 AM, Serkan Çoban <cobanser...@gmail.com>
>> > wrote:
>> >>
>> >> Hi, I cannot get an answer from user list, so asking to devel list.
>> >>
>> >> I am getting [dht-diskusage.c:277:dht_is_subvol_filled] 0-v0-dht:
>> >> inodes on subvolume 'v0-disperse-56' are at (100.00 %), consider
>> >> adding more bricks.
>> >>
>> >> message on client logs.My cluster is empty there are only a couple of
>> >> GB files for testing. Why this message appear in syslog?
>> >
>> >
>> > dht uses disk usage information from backend export.
>> >
>> > 1. What is the out put of du -hs ? Please get this
>> > information for each of the brick that are part of disperse.
>> > 2. Once you get du information from each brick, the value seen by dht
>> > will
>> > be based on how cluster/disperse aggregates du info (basically statfs
>> > fop).
>> >
>> > The reason for 100% disk usage may be,
>> > In case of 1, backend fs might be shared by data other than brick.
>> > In case of 2, some issues with aggregation.
>> >
>> >> Is is safe to
>> >> ignore it?
>> >
>> >
>> > dht will try not to have data files on the subvol in question
>> > (v0-disperse-56). Hence lookup cost will be two hops for files hashing
>> > to
>> > disperse-56 (note that other fops like read/write/open still have the
>> > cost
>> > of single hop and dont suffer from this penalty). Other than that there
>> > is
>> > no significant harm unless disperse-56 is really running out of space.
>> >
>> > regards,
>> > Raghavendra
>> >
>> >> ___
>> >> Gluster-devel mailing list
>> >> Gluster-devel@gluster.org
>> >> http://www.gluster.org/mailman/listinfo/gluster-devel
>> >
>> >
>> >
>> >
>> > --
>> > Raghavendra G
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-devel
>
>
>
>
> --
> Raghavendra G
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Fwd: dht_is_subvol_filled messages on client

[Serkan Çoban]

Hi,

I checked with this(1) C program if FS is returning something wrong
but that is not the case.
I also try to understand the source code, somehow in below statement
subvol_filled_inodes sets to true.

if (conf->du_stats[i].avail_inodes <
conf->min_free_inodes) {
subvol_filled_inodes = _gf_true;
break;
}

Can I skip this if statement if I change "conf->min_free_inodes" to 0?



1- 
https://raw.githubusercontent.com/perusio/linux-programming-by-example/master/book/ch08/ch08-statvfs.c

On Mon, May 2, 2016 at 12:00 PM, Serkan Çoban <cobanser...@gmail.com> wrote:
> I started a rebalace and it did not fix the issue...
>
> On Mon, May 2, 2016 at 9:11 AM, Serkan Çoban <cobanser...@gmail.com> wrote:
>>>1. What is the out put of du -hs ? Please get this 
>>>information for each of the brick that are part of disperse.
>> There are 20 bricks in disperse-56 and the du -hs output is like:
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 1.8M /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>> 80K /bricks/20
>>
>> I see that gluster is not writing to this disperse set. All other
>> disperse sets are filled 13GB but this one is empty. I see directory
>> structure created but no files in directories.
>> How can I fix the issue? I will try to rebalance but I don't think it
>> will write to this disperse set...
>>
>>
>>
>> On Sat, Apr 30, 2016 at 9:22 AM, Raghavendra G <raghaven...@gluster.com> 
>> wrote:
>>>
>>>
>>> On Fri, Apr 29, 2016 at 12:32 AM, Serkan Çoban <cobanser...@gmail.com>
>>> wrote:
>>>>
>>>> Hi, I cannot get an answer from user list, so asking to devel list.
>>>>
>>>> I am getting [dht-diskusage.c:277:dht_is_subvol_filled] 0-v0-dht:
>>>> inodes on subvolume 'v0-disperse-56' are at (100.00 %), consider
>>>> adding more bricks.
>>>>
>>>> message on client logs.My cluster is empty there are only a couple of
>>>> GB files for testing. Why this message appear in syslog?
>>>
>>>
>>> dht uses disk usage information from backend export.
>>>
>>> 1. What is the out put of du -hs ? Please get this
>>> information for each of the brick that are part of disperse.
>>> 2. Once you get du information from each brick, the value seen by dht will
>>> be based on how cluster/disperse aggregates du info (basically statfs fop).
>>>
>>> The reason for 100% disk usage may be,
>>> In case of 1, backend fs might be shared by data other than brick.
>>> In case of 2, some issues with aggregation.
>>>
>>>> Is is safe to
>>>> ignore it?
>>>
>>>
>>> dht will try not to have data files on the subvol in question
>>> (v0-disperse-56). Hence lookup cost will be two hops for files hashing to
>>> disperse-56 (note that other fops like read/write/open still have the cost
>>> of single hop and dont suffer from this penalty). Other than that there is
>>> no significant harm unless disperse-56 is really running out of space.
>>>
>>> regards,
>>> Raghavendra
>>>
>>>> ___
>>>> Gluster-devel mailing list
>>>> Gluster-devel@gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>
>>>
>>>
>>>
>>> --
>>> Raghavendra G
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Fwd: disperse volume file to subvolume mapping

[Serkan Çoban]

Hi, I just reinstalled fresh 3.7.11 and I am seeing the same behavior.
50 clients copying part-0- named files using mapreduce to gluster
using one thread per server and they are using only 20 servers out of
60. On the other hand fio tests use all the servers. Anything I can do
to solve the issue?

Thanks,
Serkan


-- Forwarded message --
From: Serkan Çoban <cobanser...@gmail.com>
Date: Mon, Apr 18, 2016 at 2:39 PM
Subject: disperse volume file to subvolume mapping
To: Gluster Users <gluster-us...@gluster.org>


Hi, I have a problem where clients are using only 1/3 of nodes in
disperse volume for writing.
I am testing from 50 clients using 1 to 10 threads with file names part-0-.
What I see is clients only use 20 nodes for writing. How is the file
name to sub volume hashing is done? Is this related to file names are
similar?

My cluster is 3.7.10 with 60 nodes each has 26 disks. Disperse volume
is 78 x (16+4). Only 26 out of 78 sub volumes used during writes..
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] disperse volume file to subvolume mapping

[Serkan Çoban]

I am copying 10.000 files to gluster volume using mapreduce on
clients. Each map process took one file at a time and copy it to
gluster volume.
My disperse volume consist of 78 subvolumes of 16+4 disk each. So If I
copy >78 files parallel I expect each file goes to different subvolume
right?
In my tests during tests with fio I can see every file goes to
different subvolume, but when I start mapreduce process from clients
only 78/3=26 subvolumes used for writing files.
I see that clearly from network traffic. Mapreduce on client side can
be run multi thread. I tested with 1-5-10 threads on each client but
every time only 26 subvolumes used.
How can I debug the issue further?

On Tue, Apr 19, 2016 at 11:22 AM, Xavier Hernandez
<xhernan...@datalab.es> wrote:
> Hi Serkan,
>
> On 19/04/16 09:18, Serkan Çoban wrote:
>>
>> Hi, I just reinstalled fresh 3.7.11 and I am seeing the same behavior.
>> 50 clients copying part-0- named files using mapreduce to gluster
>> using one thread per server and they are using only 20 servers out of
>> 60. On the other hand fio tests use all the servers. Anything I can do
>> to solve the issue?
>
>
> Distribution of files to ec sets is done by dht. In theory if you create
> many files each ec set will receive the same amount of files. However when
> the number of files is small enough, statistics can fail.
>
> Not sure what you are doing exactly, but a mapreduce procedure generally
> only creates a single output. In that case it makes sense that only one ec
> set is used. If you want to use all ec sets for a single file, you should
> enable sharding (I haven't tested that) or split the result in multiple
> files.
>
> Xavi
>
>
>>
>> Thanks,
>> Serkan
>>
>>
>> -- Forwarded message --
>> From: Serkan Çoban <cobanser...@gmail.com>
>> Date: Mon, Apr 18, 2016 at 2:39 PM
>> Subject: disperse volume file to subvolume mapping
>> To: Gluster Users <gluster-us...@gluster.org>
>>
>>
>> Hi, I have a problem where clients are using only 1/3 of nodes in
>> disperse volume for writing.
>> I am testing from 50 clients using 1 to 10 threads with file names
>> part-0-.
>> What I see is clients only use 20 nodes for writing. How is the file
>> name to sub volume hashing is done? Is this related to file names are
>> similar?
>>
>> My cluster is 3.7.10 with 60 nodes each has 26 disks. Disperse volume
>> is 78 x (16+4). Only 26 out of 78 sub volumes used during writes..
>>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Assertion failed: ec_get_inode_size

[Serkan Çoban]

I did one upgrade only from 3.7.9 to 3.7.10 and it is not a rolling
upgrade,  I stopped volume and then upgrade all the components.
I will try restarting the volume and see if it helps..

On Mon, Apr 18, 2016 at 10:17 AM, Ashish Pandey <aspan...@redhat.com> wrote:
> Hi Serkan,
>
> I have gone through the logs and can see there are some blocked inode lock
> requests.
> We have observed that some other user have also faced this issue with
> similar logs.
> I think you  have tried some rolling update on your setup or some NODES , on
> which you have collected these statedumps, must have gone down for one or
> other reason.
>
> We will further dig it up and will try to find out the root cause. Till than
> you can resolve this issue by restarting the volume which will restart nfs
> and shd and will release any locks taken by these process.
>
> "gluster volume start  force" will do the same.
>
> Regards,
> Ashish
>
>
> 
> From: "Serkan Çoban" <cobanser...@gmail.com>
> To: "Ashish Pandey" <aspan...@redhat.com>
> Cc: "Gluster Users" <gluster-us...@gluster.org>, "Gluster Devel"
> <gluster-devel@gluster.org>
> Sent: Monday, April 18, 2016 11:51:37 AM
>
> Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size
>
> You can find the statedumps of server and client in below link.
> Gluster version is 3.7.10, 78x(16+4) disperse setup. 60 nodes named
> node185..node244
> https://www.dropbox.com/s/cc2dgsxwuk48mba/gluster_statedumps.zip?dl=0
>
>
> On Fri, Apr 15, 2016 at 9:52 PM, Ashish Pandey <aspan...@redhat.com> wrote:
>>
>> Actually it was my mistake I overlooked the configuration you provided..It
>> will be huge.
>> I would suggest to take statedump on all the nodes and try to grep for
>> "BLOCKED" in statedump files on all the nodes.
>> See if you can see any such line in any file and send those files. No need
>> to send statedump of all the bricks..
>>
>>
>>
>>
>> 
>> From: "Serkan Çoban" <cobanser...@gmail.com>
>> To: "Ashish Pandey" <aspan...@redhat.com>
>> Cc: "Gluster Users" <gluster-us...@gluster.org>, "Gluster Devel"
>> <gluster-devel@gluster.org>
>> Sent: Friday, April 15, 2016 6:07:00 PM
>>
>> Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size
>>
>> Hi Asish,
>>
>> Sorry for the question but do you want all brick statedumps from all
>> servers or all brick dumps from one server?
>> All server brick dumps is nearly 700MB zipped..
>>
>> On Fri, Apr 15, 2016 at 2:16 PM, Ashish Pandey <aspan...@redhat.com>
>> wrote:
>>>
>>> To get the state dump of fuse client-
>>> 1 - get the PID of fuse mount process
>>> 2 - kill -USR1 
>>>
>>> statedump can be found in the same directory where u get for brick
>>> process.
>>>
>>> Following link could be helpful for future reference -
>>>
>>>
>>> https://github.com/gluster/glusterfs/blob/master/doc/debugging/statedump.md
>>>
>>> Ashish
>>>
>>> 
>>> From: "Serkan Çoban" <cobanser...@gmail.com>
>>> To: "Ashish Pandey" <aspan...@redhat.com>
>>> Cc: "Gluster Users" <gluster-us...@gluster.org>, "Gluster Devel"
>>> <gluster-devel@gluster.org>
>>> Sent: Friday, April 15, 2016 4:02:20 PM
>>> Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size
>>>
>>> Yes it is only one brick which error appears. I can send all other
>>> brick dumps too..
>>> How can I get state dump in fuse client? There is no gluster command
>>> there..
>>> ___
>>> Gluster-users mailing list
>>> gluster-us...@gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>> ___
>> Gluster-users mailing list
>> gluster-us...@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Assertion failed: ec_get_inode_size

[Serkan Çoban]

You can find the statedumps of server and client in below link.
Gluster version is 3.7.10, 78x(16+4) disperse setup. 60 nodes named
node185..node244
https://www.dropbox.com/s/cc2dgsxwuk48mba/gluster_statedumps.zip?dl=0


On Fri, Apr 15, 2016 at 9:52 PM, Ashish Pandey <aspan...@redhat.com> wrote:
>
> Actually it was my mistake I overlooked the configuration you provided..It
> will be huge.
> I would suggest to take statedump on all the nodes and try to grep for
> "BLOCKED" in statedump files on all the nodes.
> See if you can see any such line in any file and send those files. No need
> to send statedump of all the bricks..
>
>
>
>
> ____
> From: "Serkan Çoban" <cobanser...@gmail.com>
> To: "Ashish Pandey" <aspan...@redhat.com>
> Cc: "Gluster Users" <gluster-us...@gluster.org>, "Gluster Devel"
> <gluster-devel@gluster.org>
> Sent: Friday, April 15, 2016 6:07:00 PM
>
> Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size
>
> Hi Asish,
>
> Sorry for the question but do you want all brick statedumps from all
> servers or all brick dumps from one server?
> All server brick dumps is nearly 700MB zipped..
>
> On Fri, Apr 15, 2016 at 2:16 PM, Ashish Pandey <aspan...@redhat.com> wrote:
>>
>> To get the state dump of fuse client-
>> 1 - get the PID of fuse mount process
>> 2 - kill -USR1 
>>
>> statedump can be found in the same directory where u get for brick
>> process.
>>
>> Following link could be helpful for future reference -
>>
>> https://github.com/gluster/glusterfs/blob/master/doc/debugging/statedump.md
>>
>> Ashish
>>
>> 
>> From: "Serkan Çoban" <cobanser...@gmail.com>
>> To: "Ashish Pandey" <aspan...@redhat.com>
>> Cc: "Gluster Users" <gluster-us...@gluster.org>, "Gluster Devel"
>> <gluster-devel@gluster.org>
>> Sent: Friday, April 15, 2016 4:02:20 PM
>> Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size
>>
>> Yes it is only one brick which error appears. I can send all other
>> brick dumps too..
>> How can I get state dump in fuse client? There is no gluster command
>> there..
>> ___
>> Gluster-users mailing list
>> gluster-us...@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Assertion failed: ec_get_inode_size

[Serkan Çoban]

Hi Asish,

Sorry for the question but do you want all brick statedumps from all
servers or all brick dumps from one server?
All server brick dumps is nearly 700MB zipped..

On Fri, Apr 15, 2016 at 2:16 PM, Ashish Pandey <aspan...@redhat.com> wrote:
>
> To get the state dump of fuse client-
> 1 - get the PID of fuse mount process
> 2 - kill -USR1 
>
> statedump can be found in the same directory where u get for brick process.
>
> Following link could be helpful for future reference -
> https://github.com/gluster/glusterfs/blob/master/doc/debugging/statedump.md
>
> Ashish
>
> ________
> From: "Serkan Çoban" <cobanser...@gmail.com>
> To: "Ashish Pandey" <aspan...@redhat.com>
> Cc: "Gluster Users" <gluster-us...@gluster.org>, "Gluster Devel"
> <gluster-devel@gluster.org>
> Sent: Friday, April 15, 2016 4:02:20 PM
> Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size
>
> Yes it is only one brick which error appears. I can send all other
> brick dumps too..
> How can I get state dump in fuse client? There is no gluster command there..
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Assertion failed: ec_get_inode_size

[Serkan Çoban]

Here is the related brick log:

/var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700556] E
[inodelk.c:309:__inode_unlock_lock] 0-v0-locks:  Matching lock not
found for unlock 0-9223372036854775807, by 94d29e885e7f on
0x7f037413b990
/var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700639] E
[MSGID: 115053] [server-rpc-fops.c:276:server_inodelk_cbk]
0-v0-server: 712984: INODELK
/workdir/raw_output/xxx/yyy/zzz.dat.gz.snappy1460474606605
(1191e32e-44ba-4e20-87ca-35ace8519c19) ==> (Invalid argument) [Invalid
argument]

On Thu, Apr 14, 2016 at 3:25 PM, Serkan Çoban <cobanser...@gmail.com> wrote:
> Hi,
>
> During read/write tests to a 78x(16+4) distributed disperse volume
> from 50 clients, One clients hangs on read/write with the following
> logs:
>
> [2016-04-14 11:11:04.728580] W [MSGID: 122056]
> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-6: Mismatching xdata
> in answers of 'LOOKUP'
> [2016-04-14 11:11:04.728624] W [MSGID: 122053]
> [ec-common.c:116:ec_check_status] 0-v0-disperse-6: Operation failed on
> some subvolumes (up=F, mask=F, remaining=0, good=D,
> bad=2)
> [2016-04-14 11:11:04.736689] I [MSGID: 122058]
> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-6: /workdir/raw_output2:
> name heal successful on F
> [2016-04-14 11:29:26.718036] W [MSGID: 122056]
> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-1: Mismatching xdata
> in answers of 'LOOKUP'
> [2016-04-14 11:29:26.718121] W [MSGID: 122053]
> [ec-common.c:116:ec_check_status] 0-v0-disperse-1: Operation failed on
> some subvolumes (up=F, mask=F, remaining=0, good=E,
> bad=1)
> [2016-04-14 11:29:42.501760] I [MSGID: 122058]
> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-1: /workdir/raw_output2:
> name heal successful on F
> [2016-04-14 11:31:25.714812] E [ec-inode-read.c:1612:ec_manager_stat]
> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_resume+0x91)
> [0x7f5ec9f942b1]
> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(__ec_manager+0x57)
> [0x7f5ec9f94497]
> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_stat+0x2c4)
> [0x7f5ec9faaed4] ) 0-: Assertion failed: ec_get_inode_size(fop,
> fop->locks[0].lock->loc.inode, >iatt[0].ia_size)
> [2016-04-14 11:31:25.722372] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-40: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722411] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-41: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722450] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-44: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722477] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-42: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722503] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-43: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722577] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-45: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722605] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-46: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722742] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-49: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722794] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-47: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722818] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-48: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722840] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-50: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722883] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-52: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722906] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-54: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722958] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-51: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.722983] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-55: remote
> operation failed [Invalid argument]
> [2016-04-14 11:31:25.723037] E [MSGID: 114031]
> [client-rpc-fops.c:1624:client3_3_inodelk_cb

Re: [Gluster-devel] [Gluster-users] Assertion failed: ec_get_inode_size

[Serkan Çoban]

Hi I reproduce the problem, brick log file is in below link:
https://www.dropbox.com/s/iy09j7mm2hrsf03/bricks-02.5677.dump.1460705370.gz?dl=0


On Thu, Apr 14, 2016 at 8:07 PM, Ashish Pandey <aspan...@redhat.com> wrote:
> Hi Serkan,
>
> Could you also provide us the statedump of all the brick processes and
> clients?
>
> Commands to generate statedumps for brick processes/nfs server/quotad
>
> For bricks: gluster volume statedump 
>
> For nfs server: gluster volume statedump  nfs
>
>
> We can find the directory where statedump files are created using 'gluster
> --print-statedumpdir'
> Also, the mount logs would help us to debug the issue.
>
> Ashish
>
> 
> From: "Serkan Çoban" <cobanser...@gmail.com>
> To: "Gluster Users" <gluster-us...@gluster.org>, "Gluster Devel"
> <gluster-devel@gluster.org>
> Sent: Thursday, April 14, 2016 6:27:10 PM
> Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size
>
>
> Here is the related brick log:
>
> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700556] E
> [inodelk.c:309:__inode_unlock_lock] 0-v0-locks:  Matching lock not
> found for unlock 0-9223372036854775807, by 94d29e885e7f on
> 0x7f037413b990
> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700639] E
> [MSGID: 115053] [server-rpc-fops.c:276:server_inodelk_cbk]
> 0-v0-server: 712984: INODELK
> /workdir/raw_output/xxx/yyy/zzz.dat.gz.snappy1460474606605
> (1191e32e-44ba-4e20-87ca-35ace8519c19) ==> (Invalid argument) [Invalid
> argument]
>
> On Thu, Apr 14, 2016 at 3:25 PM, Serkan Çoban <cobanser...@gmail.com> wrote:
>> Hi,
>>
>> During read/write tests to a 78x(16+4) distributed disperse volume
>> from 50 clients, One clients hangs on read/write with the following
>> logs:
>>
>> [2016-04-14 11:11:04.728580] W [MSGID: 122056]
>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-6: Mismatching xdata
>> in answers of 'LOOKUP'
>> [2016-04-14 11:11:04.728624] W [MSGID: 122053]
>> [ec-common.c:116:ec_check_status] 0-v0-disperse-6: Operation failed on
>> some subvolumes (up=F, mask=F, remaining=0, good=D,
>> bad=2)
>> [2016-04-14 11:11:04.736689] I [MSGID: 122058]
>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-6: /workdir/raw_output2:
>> name heal successful on F
>> [2016-04-14 11:29:26.718036] W [MSGID: 122056]
>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-1: Mismatching xdata
>> in answers of 'LOOKUP'
>> [2016-04-14 11:29:26.718121] W [MSGID: 122053]
>> [ec-common.c:116:ec_check_status] 0-v0-disperse-1: Operation failed on
>> some subvolumes (up=F, mask=F, remaining=0, good=E,
>> bad=1)
>> [2016-04-14 11:29:42.501760] I [MSGID: 122058]
>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-1: /workdir/raw_output2:
>> name heal successful on F
>> [2016-04-14 11:31:25.714812] E [ec-inode-read.c:1612:ec_manager_stat]
>> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_resume+0x91)
>> [0x7f5ec9f942b1]
>>
>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(__ec_manager+0x57)
>> [0x7f5ec9f94497]
>>
>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_stat+0x2c4)
>> [0x7f5ec9faaed4] ) 0-: Assertion failed: ec_get_inode_size(fop,
>> fop->locks[0].lock->loc.inode, >iatt[0].ia_size)
>> [2016-04-14 11:31:25.722372] E [MSGID: 114031]
>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-40: remote
>> operation failed [Invalid argument]
>> [2016-04-14 11:31:25.722411] E [MSGID: 114031]
>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-41: remote
>> operation failed [Invalid argument]
>> [2016-04-14 11:31:25.722450] E [MSGID: 114031]
>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-44: remote
>> operation failed [Invalid argument]
>> [2016-04-14 11:31:25.722477] E [MSGID: 114031]
>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-42: remote
>> operation failed [Invalid argument]
>> [2016-04-14 11:31:25.722503] E [MSGID: 114031]
>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-43: remote
>> operation failed [Invalid argument]
>> [2016-04-14 11:31:25.722577] E [MSGID: 114031]
>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-45: remote
>> operation failed [Invalid argument]
>> [2016-04-14 11:31:25.722605] E [MSGID: 114031]
>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-46: remote
>> operation failed [Invalid argument]
>> [2016-04-14 11:31:25.722742] E [MSGID: 114031]
&

Re: [Gluster-devel] [Gluster-users] Assertion failed: ec_get_inode_size

[Serkan Çoban]

Sorry for typo, brick state dump file.

On Fri, Apr 15, 2016 at 11:41 AM, Serkan Çoban <cobanser...@gmail.com> wrote:
> Hi I reproduce the problem, brick log file is in below link:
> https://www.dropbox.com/s/iy09j7mm2hrsf03/bricks-02.5677.dump.1460705370.gz?dl=0
>
>
> On Thu, Apr 14, 2016 at 8:07 PM, Ashish Pandey <aspan...@redhat.com> wrote:
>> Hi Serkan,
>>
>> Could you also provide us the statedump of all the brick processes and
>> clients?
>>
>> Commands to generate statedumps for brick processes/nfs server/quotad
>>
>> For bricks: gluster volume statedump 
>>
>> For nfs server: gluster volume statedump  nfs
>>
>>
>> We can find the directory where statedump files are created using 'gluster
>> --print-statedumpdir'
>> Also, the mount logs would help us to debug the issue.
>>
>> Ashish
>>
>> 
>> From: "Serkan Çoban" <cobanser...@gmail.com>
>> To: "Gluster Users" <gluster-us...@gluster.org>, "Gluster Devel"
>> <gluster-devel@gluster.org>
>> Sent: Thursday, April 14, 2016 6:27:10 PM
>> Subject: Re: [Gluster-users] Assertion failed: ec_get_inode_size
>>
>>
>> Here is the related brick log:
>>
>> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700556] E
>> [inodelk.c:309:__inode_unlock_lock] 0-v0-locks:  Matching lock not
>> found for unlock 0-9223372036854775807, by 94d29e885e7f on
>> 0x7f037413b990
>> /var/log/glusterfs/bricks/bricks-02.log:[2016-04-14 11:31:25.700639] E
>> [MSGID: 115053] [server-rpc-fops.c:276:server_inodelk_cbk]
>> 0-v0-server: 712984: INODELK
>> /workdir/raw_output/xxx/yyy/zzz.dat.gz.snappy1460474606605
>> (1191e32e-44ba-4e20-87ca-35ace8519c19) ==> (Invalid argument) [Invalid
>> argument]
>>
>> On Thu, Apr 14, 2016 at 3:25 PM, Serkan Çoban <cobanser...@gmail.com> wrote:
>>> Hi,
>>>
>>> During read/write tests to a 78x(16+4) distributed disperse volume
>>> from 50 clients, One clients hangs on read/write with the following
>>> logs:
>>>
>>> [2016-04-14 11:11:04.728580] W [MSGID: 122056]
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-6: Mismatching xdata
>>> in answers of 'LOOKUP'
>>> [2016-04-14 11:11:04.728624] W [MSGID: 122053]
>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-6: Operation failed on
>>> some subvolumes (up=F, mask=F, remaining=0, good=D,
>>> bad=2)
>>> [2016-04-14 11:11:04.736689] I [MSGID: 122058]
>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-6: /workdir/raw_output2:
>>> name heal successful on F
>>> [2016-04-14 11:29:26.718036] W [MSGID: 122056]
>>> [ec-combine.c:866:ec_combine_check] 0-v0-disperse-1: Mismatching xdata
>>> in answers of 'LOOKUP'
>>> [2016-04-14 11:29:26.718121] W [MSGID: 122053]
>>> [ec-common.c:116:ec_check_status] 0-v0-disperse-1: Operation failed on
>>> some subvolumes (up=F, mask=F, remaining=0, good=E,
>>> bad=1)
>>> [2016-04-14 11:29:42.501760] I [MSGID: 122058]
>>> [ec-heal.c:2340:ec_heal_do] 0-v0-disperse-1: /workdir/raw_output2:
>>> name heal successful on F
>>> [2016-04-14 11:31:25.714812] E [ec-inode-read.c:1612:ec_manager_stat]
>>> (-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_resume+0x91)
>>> [0x7f5ec9f942b1]
>>>
>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(__ec_manager+0x57)
>>> [0x7f5ec9f94497]
>>>
>>> -->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_stat+0x2c4)
>>> [0x7f5ec9faaed4] ) 0-: Assertion failed: ec_get_inode_size(fop,
>>> fop->locks[0].lock->loc.inode, >iatt[0].ia_size)
>>> [2016-04-14 11:31:25.722372] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-40: remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722411] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-41: remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722450] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-44: remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722477] E [MSGID: 114031]
>>> [client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-42: remote
>>> operation failed [Invalid argument]
>>> [2016-04-14 11:31:25.722503] E [MSGID: 114031]
>>> [client-rpc-fops.c:

Re: [Gluster-devel] [Gluster-users] Assertion failed: ec_get_inode_size

[Serkan Çoban]

Yes it is only one brick which error appears. I can send all other
brick dumps too..
How can I get state dump in fuse client? There is no gluster command there..
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Assertion failed: ec_get_inode_size

[Serkan Çoban]

Hi,

During read/write tests to a 78x(16+4) distributed disperse volume
from 50 clients, One clients hangs on read/write with the following
logs:

[2016-04-14 11:11:04.728580] W [MSGID: 122056]
[ec-combine.c:866:ec_combine_check] 0-v0-disperse-6: Mismatching xdata
in answers of 'LOOKUP'
[2016-04-14 11:11:04.728624] W [MSGID: 122053]
[ec-common.c:116:ec_check_status] 0-v0-disperse-6: Operation failed on
some subvolumes (up=F, mask=F, remaining=0, good=D,
bad=2)
[2016-04-14 11:11:04.736689] I [MSGID: 122058]
[ec-heal.c:2340:ec_heal_do] 0-v0-disperse-6: /workdir/raw_output2:
name heal successful on F
[2016-04-14 11:29:26.718036] W [MSGID: 122056]
[ec-combine.c:866:ec_combine_check] 0-v0-disperse-1: Mismatching xdata
in answers of 'LOOKUP'
[2016-04-14 11:29:26.718121] W [MSGID: 122053]
[ec-common.c:116:ec_check_status] 0-v0-disperse-1: Operation failed on
some subvolumes (up=F, mask=F, remaining=0, good=E,
bad=1)
[2016-04-14 11:29:42.501760] I [MSGID: 122058]
[ec-heal.c:2340:ec_heal_do] 0-v0-disperse-1: /workdir/raw_output2:
name heal successful on F
[2016-04-14 11:31:25.714812] E [ec-inode-read.c:1612:ec_manager_stat]
(-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_resume+0x91)
[0x7f5ec9f942b1]
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(__ec_manager+0x57)
[0x7f5ec9f94497]
-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_stat+0x2c4)
[0x7f5ec9faaed4] ) 0-: Assertion failed: ec_get_inode_size(fop,
fop->locks[0].lock->loc.inode, >iatt[0].ia_size)
[2016-04-14 11:31:25.722372] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-40: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722411] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-41: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722450] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-44: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722477] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-42: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722503] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-43: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722577] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-45: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722605] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-46: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722742] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-49: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722794] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-47: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722818] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-48: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722840] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-50: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722883] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-52: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722906] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-54: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722958] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-51: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.722983] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-55: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.723037] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-56: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.723045] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-53: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.725044] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-59: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.741338] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-58: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.746602] E [MSGID: 114031]
[client-rpc-fops.c:1624:client3_3_inodelk_cbk] 0-v0-client-57: remote
operation failed [Invalid argument]
[2016-04-14 11:31:25.746629] W [MSGID: 122015]
[ec-common.c:1675:ec_unlocked] 0-v0-disperse-2: entry/inode unlocking
failed (FSTAT) [Invalid argument]
[2016-04-14 11:31:25.746687] E [ec-common.c:1639:ec_lock_unfreeze]
(-->/usr/lib64/glusterfs/3.7.10/xlator/cluster/disperse.so(ec_manager_inodelk+0x2ae)
[0x7f5ec9fa009e]

Re: [Gluster-devel] [Gluster-users] GlusterFS FUSE client leaks summary — part I

[Serkan Çoban]

Will those patches be available in 3.7.7?

On Mon, Feb 1, 2016 at 1:28 PM, Soumya Koduri  wrote:
>
>
> On 02/01/2016 02:48 PM, Xavier Hernandez wrote:
>>
>> Hi,
>>
>> On 01/02/16 09:54, Soumya Koduri wrote:
>>>
>>>
>>>
>>> On 02/01/2016 01:39 PM, Oleksandr Natalenko wrote:

 Wait. It seems to be my bad.

 Before unmounting I do drop_caches (2), and glusterfs process CPU usage
 goes to 100% for a while. I haven't waited for it to drop to 0%, and
 instead perform unmount. It seems glusterfs is purging inodes and that's
 why it uses 100% of CPU. I've re-tested it, waiting for CPU usage to
 become normal, and got no leaks.

 Will verify this once again and report more.

 BTW, if that works, how could I limit inode cache for FUSE client? I do
 not want it to go beyond 1G, for example, even if I have 48G of RAM on
 my server.
>>>
>>>
>>> Its hard-coded for now. For fuse the lru limit (of the inodes which are
>>> not active) is (32*1024).
>>
>>
>> This is not exact for current implementation. The inode memory pool is
>> configured with 32*1024 entries, but the lru limit is set to infinite:
>> currently inode_table_prune() takes lru_limit == 0 as infinite, and the
>> inode table created by fuse is initialized with 0.
>>
>> Anyway this should not be a big problem in normal conditions. After
>> having fixed the incorrect nlookup count for "." and ".." directory
>> entries, when the kernel detects memory pressure and sends inode
>> forgets, the memory will be released.
>>
>>> One of the ways to address this (which we were discussing earlier) is to
>>> have an option to configure inode cache limit.
>>
>>
>> I think this will need more thinking. I've made a fast test forcing
>> lru_limit to a small value and weird errors have appeared (probably from
>> inodes being expected to exist when kernel sends new requests). Anyway I
>> haven't spent time on this. I haven't tested in on master either.
>
>
> Oh okay. Thanks for checking.
>
> -Soumya
>
>
>>
>> Xavi
>>
>>> If that sounds good, we
>>> can then check on if it has to be global/volume-level,
>>> client/server/both.
>>>
>>> Thanks,
>>> Soumya
>>>

 01.02.2016 09:54, Soumya Koduri написав:
>
> On 01/31/2016 03:05 PM, Oleksandr Natalenko wrote:
>>
>> Unfortunately, this patch doesn't help.
>>
>> RAM usage on "find" finish is ~9G.
>>
>> Here is statedump before drop_caches: https://gist.github.com/
>> fc1647de0982ab447e20
>
>
> [mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
> size=706766688
> num_allocs=2454051
>
>>
>> And after drop_caches: https://gist.github.com/5eab63bc13f78787ed19
>
>
> [mount/fuse.fuse - usage-type gf_common_mt_inode_ctx memusage]
> size=550996416
> num_allocs=1913182
>
> There isn't much significant drop in inode contexts. One of the
> reasons could be because of dentrys holding a refcount on the inodes
> which shall result in inodes not getting purged even after
> fuse_forget.
>
>
> pool-name=fuse:dentry_t
> hot-count=32761
>
> if  '32761' is the current active dentry count, it still doesn't seem
> to match up to inode count.
>
> Thanks,
> Soumya
>>
>>
>> And here is Valgrind output:
>> https://gist.github.com/2490aeac448320d98596
>>
>> On субота, 30 січня 2016 р. 22:56:37 EET Xavier Hernandez wrote:
>>>
>>> There's another inode leak caused by an incorrect counting of
>>> lookups on directory reads.
>>>
>>> Here's a patch that solves the problem for
>>> 3.7:
>>>
>>> http://review.gluster.org/13324
>>>
>>> Hopefully with this patch the
>>> memory leaks should disapear.
>>>
>>> Xavi
>>>
>>> On 29.01.2016 19:09, Oleksandr
>>>
>>> Natalenko wrote:

 Here is intermediate summary of current memory
>>>
>>>
>>> leaks in FUSE client
>>>
 investigation.

 I use GlusterFS v3.7.6
>>>
>>>
>>> release with the following patches:

 ===
>>>
>>>
 Kaleb S KEITHLEY (1):
>>>
>>> fuse: use-after-free fix in fuse-bridge, revisited
>>>
 Pranith Kumar K
>>>
>>>
>>> (1):

 mount/fuse: Fix use-after-free crash
>>>
>>>
 Soumya Koduri (3):
>>>
>>> gfapi: Fix inode nlookup counts
>>>
 inode: Retire the inodes from the lru
>>>
>>>
>>> list in inode_table_destroy
>>>
 upcall: free the xdr* allocations
 ===


 With those patches we got API leaks fixed (I hope, brief tests show
>>>
>>>
>>> that) and
>>>
 got rid of "kernel notifier loop terminated" message.
>>>
>>>
>>> Nevertheless, FUSE
>>>
 client still leaks.

 I have several test
>>>