Re: [Gluster-users] Proposal to mark few features as Deprecated / SunSet from Version 5.0

[Hans Henrik Happe]

On 19/03/2019 14.10, Amar Tumballi Suryanarayan wrote:
> Hi Hans,
>
> Thanks for the honest feedback. Appreciate this.
>
> On Tue, Mar 19, 2019 at 5:39 PM Hans Henrik Happe  <mailto:ha...@nbi.dk>> wrote:
>
> Hi,
>
> Looking into something else I fell over this proposal. Being a
> shop that are going into "Leaving GlusterFS" mode, I thought I
> would give my two cents.
>
> While being partially an HPC shop with a few Lustre filesystems, 
> we chose GlusterFS for an archiving solution (2-3 PB), because we
> could find files in the underlying ZFS filesystems if GlusterFS
> went sour.
>
> We have used the access to the underlying files plenty, because of
> the continuous instability of GlusterFS'. Meanwhile, Lustre have
> been almost effortless to run and mainly for that reason we are
> planning to move away from GlusterFS.
>
> Reading this proposal kind of underlined that "Leaving GluserFS"
> is the right thing to do. While I never understood why GlusterFS
> has been in feature crazy mode instead of stabilizing mode, taking
> away crucial features I don't get. With RoCE, RDMA is getting
> mainstream. Quotas are very useful, even though the current
> implementation are not perfect. Tiering also makes so much sense,
> but, for large files, not on a per-file level.
>
>
> It is a right concern to raise, and removing the existing features is
> not a good thing most of the times. But, one thing we noticed over the
> years is, the features which we develop, and not take to completion
> cause the major heart-burn. People think it is present, and it is
> already few years since its introduced, but if the developers are not
> working on it, users would always feel that the product doesn't work,
> because that one feature didn't work. 
>
> Other than Quota in the proposal email, for all other features, even
> though we have *some* users, we are inclined towards deprecating them,
> considering projects overall goals of stability in the longer run.
>  
>
> To be honest we only use quotas. We got scared of trying out new
> performance features that potentially would open up a new back of
> issues.
>
> About Quota, we heard enough voices, so we will make sure we keep it.
> The original email was 'Proposal', and hence these opinions matter for
> decision.
>
> Sorry for being such a buzzkill. I really wanted it to be different.
>
> We hear you. Please let us know one thing, which were the versions you
> tried ?
>
We started at 3.6 4 years ago. Now we are at 3.12.15, working towards
moving to 4.1.latest.
> We hope in coming months, our recent focus on Stability and Technical
> debt reduction will help you to re-look at Gluster after sometime.
That's great to hear.
>
> Cheers,
> Hans Henrik
>
> On 19/07/2018 08.56, Amar Tumballi wrote:
>> *
>>
>> Hi all,
>>
>> Over last 12 years of Gluster, we have developed many features,
>> and continue to support most of it till now. But along the way,
>> we have figured out better methods of doing things. Also we are
>> not actively maintaining some of these features.
>>
>> We are now thinking of cleaning up some of these ‘unsupported’
>> features, and mark them as ‘SunSet’ (i.e., would be totally taken
>> out of codebase in following releases) in next upcoming release,
>> v5.0. The release notes will provide options for smoothly
>> migrating to the supported configurations.
>>
>> If you are using any of these features, do let us know, so that
>> we can help you with ‘migration’.. Also, we are happy to guide
>> new developers to work on those components which are not actively
>> being maintained by current set of developers.
>>
>>
>>   List of features hitting sunset:
>>
>>
>> ‘cluster/stripe’ translator:
>>
>> This translator was developed very early in the evolution of
>> GlusterFS, and addressed one of the very common question of
>> Distributed FS, which is “What happens if one of my file is
>> bigger than the available brick. Say, I have 2 TB hard drive,
>> exported in glusterfs, my file is 3 TB”. While it solved the
>> purpose, it was very hard to handle failure scenarios, and give a
>> real good experience to our users with this feature. Over the
>> time, Gluster solved the problem with it’s ‘Shard’ feature, which
>> solves the problem in much better way, and provides much better
>> solution with existing well supported stack. H

Re: [Gluster-users] Proposal to mark few features as Deprecated / SunSet from Version 5.0

[Hans Henrik Happe]

Hi,

Looking into something else I fell over this proposal. Being a shop that
are going into "Leaving GlusterFS" mode, I thought I would give my two
cents.

While being partially an HPC shop with a few Lustre filesystems,  we
chose GlusterFS for an archiving solution (2-3 PB), because we could
find files in the underlying ZFS filesystems if GlusterFS went sour.

We have used the access to the underlying files plenty, because of the
continuous instability of GlusterFS'. Meanwhile, Lustre have been almost
effortless to run and mainly for that reason we are planning to move
away from GlusterFS.

Reading this proposal kind of underlined that "Leaving GluserFS" is the
right thing to do. While I never understood why GlusterFS has been in
feature crazy mode instead of stabilizing mode, taking away crucial
features I don't get. With RoCE, RDMA is getting mainstream. Quotas are
very useful, even though the current implementation are not perfect.
Tiering also makes so much sense, but, for large files, not on a
per-file level.

To be honest we only use quotas. We got scared of trying out new
performance features that potentially would open up a new back of issues.

Sorry for being such a buzzkill. I really wanted it to be different.

Cheers,
Hans Henrik

On 19/07/2018 08.56, Amar Tumballi wrote:
> *
>
> Hi all,
>
> Over last 12 years of Gluster, we have developed many features, and
> continue to support most of it till now. But along the way, we have
> figured out better methods of doing things. Also we are not actively
> maintaining some of these features.
>
> We are now thinking of cleaning up some of these ‘unsupported’
> features, and mark them as ‘SunSet’ (i.e., would be totally taken out
> of codebase in following releases) in next upcoming release, v5.0. The
> release notes will provide options for smoothly migrating to the
> supported configurations.
>
> If you are using any of these features, do let us know, so that we can
> help you with ‘migration’.. Also, we are happy to guide new developers
> to work on those components which are not actively being maintained by
> current set of developers.
>
>
>   List of features hitting sunset:
>
>
> ‘cluster/stripe’ translator:
>
> This translator was developed very early in the evolution of
> GlusterFS, and addressed one of the very common question of
> Distributed FS, which is “What happens if one of my file is bigger
> than the available brick. Say, I have 2 TB hard drive, exported in
> glusterfs, my file is 3 TB”. While it solved the purpose, it was very
> hard to handle failure scenarios, and give a real good experience to
> our users with this feature. Over the time, Gluster solved the problem
> with it’s ‘Shard’ feature, which solves the problem in much better
> way, and provides much better solution with existing well supported
> stack. Hence the proposal for Deprecation.
>
> If you are using this feature, then do write to us, as it needs a
> proper migration from existing volume to a new full supported volume
> type before you upgrade.
>
>
> ‘storage/bd’ translator:
>
> This feature got into the code base 5 years back with this patch
> [1]. Plan was to use a block device
> directly as a brick, which would help to handle disk-image storage
> much easily in glusterfs.
>
> As the feature is not getting more contribution, and we are not seeing
> any user traction on this, would like to propose for Deprecation.
>
> If you are using the feature, plan to move to a supported gluster
> volume configuration, and have your setup ‘supported’ before upgrading
> to your new gluster version.
>
>
> ‘RDMA’ transport support:
>
> Gluster started supporting RDMA while ib-verbs was still new, and very
> high-end infra around that time were using Infiniband. Engineers did
> work with Mellanox, and got the technology into GlusterFS for better
> data migration, data copy. While current day kernels support very good
> speed with IPoIB module itself, and there are no more bandwidth for
> experts in these area to maintain the feature, we recommend migrating
> over to TCP (IP based) network for your volume.
>
> If you are successfully using RDMA transport, do get in touch with us
> to prioritize the migration plan for your volume. Plan is to work on
> this after the release, so by version 6.0, we will have a cleaner
> transport code, which just needs to support one type.
>
>
> ‘Tiering’ feature
>
> Gluster’s tiering feature which was planned to be providing an option
> to keep your ‘hot’ data in different location than your cold data, so
> one can get better performance. While we saw some users for the
> feature, it needs much more attention to be completely bug free. At
> the time, we are not having any active maintainers for the feature,
> and hence suggesting to take it out of the ‘supported’ tag.
>
> If you are willing to take it up, and maintain it, do let us know, and
> we are happy to assist you.
>
> If you 

Re: [Gluster-users] Files from one brick missing from readdir

[Hans Henrik Happe]

Hi Nithya,

Now, we have the same situation as we did the last time. It fixed itself.

Do you have any insights into what might trigger a fix. It must be
related to use of the dirs, but it's almost 24 hours since we started
poking around in that path.

According the the rebalance log the dir has not been touched.

Cheers,
Hans Henrik

On 09-07-2018 11:25, Nithya Balachandran wrote:
> Hi Hans,
> 
> Never mind - I found it. It looks like the same problem as reported by
> the other user. In both cases, this is a pure distribute volume.
> 
> See packet 154.The iatt is null for all entries. It looks like a
> .glusterfs gfid link is missing on that brick.
> 
> Would you prefer that I send the steps to recover in a private email?
> 
> Regards,
> Nithya
> 
> 
> On 9 July 2018 at 14:49, Nithya Balachandran  <mailto:nbala...@redhat.com>> wrote:
> 
> Or even better, the brick on which those files exist and the gluster
> volume status output for the volume.
> 
> Thanks,
> Nithya
> 
> On 9 July 2018 at 14:42, Nithya Balachandran  <mailto:nbala...@redhat.com>> wrote:
> 
> Thanks Hans. What are the names of the "missing" files?
> 
> Regards,
> Nithya
> 
> On 9 July 2018 at 13:30, Nithya Balachandran
> mailto:nbala...@redhat.com>> wrote:
> 
> Hi Hans,
> 
> Another user has reported something similar and we are still
> debugging this.
> 
> Would you mind taking a tcpdump of the client while listing
> the directory from a FUSE client and sending it to me?
> Please use 
> tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22
> 
> 
> Also, please send the output of gluster volume info and
> gluster volume get  all.
> 
> Thanks,
> Nithya
> 
> 
> On 9 July 2018 at 12:51, Hans Henrik Happe  <mailto:ha...@nbi.dk>> wrote:
> 
> Hi,
> 
> After an upgrade from 3.7 -> 3.10 -> 3.12.9 that seemed
> to go smoothly,
> we have experienced missing files and dirs when listing
> directories.
> 
> We are using a distributed setup with 20 bricks (no
> redundance from
> glusterfs).
> 
> The dirs and files can be referenced directly, but does
> not show up in
> listings (readdir, i.e. ls). Renaming them works, but
> they still does
> not show up.
> 
> The first time we discovered this, we noticed that files
> slowly
> reappeared and finally all were there. After that we
> started a
> fix-layout which is still running (5mio dirs). After
> this we would
> compare brick files to the mounted fs.
> 
> Yesterday we again discovered some missing files in a
> dir. After some
> poking around we found that all missing files were
> located on the same
> brick.
> 
> Comparing dir xattr did not give us a clue:
> 
> 
> Brick with missing files:
> 
> # getfattr  -m . -d -e hex backup
> # file: backup
> trusted.gfid=0x8613f6e0317141918b42d8c8063ffbce
> trusted.glusterfs.dht=0x0001b2169fa3bf11b4e7
> 
> trusted.glusterfs.quota.6e0ab807-6eed-4af1-92b8-0db3ca7a19e0.contri.1=0xd65e34750001
> trusted.glusterfs.quota.dirty=0x3000
> 
> trusted.glusterfs.quota.size.1=0xd65e34750001
> 
> Other brick:
> 
> # getfattr  -m . -d -e hex backup
> # file: backup
> trusted.gfid=0x8613f6e0317141918b42d8c8063ffbce
> trusted.glusterfs.dht=0x0001bf11b4e8cbe5e488
> 
> trusted.glusterfs.quota.6e0ab807-6eed-4af1-92b8-0db3ca7a19e0.contri.1=0xb03aa871
> trusted.glusterfs.quota.dirty=0x3000
> 
> trusted.glusterfs.quota.size.1=0xb03aa871
> 
> 
> Anyone who experienced this or have some clues to what
> might be wrong?
> 
> Cheers,
> Hans Henrik
> _

[Gluster-users] Files from one brick missing from readdir

[Hans Henrik Happe]

Hi,

After an upgrade from 3.7 -> 3.10 -> 3.12.9 that seemed to go smoothly,
we have experienced missing files and dirs when listing directories.

We are using a distributed setup with 20 bricks (no redundance from
glusterfs).

The dirs and files can be referenced directly, but does not show up in
listings (readdir, i.e. ls). Renaming them works, but they still does
not show up.

The first time we discovered this, we noticed that files slowly
reappeared and finally all were there. After that we started a
fix-layout which is still running (5mio dirs). After this we would
compare brick files to the mounted fs.

Yesterday we again discovered some missing files in a dir. After some
poking around we found that all missing files were located on the same
brick.

Comparing dir xattr did not give us a clue:


Brick with missing files:

# getfattr  -m . -d -e hex backup
# file: backup
trusted.gfid=0x8613f6e0317141918b42d8c8063ffbce
trusted.glusterfs.dht=0x0001b2169fa3bf11b4e7
trusted.glusterfs.quota.6e0ab807-6eed-4af1-92b8-0db3ca7a19e0.contri.1=0xd65e34750001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size.1=0xd65e34750001

Other brick:

# getfattr  -m . -d -e hex backup
# file: backup
trusted.gfid=0x8613f6e0317141918b42d8c8063ffbce
trusted.glusterfs.dht=0x0001bf11b4e8cbe5e488
trusted.glusterfs.quota.6e0ab807-6eed-4af1-92b8-0db3ca7a19e0.contri.1=0xb03aa871
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size.1=0xb03aa871


Anyone who experienced this or have some clues to what might be wrong?

Cheers,
Hans Henrik
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Release 3.12.5: Scheduled for the 12th of January

[Hans Henrik Happe]

Hi,

I wonder how this procedure works. I could add a bug that I think is a
*blocker*, but there might not be consensus.

Cheers,
Hans Henrik

On 11-01-2018 07:02, Jiffin Tony Thottan wrote:
> Hi,
> 
> It's time to prepare the 3.12.5 release, which falls on the 10th of
> each month, and hence would be 12-01-2018 this time around.
> 
> This mail is to call out the following,
> 
> 1) Are there any pending *blocker* bugs that need to be tracked for
> 3.12.5? If so mark them against the provided tracker [1] as blockers
> for the release, or at the very least post them as a response to this
> mail
> 
> 2) Pending reviews in the 3.12 dashboard will be part of the release,
> *iff* they pass regressions and have the review votes, so use the
> dashboard [2] to check on the status of your patches to 3.12 and get
> these going
> 
> 3) I have made checks on what went into 3.10 post 3.12 release and if
> these fixes are already included in 3.12 branch, then status on this is
> *green*
> as all fixes ported to 3.10, are ported to 3.12 as well.
> 
> Thanks,
> Jiffin
> 
> [1] Release bug tracker:
> https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-3.12.5
> 
> [2] 3.12 review dashboard:
> https://review.gluster.org/#/projects/glusterfs,dashboards/dashboard:3-12-dashboard
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Subvolume failure

[Hans Henrik Happe]

Hi,

I have always found the default behaviour of subvolume failure a bit
confusing for users or applications that need deterministic failure
behaviour.

When a subvolume fails the clients can still use and access files on
other subvolumes. However, accessing files on a failed subvolume returns
an error saying it cannot find the files.

It would be better if it would block until a subvolume comes back online.

An alternative would be that all access would fail when one subvolume is
down.

Is there a way to configure this?

Cheers,
Hans Henrik
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Memory Leakage in Gluster 3.10.2-1

[Hans Henrik Happe]

Hi,

I just filled this bug, which seems to be related:

https://bugzilla.redhat.com/show_bug.cgi?id=1509071

Cheers,
Hans Henrik

On 27-07-2017 15:53, Mohammed Rafi K C wrote:
> Are you still facing the problem ? If so, Can you please provide the
> workload , cmd_log_history file, log files , etc ?
> 
> 
> Regards
> 
> Rafi KC
> 
> 
> On 06/23/2017 02:06 PM, shridhar s n wrote:
>> Hi All,
>>
>> We are using GlusterFS 3.10.2 (upgraded from 3.7.0 last week) on
>> CentOS 7.x .
>>
>> We continue to see memory utilization going up once every 3 days. The
>> memory utilization of the server demon(glusterd) in  server is keep on
>> increasing. In about 30+ hours the Memory utilization of glusterd
>> service alone will reach 70% of memory available. Since we have alarms
>> for this threshold, we get notified and only way to stop it so far is
>> to restart the glusterd. 
>>
>> The GlusterFS is configured in the two server nodes with replica option.
>>
>> Kindly let us know how to fix this memory leakage.
>>
>> Thanks in advance,
>>
>> Shridhar S N
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [URGENT] Add-bricks to a volume corrupted the files

[Hans Henrik Happe]

Hi,

This is scary stuff. While not as scary, you might confirm a bug that I 
reported a while back on your test systems:


https://bugzilla.redhat.com/show_bug.cgi?id=1370832

Cheers,
Hans Henrik



On 19-10-2016 08:40, Krutika Dhananjay wrote:

Agreed.
I will run the same test on an actual vm setup one of these days and
see if I manage to recreate the issue (after I have completed some
of my long pending tasks). Meanwhile if any of you find a consistent simpler
test case to hit the issue, feel free to reply on this thread. At least
I had no success
in recreating the bug in a non-VM-store setup.

-Krutika

On Mon, Oct 17, 2016 at 12:50 PM, Gandalf Corvotempesta
> wrote:

Il 14 ott 2016 17:37, "David Gossage" > ha scritto:
>
> Sorry to resurrect an old email but did any resolution occur for this or 
a cause found?  I just see this as a potential task I may need to also run through 
some day and if their are pitfalls to watch for would be good to know.
>

I think that the issue wrote in these emails must be addressed in
some way.
It's really bad that adding bricks to a cluster lead to data
corruption as adding bricks is a standard administration task

I hope that the issue will be detected and fixed asap.


___
Gluster-users mailing list
Gluster-users@gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users





___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Directory c+mtime changed after add-brick and fix-layout

[Hans Henrik Happe]

Hi,

I have now tried on a test fs, to see if I could recreate. First a bit 
more info.


OS: CentOS 6.8
Gluster: 3.7.13 using centos-release-gluster37
Underlying FS (1st report): ZFS 0.6.5.7
Underlying FS (test): EXT4

# gluster vol info test

Volume Name: test
Type: Distribute
Volume ID: 291cab24-269a-4f99-a8f3-81e2c7ad6dd7
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: scistor01.science:/gluster/brick
Options Reconfigured:
performance.readdir-ahead: on


Procedure to recreate:

server: gluster vol create test 
server: gluster vol start test
client: (mount in /mnt/test)
client: mkdir foo
client: (check foo mtime)
server: gluster vol add-brick test 
client: (foo mtime now changed)
server: gluster vol remove-brick test  start
server: gluster vol remove-brick test  commit
client: (foo mtime now back to expected time)

So removing the brick again give us the right mtime again. Also,
doing this with one or two servers does not matter.

Cheers,
Hans Henrik

On 22-08-2016 15:47, Hans Henrik Happe wrote:

Hi,

We see a lot of directories that got their c+mtime updated after we
added bricks and ran a rebalance fix-layout.

On the new bricks the dirs have the current time from when fix-layout
ran. On the old they seem to be as they are supposed to. On clients some
dirs are showing the new time.

Is this a known issue and is there anything we can do about it.

Version: 3.7.13

Cheers,
Hans Henrik Happe


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Directory c+mtime changed after add-brick and fix-layout

[Hans Henrik Happe]

Hi,

We see a lot of directories that got their c+mtime updated after we 
added bricks and ran a rebalance fix-layout.


On the new bricks the dirs have the current time from when fix-layout 
ran. On the old they seem to be as they are supposed to. On clients some 
dirs are showing the new time.


Is this a known issue and is there anything we can do about it.

Version: 3.7.13

Cheers,
Hans Henrik Happe


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Missing brick behaviour

[Hans Henrik Happe]

Hi,

I'm having a distributed volume, so no replication *.

When a brick becomes missing for some reason, it is still possible to 
access files on remaining bricks, while writes that hash to the missing 
will fail. It is kind of sane, but can be a bit confusing to users.


A few other ways to handle this scenario would be nice:

- Make the whole volume read-only.
- Block I/O to missing brick until it becomes available again.
- Just block all I/O until missing brick comes back.

Is any of these possible?

Cheers,
Hans Henrik

*) Motivation: Replication is to wasteful. Also, we have 3PB behind 6 
servers, so dispersed volumes do not scale down in our case (still 
wasteful). We are okay with the occasional downtime and happy that the 
files are directly available on the underlying file system in case 
gluster blows up.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] option transport.socket.bind-address

[Hans Henrik Happe]

Hi,

If I set,

option transport.socket.bind-address x.x.x.x

in glusterd.vol, then rebalance will fail. This old thread is suggesting 
a patch to solve is:


http://www.gluster.org/pipermail/gluster-users/2013-November/014940.html

Basically, the problem is that the daemons has localhost hard-coded. 
Also, the --remote-host option does not help (silently exiting with the 
rebalance command). I'm using 3.6.2-1, so I'm wondering what the status 
of this issue is?


Cheers,
Hans Henrik
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users