Re: [Gluster-Maintainers] Fwd: Build failed in Jenkins: regression-test-with-multiplex #1359

[Anoop C S]

On Fri, 2019-06-07 at 10:06 +0530, Amar Tumballi Suryanarayan wrote:
> Got time to test subdir-mount.t failing in brick-mux scenario.
> 
> I noticed some issues, where I need further help from glusterd team.
> 
> subdir-mount.t expects 'hook' script to run after add-brick to make
> sure the required subdirectories are healed and are present in new
> bricks. This is important as subdir mount expects the subdirs to
> exist for successful mount.
> 
> But in case of brick-mux setup, I see that in some cases (6/10), hook
> script (add-brick/post-hook/S13-create-subdir-mount.sh) started
> getting executed after 20second of finishing the add-brick command.
> Due to this, the mount which we execute after add-brick failed.
> 
> My question is, what is making post hook script to run so late ??

Note:-
I would like to add that I have a patch[1] under review adding another
post add-brick hook script which might further result in delayed
execution of S13create-subdir-mounts.sh. Because S10selinux-label-
brick.sh from my change comes before the existing hook script.

[1] https://review.gluster.org/c/glusterfs/+/22834

> I can recreate the issues locally on my laptop too. 
> 
> 
> On Sat, Jun 1, 2019 at 4:55 PM Atin Mukherjee 
> wrote:
> > subdir-mount.t has started failing in brick mux regression nightly.
> > This needs to be fixed.
> > 
> > Raghavendra - did we manage to get any further clue on uss.t
> > failure?
> > 
> > -- Forwarded message -
> > From: 
> > Date: Fri, 31 May 2019 at 23:34
> > Subject: [Gluster-Maintainers] Build failed in Jenkins: regression-
> > test-with-multiplex #1359
> > To: , , <
> > ama...@redhat.com>, , 
> > 
> > 
> > See <
> > https://build.gluster.org/job/regression-test-with-multiplex/1359/display/redirect?page=changes
> > >
> > 
> > Changes:
> > 
> > [atin] glusterd: add an op-version check
> > 
> > [atin] glusterd/svc: glusterd_svcs_stop should call individual
> > wrapper function
> > 
> > [atin] glusterd/svc: Stop stale process using the
> > glusterd_proc_stop
> > 
> > [Amar Tumballi] lcov: more coverage to shard, old-protocol, sdfs
> > 
> > [Kotresh H R] tests/geo-rep: Add EC volume test case
> > 
> > [Amar Tumballi] glusterfsd/cleanup: Protect graph object under a
> > lock
> > 
> > [Mohammed Rafi KC] glusterd/shd: Optimize the glustershd manager to
> > send reconfigure
> > 
> > [Kotresh H R] tests/geo-rep: Add tests to cover glusterd geo-rep
> > 
> > [atin] glusterd: Optimize code to copy dictionary in handshake code
> > path
> > 
> > --
> > [...truncated 3.18 MB...]
> > ./tests/basic/afr/stale-file-lookup.t  -  9 second
> > ./tests/basic/afr/granular-esh/replace-brick.t  -  9 second
> > ./tests/basic/afr/granular-esh/add-brick.t  -  9 second
> > ./tests/basic/afr/gfid-mismatch.t  -  9 second
> > ./tests/performance/open-behind.t  -  8 second
> > ./tests/features/ssl-authz.t  -  8 second
> > ./tests/features/readdir-ahead.t  -  8 second
> > ./tests/bugs/upcall/bug-1458127.t  -  8 second
> > ./tests/bugs/transport/bug-873367.t  -  8 second
> > ./tests/bugs/replicate/bug-1498570-client-iot-graph-check.t  -  8
> > second
> > ./tests/bugs/replicate/bug-1132102.t  -  8 second
> > ./tests/bugs/quota/bug-1250582-volume-reset-should-not-remove-
> > quota-quota-deem-statfs.t  -  8 second
> > ./tests/bugs/quota/bug-1104692.t  -  8 second
> > ./tests/bugs/posix/bug-1360679.t  -  8 second
> > ./tests/bugs/posix/bug-1122028.t  -  8 second
> > ./tests/bugs/nfs/bug-1157223-symlink-mounting.t  -  8 second
> > ./tests/bugs/glusterfs/bug-861015-log.t  -  8 second
> > ./tests/bugs/glusterd/sync-post-glusterd-restart.t  -  8 second
> > ./tests/bugs/glusterd/bug-1696046.t  -  8 second
> > ./tests/bugs/fuse/bug-983477.t  -  8 second
> > ./tests/bugs/ec/bug-1227869.t  -  8 second
> > ./tests/bugs/distribute/bug-1088231.t  -  8 second
> > ./tests/bugs/distribute/bug-1086228.t  -  8 second
> > ./tests/bugs/cli/bug-1087487.t  -  8 second
> > ./tests/bugs/cli/bug-1022905.t  -  8 second
> > ./tests/bugs/bug-1258069.t  -  8 second
> > ./tests/bugs/bitrot/1209752-volume-status-should-show-bitrot-scrub-
> > info.t  -  8 second
> > ./tests/basic/xlator-pass-through-sanity.t  -  8 second
> > ./tests/basic/quota-nfs.t  -  8 second
> > ./tests/basic/glusterd/arbiter-volume.t  -  8 second
> > ./tests/basic/ctime/ctime-noatime.t  -  8 second
> > ./tests/line-coverage/cli-peer-and-volume-operations.t  -  7 second
> > ./tests/gfid2path/get-gfid-to-path.t  -  7 second
> > ./tests/bugs/upcall/bug-1369430.t  -  7 second
> > ./tests/bugs/snapshot/bug-1260848.t  -  7 second
> > ./tests/bugs/shard/shard-inode-refcount-test.t  -  7 second
> > ./tests/bugs/shard/bug-1258334.t  -  7 second
> > ./tests/bugs/replicate/bug-767585-gfid.t  -  7 second
> > ./tests/bugs/replicate/bug-1448804-check-quorum-type-values.t  -  7
> > second
> > ./tests/bugs/replicate/bug-1250170-fsync.t  -  7 second
> > ./tests/bugs/posix/bug-1175711.t  -  7 second
> > ./tests/bugs/nfs/bug-915280.t  -  7 

[Gluster-Maintainers] Build failed in Jenkins: regression-test-with-multiplex #1374

[jenkins]

See 


Changes:

[Xavi Hernandez] features/shard: Fix extra unref when inode object is lru'd out 
and added

--
[...truncated 3.85 MB...]
./tests/bugs/distribute/bug-1667804.t  -  10 second
./tests/bugs/distribute/bug-1086228.t  -  10 second
./tests/bugs/cli/bug-1022905.t  -  10 second
./tests/bugs/bitrot/1209752-volume-status-should-show-bitrot-scrub-info.t  -  
10 second
./tests/basic/stats-dump.t  -  10 second
./tests/basic/quota-nfs.t  -  10 second
./tests/basic/ios-dump.t  -  10 second
./tests/basic/fop-sampling.t  -  10 second
./tests/basic/ec/statedump.t  -  10 second
./tests/basic/ec/ec-root-heal.t  -  10 second
./tests/basic/afr/ta-read.t  -  10 second
./tests/basic/afr/gfid-mismatch.t  -  10 second
./tests/basic/afr/add-brick-self-heal.t  -  10 second
./tests/performance/open-behind.t  -  9 second
./tests/features/ssl-authz.t  -  9 second
./tests/features/readdir-ahead.t  -  9 second
./tests/bugs/upcall/bug-1458127.t  -  9 second
./tests/bugs/transport/bug-873367.t  -  9 second
./tests/bugs/replicate/bug-1221481-allow-fops-on-dir-split-brain.t  -  9 second
./tests/bugs/replicate/bug-1132102.t  -  9 second
./tests/bugs/replicate/bug-1037501.t  -  9 second
./tests/bugs/quota/bug-1104692.t  -  9 second
./tests/bugs/posix/bug-1122028.t  -  9 second
./tests/bugs/md-cache/setxattr-prepoststat.t  -  9 second
./tests/bugs/glusterfs/bug-872923.t  -  9 second
./tests/bugs/glusterfs/bug-848251.t  -  9 second
./tests/bugs/glusterd/bug-1696046.t  -  9 second
./tests/bugs/fuse/bug-983477.t  -  9 second
./tests/bugs/distribute/bug-1122443.t  -  9 second
./tests/bugs/cli/bug-1087487.t  -  9 second
./tests/bugs/changelog/bug-1208470.t  -  9 second
./tests/bugs/bug-1371806_2.t  -  9 second
./tests/bugs/bitrot/1209818-vol-info-show-scrub-process-properly.t  -  9 second
./tests/bugs/bitrot/1209751-bitrot-scrub-tunable-reset.t  -  9 second
./tests/basic/xlator-pass-through-sanity.t  -  9 second
./tests/basic/glusterd/arbiter-volume-probe.t  -  9 second
./tests/basic/ctime/ctime-noatime.t  -  9 second
./tests/basic/afr/ta.t  -  9 second
./tests/basic/afr/granular-esh/add-brick.t  -  9 second
./tests/basic/afr/afr-read-hash-mode.t  -  9 second
./tests/bugs/upcall/bug-1369430.t  -  8 second
./tests/bugs/snapshot/bug-1260848.t  -  8 second
./tests/bugs/shard/shard-inode-refcount-test.t  -  8 second
./tests/bugs/shard/bug-1258334.t  -  8 second
./tests/bugs/replicate/bug-767585-gfid.t  -  8 second
./tests/bugs/replicate/bug-1686568-send-truncate-on-arbiter-from-shd.t  -  8 
second
./tests/bugs/replicate/bug-1626994-info-split-brain.t  -  8 second
./tests/bugs/replicate/bug-1250170-fsync.t  -  8 second
./tests/bugs/replicate/bug-1101647.t  -  8 second
./tests/bugs/protocol/bug-1321578.t  -  8 second
./tests/bugs/posix/bug-1175711.t  -  8 second
./tests/bugs/nfs/bug-915280.t  -  8 second
./tests/bugs/md-cache/afr-stale-read.t  -  8 second
./tests/bugs/glusterfs/bug-895235.t  -  8 second
./tests/bugs/glusterd/quorum-value-check.t  -  8 second
./tests/bugs/ec/bug-1227869.t  -  8 second
./tests/bugs/distribute/bug-884597.t  -  8 second
./tests/bugs/distribute/bug-1368012.t  -  8 second
./tests/bugs/core/bug-986429.t  -  8 second
./tests/bugs/core/bug-1699025-brick-mux-detach-brick-fd-issue.t  -  8 second
./tests/bugs/bug-1258069.t  -  8 second
./tests/bugs/bitrot/1207029-bitrot-daemon-should-start-on-valid-node.t  -  8 
second
./tests/bitrot/br-stub.t  -  8 second
./tests/basic/md-cache/bug-1317785.t  -  8 second
./tests/basic/hardlink-limit.t  -  8 second
./tests/basic/gfapi/libgfapi-fini-hang.t  -  8 second
./tests/basic/fencing/fencing-crash-conistency.t  -  8 second
./tests/basic/ec/ec-read-policy.t  -  8 second
./tests/basic/distribute/file-create.t  -  8 second
./tests/basic/changelog/changelog-rename.t  -  8 second
./tests/basic/afr/ta-write-on-bad-brick.t  -  8 second
./tests/basic/afr/ta-shd.t  -  8 second
./tests/basic/afr/tarissue.t  -  8 second
./tests/basic/afr/heal-info.t  -  8 second
./tests/basic/afr/gfid-heal.t  -  8 second
./tests/gfid2path/get-gfid-to-path.t  -  7 second
./tests/bugs/upcall/bug-upcall-stat.t  -  7 second
./tests/bugs/snapshot/bug-1178079.t  -  7 second
./tests/bugs/snapshot/bug-1064768.t  -  7 second
./tests/bugs/shard/bug-1342298.t  -  7 second
./tests/bugs/shard/bug-1259651.t  -  7 second
./tests/bugs/replicate/bug-1325792.t  -  7 second
./tests/bugs/readdir-ahead/bug-1670253-consistent-metadata.t  -  7 second
./tests/bugs/quota/bug-1243798.t  -  7 second
./tests/bugs/nfs/bug-877885.t  -  7 second
./tests/bugs/nfs/bug-1143880-fix-gNFSd-auth-crash.t  -  7 second
./tests/bugs/md-cache/bug-1211863_unlink.t  -  7 second
./tests/bugs/io-cache/bug-858242.t  -  7 second
./tests/bugs/glusterfs/bug-893378.t  -  7 second
./tests/bugs/glusterd/bug-948729/bug-948729-force.t  -  7 second
./tests/bugs/glusterd/bug-1091935-brick-order-check-from-cli-to-glusterd.t  -  

Re: [Gluster-Maintainers] Fwd: Build failed in Jenkins: regression-test-with-multiplex #1359

[Atin Mukherjee]

On Fri, Jun 7, 2019 at 10:07 AM Amar Tumballi Suryanarayan <
atumb...@redhat.com> wrote:

> Got time to test subdir-mount.t failing in brick-mux scenario.
>
> I noticed some issues, where I need further help from glusterd team.
>
> subdir-mount.t expects 'hook' script to run after add-brick to make sure
> the required subdirectories are healed and are present in new bricks. This
> is important as subdir mount expects the subdirs to exist for successful
> mount.
>
> But in case of brick-mux setup, I see that in some cases (6/10), hook
> script (add-brick/post-hook/S13-create-subdir-mount.sh) started getting
> executed after 20second of finishing the add-brick command. Due to this,
> the mount which we execute after add-brick failed.
>
> My question is, what is making post hook script to run so late ??
>

It's not only the add-brick in the post hook. Given post hook scripts are
async in nature, I see the respective hook scripts of create/start/set
volume operation have executed quite a late which is very surprising until
and unless some thread has been stuck for quite a while. Unfortunately for
both Mohit and I, the issue isn't reproducible locally. Mohit would give it
a try in softserve infra but at this point of time, there's no conclusive
evidence, the analysis continues.

Amar - would it be possible for you to do a git blame given you can
reproduce this? May 31 nightly (
https://build.gluster.org/job/regression-test-with-multiplex/1359/) is when
this test started failing.


> I can recreate the issues locally on my laptop too.
>
>
> On Sat, Jun 1, 2019 at 4:55 PM Atin Mukherjee  wrote:
>
>> subdir-mount.t has started failing in brick mux regression nightly. This
>> needs to be fixed.
>>
>> Raghavendra - did we manage to get any further clue on uss.t failure?
>>
>> -- Forwarded message -
>> From: 
>> Date: Fri, 31 May 2019 at 23:34
>> Subject: [Gluster-Maintainers] Build failed in Jenkins:
>> regression-test-with-multiplex #1359
>> To: , , ,
>> , 
>>
>>
>> See <
>> https://build.gluster.org/job/regression-test-with-multiplex/1359/display/redirect?page=changes
>> >
>>
>> Changes:
>>
>> [atin] glusterd: add an op-version check
>>
>> [atin] glusterd/svc: glusterd_svcs_stop should call individual wrapper
>> function
>>
>> [atin] glusterd/svc: Stop stale process using the glusterd_proc_stop
>>
>> [Amar Tumballi] lcov: more coverage to shard, old-protocol, sdfs
>>
>> [Kotresh H R] tests/geo-rep: Add EC volume test case
>>
>> [Amar Tumballi] glusterfsd/cleanup: Protect graph object under a lock
>>
>> [Mohammed Rafi KC] glusterd/shd: Optimize the glustershd manager to send
>> reconfigure
>>
>> [Kotresh H R] tests/geo-rep: Add tests to cover glusterd geo-rep
>>
>> [atin] glusterd: Optimize code to copy dictionary in handshake code path
>>
>> --
>> [...truncated 3.18 MB...]
>> ./tests/basic/afr/stale-file-lookup.t  -  9 second
>> ./tests/basic/afr/granular-esh/replace-brick.t  -  9 second
>> ./tests/basic/afr/granular-esh/add-brick.t  -  9 second
>> ./tests/basic/afr/gfid-mismatch.t  -  9 second
>> ./tests/performance/open-behind.t  -  8 second
>> ./tests/features/ssl-authz.t  -  8 second
>> ./tests/features/readdir-ahead.t  -  8 second
>> ./tests/bugs/upcall/bug-1458127.t  -  8 second
>> ./tests/bugs/transport/bug-873367.t  -  8 second
>> ./tests/bugs/replicate/bug-1498570-client-iot-graph-check.t  -  8 second
>> ./tests/bugs/replicate/bug-1132102.t  -  8 second
>> ./tests/bugs/quota/bug-1250582-volume-reset-should-not-remove-quota-quota-deem-statfs.t
>> -  8 second
>> ./tests/bugs/quota/bug-1104692.t  -  8 second
>> ./tests/bugs/posix/bug-1360679.t  -  8 second
>> ./tests/bugs/posix/bug-1122028.t  -  8 second
>> ./tests/bugs/nfs/bug-1157223-symlink-mounting.t  -  8 second
>> ./tests/bugs/glusterfs/bug-861015-log.t  -  8 second
>> ./tests/bugs/glusterd/sync-post-glusterd-restart.t  -  8 second
>> ./tests/bugs/glusterd/bug-1696046.t  -  8 second
>> ./tests/bugs/fuse/bug-983477.t  -  8 second
>> ./tests/bugs/ec/bug-1227869.t  -  8 second
>> ./tests/bugs/distribute/bug-1088231.t  -  8 second
>> ./tests/bugs/distribute/bug-1086228.t  -  8 second
>> ./tests/bugs/cli/bug-1087487.t  -  8 second
>> ./tests/bugs/cli/bug-1022905.t  -  8 second
>> ./tests/bugs/bug-1258069.t  -  8 second
>> ./tests/bugs/bitrot/1209752-volume-status-should-show-bitrot-scrub-info.t
>> -  8 second
>> ./tests/basic/xlator-pass-through-sanity.t  -  8 second
>> ./tests/basic/quota-nfs.t  -  8 second
>> ./tests/basic/glusterd/arbiter-volume.t  -  8 second
>> ./tests/basic/ctime/ctime-noatime.t  -  8 second
>> ./tests/line-coverage/cli-peer-and-volume-operations.t  -  7 second
>> ./tests/gfid2path/get-gfid-to-path.t  -  7 second
>> ./tests/bugs/upcall/bug-1369430.t  -  7 second
>> ./tests/bugs/snapshot/bug-1260848.t  -  7 second
>> ./tests/bugs/shard/shard-inode-refcount-test.t  -  7 second
>> ./tests/bugs/shard/bug-1258334.t  -  7 second
>> ./tests/bugs/replicate/bug-767585-gfid.t  -  7 second

[Gluster-Maintainers] glusterfs-4.1.9 released

[jenkins]

SRC: 
https://build.gluster.org/job/release-new/90/artifact/glusterfs-4.1.9.tar.gz
HASH: 
https://build.gluster.org/job/release-new/90/artifact/glusterfs-4.1.9.sha512sum

This release is made off jenkins-release-90___
maintainers mailing list
maintainers@gluster.org
https://lists.gluster.org/mailman/listinfo/maintainers