Re: [Gluster-devel] Core generated by trash.t

[Pranith Kumar Karampuri]

+Krutika

- Original Message -
> From: "Anoop C S" <anoo...@redhat.com>
> To: "Atin Mukherjee" <amukh...@redhat.com>
> Cc: "Pranith Kumar Karampuri" <pkara...@redhat.com>, "Ravishankar N" 
> <ravishan...@redhat.com>, "Anuradha Talur"
> <ata...@redhat.com>, gluster-devel@gluster.org
> Sent: Friday, April 22, 2016 2:14:28 PM
> Subject: Re: [Gluster-devel] Core generated by trash.t
> 
> On Wed, 2016-04-20 at 16:24 +0530, Atin Mukherjee wrote:
> > I should have said the regression link is irrelevant here. Try
> > running
> > this test on your local setup multiple times on mainline. I do
> > believe
> > you should see the crash.
> > 
> 
> I could see coredump on running trash.t multiple times in a while loop.
> Info from coredump:
> 
> Core was generated by `/usr/local/sbin/glusterfs -s localhost --
> volfile-id gluster/glustershd -p /var/'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x0040bd31 in glusterfs_handle_translator_op
> (req=0x7feab8001dec) at glusterfsd-mgmt.c:590
> 590   any = active->first;
> [Current thread is 1 (Thread 0x7feac1657700 (LWP 12050))]
> (gdb) l
> 585   goto out;
> 586   }
> 587
> 588   ctx = glusterfsd_ctx;
> 589   active = ctx->active;
> 590   any = active->first;
> 591   input = dict_new ();
> 592   ret = dict_unserialize (xlator_req.input.input_val,
> 593   xlator_req.input.input_len,
> 594   );
> (gdb) p ctx
> $1 = (glusterfs_ctx_t *) 0x7fa010
> (gdb) p ctx->active
> $2 = (glusterfs_graph_t *) 0x0

I think this is because the request came to shd even before the graph is 
intialized? Thanks for the test case. I will take a look at this.

Pranith
> (gdb) p *req
> $1 = {trans = 0x7feab8000e20, svc = 0x83ca50, prog = 0x874810, xid = 1,
> prognum = 4867634, progver = 2, procnum = 3, type = 0, uid = 0, gid =
> 0, pid = 0, lk_owner = {len = 4,
> data = '\000' }, gfs_id = 0, auxgids =
> 0x7feab800223c, auxgidsmall = {0 }, auxgidlarge =
> 0x0, auxgidcount = 0, msg = {{iov_base = 0x7feacc253840,
>   iov_len = 488}, {iov_base = 0x0, iov_len = 0}  times>}, count = 1, iobref = 0x7feab8000c40, rpc_status = 0, rpc_err =
> 0, auth_err = 0, txlist = {next = 0x7feab800256c,
> prev = 0x7feab800256c}, payloadsize = 0, cred = {flavour = 390039,
> datalen = 24, authdata = '\000' , "\004", '\000'
> }, verf = {flavour = 0,
> datalen = 0, authdata = '\000' }, synctask =
> _gf_true, private = 0x0, trans_private = 0x0, hdr_iobuf = 0x82b038,
> reply = 0x0}
> (gdb) p req->procnum
> $3 = 3 <== GLUSTERD_BRICK_XLATOR_OP
> (gdb) t a a bt
> 
> Thread 6 (Thread 0x7feabf178700 (LWP 12055)):
> #0  0x7feaca522043 in epoll_wait () at ../sysdeps/unix/syscall-
> template.S:84
> #1  0x7feacbe5076f in event_dispatch_epoll_worker (data=0x878130)
> at event-epoll.c:664
> #2  0x7feacac4560a in start_thread (arg=0x7feabf178700) at
> pthread_create.c:334
> #3  0x7feaca521a4d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
> 
> Thread 5 (Thread 0x7feac2659700 (LWP 12048)):
> #0  do_sigwait (sig=0x7feac2658e3c, set=) at
> ../sysdeps/unix/sysv/linux/sigwait.c:64
> #1  __sigwait (set=, sig=0x7feac2658e3c) at
> ../sysdeps/unix/sysv/linux/sigwait.c:96
> #2  0x00409895 in glusterfs_sigwaiter (arg=0x7ffe3debbf00) at
> glusterfsd.c:2032
> #3  0x7feacac4560a in start_thread (arg=0x7feac2659700) at
> pthread_create.c:334
> #4  0x7feaca521a4d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
> 
> Thread 4 (Thread 0x7feacc2b4780 (LWP 12046)):
> #0  0x7feacac466ad in pthread_join (threadid=140646205064960,
> thread_return=0x0) at pthread_join.c:90
> #1  0x7feacbe509bb in event_dispatch_epoll (event_pool=0x830b80) at
> event-epoll.c:758
> #2  0x7feacbe17a91 in event_dispatch (event_pool=0x830b80) at
> event.c:124
> #3  0x0040a3c8 in main (argc=13, argv=0x7ffe3debd0f8) at
> glusterfsd.c:2376
> 
> Thread 3 (Thread 0x7feac2e5a700 (LWP 12047)):
> #0  0x7feacac4e27d in nanosleep () at ../sysdeps/unix/syscall-
> template.S:84
> #1  0x7feacbdfc152 in gf_timer_proc (ctx=0x7fa010) at timer.c:188
> #2  0x7feacac4560a in start_thread (arg=0x7feac2e5a700) at
> pthread_create.c:334
> #3  0x7feaca521a4d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
> 
> Thread 2 (Thread 0x7feac1e58700 (LWP 12049)):
> #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at
> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
> #

Re: [Gluster-devel] Core generated by trash.t

[Anoop C S]

On Wed, 2016-04-20 at 16:24 +0530, Atin Mukherjee wrote:
> I should have said the regression link is irrelevant here. Try
> running
> this test on your local setup multiple times on mainline. I do
> believe
> you should see the crash.
> 

I could see coredump on running trash.t multiple times in a while loop.
Info from coredump:

Core was generated by `/usr/local/sbin/glusterfs -s localhost --
volfile-id gluster/glustershd -p /var/'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0040bd31 in glusterfs_handle_translator_op
(req=0x7feab8001dec) at glusterfsd-mgmt.c:590
590 any = active->first;
[Current thread is 1 (Thread 0x7feac1657700 (LWP 12050))]
(gdb) l
585 goto out;
586 }
587 
588 ctx = glusterfsd_ctx;
589 active = ctx->active;
590 any = active->first;
591 input = dict_new ();
592 ret = dict_unserialize (xlator_req.input.input_val,
593 xlator_req.input.input_len,
594 );
(gdb) p ctx
$1 = (glusterfs_ctx_t *) 0x7fa010
(gdb) p ctx->active
$2 = (glusterfs_graph_t *) 0x0
(gdb) p *req
$1 = {trans = 0x7feab8000e20, svc = 0x83ca50, prog = 0x874810, xid = 1,
prognum = 4867634, progver = 2, procnum = 3, type = 0, uid = 0, gid =
0, pid = 0, lk_owner = {len = 4, 
data = '\000' }, gfs_id = 0, auxgids =
0x7feab800223c, auxgidsmall = {0 }, auxgidlarge =
0x0, auxgidcount = 0, msg = {{iov_base = 0x7feacc253840, 
  iov_len = 488}, {iov_base = 0x0, iov_len = 0} }, count = 1, iobref = 0x7feab8000c40, rpc_status = 0, rpc_err =
0, auth_err = 0, txlist = {next = 0x7feab800256c, 
prev = 0x7feab800256c}, payloadsize = 0, cred = {flavour = 390039,
datalen = 24, authdata = '\000' , "\004", '\000'
}, verf = {flavour = 0, 
datalen = 0, authdata = '\000' }, synctask =
_gf_true, private = 0x0, trans_private = 0x0, hdr_iobuf = 0x82b038,
reply = 0x0}
(gdb) p req->procnum
$3 = 3 <== GLUSTERD_BRICK_XLATOR_OP
(gdb) t a a bt

Thread 6 (Thread 0x7feabf178700 (LWP 12055)):
#0  0x7feaca522043 in epoll_wait () at ../sysdeps/unix/syscall-
template.S:84
#1  0x7feacbe5076f in event_dispatch_epoll_worker (data=0x878130)
at event-epoll.c:664
#2  0x7feacac4560a in start_thread (arg=0x7feabf178700) at
pthread_create.c:334
#3  0x7feaca521a4d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 5 (Thread 0x7feac2659700 (LWP 12048)):
#0  do_sigwait (sig=0x7feac2658e3c, set=) at
../sysdeps/unix/sysv/linux/sigwait.c:64
#1  __sigwait (set=, sig=0x7feac2658e3c) at
../sysdeps/unix/sysv/linux/sigwait.c:96
#2  0x00409895 in glusterfs_sigwaiter (arg=0x7ffe3debbf00) at
glusterfsd.c:2032
#3  0x7feacac4560a in start_thread (arg=0x7feac2659700) at
pthread_create.c:334
#4  0x7feaca521a4d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 4 (Thread 0x7feacc2b4780 (LWP 12046)):
#0  0x7feacac466ad in pthread_join (threadid=140646205064960,
thread_return=0x0) at pthread_join.c:90
#1  0x7feacbe509bb in event_dispatch_epoll (event_pool=0x830b80) at
event-epoll.c:758
#2  0x7feacbe17a91 in event_dispatch (event_pool=0x830b80) at
event.c:124
#3  0x0040a3c8 in main (argc=13, argv=0x7ffe3debd0f8) at
glusterfsd.c:2376

Thread 3 (Thread 0x7feac2e5a700 (LWP 12047)):
#0  0x7feacac4e27d in nanosleep () at ../sysdeps/unix/syscall-
template.S:84
#1  0x7feacbdfc152 in gf_timer_proc (ctx=0x7fa010) at timer.c:188
#2  0x7feacac4560a in start_thread (arg=0x7feac2e5a700) at
pthread_create.c:334
#3  0x7feaca521a4d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7feac1e58700 (LWP 12049)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x7feacbe2d73d in syncenv_task (proc=0x838310) at syncop.c:603
#2  0x7feacbe2d9dd in syncenv_processor (thdata=0x838310) at
syncop.c:695
#3  0x7feacac4560a in start_thread (arg=0x7feac1e58700) at
pthread_create.c:334
#4  0x7feaca521a4d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x7feac1657700 (LWP 12050)):
#0  0x0040bd31 in glusterfs_handle_translator_op
(req=0x7feab8001dec) at glusterfsd-mgmt.c:590
#1  0x7feacbe2cf04 in synctask_wrap (old_task=0x7feab80031c0) at
syncop.c:375
#2  0x7feaca467f30 in ?? () from /lib64/libc.so.6
#3  0x in ?? ()

Looking at the core, crash was seen from
glusterfs_handle_translator_op() routine while doing a 'volume heal'
command. I could then easily create a small test case to re-produce the
issue. Please find the attachment for the same.

--Anoop C S.


core-reprod.t
Description: Perl program
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Core generated by trash.t

[Atin Mukherjee]

I should have said the regression link is irrelevant here. Try running
this test on your local setup multiple times on mainline. I do believe
you should see the crash.

~Atin

On 04/20/2016 03:36 PM, Anoop C S wrote:
> On Wed, 2016-04-20 at 13:25 +0530, Anoop C S wrote:
>> On Wed, 2016-04-20 at 01:21 +0530, Atin Mukherjee wrote:
>>>
>>> Regression run [1] failed from trash.t, however the same doesn't
>>> talk
>>> about any core file, but when I run it with and with out my changes
>>> the same generates a core.
>> on which platform you ran trash.t, NetBSD or Linux?
>>
>>>
>>>
>>> [1]
>>> https://build.gluster.org/job/rackspace-netbsd7-regression-triggere
>>> d/
>>> 15971/consoleFull
>>>
>> not ok 62 , LINENUM:245
>> FAILED COMMAND: start_vol patchy /mnt/glusterfs/0
>> /mnt/glusterfs/0/abc
>>
>> start_vol function basically performs volume start command and
>> repeatedly checks for the presence of directory named 'abc' under
>> root
>> of the volume. I see the following glusterd errors from archived
>> logs(build-install-etc-glusterfs-glusterd.vol.log):
>>
>>>
>>> [2016-04-19 15:56:48.722283] W [common-
>>> utils.c:1805:gf_string2boolean] (-->0xb9c31f47
>>>  at
>>> /build/install/lib/glusterfs/3.8dev/xlator/mgmt/glusterd.so --

 0xbb733b56  at
>>> /build/install/lib/libglusterfs.so.0 ) 0-management: argument
>>> invalid
>>> [Invalid argument]
>>> [2016-04-19 15:56:48.766453] I [MSGID: 106144] [glusterd-
>>> pmap.c:270:pmap_registry_remove] 0-pmap: removing brick
>>> /d/backends/patchy11 on port 49153
>>> [2016-04-19 15:56:48.771041] E [MSGID: 106005] [glusterd-
>>> utils.c:4689:glusterd_brick_start] 0-management: Unable to start
>>> brick nbslave75.cloud.gluster.org:/d/backends/patchy1
>>> [2016-04-19 15:56:48.771132] E [MSGID: 106123] [glusterd-
>>> mgmt.c:306:gd_mgmt_v3_commit_fn] 0-management: Volume start commit
>>> failed.
>>> [2016-04-19 15:56:48.771161] E [MSGID: 106123] [glusterd-
>>> mgmt.c:1423:glusterd_mgmt_v3_commit] 0-management: Commit failed
>>> for
>>> operation Start on local node
>>> [2016-04-19 15:56:48.771188] E [MSGID: 106123] [glusterd-
>>> mgmt.c:2014:glusterd_mgmt_v3_initiate_all_phases] 0-management:
>>> Commit Op Failed
>> Brick errors from bricks/d-backends-patchy1.log:
>>
>>>
>>> [2016-04-19 15:56:48.763066] I
>>> [rpcsvc.c:2218:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service:
>>> Configured rpc.outstanding-rpc-limit with value 64
>>> [2016-04-19 15:56:48.763127] W [MSGID: 101002]
>>> [options.c:954:xl_opt_validate] 0-patchy-server: option 'listen-
>>> port' 
>>> is deprecated, preferred is 'transport.socket.listen-port',
>>> continuing with correction
>>> [2016-04-19 15:56:48.763273] E [socket.c:765:__socket_server_bind]
>>> 0-
>>> tcp.patchy-server: binding to  failed: Address already in use
>>> [2016-04-19 15:56:48.763293] E [socket.c:768:__socket_server_bind]
>>> 0-
>>> tcp.patchy-server: Port is already in use
>>> [2016-04-19 15:56:48.763314] W
>>> [rpcsvc.c:1600:rpcsvc_transport_create] 0-rpc-service: listening on
>>> transport failed
>>> [2016-04-19 15:56:48.763332] W [MSGID: 115045] [server.c:1061:init]
>>> 0-patchy-server: creation of listener failed
>>> [2016-04-19 15:56:48.763351] E [MSGID: 101019]
>>> [xlator.c:430:xlator_init] 0-patchy-server: Initialization of
>>> volume
>>> 'patchy-server' failed, review your volfile again
>>> [2016-04-19 15:56:48.763368] E [MSGID: 101066]
>>> [graph.c:324:glusterfs_graph_init] 0-patchy-server: initializing
>>> translator failed
>>> [2016-04-19 15:56:48.763383] E [MSGID: 101176]
>>> [graph.c:670:glusterfs_graph_activate] 0-graph: init failed
>>> [2016-04-19 15:56:48.766235] W [glusterfsd.c:1265:cleanup_and_exit]
>>> (-->0x8050b83  at
>>> /build/install/sbin/glusterfsd -->0x804e8e7 
>>> at /build/install/sbin/glusterfsd ) 0-: received signum (0),
>>> shutting
>>> down
> 
> I found the following BZ with exactly similar brick error messages. Can
> you please confirm whether they are related or not?
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1322805
> 
>> Is this something related to the
>> patch(http://review.gluster.org/#/c/10
>> 785/) for which the this regression was run against? Because I don't
>> expect volume start to fail.
>>
>> If you have the coredump, can you please share the back trace for
>> further analysis?
>>
>> Thanks,
>> --Anoop C S. 
>>
>>>
>>> ~Atin
>>> ___
>>> Gluster-devel mailing list
>>> Gluster-devel@gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Core generated by trash.t

[Anoop C S]

On Wed, 2016-04-20 at 13:25 +0530, Anoop C S wrote:
> On Wed, 2016-04-20 at 01:21 +0530, Atin Mukherjee wrote:
> > 
> > Regression run [1] failed from trash.t, however the same doesn't
> > talk
> > about any core file, but when I run it with and with out my changes
> > the same generates a core.
> on which platform you ran trash.t, NetBSD or Linux?
> 
> > 
> > 
> > [1]
> > https://build.gluster.org/job/rackspace-netbsd7-regression-triggere
> > d/
> > 15971/consoleFull
> > 
> not ok 62 , LINENUM:245
> FAILED COMMAND: start_vol patchy /mnt/glusterfs/0
> /mnt/glusterfs/0/abc
> 
> start_vol function basically performs volume start command and
> repeatedly checks for the presence of directory named 'abc' under
> root
> of the volume. I see the following glusterd errors from archived
> logs(build-install-etc-glusterfs-glusterd.vol.log):
> 
> > 
> > [2016-04-19 15:56:48.722283] W [common-
> > utils.c:1805:gf_string2boolean] (-->0xb9c31f47
> >  at
> > /build/install/lib/glusterfs/3.8dev/xlator/mgmt/glusterd.so --
> > > 
> > > 0xbb733b56  at
> > /build/install/lib/libglusterfs.so.0 ) 0-management: argument
> > invalid
> > [Invalid argument]
> > [2016-04-19 15:56:48.766453] I [MSGID: 106144] [glusterd-
> > pmap.c:270:pmap_registry_remove] 0-pmap: removing brick
> > /d/backends/patchy11 on port 49153
> > [2016-04-19 15:56:48.771041] E [MSGID: 106005] [glusterd-
> > utils.c:4689:glusterd_brick_start] 0-management: Unable to start
> > brick nbslave75.cloud.gluster.org:/d/backends/patchy1
> > [2016-04-19 15:56:48.771132] E [MSGID: 106123] [glusterd-
> > mgmt.c:306:gd_mgmt_v3_commit_fn] 0-management: Volume start commit
> > failed.
> > [2016-04-19 15:56:48.771161] E [MSGID: 106123] [glusterd-
> > mgmt.c:1423:glusterd_mgmt_v3_commit] 0-management: Commit failed
> > for
> > operation Start on local node
> > [2016-04-19 15:56:48.771188] E [MSGID: 106123] [glusterd-
> > mgmt.c:2014:glusterd_mgmt_v3_initiate_all_phases] 0-management:
> > Commit Op Failed
> Brick errors from bricks/d-backends-patchy1.log:
> 
> > 
> > [2016-04-19 15:56:48.763066] I
> > [rpcsvc.c:2218:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service:
> > Configured rpc.outstanding-rpc-limit with value 64
> > [2016-04-19 15:56:48.763127] W [MSGID: 101002]
> > [options.c:954:xl_opt_validate] 0-patchy-server: option 'listen-
> > port' 
> > is deprecated, preferred is 'transport.socket.listen-port',
> > continuing with correction
> > [2016-04-19 15:56:48.763273] E [socket.c:765:__socket_server_bind]
> > 0-
> > tcp.patchy-server: binding to  failed: Address already in use
> > [2016-04-19 15:56:48.763293] E [socket.c:768:__socket_server_bind]
> > 0-
> > tcp.patchy-server: Port is already in use
> > [2016-04-19 15:56:48.763314] W
> > [rpcsvc.c:1600:rpcsvc_transport_create] 0-rpc-service: listening on
> > transport failed
> > [2016-04-19 15:56:48.763332] W [MSGID: 115045] [server.c:1061:init]
> > 0-patchy-server: creation of listener failed
> > [2016-04-19 15:56:48.763351] E [MSGID: 101019]
> > [xlator.c:430:xlator_init] 0-patchy-server: Initialization of
> > volume
> > 'patchy-server' failed, review your volfile again
> > [2016-04-19 15:56:48.763368] E [MSGID: 101066]
> > [graph.c:324:glusterfs_graph_init] 0-patchy-server: initializing
> > translator failed
> > [2016-04-19 15:56:48.763383] E [MSGID: 101176]
> > [graph.c:670:glusterfs_graph_activate] 0-graph: init failed
> > [2016-04-19 15:56:48.766235] W [glusterfsd.c:1265:cleanup_and_exit]
> > (-->0x8050b83  at
> > /build/install/sbin/glusterfsd -->0x804e8e7 
> > at /build/install/sbin/glusterfsd ) 0-: received signum (0),
> > shutting
> > down

I found the following BZ with exactly similar brick error messages. Can
you please confirm whether they are related or not?

https://bugzilla.redhat.com/show_bug.cgi?id=1322805

> Is this something related to the
> patch(http://review.gluster.org/#/c/10
> 785/) for which the this regression was run against? Because I don't
> expect volume start to fail.
> 
> If you have the coredump, can you please share the back trace for
> further analysis?
> 
> Thanks,
> --Anoop C S. 
> 
> > 
> > ~Atin
> > ___
> > Gluster-devel mailing list
> > Gluster-devel@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-devel
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Core generated by trash.t

[Anoop C S]

On Wed, 2016-04-20 at 01:21 +0530, Atin Mukherjee wrote:
> Regression run [1] failed from trash.t, however the same doesn't talk
> about any core file, but when I run it with and with out my changes
> the same generates a core.

on which platform you ran trash.t, NetBSD or Linux?

> 
> [1]
> https://build.gluster.org/job/rackspace-netbsd7-regression-triggered/
> 15971/consoleFull
> 

not ok 62 , LINENUM:245
FAILED COMMAND: start_vol patchy /mnt/glusterfs/0 /mnt/glusterfs/0/abc

start_vol function basically performs volume start command and
repeatedly checks for the presence of directory named 'abc' under root
of the volume. I see the following glusterd errors from archived
logs(build-install-etc-glusterfs-glusterd.vol.log):

> [2016-04-19 15:56:48.722283] W [common-
> utils.c:1805:gf_string2boolean] (-->0xb9c31f47
>  at
> /build/install/lib/glusterfs/3.8dev/xlator/mgmt/glusterd.so --
> >0xbb733b56  at
> /build/install/lib/libglusterfs.so.0 ) 0-management: argument invalid
> [Invalid argument]
> [2016-04-19 15:56:48.766453] I [MSGID: 106144] [glusterd-
> pmap.c:270:pmap_registry_remove] 0-pmap: removing brick
> /d/backends/patchy11 on port 49153
> [2016-04-19 15:56:48.771041] E [MSGID: 106005] [glusterd-
> utils.c:4689:glusterd_brick_start] 0-management: Unable to start
> brick nbslave75.cloud.gluster.org:/d/backends/patchy1
> [2016-04-19 15:56:48.771132] E [MSGID: 106123] [glusterd-
> mgmt.c:306:gd_mgmt_v3_commit_fn] 0-management: Volume start commit
> failed.
> [2016-04-19 15:56:48.771161] E [MSGID: 106123] [glusterd-
> mgmt.c:1423:glusterd_mgmt_v3_commit] 0-management: Commit failed for
> operation Start on local node
> [2016-04-19 15:56:48.771188] E [MSGID: 106123] [glusterd-
> mgmt.c:2014:glusterd_mgmt_v3_initiate_all_phases] 0-management:
> Commit Op Failed

Brick errors from bricks/d-backends-patchy1.log:

> [2016-04-19 15:56:48.763066] I
> [rpcsvc.c:2218:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service:
> Configured rpc.outstanding-rpc-limit with value 64
> [2016-04-19 15:56:48.763127] W [MSGID: 101002]
> [options.c:954:xl_opt_validate] 0-patchy-server: option 'listen-port' 
> is deprecated, preferred is 'transport.socket.listen-port',
> continuing with correction
> [2016-04-19 15:56:48.763273] E [socket.c:765:__socket_server_bind] 0-
> tcp.patchy-server: binding to  failed: Address already in use
> [2016-04-19 15:56:48.763293] E [socket.c:768:__socket_server_bind] 0-
> tcp.patchy-server: Port is already in use
> [2016-04-19 15:56:48.763314] W
> [rpcsvc.c:1600:rpcsvc_transport_create] 0-rpc-service: listening on
> transport failed
> [2016-04-19 15:56:48.763332] W [MSGID: 115045] [server.c:1061:init]
> 0-patchy-server: creation of listener failed
> [2016-04-19 15:56:48.763351] E [MSGID: 101019]
> [xlator.c:430:xlator_init] 0-patchy-server: Initialization of volume
> 'patchy-server' failed, review your volfile again
> [2016-04-19 15:56:48.763368] E [MSGID: 101066]
> [graph.c:324:glusterfs_graph_init] 0-patchy-server: initializing
> translator failed
> [2016-04-19 15:56:48.763383] E [MSGID: 101176]
> [graph.c:670:glusterfs_graph_activate] 0-graph: init failed
> [2016-04-19 15:56:48.766235] W [glusterfsd.c:1265:cleanup_and_exit]
> (-->0x8050b83  at
> /build/install/sbin/glusterfsd -->0x804e8e7 
> at /build/install/sbin/glusterfsd ) 0-: received signum (0), shutting
> down

Is this something related to the patch(http://review.gluster.org/#/c/10
785/) for which the this regression was run against? Because I don't
expect volume start to fail.

If you have the coredump, can you please share the back trace for
further analysis?

Thanks,
--Anoop C S. 

> ~Atin
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel