Re: [Gluster-devel] Core from gNFS process

2016-01-18 Thread Raghavendra Talur 
On Sat, Jan 16, 2016 at 2:25 AM, Vijay Bellur  wrote:

> On 01/15/2016 08:38 AM, Soumya Koduri wrote:
>
>>
>>
>>
>> On 01/15/2016 06:52 PM, Soumya Koduri wrote:
>>
>>>
>>>
>>> On 01/14/2016 08:41 PM, Vijay Bellur wrote:
>>>
 On 01/14/2016 04:11 AM, Jiffin Tony Thottan wrote:

>
>
> On 14/01/16 14:28, Jiffin Tony Thottan wrote:
>
>> Hi,
>>
>> The core generated when encryption xlator is enabled
>>
>> [2016-01-14 08:13:15.740835] E
>> [crypt.c:4298:master_set_master_vol_key] 0-test1-crypt: FATAL: missing
>> master key
>> [2016-01-14 08:13:15.740859] E [MSGID: 101019]
>> [xlator.c:429:xlator_init] 0-test1-crypt: Initialization of volume
>> 'test1-crypt' failed, review your volfile again
>> [2016-01-14 08:13:15.740890] E [MSGID: 101066]
>> [graph.c:324:glusterfs_graph_init] 0-test1-crypt: initializing
>> translator failed
>> [2016-01-14 08:13:15.740904] E [MSGID: 101176]
>> [graph.c:670:glusterfs_graph_activate] 0-graph: init failed
>> [2016-01-14 08:13:15.741676] W [glusterfsd.c:1231:cleanup_and_exit]
>> (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x307) [0x40d287]
>> -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x117) [0x4086c7]
>> -->/usr/sbin/glusterfs(cleanup_and_exit+0x4d) [0x407e1d] ) 0-:
>> received signum (0), shutting down
>>
>>
>>
> Forgot to mention this last mail,  for crypt xlator needs master key
> before enabling the translator which cause the issue
> --
>

 Irrespective of the problem, the nfs process should not crash. Can we
 check why there is a memory corruption during cleanup_and_exit()?

 That's right. This issue was reported quite a few times earlier in
>>> gluster-devel and it is not specific to gluster-nfs process. As updated
>>> in [1], we have raised bug1293594[2] against lib-gcc team to further
>>> investigate this.
>>>
>>
> The segmentation fault in gcc is while attempting to print a backtrace
> upon glusterfs receiving a SIGSEGV. It would be good to isolate the reason
> for the initial SIGSEGV whose signal handler causes the further crash.


I wasn't able to check this today. Will check tomorrow.


>
>
> -Vijay
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Core from gNFS process

2016-01-15 Thread Soumya Koduri 



On 01/14/2016 08:41 PM, Vijay Bellur wrote:

On 01/14/2016 04:11 AM, Jiffin Tony Thottan wrote:



On 14/01/16 14:28, Jiffin Tony Thottan wrote:

Hi,

The core generated when encryption xlator is enabled

[2016-01-14 08:13:15.740835] E
[crypt.c:4298:master_set_master_vol_key] 0-test1-crypt: FATAL: missing
master key
[2016-01-14 08:13:15.740859] E [MSGID: 101019]
[xlator.c:429:xlator_init] 0-test1-crypt: Initialization of volume
'test1-crypt' failed, review your volfile again
[2016-01-14 08:13:15.740890] E [MSGID: 101066]
[graph.c:324:glusterfs_graph_init] 0-test1-crypt: initializing
translator failed
[2016-01-14 08:13:15.740904] E [MSGID: 101176]
[graph.c:670:glusterfs_graph_activate] 0-graph: init failed
[2016-01-14 08:13:15.741676] W [glusterfsd.c:1231:cleanup_and_exit]
(-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x307) [0x40d287]
-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x117) [0x4086c7]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x4d) [0x407e1d] ) 0-:
received signum (0), shutting down




Forgot to mention this last mail,  for crypt xlator needs master key
before enabling the translator which cause the issue
--


Irrespective of the problem, the nfs process should not crash. Can we
check why there is a memory corruption during cleanup_and_exit()?

That's right. This issue was reported quite a few times earlier in 
gluster-devel and it is not specific to gluster-nfs process. As updated 
in [1], we have raised bug1293594[2] against lib-gcc team to further 
investigate this.


As requested in [1], kindly upload the core in the bug along with bt 
taken with gcc debuginfo packages installed. Might help to get their 
attention and get a closure on this issue sooner.


Thanks,
Soumya
[1] http://article.gmane.org/gmane.comp.file-systems.gluster.devel/13298


-Vijay

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Core from gNFS process

2016-01-15 Thread Soumya Koduri 




On 01/15/2016 06:52 PM, Soumya Koduri wrote:



On 01/14/2016 08:41 PM, Vijay Bellur wrote:

On 01/14/2016 04:11 AM, Jiffin Tony Thottan wrote:



On 14/01/16 14:28, Jiffin Tony Thottan wrote:

Hi,

The core generated when encryption xlator is enabled

[2016-01-14 08:13:15.740835] E
[crypt.c:4298:master_set_master_vol_key] 0-test1-crypt: FATAL: missing
master key
[2016-01-14 08:13:15.740859] E [MSGID: 101019]
[xlator.c:429:xlator_init] 0-test1-crypt: Initialization of volume
'test1-crypt' failed, review your volfile again
[2016-01-14 08:13:15.740890] E [MSGID: 101066]
[graph.c:324:glusterfs_graph_init] 0-test1-crypt: initializing
translator failed
[2016-01-14 08:13:15.740904] E [MSGID: 101176]
[graph.c:670:glusterfs_graph_activate] 0-graph: init failed
[2016-01-14 08:13:15.741676] W [glusterfsd.c:1231:cleanup_and_exit]
(-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x307) [0x40d287]
-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x117) [0x4086c7]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x4d) [0x407e1d] ) 0-:
received signum (0), shutting down




Forgot to mention this last mail,  for crypt xlator needs master key
before enabling the translator which cause the issue
--


Irrespective of the problem, the nfs process should not crash. Can we
check why there is a memory corruption during cleanup_and_exit()?


That's right. This issue was reported quite a few times earlier in
gluster-devel and it is not specific to gluster-nfs process. As updated
in [1], we have raised bug1293594[2] against lib-gcc team to further
investigate this.

As requested in [1], kindly upload the core in the bug along with bt
taken with gcc debuginfo packages installed. Might help to get their
attention and get a closure on this issue sooner.


Here is the bug link -
https://bugzilla.redhat.com/show_bug.cgi?id=1293594

Request Raghavendra/Ravi to update it.

Thanks,
Soumya


Thanks,
Soumya
[1] http://article.gmane.org/gmane.comp.file-systems.gluster.devel/13298


-Vijay

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Core from gNFS process

2016-01-15 Thread Vijay Bellur 

On 01/15/2016 08:38 AM, Soumya Koduri wrote:




On 01/15/2016 06:52 PM, Soumya Koduri wrote:



On 01/14/2016 08:41 PM, Vijay Bellur wrote:

On 01/14/2016 04:11 AM, Jiffin Tony Thottan wrote:



On 14/01/16 14:28, Jiffin Tony Thottan wrote:

Hi,

The core generated when encryption xlator is enabled

[2016-01-14 08:13:15.740835] E
[crypt.c:4298:master_set_master_vol_key] 0-test1-crypt: FATAL: missing
master key
[2016-01-14 08:13:15.740859] E [MSGID: 101019]
[xlator.c:429:xlator_init] 0-test1-crypt: Initialization of volume
'test1-crypt' failed, review your volfile again
[2016-01-14 08:13:15.740890] E [MSGID: 101066]
[graph.c:324:glusterfs_graph_init] 0-test1-crypt: initializing
translator failed
[2016-01-14 08:13:15.740904] E [MSGID: 101176]
[graph.c:670:glusterfs_graph_activate] 0-graph: init failed
[2016-01-14 08:13:15.741676] W [glusterfsd.c:1231:cleanup_and_exit]
(-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x307) [0x40d287]
-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x117) [0x4086c7]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x4d) [0x407e1d] ) 0-:
received signum (0), shutting down




Forgot to mention this last mail,  for crypt xlator needs master key
before enabling the translator which cause the issue
--


Irrespective of the problem, the nfs process should not crash. Can we
check why there is a memory corruption during cleanup_and_exit()?


That's right. This issue was reported quite a few times earlier in
gluster-devel and it is not specific to gluster-nfs process. As updated
in [1], we have raised bug1293594[2] against lib-gcc team to further
investigate this.


The segmentation fault in gcc is while attempting to print a backtrace 
upon glusterfs receiving a SIGSEGV. It would be good to isolate the 
reason for the initial SIGSEGV whose signal handler causes the further 
crash.


-Vijay

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Core from gNFS process

2016-01-14 Thread Jiffin Tony Thottan 



On 14/01/16 14:28, Jiffin Tony Thottan wrote:

Hi,

The core generated when encryption xlator is enabled

[2016-01-14 08:13:15.740835] E 
[crypt.c:4298:master_set_master_vol_key] 0-test1-crypt: FATAL: missing 
master key
[2016-01-14 08:13:15.740859] E [MSGID: 101019] 
[xlator.c:429:xlator_init] 0-test1-crypt: Initialization of volume 
'test1-crypt' failed, review your volfile again
[2016-01-14 08:13:15.740890] E [MSGID: 101066] 
[graph.c:324:glusterfs_graph_init] 0-test1-crypt: initializing 
translator failed
[2016-01-14 08:13:15.740904] E [MSGID: 101176] 
[graph.c:670:glusterfs_graph_activate] 0-graph: init failed
[2016-01-14 08:13:15.741676] W [glusterfsd.c:1231:cleanup_and_exit] 
(-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x307) [0x40d287] 
-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x117) [0x4086c7] 
-->/usr/sbin/glusterfs(cleanup_and_exit+0x4d) [0x407e1d] ) 0-: 
received signum (0), shutting down





Forgot to mention this last mail,  for crypt xlator needs master key 
before enabling the translator which cause the issue

--
Jiffin

With regards,
Jiffin


On 14/01/16 12:28, Raghavendra Talur wrote:

Hi Jiffin and Soumya,

Ravishankar told me about core generated by gNFS process during 
./tests/bugs/snapshot/bug-1140162-file-snapshot-features-encrypt-opts-validation.t. 



Here is console output:
https://build.gluster.org/job/rackspace-regression-2GB-triggered/17525/console 



And here is the backtrace for convenience

(gdb) thread apply all bt

Thread 9 (LWP 12499):
#0  0x7f622f4fda0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
./lib64/libpthread.so.0

#1  0x7f6230258a61 in syncenv_task (proc=0x7f621c0332f0)
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/syncop.c:603

#2  0x7f6230258d08 in syncenv_processor (thdata=0x7f621c0332f0)
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/syncop.c:695

#3  0x7f622f4f9a51 in start_thread () from ./lib64/libpthread.so.0
#4  0x7f622ee6393d in clone () from ./lib64/libc.so.6

Thread 8 (LWP 12497):
#0  0x7f622edc2e2c in vfprintf () from ./lib64/libc.so.6
#1  0x7f622edea752 in vsnprintf () from ./lib64/libc.so.6
#2  0x7f6230243f70 in gf_vasprintf (string_ptr=0x7f6220a66ba8,
format=0x7f62302aeacd "[%s] %s [%s:%d:%s] %d-%s: ", 
arg=0x7f6220a66a70)
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/mem-pool.c:219

#3  0x7f62302440ad in gf_asprintf (string_ptr=0x7f6220a66ba8,
format=0x7f62302aeacd "[%s] %s [%s:%d:%s] %d-%s: ")
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/mem-pool.c:239
#4  0x7f623021d387 in _gf_log (domain=0x7f621c00cde0 
"d_exit+0x87) [0x407cdf]",
file=0x7f622272b468 0x7f622272b468>,
function=0x7f622272d130 0x7f622272d130>, line=2895,
level=GF_LOG_INFO, fmt=0x7f622272c690 memory at address 0x7f622272c690>)
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/logging.c:2216

#5  0x7f6222725d99 in ?? ()
#6  0x0005 in ?? ()
#7  0x in ?? ()

Thread 7 (LWP 12460):
#0  0x7f622f4fda0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from 
./lib64/libpthread.so.0

#1  0x7f6230258a61 in syncenv_task (proc=0x241b210)
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/syncop.c:603

#2  0x7f6230258d08 in syncenv_processor (thdata=0x241b210)
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/syncop.c:695

#3  0x7f622f4f9a51 in start_thread () from ./lib64/libpthread.so.0
#4  0x7f622ee6393d in clone () from ./lib64/libc.so.6

Thread 6 (LWP 12476):
#0  0x7f622f5002e4 in __lll_lock_wait () from 
./lib64/libpthread.so.0

#1  0x7f622f4fb588 in _L_lock_854 () from ./lib64/libpthread.so.0
#2  0x7f622f4fb457 in pthread_mutex_lock () from 
./lib64/libpthread.so.0
#3  0x7f623021ca6c in _gf_msg (domain=0x4117ef access memory at address 0x4117ef>,

---Type  to continue, or q  to quit---
file=0x411468 ,
function=0x4125d0 <__FUNCTION__.18918> memory at address 0x4125d0>, line=1231,

level=GF_LOG_WARNING, errnum=0, trace=1, msgid=100032,
fmt=0x411bb0 )
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/logging.c:2055

#4  0x00407cdf in cleanup_and_exit (signum=0)
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/glusterfsd/src/glusterfsd.c:1231
#5  0x00409ee4 in glusterfs_process_volfp (ctx=0x23f6010, 
fp=0x7f621c001400)
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/glusterfsd/src/glusterfsd.c:2202
#6  0x0040e71d in mgmt_getspec_cbk (req=0x7f621c001a4c, 
iov=0x7f621c001a8c, count=1,

myframe=0x7f621c00135c)
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/glusterfsd/src/glusterfsd-mgmt.c:1640
#7  0x7f622ffe242a in

Re: [Gluster-devel] Core from gNFS process

2016-01-14 Thread Vijay Bellur 

On 01/14/2016 04:11 AM, Jiffin Tony Thottan wrote:



On 14/01/16 14:28, Jiffin Tony Thottan wrote:

Hi,

The core generated when encryption xlator is enabled

[2016-01-14 08:13:15.740835] E
[crypt.c:4298:master_set_master_vol_key] 0-test1-crypt: FATAL: missing
master key
[2016-01-14 08:13:15.740859] E [MSGID: 101019]
[xlator.c:429:xlator_init] 0-test1-crypt: Initialization of volume
'test1-crypt' failed, review your volfile again
[2016-01-14 08:13:15.740890] E [MSGID: 101066]
[graph.c:324:glusterfs_graph_init] 0-test1-crypt: initializing
translator failed
[2016-01-14 08:13:15.740904] E [MSGID: 101176]
[graph.c:670:glusterfs_graph_activate] 0-graph: init failed
[2016-01-14 08:13:15.741676] W [glusterfsd.c:1231:cleanup_and_exit]
(-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x307) [0x40d287]
-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x117) [0x4086c7]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x4d) [0x407e1d] ) 0-:
received signum (0), shutting down




Forgot to mention this last mail,  for crypt xlator needs master key
before enabling the translator which cause the issue
--


Irrespective of the problem, the nfs process should not crash. Can we 
check why there is a memory corruption during cleanup_and_exit()?


-Vijay

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel