Re: [Gluster-devel] [master][FAILED] brick-mux-regression

2018-12-02 Thread Milind Changire 
On Mon, Dec 3, 2018 at 8:32 AM Raghavendra Gowdappa 
wrote:

> On Mon, Dec 3, 2018 at 8:25 AM Raghavendra Gowdappa 
> wrote:
>
>> On Sat, Dec 1, 2018 at 11:02 AM Milind Changire 
>> wrote:
>>
>>> failed brick-mux-regression job:
>>> https://build.gluster.org/job/regression-on-demand-multiplex/411/console
>>>
>>> patch:
>>> https://review.gluster.org/c/glusterfs/+/21719
>>>
>>
>> Does this happen only with the above patch? Does brick-mux regression
>> succeed on current master without this patch? Wondering whether the
>> parallelism introduced by bumping up event-threads to 2, is opening up some
>> races in multiplexed environment (though there were always more than one
>> event-thread when more than one brick is multiplexed).
>>
>
> Also, is this bug locally reproducible on your setup if you run test
> following test with brick-mux enabled (with and without your patch)?
>
> ./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t
>
> running the above test with brick-mux enabled works well with and without
the "event-threads bump" patch on my centos7 setup
so, the issue is not reproducible on my setup


>>
>>>
>>> stack trace:
>>> $ gdb -ex 'set sysroot ./' -ex 'core-file
>>> ./build/install/cores/glfs_epoll000-964.core'
>>> ./build/install/sbin/glusterfsd
>>> GNU gdb (GDB) Fedora 8.2-4.fc29
>>> Copyright (C) 2018 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later <
>>> http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law.
>>> Type "show copying" and "show warranty" for details.
>>> This GDB was configured as "x86_64-redhat-linux-gnu".
>>> Type "show configuration" for configuration details.
>>> For bug reporting instructions, please see:
>>> .
>>> Find the GDB manual and other documentation resources online at:
>>> .
>>>
>>> For help, type "help".
>>> Type "apropos word" to search for commands related to "word"...
>>> Reading symbols from ./build/install/sbin/glusterfsd...done.
>>> [New LWP 970]
>>> [New LWP 992]
>>> [New LWP 993]
>>> [New LWP 1005]
>>> [New LWP 1241]
>>> [New LWP 964]
>>> [New LWP 968]
>>> [New LWP 996]
>>> [New LWP 995]
>>> [New LWP 994]
>>> [New LWP 967]
>>> [New LWP 969]
>>> [New LWP 1003]
>>> [New LWP 1181]
>>> [New LWP 1242]
>>> [New LWP 966]
>>> [New LWP 965]
>>> [New LWP 999]
>>> [New LWP 1000]
>>> [New LWP 1002]
>>> [New LWP 989]
>>> [New LWP 990]
>>> [New LWP 991]
>>> [New LWP 971]
>>> warning: Ignoring non-absolute filename: <./lib64/libz.so.1>
>>> Missing separate debuginfo for ./lib64/libz.so.1
>>> Try: dnf --enablerepo='*debug*' install
>>> /usr/lib/debug/.build-id/ea/8e45dc8e395cc5e26890470112d97a1f1e0b65.debug
>>> warning: Ignoring non-absolute filename: <./lib64/libuuid.so.1>
>>> Missing separate debuginfo for ./lib64/libuuid.so.1
>>> Try: dnf --enablerepo='*debug*' install
>>> /usr/lib/debug/.build-id/71/de190dc0c93504abacc17b9747cd772a1e4b0d.debug
>>> warning: Ignoring non-absolute filename: <./lib64/libm.so.6>
>>> Missing separate debuginfo for ./lib64/libm.so.6
>>> Try: dnf --enablerepo='*debug*' install
>>> /usr/lib/debug/.build-id/f4/cae74047f9aa2d5a71fdec67c4285d75753eba.debug
>>> warning: Ignoring non-absolute filename: <./lib64/librt.so.1>
>>> Missing separate debuginfo for ./lib64/librt.so.1
>>> Try: dnf --enablerepo='*debug*' install
>>> /usr/lib/debug/.build-id/d3/3989ec31efe745eb0d3b68a92d19e77d7ddfda.debug
>>> warning: Ignoring non-absolute filename: <./lib64/libdl.so.2>
>>> Missing separate debuginfo for ./lib64/libdl.so.2
>>> Try: dnf --enablerepo='*debug*' install
>>> /usr/lib/debug/.build-id/5c/db5a56336e7e2bd14ffa189411e44a834afcd8.debug
>>> warning: Ignoring non-absolute filename: <./lib64/libpthread.so.0>
>>> Missing separate debuginfo for ./lib64/libpthread.so.0
>>> Try: dnf --enablerepo='*debug*' install
>>> /usr/lib/debug/.build-id/f4/c04bce85d2d269d0a2af4972fc69805b50345b.debug
>>> warning: Expected absolute pathname for libpthread in the inferior, but
>>> got ./lib64/libpthread.so.0.
>>> warning: Unable to find libthread_db matching inferior's thread library,
>>> thread debugging will not be available.
>>> warning: Ignoring non-absolute filename: <./lib64/libcrypto.so.10>
>>> Missing separate debuginfo for ./lib64/libcrypto.so.10
>>> Try: dnf --enablerepo='*debug*' install
>>> /usr/lib/debug/.build-id/67/ceb4edd36bfe0eb31cd92da2694aca5377a599.debug
>>> warning: Ignoring non-absolute filename: <./lib64/libc.so.6>
>>> Missing separate debuginfo for ./lib64/libc.so.6
>>> Try: dnf --enablerepo='*debug*' install
>>> /usr/lib/debug/.build-id/cb/4b7554d1adbef2f001142dd6f0a5139fc9aa69.debug
>>> warning: Ignoring non-absolute filename: <./lib64/ld-linux-x86-64.so.2>
>>> Missing separate debuginfo for ./lib64/ld-linux-x86-64.so.2
>>> Try: dnf --enablerepo='*debug*' install
>>>

Re: [Gluster-devel] [master][FAILED] brick-mux-regression

2018-12-02 Thread Raghavendra Gowdappa 
On Mon, Dec 3, 2018 at 8:25 AM Raghavendra Gowdappa 
wrote:

>
>
> On Sat, Dec 1, 2018 at 11:02 AM Milind Changire 
> wrote:
>
>> failed brick-mux-regression job:
>> https://build.gluster.org/job/regression-on-demand-multiplex/411/console
>>
>> patch:
>> https://review.gluster.org/c/glusterfs/+/21719
>>
>
> Does this happen only with the above patch? Does brick-mux regression
> succeed on current master without this patch? Wondering whether the
> parallelism introduced by bumping up event-threads to 2, is opening up some
> races in multiplexed environment (though there were always more than one
> event-thread when more than one brick is multiplexed).
>

Also, is this bug locally reproducible on your setup if you run test
following test with brick-mux enabled (with and without your patch)?

./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t


>
>>
>> stack trace:
>> $ gdb -ex 'set sysroot ./' -ex 'core-file
>> ./build/install/cores/glfs_epoll000-964.core'
>> ./build/install/sbin/glusterfsd
>> GNU gdb (GDB) Fedora 8.2-4.fc29
>> Copyright (C) 2018 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <
>> http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.
>> Type "show copying" and "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux-gnu".
>> Type "show configuration" for configuration details.
>> For bug reporting instructions, please see:
>> .
>> Find the GDB manual and other documentation resources online at:
>> .
>>
>> For help, type "help".
>> Type "apropos word" to search for commands related to "word"...
>> Reading symbols from ./build/install/sbin/glusterfsd...done.
>> [New LWP 970]
>> [New LWP 992]
>> [New LWP 993]
>> [New LWP 1005]
>> [New LWP 1241]
>> [New LWP 964]
>> [New LWP 968]
>> [New LWP 996]
>> [New LWP 995]
>> [New LWP 994]
>> [New LWP 967]
>> [New LWP 969]
>> [New LWP 1003]
>> [New LWP 1181]
>> [New LWP 1242]
>> [New LWP 966]
>> [New LWP 965]
>> [New LWP 999]
>> [New LWP 1000]
>> [New LWP 1002]
>> [New LWP 989]
>> [New LWP 990]
>> [New LWP 991]
>> [New LWP 971]
>> warning: Ignoring non-absolute filename: <./lib64/libz.so.1>
>> Missing separate debuginfo for ./lib64/libz.so.1
>> Try: dnf --enablerepo='*debug*' install
>> /usr/lib/debug/.build-id/ea/8e45dc8e395cc5e26890470112d97a1f1e0b65.debug
>> warning: Ignoring non-absolute filename: <./lib64/libuuid.so.1>
>> Missing separate debuginfo for ./lib64/libuuid.so.1
>> Try: dnf --enablerepo='*debug*' install
>> /usr/lib/debug/.build-id/71/de190dc0c93504abacc17b9747cd772a1e4b0d.debug
>> warning: Ignoring non-absolute filename: <./lib64/libm.so.6>
>> Missing separate debuginfo for ./lib64/libm.so.6
>> Try: dnf --enablerepo='*debug*' install
>> /usr/lib/debug/.build-id/f4/cae74047f9aa2d5a71fdec67c4285d75753eba.debug
>> warning: Ignoring non-absolute filename: <./lib64/librt.so.1>
>> Missing separate debuginfo for ./lib64/librt.so.1
>> Try: dnf --enablerepo='*debug*' install
>> /usr/lib/debug/.build-id/d3/3989ec31efe745eb0d3b68a92d19e77d7ddfda.debug
>> warning: Ignoring non-absolute filename: <./lib64/libdl.so.2>
>> Missing separate debuginfo for ./lib64/libdl.so.2
>> Try: dnf --enablerepo='*debug*' install
>> /usr/lib/debug/.build-id/5c/db5a56336e7e2bd14ffa189411e44a834afcd8.debug
>> warning: Ignoring non-absolute filename: <./lib64/libpthread.so.0>
>> Missing separate debuginfo for ./lib64/libpthread.so.0
>> Try: dnf --enablerepo='*debug*' install
>> /usr/lib/debug/.build-id/f4/c04bce85d2d269d0a2af4972fc69805b50345b.debug
>> warning: Expected absolute pathname for libpthread in the inferior, but
>> got ./lib64/libpthread.so.0.
>> warning: Unable to find libthread_db matching inferior's thread library,
>> thread debugging will not be available.
>> warning: Ignoring non-absolute filename: <./lib64/libcrypto.so.10>
>> Missing separate debuginfo for ./lib64/libcrypto.so.10
>> Try: dnf --enablerepo='*debug*' install
>> /usr/lib/debug/.build-id/67/ceb4edd36bfe0eb31cd92da2694aca5377a599.debug
>> warning: Ignoring non-absolute filename: <./lib64/libc.so.6>
>> Missing separate debuginfo for ./lib64/libc.so.6
>> Try: dnf --enablerepo='*debug*' install
>> /usr/lib/debug/.build-id/cb/4b7554d1adbef2f001142dd6f0a5139fc9aa69.debug
>> warning: Ignoring non-absolute filename: <./lib64/ld-linux-x86-64.so.2>
>> Missing separate debuginfo for ./lib64/ld-linux-x86-64.so.2
>> Try: dnf --enablerepo='*debug*' install
>> /usr/lib/debug/.build-id/d2/66b1f6650927e18108323bcca8f7b68e68eb92.debug
>> warning: Ignoring non-absolute filename: <./lib64/libssl.so.10>
>> Missing separate debuginfo for ./lib64/libssl.so.10
>> Try: dnf --enablerepo='*debug*' install
>> /usr/lib/debug/.build-id/64/68a4e28a19cdd885a3cbc30e009589ca4c2e92.debug
>> warning: Ignoring non-absolute filename: <./lib64/libgssapi_krb5.so.2>

Re: [Gluster-devel] [master][FAILED] brick-mux-regression

2018-12-02 Thread Raghavendra Gowdappa 
On Sat, Dec 1, 2018 at 11:02 AM Milind Changire  wrote:

> failed brick-mux-regression job:
> https://build.gluster.org/job/regression-on-demand-multiplex/411/console
>
> patch:
> https://review.gluster.org/c/glusterfs/+/21719
>

Does this happen only with the above patch? Does brick-mux regression
succeed on current master without this patch? Wondering whether the
parallelism introduced by bumping up event-threads to 2, is opening up some
races in multiplexed environment (though there were always more than one
event-thread when more than one brick is multiplexed).


>
> stack trace:
> $ gdb -ex 'set sysroot ./' -ex 'core-file
> ./build/install/cores/glfs_epoll000-964.core'
> ./build/install/sbin/glusterfsd
> GNU gdb (GDB) Fedora 8.2-4.fc29
> Copyright (C) 2018 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> .
> Find the GDB manual and other documentation resources online at:
> .
>
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from ./build/install/sbin/glusterfsd...done.
> [New LWP 970]
> [New LWP 992]
> [New LWP 993]
> [New LWP 1005]
> [New LWP 1241]
> [New LWP 964]
> [New LWP 968]
> [New LWP 996]
> [New LWP 995]
> [New LWP 994]
> [New LWP 967]
> [New LWP 969]
> [New LWP 1003]
> [New LWP 1181]
> [New LWP 1242]
> [New LWP 966]
> [New LWP 965]
> [New LWP 999]
> [New LWP 1000]
> [New LWP 1002]
> [New LWP 989]
> [New LWP 990]
> [New LWP 991]
> [New LWP 971]
> warning: Ignoring non-absolute filename: <./lib64/libz.so.1>
> Missing separate debuginfo for ./lib64/libz.so.1
> Try: dnf --enablerepo='*debug*' install
> /usr/lib/debug/.build-id/ea/8e45dc8e395cc5e26890470112d97a1f1e0b65.debug
> warning: Ignoring non-absolute filename: <./lib64/libuuid.so.1>
> Missing separate debuginfo for ./lib64/libuuid.so.1
> Try: dnf --enablerepo='*debug*' install
> /usr/lib/debug/.build-id/71/de190dc0c93504abacc17b9747cd772a1e4b0d.debug
> warning: Ignoring non-absolute filename: <./lib64/libm.so.6>
> Missing separate debuginfo for ./lib64/libm.so.6
> Try: dnf --enablerepo='*debug*' install
> /usr/lib/debug/.build-id/f4/cae74047f9aa2d5a71fdec67c4285d75753eba.debug
> warning: Ignoring non-absolute filename: <./lib64/librt.so.1>
> Missing separate debuginfo for ./lib64/librt.so.1
> Try: dnf --enablerepo='*debug*' install
> /usr/lib/debug/.build-id/d3/3989ec31efe745eb0d3b68a92d19e77d7ddfda.debug
> warning: Ignoring non-absolute filename: <./lib64/libdl.so.2>
> Missing separate debuginfo for ./lib64/libdl.so.2
> Try: dnf --enablerepo='*debug*' install
> /usr/lib/debug/.build-id/5c/db5a56336e7e2bd14ffa189411e44a834afcd8.debug
> warning: Ignoring non-absolute filename: <./lib64/libpthread.so.0>
> Missing separate debuginfo for ./lib64/libpthread.so.0
> Try: dnf --enablerepo='*debug*' install
> /usr/lib/debug/.build-id/f4/c04bce85d2d269d0a2af4972fc69805b50345b.debug
> warning: Expected absolute pathname for libpthread in the inferior, but
> got ./lib64/libpthread.so.0.
> warning: Unable to find libthread_db matching inferior's thread library,
> thread debugging will not be available.
> warning: Ignoring non-absolute filename: <./lib64/libcrypto.so.10>
> Missing separate debuginfo for ./lib64/libcrypto.so.10
> Try: dnf --enablerepo='*debug*' install
> /usr/lib/debug/.build-id/67/ceb4edd36bfe0eb31cd92da2694aca5377a599.debug
> warning: Ignoring non-absolute filename: <./lib64/libc.so.6>
> Missing separate debuginfo for ./lib64/libc.so.6
> Try: dnf --enablerepo='*debug*' install
> /usr/lib/debug/.build-id/cb/4b7554d1adbef2f001142dd6f0a5139fc9aa69.debug
> warning: Ignoring non-absolute filename: <./lib64/ld-linux-x86-64.so.2>
> Missing separate debuginfo for ./lib64/ld-linux-x86-64.so.2
> Try: dnf --enablerepo='*debug*' install
> /usr/lib/debug/.build-id/d2/66b1f6650927e18108323bcca8f7b68e68eb92.debug
> warning: Ignoring non-absolute filename: <./lib64/libssl.so.10>
> Missing separate debuginfo for ./lib64/libssl.so.10
> Try: dnf --enablerepo='*debug*' install
> /usr/lib/debug/.build-id/64/68a4e28a19cdd885a3cbc30e009589ca4c2e92.debug
> warning: Ignoring non-absolute filename: <./lib64/libgssapi_krb5.so.2>
> Missing separate debuginfo for ./lib64/libgssapi_krb5.so.2
> Try: dnf --enablerepo='*debug*' install
> /usr/lib/debug/.build-id/16/fe0dc6cefc5f444bc876516d02efe9cc2d432f.debug
> warning: Ignoring non-absolute filename: <./lib64/libkrb5.so.3>
> Missing separate debuginfo for ./lib64/libkrb5.so.3
> Try: dnf --enablerepo='*debug*' install
>

[Gluster-devel] [master][FAILED] brick-mux-regression

2018-11-30 Thread Milind Changire 
failed brick-mux-regression job:
https://build.gluster.org/job/regression-on-demand-multiplex/411/console

patch:
https://review.gluster.org/c/glusterfs/+/21719

stack trace:
$ gdb -ex 'set sysroot ./' -ex 'core-file
./build/install/cores/glfs_epoll000-964.core'
./build/install/sbin/glusterfsd
GNU gdb (GDB) Fedora 8.2-4.fc29
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build/install/sbin/glusterfsd...done.
[New LWP 970]
[New LWP 992]
[New LWP 993]
[New LWP 1005]
[New LWP 1241]
[New LWP 964]
[New LWP 968]
[New LWP 996]
[New LWP 995]
[New LWP 994]
[New LWP 967]
[New LWP 969]
[New LWP 1003]
[New LWP 1181]
[New LWP 1242]
[New LWP 966]
[New LWP 965]
[New LWP 999]
[New LWP 1000]
[New LWP 1002]
[New LWP 989]
[New LWP 990]
[New LWP 991]
[New LWP 971]
warning: Ignoring non-absolute filename: <./lib64/libz.so.1>
Missing separate debuginfo for ./lib64/libz.so.1
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/ea/8e45dc8e395cc5e26890470112d97a1f1e0b65.debug
warning: Ignoring non-absolute filename: <./lib64/libuuid.so.1>
Missing separate debuginfo for ./lib64/libuuid.so.1
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/71/de190dc0c93504abacc17b9747cd772a1e4b0d.debug
warning: Ignoring non-absolute filename: <./lib64/libm.so.6>
Missing separate debuginfo for ./lib64/libm.so.6
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/f4/cae74047f9aa2d5a71fdec67c4285d75753eba.debug
warning: Ignoring non-absolute filename: <./lib64/librt.so.1>
Missing separate debuginfo for ./lib64/librt.so.1
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/d3/3989ec31efe745eb0d3b68a92d19e77d7ddfda.debug
warning: Ignoring non-absolute filename: <./lib64/libdl.so.2>
Missing separate debuginfo for ./lib64/libdl.so.2
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/5c/db5a56336e7e2bd14ffa189411e44a834afcd8.debug
warning: Ignoring non-absolute filename: <./lib64/libpthread.so.0>
Missing separate debuginfo for ./lib64/libpthread.so.0
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/f4/c04bce85d2d269d0a2af4972fc69805b50345b.debug
warning: Expected absolute pathname for libpthread in the inferior, but got
./lib64/libpthread.so.0.
warning: Unable to find libthread_db matching inferior's thread library,
thread debugging will not be available.
warning: Ignoring non-absolute filename: <./lib64/libcrypto.so.10>
Missing separate debuginfo for ./lib64/libcrypto.so.10
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/67/ceb4edd36bfe0eb31cd92da2694aca5377a599.debug
warning: Ignoring non-absolute filename: <./lib64/libc.so.6>
Missing separate debuginfo for ./lib64/libc.so.6
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/cb/4b7554d1adbef2f001142dd6f0a5139fc9aa69.debug
warning: Ignoring non-absolute filename: <./lib64/ld-linux-x86-64.so.2>
Missing separate debuginfo for ./lib64/ld-linux-x86-64.so.2
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/d2/66b1f6650927e18108323bcca8f7b68e68eb92.debug
warning: Ignoring non-absolute filename: <./lib64/libssl.so.10>
Missing separate debuginfo for ./lib64/libssl.so.10
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/64/68a4e28a19cdd885a3cbc30e009589ca4c2e92.debug
warning: Ignoring non-absolute filename: <./lib64/libgssapi_krb5.so.2>
Missing separate debuginfo for ./lib64/libgssapi_krb5.so.2
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/16/fe0dc6cefc5f444bc876516d02efe9cc2d432f.debug
warning: Ignoring non-absolute filename: <./lib64/libkrb5.so.3>
Missing separate debuginfo for ./lib64/libkrb5.so.3
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/d1/cd1b94855a85fbc735c745db39bc096f7d8cc3.debug
warning: Ignoring non-absolute filename: <./lib64/libcom_err.so.2>
Missing separate debuginfo for ./lib64/libcom_err.so.2
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/2c/7ef64ef0c5af8bcfa8f9e628e5605a7d8c52d3.debug
warning: Ignoring non-absolute filename: <./lib64/libk5crypto.so.3>
Missing separate debuginfo for ./lib64/libk5crypto.so.3
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/a2/0f715c514b3ea873f4cc77d585a50cb670e266.debug
warning: Ignoring non-absolute filename: <./lib64/libkrb5support.so.0>
Missing separate debuginfo for