Hi,

I was about to say I'm sure and then realized that I've had two lustreapi 
libraries laying around on the system, one being leftover from a previous 
lustre client rpm.
Obviously robinhood was picking up the wrong one.

-------------------------------------------------------------------------------
lorand@robinhood:~ $ rpmbuild -tb --with lustre 
rpmbuild/SOURCES/robinhood-3.1.4.tar.gz >& compile-$(date +%F).log

lorand@robinhood:~ $ grep liblustreapi compile-2018-11-29.log
Requires: /bin/bash /bin/sh ld-linux-x86-64.so.2()(64bit) 
ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) libc.so.6()(64bit) 
libc.so.6(GLIBC_2.10)(64bit) libc.so.6(GLIBC_2.14)(64bit) 
libc.so.6(GLIBC_2.17)(64bit) libc.so.6(GLIBC_2.2.5)(64bit) 
libc.so.6(GLIBC_2.3)(64bit) libc.so.6(GLIBC_2.4)(64bit) 
libcrypto.so.10()(64bit) libdl.so.2()(64bit) libdl.so.2(GLIBC_2.2.5)(64bit) 
libglib-2.0.so.0()(64bit) libgthread-2.0.so.0()(64bit) 
libjemalloc.so.1()(64bit) liblustreapi.so()(64bit) libm.so.6()(64bit) 
libmysqlclient.so.18()(64bit) libmysqlclient.so.18(libmysqlclient_18)(64bit) 
libpthread.so.0()(64bit) libpthread.so.0(GLIBC_2.2.5)(64bit) 
libpthread.so.0(GLIBC_2.3.2)(64bit) librbh_mod_alerter.so.0()(64bit) 
librbh_mod_backup.so.0()(64bit) librbh_mod_basic.so.0()(64bit) 
librbh_mod_checker.so.0()(64bit) librbh_mod_common.so.0()(64bit) 
librbh_mod_lhsm.so.0()(64bit) librbh_mod_modeguard.so.0()(64bit) 
librbh_mod_test.so.0()(64bit) libssl.so.10()(64bit) libz.so.1()(64bit) 
rtld(GNU_HASH)
Requires: libc.so.6()(64bit) libc.so.6(GLIBC_2.14)(64bit) 
libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit) 
libglib-2.0.so.0()(64bit) libgthread-2.0.so.0()(64bit) 
libjemalloc.so.1()(64bit) liblustreapi.so()(64bit) libpthread.so.0()(64bit) 
libpthread.so.0(GLIBC_2.2.5)(64bit) librt.so.1()(64bit) libz.so.1()(64bit) 
rtld(GNU_HASH)

lorand@robinhood:~ $ rpm -qa |egrep "lustre|robinhood"
lustre-client-debuginfo-2.10.5-1.el7.x86_64
kmod-lustre-client-tests-2.10.5-1.el7.x86_64
robinhood-tools-3.1.4-1.lustre.el7.x86_64
lustre-iokit-2.10.5-1.el7.x86_64
robinhood-tests-3.1.4-1.x86_64
kmod-lustre-client-2.10.5-1.el7.x86_64
lustre-client-tests-2.10.5-1.el7.x86_64
robinhood-debuginfo-3.1.4-1.el7.x86_64
lustre-client-2.10.5-1.el7.x86_64
robinhood-lustre-3.1.4-1.lustre.el7.x86_64
robinhood-adm-3.1.4-1.x86_64

lorand@robinhood:~ $ ldd $(which robinhood)
        linux-vdso.so.1 =>  (0x00007ffc591f0000)
        libmysqlclient.so.18 => /usr/lib64/mysql/libmysqlclient.so.18 
(0x00007f82f3278000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f82f2f76000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f82f2d72000)
        libssl.so.10 => /lib64/libssl.so.10 (0x00007f82f2b00000)
        libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f82f269f000)
        liblustreapi.so => /lib/liblustreapi.so (0x00007f82f2485000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f82f2269000)
        libgthread-2.0.so.0 => /lib64/libgthread-2.0.so.0 (0x00007f82f2067000)
        libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f82f1d53000)
        libjemalloc.so.1 => /lib64/libjemalloc.so.1 (0x00007f82f1b1f000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f82f1909000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f82f153c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f82f3778000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f82f1235000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f82f0fe8000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f82f0d00000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f82f0afc000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f82f08c9000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f82f0667000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f82f0451000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f82f0243000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f82f003f000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f82efe26000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f82efbff000)

lorand@robinhood:~ $ md5sum /lib/liblustreapi.so
c58640c0bfe0c7551e917db122f2f42d  /lib/liblustreapi.so

lorand@robinhood:~ $ md5sum /usr/lib64/liblustreapi.so
1e649f4b0a541a8f62c1bf0c0402ae7f  /usr/lib64/liblustreapi.so

lorand@robinhood:~ $ rpm -qf /usr/lib64/liblustreapi.so
lustre-client-2.10.5-1.el7.x86_64

lorand@robinhood:~ $ ls -la /lib/liblustreapi.so
-rw-r--r-- 1 root root 412152 Sep 18  2017 /lib/liblustreapi.so

[root@robinhood x86_64]# mv  /lib/liblustreapi.so{,.dis}
-------------------------------------------------------------------------------


Thank you!
Lorand



> On 29 Nov 2018, at 11:46, quentin.bou...@cea.fr wrote:
> 
> Hi again,
> 
> I am pretty sure this is it though. I remember having to deal with it while 
> working on lustre.
> 
> > I initially installed robinhood from rpms, but now fetched the sources and 
> > freshly compiled, built new rpms.
> 
> And are you _sure_ lustre 2.10 was installed on the node you compiled/built 
> robinhood's rpms?
> 
> Quentin
> 
> Le 29/11/2018 à 10:51, Lorand Szentannai a écrit :
>> Hi Quentin,
>> 
>> I initially installed robinhood from rpms, but now fetched the sources and 
>> freshly compiled, built new rpms.
>> Unfortunately the behavior is still the same.
>> 
>> 
>> Thanks,
>> Lorand
>> 
>>> On 28 Nov 2018, at 15:46, quentin.bou...@cea.fr wrote:
>>> 
>>> Hi Lorand,
>>> 
>>> I seem to remember this is caused by an update of lustre's changelog API.
>>> 
>>> Did compile robinhood against the right version of liblustreapi? (there is 
>>> no need for ioctl in Lustre 2.10's API)
>>> 
>>> Quentin
>>> 
>>> Le 28/11/2018 à 15:04, Lorand Szentannai a écrit :
>>>> Hi Thomas,
>>>> 
>>>> I actually did that already but forgot to add the strace lines to the 
>>>> initial email. But it is still unclear to me why open function is called 
>>>> with the O_RDONLY flag.
>>>> 
>>>> [pid 2348817] open("/cluster", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 12
>>>> [pid 2348817] ioctl(12, _IOC(_IOC_WRITE, 0x66, 0x94, 0x08), 
>>>> 0x7ffef3f20240) = -1 ENOTTY (Inappropriate ioctl for device)
>>>> 
>>>> # grep lustre /proc/mounts
>>>> 10.3.5.18@o2ib,10.4.5.18@o2ib1:10.3.5.19@o2ib,10.4.5.19@o2ib1:/scratch 
>>>> /cluster lustre rw,flock,lazystatfs 0 0
>>>> 
>>>> 
>>>> Thanks,
>>>> Lorand Szentannai
>>>> 
>>>>> On 28 Nov 2018, at 12:47, LEIBOVICI Thomas <thomas.leibov...@cea.fr> 
>>>>> wrote:
>>>>> 
>>>>> Hello,
>>>>> 
>>>>> If you run "strace -f  robinhood --readlog" you should see the ioctl that 
>>>>> causes this error, and its arguments.
>>>>> 
>>>>> Thomas
>>>>> 
>>>>> On 11/28/18 07:36, Lorand Szentannai wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> We've recently upgraded to Lustre 2.10.5 and to Robinhood to 3.1.4.
>>>>>> Processing changelogs fails now with "ERROR -25 opening changelog for 
>>>>>> MDT 'scratch-MDT0000': Inappropriate ioctl for device".
>>>>>> 
>>>>>> Could you please advice on how to solve this? Note that currently MGS 
>>>>>> and MDS are running on two separate servers.
>>>>>> 
>>>>>> ----------------------------------------------------------------------------------
>>>>>> ChangeLog {
>>>>>>     MDT {
>>>>>>         mdt_name = "MDT0000";
>>>>>>         reader_id = "cl4";      # yes, it is cl4
>>>>>>     }
>>>>>>     force_polling    = ON ;
>>>>>>     polling_interval = 60s ;
>>>>>> }
>>>>>> ----------------------------------------------------------------------------------
>>>>>> 
>>>>>> ----------------------------------------------------------------------------------
>>>>>> # lfs changelog scratch-MDT0000 |head
>>>>>> 
>>>>>> 8211574766 10OPEN  22:24:39.830737089 2018.11.27 0x43 
>>>>>> t=[0x200033cc4:0x4e9d:0x0]
>>>>>> 8211574767 13TRUNC 22:24:39.831280872 2018.11.27 0xe 
>>>>>> t=[0x200033cc4:0x4e9d:0x0]
>>>>>> 8211574768 11CLOSE 22:24:39.831774075 2018.11.27 0x43 
>>>>>> t=[0x200033cc4:0x4e9d:0x0]
>>>>>> 8211574769 10OPEN  22:24:39.831950980 2018.11.27 0x43 
>>>>>> t=[0x200033cc4:0x4e9e:0x0]
>>>>>> 8211574770 13TRUNC 22:24:39.832289331 2018.11.27 0xe 
>>>>>> t=[0x200033cc4:0x4e9e:0x0]
>>>>>> 8211574771 11CLOSE 22:24:39.832765269 2018.11.27 0x43 
>>>>>> t=[0x200033cc4:0x4e9e:0x0]
>>>>>> 8211574772 10OPEN  22:24:39.832942039 2018.11.27 0x43 
>>>>>> t=[0x200033cc4:0x4e9f:0x0]
>>>>>> 8211574773 13TRUNC 22:24:39.833241628 2018.11.27 0xe 
>>>>>> t=[0x200033cc4:0x4e9f:0x0]
>>>>>> 8211574774 11CLOSE 22:24:39.833541406 2018.11.27 0x43 
>>>>>> t=[0x200033cc4:0x4e9f:0x0]
>>>>>> 8211574775 10OPEN  22:24:39.833672445 2018.11.27 0x43 
>>>>>> t=[0x200033cc4:0x4ea0:0x0]
>>>>>> ----------------------------------------------------------------------------------
>>>>>> 
>>>>>> ----------------------------------------------------------------------------------
>>>>>> # robinhood --readlog
>>>>>> 
>>>>>> 2018/11/28 07:20:54 [2070664/1] Main | EntryProcessor successfully 
>>>>>> initialized
>>>>>> 2018/11/28 07:20:54 [2070664/1] ChangeLog | mdt[0] = MDT0000
>>>>>> 2018/11/28 07:20:54 [2070664/1] ListMgr | Logged on to database 
>>>>>> 'robinhood_scratch' successfully
>>>>>> 2018/11/28 07:20:54 [2070664/1] ListMgr | SQL query: SET SESSION 
>>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED
>>>>>> 2018/11/28 07:20:54 [2070664/3] ListMgr | Logged on to database 
>>>>>> 'robinhood_scratch' successfully
>>>>>> 2018/11/28 07:20:54 [2070664/3] ListMgr | SQL query: SET SESSION 
>>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED
>>>>>> 2018/11/28 07:20:54 [2070664/5] ListMgr | Logged on to database 
>>>>>> 'robinhood_scratch' successfully
>>>>>> 2018/11/28 07:20:54 [2070664/5] ListMgr | SQL query: SET SESSION 
>>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED
>>>>>> 2018/11/28 07:20:54 [2070664/6] ListMgr | Logged on to database 
>>>>>> 'robinhood_scratch' successfully
>>>>>> 2018/11/28 07:20:54 [2070664/6] ListMgr | SQL query: SET SESSION 
>>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED
>>>>>> 2018/11/28 07:20:54 [2070664/4] ListMgr | Logged on to database 
>>>>>> 'robinhood_scratch' successfully
>>>>>> 2018/11/28 07:20:54 [2070664/4] ListMgr | SQL query: SET SESSION 
>>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED
>>>>>> 2018/11/28 07:20:54 [2070664/1] ListMgr | SQL query: SELECT value FROM 
>>>>>> VARS WHERE varname='CL_LastCommit_MDT0000'
>>>>>> 2018/11/28 07:20:54 [2070664/1] ListMgr | SQL query: SELECT value FROM 
>>>>>> VARS WHERE varname='ChangelogLastCommit_MDT0000'
>>>>>> 2018/11/28 07:20:54 [2070664/1] ChangeLog | Opening chglog for 
>>>>>> scratch-MDT0000 (start_rec=0)
>>>>>> 2018/11/28 07:20:54 [2070664/1] ChangeLog | ERROR -25 opening changelog 
>>>>>> for MDT 'scratch-MDT0000': Inappropriate ioctl for device
>>>>>> 2018/11/28 07:20:54 [2070664/1] Main | Error 25 initializing ChangeLog 
>>>>>> Reader
>>>>>> ----------------------------------------------------------------------------------
>>>>>> 
>>>>>> 
>>>>>> Thanks,
>>>>>> Lorand Szentannai
>>>>>> 
>>>>>> _______________________________________________
>>>>>> robinhood-support mailing list
>>>>>> robinhood-support@lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/robinhood-support
>>>> 
>>>> _______________________________________________
>>>> robinhood-support mailing list
>>>> robinhood-support@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/robinhood-support
>>>> 
>> 
> 



_______________________________________________
robinhood-support mailing list
robinhood-support@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/robinhood-support

Reply via email to