Hi, I was about to say I'm sure and then realized that I've had two lustreapi libraries laying around on the system, one being leftover from a previous lustre client rpm. Obviously robinhood was picking up the wrong one.
------------------------------------------------------------------------------- lorand@robinhood:~ $ rpmbuild -tb --with lustre rpmbuild/SOURCES/robinhood-3.1.4.tar.gz >& compile-$(date +%F).log lorand@robinhood:~ $ grep liblustreapi compile-2018-11-29.log Requires: /bin/bash /bin/sh ld-linux-x86-64.so.2()(64bit) ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) libc.so.6()(64bit) libc.so.6(GLIBC_2.10)(64bit) libc.so.6(GLIBC_2.14)(64bit) libc.so.6(GLIBC_2.17)(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit) libc.so.6(GLIBC_2.4)(64bit) libcrypto.so.10()(64bit) libdl.so.2()(64bit) libdl.so.2(GLIBC_2.2.5)(64bit) libglib-2.0.so.0()(64bit) libgthread-2.0.so.0()(64bit) libjemalloc.so.1()(64bit) liblustreapi.so()(64bit) libm.so.6()(64bit) libmysqlclient.so.18()(64bit) libmysqlclient.so.18(libmysqlclient_18)(64bit) libpthread.so.0()(64bit) libpthread.so.0(GLIBC_2.2.5)(64bit) libpthread.so.0(GLIBC_2.3.2)(64bit) librbh_mod_alerter.so.0()(64bit) librbh_mod_backup.so.0()(64bit) librbh_mod_basic.so.0()(64bit) librbh_mod_checker.so.0()(64bit) librbh_mod_common.so.0()(64bit) librbh_mod_lhsm.so.0()(64bit) librbh_mod_modeguard.so.0()(64bit) librbh_mod_test.so.0()(64bit) libssl.so.10()(64bit) libz.so.1()(64bit) rtld(GNU_HASH) Requires: libc.so.6()(64bit) libc.so.6(GLIBC_2.14)(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit) libglib-2.0.so.0()(64bit) libgthread-2.0.so.0()(64bit) libjemalloc.so.1()(64bit) liblustreapi.so()(64bit) libpthread.so.0()(64bit) libpthread.so.0(GLIBC_2.2.5)(64bit) librt.so.1()(64bit) libz.so.1()(64bit) rtld(GNU_HASH) lorand@robinhood:~ $ rpm -qa |egrep "lustre|robinhood" lustre-client-debuginfo-2.10.5-1.el7.x86_64 kmod-lustre-client-tests-2.10.5-1.el7.x86_64 robinhood-tools-3.1.4-1.lustre.el7.x86_64 lustre-iokit-2.10.5-1.el7.x86_64 robinhood-tests-3.1.4-1.x86_64 kmod-lustre-client-2.10.5-1.el7.x86_64 lustre-client-tests-2.10.5-1.el7.x86_64 robinhood-debuginfo-3.1.4-1.el7.x86_64 lustre-client-2.10.5-1.el7.x86_64 robinhood-lustre-3.1.4-1.lustre.el7.x86_64 robinhood-adm-3.1.4-1.x86_64 lorand@robinhood:~ $ ldd $(which robinhood) linux-vdso.so.1 => (0x00007ffc591f0000) libmysqlclient.so.18 => /usr/lib64/mysql/libmysqlclient.so.18 (0x00007f82f3278000) libm.so.6 => /lib64/libm.so.6 (0x00007f82f2f76000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f82f2d72000) libssl.so.10 => /lib64/libssl.so.10 (0x00007f82f2b00000) libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f82f269f000) liblustreapi.so => /lib/liblustreapi.so (0x00007f82f2485000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f82f2269000) libgthread-2.0.so.0 => /lib64/libgthread-2.0.so.0 (0x00007f82f2067000) libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f82f1d53000) libjemalloc.so.1 => /lib64/libjemalloc.so.1 (0x00007f82f1b1f000) libz.so.1 => /lib64/libz.so.1 (0x00007f82f1909000) libc.so.6 => /lib64/libc.so.6 (0x00007f82f153c000) /lib64/ld-linux-x86-64.so.2 (0x00007f82f3778000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f82f1235000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f82f0fe8000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f82f0d00000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f82f0afc000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f82f08c9000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f82f0667000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f82f0451000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f82f0243000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f82f003f000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f82efe26000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f82efbff000) lorand@robinhood:~ $ md5sum /lib/liblustreapi.so c58640c0bfe0c7551e917db122f2f42d /lib/liblustreapi.so lorand@robinhood:~ $ md5sum /usr/lib64/liblustreapi.so 1e649f4b0a541a8f62c1bf0c0402ae7f /usr/lib64/liblustreapi.so lorand@robinhood:~ $ rpm -qf /usr/lib64/liblustreapi.so lustre-client-2.10.5-1.el7.x86_64 lorand@robinhood:~ $ ls -la /lib/liblustreapi.so -rw-r--r-- 1 root root 412152 Sep 18 2017 /lib/liblustreapi.so [root@robinhood x86_64]# mv /lib/liblustreapi.so{,.dis} ------------------------------------------------------------------------------- Thank you! Lorand > On 29 Nov 2018, at 11:46, quentin.bou...@cea.fr wrote: > > Hi again, > > I am pretty sure this is it though. I remember having to deal with it while > working on lustre. > > > I initially installed robinhood from rpms, but now fetched the sources and > > freshly compiled, built new rpms. > > And are you _sure_ lustre 2.10 was installed on the node you compiled/built > robinhood's rpms? > > Quentin > > Le 29/11/2018 à 10:51, Lorand Szentannai a écrit : >> Hi Quentin, >> >> I initially installed robinhood from rpms, but now fetched the sources and >> freshly compiled, built new rpms. >> Unfortunately the behavior is still the same. >> >> >> Thanks, >> Lorand >> >>> On 28 Nov 2018, at 15:46, quentin.bou...@cea.fr wrote: >>> >>> Hi Lorand, >>> >>> I seem to remember this is caused by an update of lustre's changelog API. >>> >>> Did compile robinhood against the right version of liblustreapi? (there is >>> no need for ioctl in Lustre 2.10's API) >>> >>> Quentin >>> >>> Le 28/11/2018 à 15:04, Lorand Szentannai a écrit : >>>> Hi Thomas, >>>> >>>> I actually did that already but forgot to add the strace lines to the >>>> initial email. But it is still unclear to me why open function is called >>>> with the O_RDONLY flag. >>>> >>>> [pid 2348817] open("/cluster", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 12 >>>> [pid 2348817] ioctl(12, _IOC(_IOC_WRITE, 0x66, 0x94, 0x08), >>>> 0x7ffef3f20240) = -1 ENOTTY (Inappropriate ioctl for device) >>>> >>>> # grep lustre /proc/mounts >>>> 10.3.5.18@o2ib,10.4.5.18@o2ib1:10.3.5.19@o2ib,10.4.5.19@o2ib1:/scratch >>>> /cluster lustre rw,flock,lazystatfs 0 0 >>>> >>>> >>>> Thanks, >>>> Lorand Szentannai >>>> >>>>> On 28 Nov 2018, at 12:47, LEIBOVICI Thomas <thomas.leibov...@cea.fr> >>>>> wrote: >>>>> >>>>> Hello, >>>>> >>>>> If you run "strace -f robinhood --readlog" you should see the ioctl that >>>>> causes this error, and its arguments. >>>>> >>>>> Thomas >>>>> >>>>> On 11/28/18 07:36, Lorand Szentannai wrote: >>>>>> Hi, >>>>>> >>>>>> We've recently upgraded to Lustre 2.10.5 and to Robinhood to 3.1.4. >>>>>> Processing changelogs fails now with "ERROR -25 opening changelog for >>>>>> MDT 'scratch-MDT0000': Inappropriate ioctl for device". >>>>>> >>>>>> Could you please advice on how to solve this? Note that currently MGS >>>>>> and MDS are running on two separate servers. >>>>>> >>>>>> ---------------------------------------------------------------------------------- >>>>>> ChangeLog { >>>>>> MDT { >>>>>> mdt_name = "MDT0000"; >>>>>> reader_id = "cl4"; # yes, it is cl4 >>>>>> } >>>>>> force_polling = ON ; >>>>>> polling_interval = 60s ; >>>>>> } >>>>>> ---------------------------------------------------------------------------------- >>>>>> >>>>>> ---------------------------------------------------------------------------------- >>>>>> # lfs changelog scratch-MDT0000 |head >>>>>> >>>>>> 8211574766 10OPEN 22:24:39.830737089 2018.11.27 0x43 >>>>>> t=[0x200033cc4:0x4e9d:0x0] >>>>>> 8211574767 13TRUNC 22:24:39.831280872 2018.11.27 0xe >>>>>> t=[0x200033cc4:0x4e9d:0x0] >>>>>> 8211574768 11CLOSE 22:24:39.831774075 2018.11.27 0x43 >>>>>> t=[0x200033cc4:0x4e9d:0x0] >>>>>> 8211574769 10OPEN 22:24:39.831950980 2018.11.27 0x43 >>>>>> t=[0x200033cc4:0x4e9e:0x0] >>>>>> 8211574770 13TRUNC 22:24:39.832289331 2018.11.27 0xe >>>>>> t=[0x200033cc4:0x4e9e:0x0] >>>>>> 8211574771 11CLOSE 22:24:39.832765269 2018.11.27 0x43 >>>>>> t=[0x200033cc4:0x4e9e:0x0] >>>>>> 8211574772 10OPEN 22:24:39.832942039 2018.11.27 0x43 >>>>>> t=[0x200033cc4:0x4e9f:0x0] >>>>>> 8211574773 13TRUNC 22:24:39.833241628 2018.11.27 0xe >>>>>> t=[0x200033cc4:0x4e9f:0x0] >>>>>> 8211574774 11CLOSE 22:24:39.833541406 2018.11.27 0x43 >>>>>> t=[0x200033cc4:0x4e9f:0x0] >>>>>> 8211574775 10OPEN 22:24:39.833672445 2018.11.27 0x43 >>>>>> t=[0x200033cc4:0x4ea0:0x0] >>>>>> ---------------------------------------------------------------------------------- >>>>>> >>>>>> ---------------------------------------------------------------------------------- >>>>>> # robinhood --readlog >>>>>> >>>>>> 2018/11/28 07:20:54 [2070664/1] Main | EntryProcessor successfully >>>>>> initialized >>>>>> 2018/11/28 07:20:54 [2070664/1] ChangeLog | mdt[0] = MDT0000 >>>>>> 2018/11/28 07:20:54 [2070664/1] ListMgr | Logged on to database >>>>>> 'robinhood_scratch' successfully >>>>>> 2018/11/28 07:20:54 [2070664/1] ListMgr | SQL query: SET SESSION >>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED >>>>>> 2018/11/28 07:20:54 [2070664/3] ListMgr | Logged on to database >>>>>> 'robinhood_scratch' successfully >>>>>> 2018/11/28 07:20:54 [2070664/3] ListMgr | SQL query: SET SESSION >>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED >>>>>> 2018/11/28 07:20:54 [2070664/5] ListMgr | Logged on to database >>>>>> 'robinhood_scratch' successfully >>>>>> 2018/11/28 07:20:54 [2070664/5] ListMgr | SQL query: SET SESSION >>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED >>>>>> 2018/11/28 07:20:54 [2070664/6] ListMgr | Logged on to database >>>>>> 'robinhood_scratch' successfully >>>>>> 2018/11/28 07:20:54 [2070664/6] ListMgr | SQL query: SET SESSION >>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED >>>>>> 2018/11/28 07:20:54 [2070664/4] ListMgr | Logged on to database >>>>>> 'robinhood_scratch' successfully >>>>>> 2018/11/28 07:20:54 [2070664/4] ListMgr | SQL query: SET SESSION >>>>>> TRANSACTION ISOLATION LEVEL READ COMMITTED >>>>>> 2018/11/28 07:20:54 [2070664/1] ListMgr | SQL query: SELECT value FROM >>>>>> VARS WHERE varname='CL_LastCommit_MDT0000' >>>>>> 2018/11/28 07:20:54 [2070664/1] ListMgr | SQL query: SELECT value FROM >>>>>> VARS WHERE varname='ChangelogLastCommit_MDT0000' >>>>>> 2018/11/28 07:20:54 [2070664/1] ChangeLog | Opening chglog for >>>>>> scratch-MDT0000 (start_rec=0) >>>>>> 2018/11/28 07:20:54 [2070664/1] ChangeLog | ERROR -25 opening changelog >>>>>> for MDT 'scratch-MDT0000': Inappropriate ioctl for device >>>>>> 2018/11/28 07:20:54 [2070664/1] Main | Error 25 initializing ChangeLog >>>>>> Reader >>>>>> ---------------------------------------------------------------------------------- >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Lorand Szentannai >>>>>> >>>>>> _______________________________________________ >>>>>> robinhood-support mailing list >>>>>> robinhood-support@lists.sourceforge.net >>>>>> https://lists.sourceforge.net/lists/listinfo/robinhood-support >>>> >>>> _______________________________________________ >>>> robinhood-support mailing list >>>> robinhood-support@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/robinhood-support >>>> >> > _______________________________________________ robinhood-support mailing list robinhood-support@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/robinhood-support