Re: [lustre-discuss] Robinhood exhausting RPC resources against 2.5.5 lustre file systems

[Jessica Otey]

Hi Megan (et al.),

I don't understand the behavior, either... I've worked successfully with 
changelogs in the past, and indeed it is very lightweight. (Since 
robinhood has not been running anywhere, I'd already removed all the 
changelog readers from the various MDTs for the reasons you noted.)


Whatever my problem is does not manifest as a load issue, on either 
client or MDT side. It manifests rather as some sort of connection 
failure. Here's the most recent example, which maybe will generate more 
ideas as to cause.


On our third lustre fs (one we use for backups), I was able to complete 
a file system scan to populate the database, but then when I activated 
changelogs, the client almost immediately experienced the disconnections 
we've seen on the other two systems.


Here's the log from the MDT (heinlein, 10.7.17.126). The robinhood 
client is akebono (10.7.17.122):

May 16 16:05:51 heinlein kernel: Lustre: lard-MDD: changelog on
May 16 16:05:51 heinlein kernel: Lustre: Modifying parameter 
general.mdd.lard-MDT*.changelog_mask in log params
May 16 16:13:16 heinlein kernel: Lustre: lard-MDT: Client 
2d1aedc0-1f5e-2741-689a-169922a2593b (at 10.7.17.122@o2ib) reconnecting
May 16 16:13:17 heinlein kernel: Lustre: lard-MDT: Client 
2d1aedc0-1f5e-2741-689a-169922a2593b (at 10.7.17.122@o2ib) reconnecting
May 16 16:13:17 heinlein kernel: Lustre: Skipped 7458 previous similar 
messages


Here's what akebono (10.7.17.122) reported:

May 16 16:13:16 akebono kernel: LustreError: 11-0: 
lard-MDT-mdc-880fd68d7000: Communicating with 10.7.17.126@o2ib, 
operation llog_origin_handle_destroy failed with -19.
May 16 16:13:16 akebono kernel: Lustre: 
lard-MDT-mdc-880fd68d7000: Connection to lard-MDT (at 
10.7.17.126@o2ib) was lost; in progress operations using this service 
will wait for recovery to complete
May 16 16:13:16 akebono kernel: Lustre: 
lard-MDT-mdc-880fd68d7000: Connection restored to lard-MDT 
(at 10.7.17.126@o2ib)
May 16 16:13:17 akebono kernel: LustreError: 11-0: 
lard-MDT-mdc-880fd68d7000: Communicating with 10.7.17.126@o2ib, 
operation llog_origin_handle_destroy failed with -19.
May 16 16:13:17 akebono kernel: LustreError: Skipped 7458 previous 
similar messages
May 16 16:13:17 akebono kernel: Lustre: 
lard-MDT-mdc-880fd68d7000: Connection to lard-MDT (at 
10.7.17.126@o2ib) was lost; in progress operations using this service 
will wait for recovery to complete
May 16 16:13:17 akebono kernel: Lustre: Skipped 7458 previous similar 
messages
May 16 16:13:17 akebono kernel: Lustre: 
lard-MDT-mdc-880fd68d7000: Connection restored to lard-MDT 
(at 10.7.17.126@o2ib)
May 16 16:13:17 akebono kernel: Lustre: Skipped 7458 previous similar 
messages
May 16 16:13:18 akebono kernel: LustreError: 11-0: 
lard-MDT-mdc-880fd68d7000: Communicating with 10.7.17.126@o2ib, 
operation llog_origin_handle_destroy failed with -19.
May 16 16:13:18 akebono kernel: LustreError: Skipped 14924 previous 
similar messages


Jessica

On 5/19/17 8:58 AM, Ms. Megan Larko wrote:

Greetings Jessica,

I'm not sure I am correctly understanding the behavior "robinhood 
activity floods the MDT".   The robinhood program as you (and I) are 
using it is consuming the MDT CHANGELOG via a reader_id which was 
assigned when the CHANGELOG was enabled on the MDT.   You can check 
the MDS for these readers via "lctl get_param mdd.*.changelog_users".  
Each CHANGELOG reader must either be consumed by a process or 
destroyed otherwise the CHANGELOG will grow until it consumes 
sufficient space to stop the MDT from functioning correctly.  So 
robinhood should consume and then clear the CHANGELOG via this 
reader_id.  This implementation of robinhood is actually a rather 
light-weight process as far as the MDS is concerned.   The load issues 
I encountered were on the robinhood server itself which is a separate 
server from the Lustre MGS/MDS server.


Just curious, have you checked for multiple reader_id's on your MDS 
for this Lustre file system?


P.S. My robinhood configuration file is using nb_threads = 8, just for 
a data point.


Cheers,
megan





___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Robinhood exhausting RPC resources against 2.5.5 lustre file systems

[Jessica Otey]

I think that may be a red herring related to rsyslog?  When we most 
recently rebooted the MDT, this is the log (still on the box, not on the 
log server):


May  3 14:24:22 asimov kernel: LNet: HW CPU cores: 12, npartitions: 4
May  3 14:24:30 asimov kernel: LNet: Added LNI 10.7.17.8@o2ib [8/256/0/180]

And lctl list_nids gives it once:

[root@asimov ~]# lctl list_nids
10.7.17.8@o2ib

Jessica

On 5/19/17 10:13 AM, Jeff Johnson wrote:

Jessica,

You are getting a NID registering twice. Doug noticed and pointed it 
out. I'd look to see if that is one machine doing something twice or 
two machines with the same NID.


--Jeff

On Fri, May 19, 2017 at 05:58 Ms. Megan Larko > wrote:


Greetings Jessica,

I'm not sure I am correctly understanding the behavior "robinhood
activity floods the MDT".   The robinhood program as you (and I)
are using it is consuming the MDT CHANGELOG via a reader_id which
was assigned when the CHANGELOG was enabled on the MDT. You can
check the MDS for these readers via "lctl get_param
mdd.*.changelog_users".  Each CHANGELOG reader must either be
consumed by a process or destroyed otherwise the CHANGELOG will
grow until it consumes sufficient space to stop the MDT from
functioning correctly.  So robinhood should consume and then clear
the CHANGELOG via this reader_id.  This implementation of
robinhood is actually a rather light-weight process as far as the
MDS is concerned.   The load issues I encountered were on the
robinhood server itself which is a separate server from the Lustre
MGS/MDS server.

Just curious, have you checked for multiple reader_id's on your
MDS for this Lustre file system?

P.S. My robinhood configuration file is using nb_threads = 8, just
for a data point.

Cheers,
megan

On Thu, May 18, 2017 at 2:36 PM, Jessica Otey > wrote:

Hi Megan,

Thanks for your input. We use percona, a drop-in replacement
for mysql... The robinhood activity floods the MDT, but it
does not seem to produce any excessive load on the robinhood
box...

Anyway, FWIW...

~]# mysql --version
mysql  Ver 14.14 Distrib 5.5.54-38.6, for Linux (x86_64) using
readline 5.1

Product: robinhood
Version: 3.0-1
Build:   2017-03-13 10:29:26

Compilation switches:
Lustre filesystems
Lustre Version: 2.5
Address entries by FID
MDT Changelogs supported

Database binding: MySQL

RPM: robinhood-lustre-3.0-1.lustre2.5.el6.x86_64

Lustre rpms:

lustre-client-2.5.5-2.6.32_642.15.1.el6.x86_64_g22a210f.x86_64
lustre-client-modules-2.5.5-2.6.32_642.15.1.el6.x86_64_g22a210f.x86_64


On 5/18/17 11:55 AM, Ms. Megan Larko wrote:

With regards to (WRT) Subject "Robinhood exhausting RPC
resources against 2.5.5  lustre file systems", what version
of robinhood and what version of MySQL database?   I mention
this because I have been working with robinhood-3.0-0.rc1 and
initially MySQL-5.5.32 and Lustre 2.5.42.1 on
kernel-2.6.32-573 and had issues in which the robinhood
server consumed more than the total amount of 32 CPU cores on
the robinhood server (with 128 G RAM) and would functionally
hang the robinhood server.   The issue was solved for me by
changing to MySQL-5.6.35.   It was the "sort" command in
robinhood that was not working well with the MySQL-5.5.32.

Cheers,
megan




___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org

http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

--
--
Jeff Johnson
Co-Founder
Aeon Computing

jeff.john...@aeoncomputing.com 
www.aeoncomputing.com 
t: 858-412-3810 x1001   f: 858-412-3845
m: 619-204-9061

4170 Morena Boulevard, Suite D - San Diego, CA 92117

High-Performance Computing / Lustre Filesystems / Scale-out Storage


___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Robinhood exhausting RPC resources against 2.5.5 lustre file systems

[Jeff Johnson]

Jessica,

You are getting a NID registering twice. Doug noticed and pointed it out.
I'd look to see if that is one machine doing something twice or two
machines with the same NID.

--Jeff

On Fri, May 19, 2017 at 05:58 Ms. Megan Larko  wrote:

> Greetings Jessica,
>
> I'm not sure I am correctly understanding the behavior "robinhood activity
> floods the MDT".   The robinhood program as you (and I) are using it is
> consuming the MDT CHANGELOG via a reader_id which was assigned when the
> CHANGELOG was enabled on the MDT.   You can check the MDS for these readers
> via "lctl get_param mdd.*.changelog_users".  Each CHANGELOG reader must
> either be consumed by a process or destroyed otherwise the CHANGELOG will
> grow until it consumes sufficient space to stop the MDT from functioning
> correctly.  So robinhood should consume and then clear the CHANGELOG via
> this reader_id.  This implementation of robinhood is actually a rather
> light-weight process as far as the MDS is concerned.   The load issues I
> encountered were on the robinhood server itself which is a separate server
> from the Lustre MGS/MDS server.
>
> Just curious, have you checked for multiple reader_id's on your MDS for
> this Lustre file system?
>
> P.S. My robinhood configuration file is using nb_threads = 8, just for a
> data point.
>
> Cheers,
> megan
>
> On Thu, May 18, 2017 at 2:36 PM, Jessica Otey  wrote:
>
>> Hi Megan,
>>
>> Thanks for your input. We use percona, a drop-in replacement for mysql...
>> The robinhood activity floods the MDT, but it does not seem to produce any
>> excessive load on the robinhood box...
>>
>> Anyway, FWIW...
>>
>> ~]# mysql --version
>> mysql  Ver 14.14 Distrib 5.5.54-38.6, for Linux (x86_64) using readline
>> 5.1
>>
>> Product: robinhood
>> Version: 3.0-1
>> Build:   2017-03-13 10:29:26
>>
>> Compilation switches:
>> Lustre filesystems
>> Lustre Version: 2.5
>> Address entries by FID
>> MDT Changelogs supported
>>
>> Database binding: MySQL
>>
>> RPM: robinhood-lustre-3.0-1.lustre2.5.el6.x86_64
>> Lustre rpms:
>>
>> lustre-client-2.5.5-2.6.32_642.15.1.el6.x86_64_g22a210f.x86_64
>> lustre-client-modules-2.5.5-2.6.32_642.15.1.el6.x86_64_g22a210f.x86_64
>>
>> On 5/18/17 11:55 AM, Ms. Megan Larko wrote:
>>
>> With regards to (WRT) Subject "Robinhood exhausting RPC resources against
>> 2.5.5   lustre file systems", what version of robinhood and what version of
>> MySQL database?   I mention this because I have been working with
>> robinhood-3.0-0.rc1 and initially MySQL-5.5.32 and Lustre 2.5.42.1 on
>> kernel-2.6.32-573 and had issues in which the robinhood server consumed
>> more than the total amount of 32 CPU cores on the robinhood server (with
>> 128 G RAM) and would functionally hang the robinhood server.   The issue
>> was solved for me by changing to MySQL-5.6.35.   It was the "sort" command
>> in robinhood that was not working well with the MySQL-5.5.32.
>>
>> Cheers,
>> megan
>>
>>
>>
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
-- 
--
Jeff Johnson
Co-Founder
Aeon Computing

jeff.john...@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x1001   f: 858-412-3845
m: 619-204-9061

4170 Morena Boulevard, Suite D - San Diego, CA 92117

High-Performance Computing / Lustre Filesystems / Scale-out Storage
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Robinhood exhausting RPC resources against 2.5.5 lustre file systems

[Ms. Megan Larko]

Greetings Jessica,

I'm not sure I am correctly understanding the behavior "robinhood activity
floods the MDT".   The robinhood program as you (and I) are using it is
consuming the MDT CHANGELOG via a reader_id which was assigned when the
CHANGELOG was enabled on the MDT.   You can check the MDS for these readers
via "lctl get_param mdd.*.changelog_users".  Each CHANGELOG reader must
either be consumed by a process or destroyed otherwise the CHANGELOG will
grow until it consumes sufficient space to stop the MDT from functioning
correctly.  So robinhood should consume and then clear the CHANGELOG via
this reader_id.  This implementation of robinhood is actually a rather
light-weight process as far as the MDS is concerned.   The load issues I
encountered were on the robinhood server itself which is a separate server
from the Lustre MGS/MDS server.

Just curious, have you checked for multiple reader_id's on your MDS for
this Lustre file system?

P.S. My robinhood configuration file is using nb_threads = 8, just for a
data point.

Cheers,
megan

On Thu, May 18, 2017 at 2:36 PM, Jessica Otey  wrote:

> Hi Megan,
>
> Thanks for your input. We use percona, a drop-in replacement for mysql...
> The robinhood activity floods the MDT, but it does not seem to produce any
> excessive load on the robinhood box...
>
> Anyway, FWIW...
>
> ~]# mysql --version
> mysql  Ver 14.14 Distrib 5.5.54-38.6, for Linux (x86_64) using readline 5.1
>
> Product: robinhood
> Version: 3.0-1
> Build:   2017-03-13 10:29:26
>
> Compilation switches:
> Lustre filesystems
> Lustre Version: 2.5
> Address entries by FID
> MDT Changelogs supported
>
> Database binding: MySQL
>
> RPM: robinhood-lustre-3.0-1.lustre2.5.el6.x86_64
> Lustre rpms:
>
> lustre-client-2.5.5-2.6.32_642.15.1.el6.x86_64_g22a210f.x86_64
> lustre-client-modules-2.5.5-2.6.32_642.15.1.el6.x86_64_g22a210f.x86_64
>
> On 5/18/17 11:55 AM, Ms. Megan Larko wrote:
>
> With regards to (WRT) Subject "Robinhood exhausting RPC resources against
> 2.5.5   lustre file systems", what version of robinhood and what version of
> MySQL database?   I mention this because I have been working with
> robinhood-3.0-0.rc1 and initially MySQL-5.5.32 and Lustre 2.5.42.1 on
> kernel-2.6.32-573 and had issues in which the robinhood server consumed
> more than the total amount of 32 CPU cores on the robinhood server (with
> 128 G RAM) and would functionally hang the robinhood server.   The issue
> was solved for me by changing to MySQL-5.6.35.   It was the "sort" command
> in robinhood that was not working well with the MySQL-5.5.32.
>
> Cheers,
> megan
>
>
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] seclabel

[Sebastien Buisson]

> Le 19 mai 2017 à 08:47, Robin Humble  a écrit :
> On Wed, May 17, 2017 at 02:37:31PM +, Sebastien Buisson wrote:
>> 
>> Reading the discussion in the ticket, supporting xattr at the time of Lustre 
>> 1.8 and 2.0 was causing issues on MDS side in some situations. So it was 
>> decided to discard security.capability xattr on Lustre client side. I think 
>> Andreas might have some insight, as he apparently participated in b15587.
> 
> my word that's a long time ago...
> I don't see much in the way of jira tickets about getxattr issues on
> MDS in recent times, and they're much more heavily used these days, so
> I hope that particular problem has long since been fixed.
> 
> should I open a jira ticket to track re-enabling of security.capabilities?

Yes please. At the same time, would you mind pushing to review.whamcloud.com 
the patch you sent to lustre-devel?

> 
>> In any case, it is important to make clear that file capabilities, the 
>> feature you want to use, is completely distinct from SELinux.
>> On the one hand, Capabilities are a Linux mechanism to refine permissions 
>> granted to privileged processes, by dividing the privileges traditionally 
>> associated with superuser into distinct units (known as capabilities).
>> On the other hand, SELinux is the Linux implementation of Mandatory Access 
>> Control.
>> Both Capabilities and SELinux rely on values stored into file extended 
>> attributes, but this is the only thing they have in common.
> 
> 10-4. thanks.
> 
> 'ls --color' requests the security.capability xattr so this would
> be heavily accessed. do you think this is handled well enough currently
> to not affect performance significantly?
> 
> setxattr would be minimal and not performance critical, unlike with eg.
> selinux and creat.
> 

For sure retrieving this xattr adds an overhead. But with recent versions of 
Lustre I am not aware of bugs in xattr handling.
I think it would be helpful, in the Jira ticket you would open, to have 
comments from people that participated in the resolution of Bugzilla #15587.

Thanks,
Sebastien.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] seclabel

[Robin Humble]

Hi Sebastien,

On Wed, May 17, 2017 at 02:37:31PM +, Sebastien Buisson wrote:
> Le 17 mai 2017 à 16:16, Robin Humble  a écrit :
>> I took a gander at the source and noticed that llite/xattr.c
>> deliberately filters out 'security.capability' and returns 0/-ENODATA
>> for setcap/getcap, which is indeed what strace sees. so setcap/getcap
>> is never even sent to the MDS.
>> 
>> if I remove that filter (see patch on lustre-devel) then setcap/getcap
>> works ->
...
>> 'b15587' is listed as the reason for the filtering.
>> I don't know what that refers to.
>> is it still relevant?
>b15587 refers to the old Lustre Bugzilla tracking tool:
>https://projectlava.xyratex.com/show_bug.cgi?id=15587
>
>Reading the discussion in the ticket, supporting xattr at the time of Lustre 
>1.8 and 2.0 was causing issues on MDS side in some situations. So it was 
>decided to discard security.capability xattr on Lustre client side. I think 
>Andreas might have some insight, as he apparently participated in b15587.

my word that's a long time ago...
I don't see much in the way of jira tickets about getxattr issues on
MDS in recent times, and they're much more heavily used these days, so
I hope that particular problem has long since been fixed.

should I open a jira ticket to track re-enabling of security.capabilities?

>In any case, it is important to make clear that file capabilities, the feature 
>you want to use, is completely distinct from SELinux.
>On the one hand, Capabilities are a Linux mechanism to refine permissions 
>granted to privileged processes, by dividing the privileges traditionally 
>associated with superuser into distinct units (known as capabilities).
>On the other hand, SELinux is the Linux implementation of Mandatory Access 
>Control.
>Both Capabilities and SELinux rely on values stored into file extended 
>attributes, but this is the only thing they have in common.

10-4. thanks.

'ls --color' requests the security.capability xattr so this would
be heavily accessed. do you think this is handled well enough currently
to not affect performance significantly?

setxattr would be minimal and not performance critical, unlike with eg.
selinux and creat.

cheers,
robin
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org