[389-users] Re: Problem adding users to group.

2017-08-20 Thread William Brown 
On Fri, 2017-08-18 at 13:32 +, Lucas Diedrich wrote:
> Thanks Willian and David,
> 
> The first tunning already helped on making the database more responsive,
> now i'm going to make the others changes and monitor the environment.
> 
> Thanks a lot.
> 

You are most welcome. Make sure you investigate and look at the various
cn=monitor objects in the system. The documentation describes them well
and how to read them. 

If you have further questions, we are always happy to help you! 

-- 
Sincerely,

William Brown
Software Engineer
Red Hat, Australia/Brisbane



signature.asc
Description: This is a digitally signed message part
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Problem adding users to group.

2017-08-18 Thread Lucas Diedrich 
Thanks Willian and David,

The first tunning already helped on making the database more responsive,
now i'm going to make the others changes and monitor the environment.

Thanks a lot.

Em qui, 17 de ago de 2017 às 20:50, William Brown 
escreveu:

> On Thu, 2017-08-17 at 10:53 +1000, William Brown wrote:
> > On Tue, 2017-08-15 at 13:15 +, Lucas Diedrich wrote:
> > > Willian,
> > >
> > > Cache values from cn=config + cn=userRoot:
> > > https://paste.fedoraproject.org/paste/ZqmA2wUkQDcSUaUIcpGDhg
> > > free -h + lspcu:
> > > https://paste.fedoraproject.org/paste/Br8Vz5-quxtcatiDyMy3QA
> > >
> > > I'll check it out the docs for the optimization + wait until your
> response,
> > > i think it will more secure.
> >
> > Hey mate,
> >
> > Those values are all default of 10MB. You probably want to change:
> >
> > dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config
> > nsslapd-cachememsize: 10485760
> > nsslapd-dncachememsize: 10485760
> >
> > and
> >
> > dn: cn=config,cn=ldbm database,cn=plugins,cn=config
> > nsslapd-dbcachesize: 1000
> >
> > Looking at your values of free/cpu, I would recommend something like:
> >
> > dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config
> > nsslapd-cachememsize: 268435456
> > nsslapd-dncachememsize: 67108864
> >
> > and
> >
> > dn: cn=config,cn=ldbm database,cn=plugins,cn=config
> > nsslapd-dbcachesize: 536870912
> >
> >
> > That should help you as a start, and you can tweak these up and down as
> > needed.
> >
>
> I was forwarded this from another list user too:
>
> Lucas,
>
> You may  also want to make additional modifications below in
> addition to what William provided.
>
> You currently have transactions building up and until you hit 60
> seconds the commits are being held in durability transaction log.
> Below setting will keep durability transaction log on, BUT only 1
> transaction will be held until 60 seconds
> nsslapd-db-transaction-batch-val: 0
>
> to be
>
> nsslapd-db-transaction-batch-val: 1
>
> If you are not doing ldif2db transactions as part of normal
> operation have discovered over the
> Years that disabling import-cache-autosize helps with memory
> management.  Historically 389ds will fence this
> Memory not use it and never release it back to OS.
> nsslapd-import-cache-autosize: -1
> nsslapd-import-cachesize: 2000
>
> to be
>
> nsslapd-import-cache-autosize: 0
> nsslapd-import-cachesize: 2000
>
>
> Make sure you check cachesize values for change after each directory
> restart.  They are dynamic based on available memory on OS at time of
> service start.
> If you have something else running on host with memory leak
> Directory Server slowly will be starved and eventually crash.
> Memory allocations for each directory instance you have running on
> host are cumulative so be aware what else is running on host before
> modifying the cachesize settings especially if you leave autosize
> enabled.
>
>
> David M. Partridge
> Identity Management and Security Engineer
> Tangible Security Inc
> 2010 Corporate Ridge, Suite 250
> McLean, VA 22102
>
>
>
> I hope this helps you too,
>
>
> --
> Sincerely,
>
> William Brown
> Software Engineer
> Red Hat, Australia/Brisbane
>
> ___
> 389-users mailing list -- 389-users@lists.fedoraproject.org
> To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
>
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Problem adding users to group.

2017-08-16 Thread William Brown 
On Tue, 2017-08-15 at 13:15 +, Lucas Diedrich wrote:
> Willian,
> 
> Cache values from cn=config + cn=userRoot:
> https://paste.fedoraproject.org/paste/ZqmA2wUkQDcSUaUIcpGDhg
> free -h + lspcu:
> https://paste.fedoraproject.org/paste/Br8Vz5-quxtcatiDyMy3QA
> 
> I'll check it out the docs for the optimization + wait until your response,
> i think it will more secure.

Hey mate,

Those values are all default of 10MB. You probably want to change:

dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config
nsslapd-cachememsize: 10485760
nsslapd-dncachememsize: 10485760

and

dn: cn=config,cn=ldbm database,cn=plugins,cn=config
nsslapd-dbcachesize: 1000

Looking at your values of free/cpu, I would recommend something like:

dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config
nsslapd-cachememsize: 268435456
nsslapd-dncachememsize: 67108864

and

dn: cn=config,cn=ldbm database,cn=plugins,cn=config
nsslapd-dbcachesize: 536870912


That should help you as a start, and you can tweak these up and down as
needed.


-- 
Sincerely,

William Brown
Software Engineer
Red Hat, Australia/Brisbane



signature.asc
Description: This is a digitally signed message part
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Problem adding users to group.

2017-08-15 Thread Lucas Diedrich 
Willian,

Cache values from cn=config + cn=userRoot:
https://paste.fedoraproject.org/paste/ZqmA2wUkQDcSUaUIcpGDhg
free -h + lspcu:
https://paste.fedoraproject.org/paste/Br8Vz5-quxtcatiDyMy3QA

I'll check it out the docs for the optimization + wait until your response,
i think it will more secure.

Thanks.

Em seg, 14 de ago de 2017 às 20:24, William Brown 
escreveu:

> On Mon, 2017-08-14 at 18:24 +, Lucas Diedrich wrote:
> > Willian, sorry for the time i took to answers, in the attachment there is
> > the gstack.out. I don't have a clue how to read that =/
> >
>
> No problem, it's my job to know how to read it :)
>
> THere are a lot of threads in:
>
> #11 0x7fb6487f3795 in __db_get_pp () from /lib64/libdb-5.3.so
> #12 0x7fb643a355fb in id2entry ()
> from /usr/lib64/dirsrv/plugins/libback-ldbm.so
> #13 0x7fb643a3171d in dn2entry_ext ()
> from /usr/lib64/dirsrv/plugins/libback-ldbm.so
>
> Are your dbcache, dncache and entrycache settings large enough? This
> issue could be indicative of cache thrashing,
>
>
> Besides that, nothing indicates specifically that the issue is MO.
>
>
> https://access.redhat.com/documentation/en-us/red_hat_directory_server/10/html/performance_tuning_guide/determining-req-cache-size
>
> This part of the performance tuning guide may help you. If you want, you
> can send me the output of your objects:
>
> cn=conficn=config,cn=ldbm database,cn=plugins,cn=configg,cn=ldbm
> database,cn=plugins,cn=config
>
> dn: cn=config,cn=ldbm database,cn=plugins,cn=config
> dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config
>
> And the output of free -h and lscpu.
>
>
>
> --
> Sincerely,
>
> William Brown
> Software Engineer
> Red Hat, Australia/Brisbane
>
> ___
> 389-users mailing list -- 389-users@lists.fedoraproject.org
> To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
>
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Problem adding users to group.

2017-08-14 Thread William Brown 
On Mon, 2017-08-14 at 18:24 +, Lucas Diedrich wrote:
> Willian, sorry for the time i took to answers, in the attachment there is
> the gstack.out. I don't have a clue how to read that =/
> 

No problem, it's my job to know how to read it :) 

THere are a lot of threads in:

#11 0x7fb6487f3795 in __db_get_pp () from /lib64/libdb-5.3.so
#12 0x7fb643a355fb in id2entry ()
from /usr/lib64/dirsrv/plugins/libback-ldbm.so
#13 0x7fb643a3171d in dn2entry_ext ()
from /usr/lib64/dirsrv/plugins/libback-ldbm.so

Are your dbcache, dncache and entrycache settings large enough? This
issue could be indicative of cache thrashing,


Besides that, nothing indicates specifically that the issue is MO. 

https://access.redhat.com/documentation/en-us/red_hat_directory_server/10/html/performance_tuning_guide/determining-req-cache-size

This part of the performance tuning guide may help you. If you want, you
can send me the output of your objects:

cn=conficn=config,cn=ldbm database,cn=plugins,cn=configg,cn=ldbm
database,cn=plugins,cn=config

dn: cn=config,cn=ldbm database,cn=plugins,cn=config
dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config

And the output of free -h and lscpu. 



-- 
Sincerely,

William Brown
Software Engineer
Red Hat, Australia/Brisbane



signature.asc
Description: This is a digitally signed message part
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Problem adding users to group.

2017-08-14 Thread Lucas Diedrich 
Willian, sorry for the time i took to answers, in the attachment there is
the gstack.out. I don't have a clue how to read that =/

Em qui, 10 de ago de 2017 às 06:24, Ludwig Krispenz 
escreveu:

>
> On 08/09/2017 09:48 PM, Lucas Diedrich wrote:
>
> Willian, i can make that, is there any manual on how to get the data
> trough gstack?
>
> Another thing, as i needed to resolve the issue, i had ldapmodify my group
> adding the members, using the method below it took like 16 sec. peer user
> and leading to un-reponsive system after the 30th user:
>
> *dn: cn=mygorup,cn=groups,cn=accounts,dc=domain*
> *changetype: modify*
> *add: member*
> *dn: uid=user1,cn=users,cn=accounts,dc=domain*
> *-*
>
> *dn: cn=mygorup,cn=groups,cn=accounts,dc=domain *
> *changetype: modify*
> *add: member*
> *dn: uid=user2,cn=users,cn=accounts,dc=domain*
> *-*
>
> *dn: cn=mygorup,cn=groups,cn=accounts,dc=domain *
> *changetype: modify*
> *add: member*
> *dn: uid=user3,cn=users,cn=accounts,dc=domain*
> *-*
> + 50 users
>
> But, changing to the method below, i was able to add all 600 users in only
> 16 sec with no un-responsive time:
>
> *dn: cn=mygorup,cn=groups,cn=accounts,dc=domain*
> *changetype: modify*
> *add: member*
> *dn: uid=user1,cn=users,cn=accounts,dc=domain*
> *dn: uid=user2,cn=users,cn=accounts,dc=domain*
> *dn: uid=user3,cn=users,cn=accounts,dc=domain*
> *+55*
>
> *this is no surprise. in on case you have >50 operations and over 50
> invocations of teh memberof plugin and in the other case there is only one
> operation and one plugin call*
>
> *-*
>
> Em ter, 8 de ago de 2017 às 19:57, William Brown 
> escreveu:
>
>> On Mon, 2017-08-07 at 17:30 +, Lucas Diedrich wrote:
>> > Hello guys,
>> >
>> > i have a *Freeipa 4.4.1* with 389 (389-ds-base-1.3.5.18-1.fc25.x86_64),
>> my
>> > problem is, i have a huge group with 5400 users, but i've to add more
>> 800
>> > to it, when i try to add more than 10 my dirsrv goes unresponsive.
>> >
>> > When the system starts to replicate all my Freeipa's goes unresponsive,
>> > asking over #389 some guys susgest that could be a bug with the
>> *memberof
>> > plugin*, is there anything i can do to fix this?
>>
>> Morning,
>>
>> Would you mind taking a gstack of the ns-slapd process when it goes
>> un-responsive? This would help me to see why it's showing high latency
>> like this.
>>
>> Thanks,
>>
>> --
>> Sincerely,
>>
>> William Brown
>> Software Engineer
>> Red Hat, Australia/Brisbane
>>
>> ___
>> 389-users mailing list -- 389-users@lists.fedoraproject.org
>> To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
>>
>
>
> ___
> 389-users mailing list -- 389-users@lists.fedoraproject.org
> To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
>
>
> --
> Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, 
> Eric Shander
>
> ___
> 389-users mailing list -- 389-users@lists.fedoraproject.org
> To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
>


gstack.out
Description: Binary data
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Problem adding users to group.

2017-08-10 Thread Ludwig Krispenz 


On 08/09/2017 09:48 PM, Lucas Diedrich wrote:
Willian, i can make that, is there any manual on how to get the data 
trough gstack?


Another thing, as i needed to resolve the issue, i had ldapmodify my 
group adding the members, using the method below it took like 16 sec. 
peer user and leading to un-reponsive system after the 30th user:


*/dn: cn=mygorup,cn=groups,cn=accounts,dc=domain/*
*/changetype: modify/*
*/add: member/*
*/dn: uid=user1,cn=users,cn=accounts,dc=domain/*
*/-/*
*/dn: cn=mygorup,cn=groups,cn=accounts,dc=domain
/*
*/changetype: modify/*
*/add: member/*
*/dn: uid=user2,cn=users,cn=accounts,dc=domain/*
*/-/*
*/dn: cn=mygorup,cn=groups,cn=accounts,dc=domain
/*
*/changetype: modify/*
*/add: member/*
*/dn: uid=user3,cn=users,cn=accounts,dc=domain/*
*/-/*
+ 50 users

But, changing to the method below, i was able to add all 600 users in 
only 16 sec with no un-responsive time:


*/dn: cn=mygorup,cn=groups,cn=accounts,dc=domain/*
*/changetype: modify/*
*/add: member/*
*/dn: uid=user1,cn=users,cn=accounts,dc=domain/*
*/dn: uid=user2,cn=users,cn=accounts,dc=domain/*
*/dn: uid=user3,cn=users,cn=accounts,dc=domain/*
*/+55/*
/*this is no surprise. in on case you have >50 operations and over 50 
invocations of teh memberof plugin and in the other case there is only 
one operation and one plugin call*/

*/-/*

Em ter, 8 de ago de 2017 às 19:57, William Brown > escreveu:


On Mon, 2017-08-07 at 17:30 +, Lucas Diedrich wrote:
> Hello guys,
>
> i have a *Freeipa 4.4.1* with 389
(389-ds-base-1.3.5.18-1.fc25.x86_64), my
> problem is, i have a huge group with 5400 users, but i've to add
more 800
> to it, when i try to add more than 10 my dirsrv goes unresponsive.
>
> When the system starts to replicate all my Freeipa's goes
unresponsive,
> asking over #389 some guys susgest that could be a bug with the
*memberof
> plugin*, is there anything i can do to fix this?

Morning,

Would you mind taking a gstack of the ns-slapd process when it goes
un-responsive? This would help me to see why it's showing high latency
like this.

Thanks,

--
Sincerely,

William Brown
Software Engineer
Red Hat, Australia/Brisbane

___
389-users mailing list -- 389-users@lists.fedoraproject.org

To unsubscribe send an email to
389-users-le...@lists.fedoraproject.org




___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org


--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric 
Shander

___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Problem adding users to group.

2017-08-09 Thread William Brown 
On Wed, 2017-08-09 at 19:48 +, Lucas Diedrich wrote:
> Willian, i can make that, is there any manual on how to get the data trough
> gstack?

On the affected master run:

ps aux | grep slapd

Note the pid of the ns-slapd process

sudo gstack  &> /tmp/gstack.out

Then send me the gstack.out file. 

> 
> Another thing, as i needed to resolve the issue, i had ldapmodify my group
> adding the members, using the method below it took like 16 sec. peer user
> and leading to un-reponsive system after the 30th user:
> 
...
> 
> But, changing to the method below, i was able to add all 600 users in only
> 16 sec with no un-responsive time:
> 

Could be combination of memberof and ndn cache I think. 

Let's see the gstack, and we can work out the best course of action from
there,  

-- 
Sincerely,

William Brown
Software Engineer
Red Hat, Australia/Brisbane



signature.asc
Description: This is a digitally signed message part
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Problem adding users to group.

2017-08-09 Thread Lucas Diedrich 
Willian, i can make that, is there any manual on how to get the data trough
gstack?

Another thing, as i needed to resolve the issue, i had ldapmodify my group
adding the members, using the method below it took like 16 sec. peer user
and leading to un-reponsive system after the 30th user:

*dn: cn=mygorup,cn=groups,cn=accounts,dc=domain*
*changetype: modify*
*add: member*
*dn: uid=user1,cn=users,cn=accounts,dc=domain*
*-*

*dn: cn=mygorup,cn=groups,cn=accounts,dc=domain*
*changetype: modify*
*add: member*
*dn: uid=user2,cn=users,cn=accounts,dc=domain*
*-*

*dn: cn=mygorup,cn=groups,cn=accounts,dc=domain*
*changetype: modify*
*add: member*
*dn: uid=user3,cn=users,cn=accounts,dc=domain*
*-*
+ 50 users

But, changing to the method below, i was able to add all 600 users in only
16 sec with no un-responsive time:

*dn: cn=mygorup,cn=groups,cn=accounts,dc=domain*
*changetype: modify*
*add: member*
*dn: uid=user1,cn=users,cn=accounts,dc=domain*
*dn: uid=user2,cn=users,cn=accounts,dc=domain*
*dn: uid=user3,cn=users,cn=accounts,dc=domain*
*+55*
*-*

Em ter, 8 de ago de 2017 às 19:57, William Brown 
escreveu:

> On Mon, 2017-08-07 at 17:30 +, Lucas Diedrich wrote:
> > Hello guys,
> >
> > i have a *Freeipa 4.4.1* with 389 (389-ds-base-1.3.5.18-1.fc25.x86_64),
> my
> > problem is, i have a huge group with 5400 users, but i've to add more 800
> > to it, when i try to add more than 10 my dirsrv goes unresponsive.
> >
> > When the system starts to replicate all my Freeipa's goes unresponsive,
> > asking over #389 some guys susgest that could be a bug with the *memberof
> > plugin*, is there anything i can do to fix this?
>
> Morning,
>
> Would you mind taking a gstack of the ns-slapd process when it goes
> un-responsive? This would help me to see why it's showing high latency
> like this.
>
> Thanks,
>
> --
> Sincerely,
>
> William Brown
> Software Engineer
> Red Hat, Australia/Brisbane
>
> ___
> 389-users mailing list -- 389-users@lists.fedoraproject.org
> To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org
>
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org

[389-users] Re: Problem adding users to group.

2017-08-08 Thread William Brown 
On Mon, 2017-08-07 at 17:30 +, Lucas Diedrich wrote:
> Hello guys,
> 
> i have a *Freeipa 4.4.1* with 389 (389-ds-base-1.3.5.18-1.fc25.x86_64), my
> problem is, i have a huge group with 5400 users, but i've to add more 800
> to it, when i try to add more than 10 my dirsrv goes unresponsive.
> 
> When the system starts to replicate all my Freeipa's goes unresponsive,
> asking over #389 some guys susgest that could be a bug with the *memberof
> plugin*, is there anything i can do to fix this?

Morning,

Would you mind taking a gstack of the ns-slapd process when it goes
un-responsive? This would help me to see why it's showing high latency
like this.

Thanks, 

-- 
Sincerely,

William Brown
Software Engineer
Red Hat, Australia/Brisbane



signature.asc
Description: This is a digitally signed message part
___
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-le...@lists.fedoraproject.org