[389-devel] Please review 50933
https://pagure.io/389-ds-base/pull-request/50934 https://pagure.io/389-ds-base/issue/50933 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Thoughts on swapping to rfc2307bis.ldif by default
> On 4 Mar 2020, at 00:26, thierry bordaz wrote: > > > > On 3/3/20 11:59 AM, Alexander Bokovoy wrote: >> On ti, 03 maalis 2020, Ludwig Krispenz wrote: >>> Hi, >>> >>> I have no expertise in this area, but would like to get also Alexander's >>> opinion and view from IPA >> >> I don't have much to add to what Thierry and William covered already. >> Having a new draft with clarifications would be nice. >> >> Given that both 10rfc2307.ldif and 10rfc2307bis.ldif are present in >> default 389-ds deployment, why this schema conflict isn't a problem >> right now? > > Good point :) > 10rfc2307bis.ldif is in /share/dirsrv/data and 10rfc2307.ldif in > /share/dirsrv/schema. > Only 'schema' definitions are loaded in 'cn=schema'. The definitions in > 'data' are available for users but are not part of QE scope. > > I guess most of the users choose one rfc or the other and then the entries > will conform the chosen RFC. > A risk exists if we are moving a dataset from one rfc to the other. This > either during an import or if instances in the same replicated topology > create incompatible entries. It's not a problem, because while we include both, we only actually install and load rfc2307.ldif. rfc2307bis.ldif is in a seperate directory as "example data", and thus is not loaded. In the past we made a decision to make schema load from /usr, which has caused at least one or two users to have issues with this specific schema as there was "no way" to remove rfc2307.ldif from the schema directory, but they wanted to use rfc2307bis. I think part of the reason we don't hear more complaints about this is as thierry said, we don't enforce the structural chain on posixGroup which is one of the major issues for rfc2307 that rfc2307bis resolves. So it's likely that most people don't notice that they are using rfc2307, instead thinking it's rfc2307bis as you can add posixGroup to other objectClasses. If we were to straight replace rfc2307 with rfc2307bis today I think we would have a compatability issue that may affect IPA, but the idea to have a compat that is a supported hybrid of both, would likely be non-disruptive, keep IPA working, and allow openldap imports. >>>>> >>>>> A possibility is making a rfc2307bis-compat.ldif instead that allows the >>>>> MAY of everything in rfc2307, but is based on rfc2307bis as the base. For >>>>> example, allowing "MAY description $ l $ o $ ou $ owner $ seeAlso $ >>>>> serialNumber" on ieee802Device and bootableDevice. That would make it >>>>> forward compatible, and actually quite seamless to upgrade. If we wanted >>>>> we could consider formalising it to a draft rfc given that's what >>>>> rfc2307bis is (a draft rfc). >>>>> >>>>> Thoughts? >>>> >>>> Sorry missed the end of the email ! >>>> Yes I think it is a good approach, deliver what we can that does not break >>>> existing deployment. >>>> For the remaining part of 2307bis we create a diagnostic/healthcheck tool >>>> that gives a go/no-go to apply a full 2307bis definition. I think that the compatibility I proposed wouldn't need an extra tool because both rfc2307 and rfc2307bis would be valid within it, so we could be supporting both in parallel. If anything I'd just need to code into my openldap migration tool to ignore the rfc2307 schema oids since we already have them in the compat. So, does drafting a rfc2307compat.ldif sound reasonable at this point? Do we think this is worth adding a draft rfc online as well, or is this something that really only affects us? If it's only us (i suspect it is ...) then I can also write a corresponding doc on the wiki. — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Thoughts on swapping to rfc2307bis.ldif by default
> On 3 Mar 2020, at 11:18, William Brown wrote: > > > >> On 3 Mar 2020, at 04:32, thierry bordaz wrote: >> >> >> >> On 3/2/20 7:24 AM, William Brown wrote: >>> Hi all, >>> >>> As you may know, I'm currently working on a migration utility to help move >>> from other ldap servers to 389-ds. Something that I have noticed in this >>> process is that other servers default to rfc2307bis.ldif [0] by default. As >>> part of the migration I would like to handle this situation a bit better. >>> It's likely not viable for me to simply plaster rfc2307bis into 99user.ldif >>> as part of the migration process, so I want to approach this better. >>> >>> rfc2307 and rfc2307bis are incompatible schemas that redefine the same OIDs >>> with new/different meanings. Some key examples: >>> >>> * posixGroup in rfc2307 only requires gidNumber, rfc2307bis requires cn and >>> gidNumber. >> Is not it the opposite ? > > I was reading the schema as I was reading this. I need to apologise for being so short in this answer! Thierry was correct in this case. Here is the full set of differences between the two: uidNumber: +EQUALITY integerMatch gidNumber: +EQUALITY integerMatch gecos: +EQUALITY caseIgnoreIA5Match SUBSTR caseIgnoreIA5SubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 -SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 homeDirectory: +EQUALITY caseExactIA5Match loginShell: +EQUALITY caseExactIA5Match shadowLastChange: +EQUALITY integerMatch shadowMin: +EQUALITY integerMatch shadowMax: +EQUALITY integerMatch shadowWarning: +EQUALITY integerMatch shadowInactive: +EQUALITY integerMatch shadowExpire: +EQUALITY integerMatch shadowFlag: +EQUALITY integerMatch memberUid: +EQUALITY caseExactIA5Match memberNisNetgroup: +EQUALITY caseExactIA5Match SUBSTR caseExactIA5SubstringsMatch nisNetgroupTriple: +EQUALITY caseIgnoreIA5Match ipServicePort: +EQUALITY integerMatch ipServiceProtocol: +SUP name -SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 ipProtocolNumber: +EQUALITY integerMatch oncRpcNumber: +EQUALITY integerMatch ipHostNumber: +SUP name -SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 ipNetworkNumber: +SUP name -SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 ipNetmaskNumber: +EQUALITY caseIgnoreIA5Match SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 -SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 macAddress: +EQUALITY caseIgnoreIA5Match SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 -SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 bootParameter: +EQUALITY caseExactIA5Match nisMapName: +SUP name -SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 nisMapEntry: +EQUALITY caseExactIA5Match SUBSTR caseExactIA5SubstringsMatch + attributeTypes: ( 1.3.6.1.1.1.1.28 NAME 'nisPublicKey' DESC 'NIS public key' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 SINGLE-VALUE ) + attributeTypes: ( 1.3.6.1.1.1.1.29 NAME 'nisSecretKey' DESC 'NIS secret key' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 SINGLE-VALUE ) + attributeTypes: ( 1.3.6.1.1.1.1.30 NAME 'nisDomain' DESC 'NIS domain' EQUALITY caseIgnoreIA5Match SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 ) + attributeTypes: ( 1.3.6.1.1.1.1.31 NAME 'automountMapName' DESC 'automount Map Name' EQUALITY caseExactIA5Match SUBSTR caseExactIA5SubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 SINGLE-VALUE ) + attributeTypes: ( 1.3.6.1.1.1.1.32 NAME 'automountKey' DESC 'Automount Key value' EQUALITY caseExactIA5Match SUBSTR caseExactIA5SubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 SINGLE-VALUE ) + attributeTypes: ( 1.3.6.1.1.1.1.33 NAME 'automountInformation' DESC 'Automount information' EQUALITY caseExactIA5Match SUBSTR caseExactIA5SubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 SINGLE-VALUE ) posixAccount: shadowAccount: posixGroup: +AUXILLARY -MUST cn STRUCTURAL ipService: ipProtocol: oncRpc: ipHost: - MAY o $ ou $ owner $ seeAlso $ serialNumber +MAY userPassword ipNetwork: -MUST cn +MAY cn nisNetgroup: nisMap: nisObject: ieee802Device: -MUST cn MAY description $ l $ o $ ou $ owner $ seeAlso $ serialNumber bootableDevice: -MUST cn MAY description $ l $ o $ ou $ owner $ seeAlso $ serialNumber nisMap: +OID 1.3.6.1.1.1.2.9 -OID 1.3.6.1.1.1.2.13 + objectClasses: ( 1.3.6.1.1.1.2.14 NAME 'nisKeyObject' SUP top AUXILIARY DESC 'An object with a public and secret key' MUST ( cn $ nisPublicKey $ nisSecretKey ) MAY ( uidNumber $ description ) ) + objectClasses: ( 1.3.6.1.1.1.2.15 NAME 'nisDomainObject' SUP top AUXILIARY DESC 'Associates a NIS domain with a naming context' MUST nisDomain ) + objectClasses: ( 1.3.6.1.1.1.2.16 NAME 'automountMap' SUP top STRUCTURAL MUST ( automountMapName ) MAY description ) + objectClasses: ( 1.3.6.1.1.1.2.17 NAME 'automount' SUP top STRUCTURAL DESC 'Automount information' MUST ( automountKey $ automountInformation ) MAY description ) ## namedObject is needed for groups without members + objectClass
[389-devel] Re: Thoughts on swapping to rfc2307bis.ldif by default
> On 3 Mar 2020, at 04:32, thierry bordaz wrote: > > > > On 3/2/20 7:24 AM, William Brown wrote: >> Hi all, >> >> As you may know, I'm currently working on a migration utility to help move >> from other ldap servers to 389-ds. Something that I have noticed in this >> process is that other servers default to rfc2307bis.ldif [0] by default. As >> part of the migration I would like to handle this situation a bit better. >> It's likely not viable for me to simply plaster rfc2307bis into 99user.ldif >> as part of the migration process, so I want to approach this better. >> >> rfc2307 and rfc2307bis are incompatible schemas that redefine the same OIDs >> with new/different meanings. Some key examples: >> >> * posixGroup in rfc2307 only requires gidNumber, rfc2307bis requires cn and >> gidNumber. > Is not it the opposite ? I was reading the schema as I was reading this. >> * ipServiceProtocol, ipHostNumber, ipNetworkNumber and nisMapName change >> from "sup name" to "syntax 1.3.6.1.4.1.1466.115.121.1.15". sup name is also >> syntax 1.3.6.1.4.1.1466.115.121.1.15 so this channge is minimal. >> * posixGroup and posixAccount change from structural to auxillary in >> rfc2307bis (allowing them to be combined with person or nsAccount). > Right but for 389-ds the structural requirement is not enforced, so it should > not be a problem You know, that's probably actually the best thing I've heard all day. It makes this problem much easier. >> >> Objectively, rfc2307bis is the better schema - but as with all proposals >> like this, there is always a risk of breaking customers or compatibility. > I agree on both :) >> >> I'm wondering what would be a reasonable course of action for us to move to >> rfc2307bis by default. My current thoughts: >> >> * have rfc2307bis vs rfc2307 as an option to dssetup so we use the correct >> schema in the setup. >> * default the setup option to rfc2307bis >> * Tests for handling both setup options >> * Upgrades of the server should not affect the rfc2307 vs rfc2307bis status >> * A dsctl tool to allow changing between the rfc2307/rfc2307bis. >> >> Thoughts? Concern? Ideas? Comments? > It would be interesting to have a complete list of the differences. at the > moment with the listed differences I think 2307bis would support 2307 > entries. In addition, 2307bis looks to be a superset of 2307 so that it would > be replicated in a mmr topology. Right. I'll get a list of all the differences, and knowing that structural isn't enforced does make things easier - a lot easier. It may be less disruptive to swap to 2307bis by default if that's the case. > > Because of some bug, 99user.ldif will contains all overridden definitions > not the only new/changed one. > > The idea of a dsctl tool looks good. It could be to create a task that check > all entries conform a schema. If all entries conform 2307bis we could replace > the default 2307 schema file with the 2307bis. Yeah, a task could help here too. >> >> >> [0] https://tools.ietf.org/html/draft-howard-rfc2307bis-02 >> >> — >> Sincerely, >> >> William Brown >> >> Senior Software Engineer, 389 Directory Server >> SUSE Labs >> ___ >> 389-devel mailing list -- 389-devel@lists.fedoraproject.org >> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Thoughts on swapping to rfc2307bis.ldif by default
Hi all, As you may know, I'm currently working on a migration utility to help move from other ldap servers to 389-ds. Something that I have noticed in this process is that other servers default to rfc2307bis.ldif [0] by default. As part of the migration I would like to handle this situation a bit better. It's likely not viable for me to simply plaster rfc2307bis into 99user.ldif as part of the migration process, so I want to approach this better. rfc2307 and rfc2307bis are incompatible schemas that redefine the same OIDs with new/different meanings. Some key examples: * posixGroup in rfc2307 only requires gidNumber, rfc2307bis requires cn and gidNumber. * ipServiceProtocol, ipHostNumber, ipNetworkNumber and nisMapName change from "sup name" to "syntax 1.3.6.1.4.1.1466.115.121.1.15". sup name is also syntax 1.3.6.1.4.1.1466.115.121.1.15 so this channge is minimal. * posixGroup and posixAccount change from structural to auxillary in rfc2307bis (allowing them to be combined with person or nsAccount). Objectively, rfc2307bis is the better schema - but as with all proposals like this, there is always a risk of breaking customers or compatibility. I'm wondering what would be a reasonable course of action for us to move to rfc2307bis by default. My current thoughts: * have rfc2307bis vs rfc2307 as an option to dssetup so we use the correct schema in the setup. * default the setup option to rfc2307bis * Tests for handling both setup options * Upgrades of the server should not affect the rfc2307 vs rfc2307bis status * A dsctl tool to allow changing between the rfc2307/rfc2307bis. Thoughts? Concern? Ideas? Comments? [0] https://tools.ietf.org/html/draft-howard-rfc2307bis-02 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: 50618 compiler cleanup
https://pagure.io/389-ds-base/pull-request/50913 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review 50900 fix cargo offline
https://pagure.io/389-ds-base/pull-request/50900 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Meeting?
hey everyone, I joined at UTC+10 22:00, and was in the room for 10 minutes but didn't hear from anyone. Did I miss something? — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: [PATCH] prevent slapd from hanging under unlikely circumstances
> On 6 Feb 2020, at 18:40, Ludwig Krispenz wrote: > > > On 02/06/2020 12:57 AM, William Brown wrote: >> >>> On 5 Feb 2020, at 20:08, Ludwig Krispenz wrote: >>> >>> >>> On 02/05/2020 10:53 AM, thierry bordaz wrote: >>>> >>>> On 2/5/20 2:30 AM, William Brown wrote: >>>>>> On 5 Feb 2020, at 03:10, Ludwig Krispenz wrote: >>>>>> >>>>>> I think I can agree with 1-8, 9 is one solution to fix the problem you >>>>>> reported, but not yet validate that there are no other side effects, >>>>>> there are potential postop plugins which should NOT be called for >>>>>> tombstone delete, eg retro cl, we need to investigate side effects of >>>>>> the patch and evaluate other solutions, I suggested one in anotehr reply. >>>>>> >>>>>> for 10, the claim that not crying hurrah and merging a patch without >>>>>> further investigation and testing is "doctrinal objection" is very >>>>>> strong and I have to reject this. >>>>>> >>>>>> Regards, >>>>>> Ludwig >>>>>> >>>>>> On 02/04/2020 05:00 PM, Jay Fenlason wrote: >>>>>>> Ok, let's take this from the top: >>>>>>> >>>>>>> 1: Defects that cause a server to become unresponsive are bad and must >>>>>>> be repaired as soon as possible. >>>>> I'm not objecting to this. >>>>> >>>>>>> 2: Some #1 class defects are exploitable and require a CVE. As far as >>>>>>> I can tell, this one does not, but you should keep an eye out for the >>>>>>> possibility. >>>>>>> >>>>>>> 3: The #1 class defect I have found can be triggered in at least two >>>>>>> ways. One requires ipa-replica-install and is hard to reproduce in a >>>>>>> test environment. The other requires ldapdelete and is easy to >>>>>>> reproduce in an isolated VM. >>>>> Ipa replica install and ldapdelete of a tombstone/conflict both require >>>>> cn=directory manager, which would automatically cause any reported CVE to >>>>> be void - cn=directory manager is game over. >>>>> >>>>>>> 4: The defect is a mismatch between the plugin ABI as implemented by >>>>>>> 389-ds, and the behavior the NIS plugin expects. >>>>>>> >>>>>>> 5: I have found no specification or documentation on said ABI, so I >>>>>>> can only guess what the correct behavior is here. >>>>>>> >>>>>>> 6: The ABI includes two functions in the plugin: pre_delete and >>>>>>> post_delete. >>>>>>> >>>>>>> 7: The NIS plugin expects that every call to pre_delete will be >>>>>>> followed by a call to post_delete. It takes a lock in pre_delete and >>>>>>> releases it in post_delete. >>>>> But you didn't report an issue in NIS? You reported an issue with >>>>> ldapdelete on tombstones and conflicts ... the slapi-nis plugin is >>>>> maintained by freeipa and if an issue in it exists, please report it to >>>>> them with reproduction steps. >>>> Hi Jay, >>>> >>>> Thanks for your great investigations/patch/summary. >>>> >>>> I made the change in NIS plugin to acquire/release map lock in pre/post >>>> ops. At that time I was assuming the pre/post callback were balanced. >>>> I was wrong and already hit similar issue in #1694263. I updated NIS >>>> plugin to workaround that bug. >>>> >>>> The new bug you are reporting is a new example that pre/post are sometime >>>> not balanced :( >>>> The proposed fixes (prevent preop call on tombstone, force postop call >>>> even on tombstone) looks valid but will have wider impact and need >>>> additional investigation. >>>> An other approach would be to make NIS plugin ignore operations on >>>> tombstone. It also require additional investigation. >>> I think that for tombstone deletions we should also prevent the call of the >>> preop plugins, no plugin should deal with these operations >> That sounds reasonable. Can we create an issue for this? > In fact discussing this yest
[389-devel] please review: support cgroup v2
https://pagure.io/389-ds-base/pull-request/50885 https://pagure.io/389-ds-base/issue/50618 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: [PATCH] prevent slapd from hanging under unlikely circumstances
> On 5 Feb 2020, at 20:08, Ludwig Krispenz wrote: > > > On 02/05/2020 10:53 AM, thierry bordaz wrote: >> >> >> On 2/5/20 2:30 AM, William Brown wrote: >>> >>>> On 5 Feb 2020, at 03:10, Ludwig Krispenz wrote: >>>> >>>> I think I can agree with 1-8, 9 is one solution to fix the problem you >>>> reported, but not yet validate that there are no other side effects, there >>>> are potential postop plugins which should NOT be called for tombstone >>>> delete, eg retro cl, we need to investigate side effects of the patch and >>>> evaluate other solutions, I suggested one in anotehr reply. >>>> >>>> for 10, the claim that not crying hurrah and merging a patch without >>>> further investigation and testing is "doctrinal objection" is very strong >>>> and I have to reject this. >>>> >>>> Regards, >>>> Ludwig >>>> >>>> On 02/04/2020 05:00 PM, Jay Fenlason wrote: >>>>> Ok, let's take this from the top: >>>>> >>>>> 1: Defects that cause a server to become unresponsive are bad and must >>>>> be repaired as soon as possible. >>> I'm not objecting to this. >>> >>>>> 2: Some #1 class defects are exploitable and require a CVE. As far as >>>>> I can tell, this one does not, but you should keep an eye out for the >>>>> possibility. >>>>> >>>>> 3: The #1 class defect I have found can be triggered in at least two >>>>> ways. One requires ipa-replica-install and is hard to reproduce in a >>>>> test environment. The other requires ldapdelete and is easy to >>>>> reproduce in an isolated VM. >>> Ipa replica install and ldapdelete of a tombstone/conflict both require >>> cn=directory manager, which would automatically cause any reported CVE to >>> be void - cn=directory manager is game over. >>> >>>>> 4: The defect is a mismatch between the plugin ABI as implemented by >>>>> 389-ds, and the behavior the NIS plugin expects. >>>>> >>>>> 5: I have found no specification or documentation on said ABI, so I >>>>> can only guess what the correct behavior is here. >>>>> >>>>> 6: The ABI includes two functions in the plugin: pre_delete and >>>>> post_delete. >>>>> >>>>> 7: The NIS plugin expects that every call to pre_delete will be >>>>> followed by a call to post_delete. It takes a lock in pre_delete and >>>>> releases it in post_delete. >>> But you didn't report an issue in NIS? You reported an issue with >>> ldapdelete on tombstones and conflicts ... the slapi-nis plugin is >>> maintained by freeipa and if an issue in it exists, please report it to >>> them with reproduction steps. >> >> Hi Jay, >> >> Thanks for your great investigations/patch/summary. >> >> I made the change in NIS plugin to acquire/release map lock in pre/post ops. >> At that time I was assuming the pre/post callback were balanced. >> I was wrong and already hit similar issue in #1694263. I updated NIS plugin >> to workaround that bug. >> >> The new bug you are reporting is a new example that pre/post are sometime >> not balanced :( >> The proposed fixes (prevent preop call on tombstone, force postop call even >> on tombstone) looks valid but will have wider impact and need additional >> investigation. >> An other approach would be to make NIS plugin ignore operations on >> tombstone. It also require additional investigation. > I think that for tombstone deletions we should also prevent the call of the > preop plugins, no plugin should deal with these operations That sounds reasonable. Can we create an issue for this? >> >> best regards >> thierry >> >> >>>>> 8: Under some circumstances 389-ds can call pre_delete but fail to >>>>> call post_delete. Because of #5, there is no way to tell if this is >>>>> expected behavior, but the NIS plugin clearly does not expect it. >>>>> >>>>> 9: My patch ensures that every call to pre_delete is followed by a >>>>> corresponding call to post_delete. V1 of the patch also ensures that >>>>> every call to post_delete is after a corresponding pre_delete call. >>>>> V2 relaxes the second requirement, allowing post_delete calls without >>>>
[389-devel] Re: [PATCH] prevent slapd from hanging under unlikely circumstances
> On 5 Feb 2020, at 03:10, Ludwig Krispenz wrote:
>
> I think I can agree with 1-8, 9 is one solution to fix the problem you
> reported, but not yet validate that there are no other side effects, there
> are potential postop plugins which should NOT be called for tombstone delete,
> eg retro cl, we need to investigate side effects of the patch and evaluate
> other solutions, I suggested one in anotehr reply.
>
> for 10, the claim that not crying hurrah and merging a patch without further
> investigation and testing is "doctrinal objection" is very strong and I have
> to reject this.
>
> Regards,
> Ludwig
>
> On 02/04/2020 05:00 PM, Jay Fenlason wrote:
>> Ok, let's take this from the top:
>>
>> 1: Defects that cause a server to become unresponsive are bad and must
>> be repaired as soon as possible.
I'm not objecting to this.
>>
>> 2: Some #1 class defects are exploitable and require a CVE. As far as
>> I can tell, this one does not, but you should keep an eye out for the
>> possibility.
>>
>> 3: The #1 class defect I have found can be triggered in at least two
>> ways. One requires ipa-replica-install and is hard to reproduce in a
>> test environment. The other requires ldapdelete and is easy to
>> reproduce in an isolated VM.
Ipa replica install and ldapdelete of a tombstone/conflict both require
cn=directory manager, which would automatically cause any reported CVE to be
void - cn=directory manager is game over.
>>
>> 4: The defect is a mismatch between the plugin ABI as implemented by
>> 389-ds, and the behavior the NIS plugin expects.
>>
>> 5: I have found no specification or documentation on said ABI, so I
>> can only guess what the correct behavior is here.
>>
>> 6: The ABI includes two functions in the plugin: pre_delete and
>> post_delete.
>>
>> 7: The NIS plugin expects that every call to pre_delete will be
>> followed by a call to post_delete. It takes a lock in pre_delete and
>> releases it in post_delete.
But you didn't report an issue in NIS? You reported an issue with ldapdelete on
tombstones and conflicts ... the slapi-nis plugin is maintained by freeipa and
if an issue in it exists, please report it to them with reproduction steps.
>>
>> 8: Under some circumstances 389-ds can call pre_delete but fail to
>> call post_delete. Because of #5, there is no way to tell if this is
>> expected behavior, but the NIS plugin clearly does not expect it.
>>
>> 9: My patch ensures that every call to pre_delete is followed by a
>> corresponding call to post_delete. V1 of the patch also ensures that
>> every call to post_delete is after a corresponding pre_delete call.
>> V2 relaxes the second requirement, allowing post_delete calls without
>> a corresponding pre_delete call because someone expressed worry about
>> potential regressions.
>>
>> 10: You are refusing to merge my patch because of a doctrinal
>> objection to the use of ldapdelete in the simple reproducer.
It's not really a doctrinal objection. Replication is a really complex topic,
and difficult to get correct. Ludwig knows a lot in this area, and has really
good comments to make on the topic. I understand that you have an issue you
have found in your setup, and you want to resolve it, but we have to consider
the deployments of many thousands of other users and their replication
experience too. For us it's important to be able to reproduce, and see the
issue, to understand the consequences, and to make these changes carefully.
Accepting the patch without clear understanding of it's consequences is not
something we do.
At this time we still don't have a clear reproducer from you on "how" you
constructed the invalid directory state. You reported the following:
> I found the bug by doing a series of "ipa-client-install" (with lots
> of arguments, followed by
> echo ca_host = {a not-firewalled IPA CA} >> /etc/ipa/default.conf
> echo [global] > /etc/ipa/installer.conf
> echo ca_host = {ditto} >> /etc/ipa/installer.conf
> echo {password} | kinit admin
> ipa hostgroup-add-member ipaservers --hosts $(hostname -f)
> ipa-relica-install --setup-ca --setup-dns --forwarder={ip addr}
>
> followed by the replica install failing due to network issues,
> misconfigured firewalls, etc, then
> ipa-server-install --uninstall on the host
> and ipa-replica-manage del {failed install host}
> elsewhere in the mesh, sometimes with ldapdelete of the initial
> replication agreement that ipa-replica-manage did not remove.
I'm certainly not able to produce a reproduction case from these steps ... This
is why I have questioned what ipa-replica-insta
[389-devel] Please review: portal issue 1
https://pagure.io/389-ds-portal/pull-request/13 https://pagure.io/389-ds-portal/issue/1 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: [PATCH] prevent slapd from hanging under unlikely circumstances
> On 3 Feb 2020, at 23:43, Jay Fenlason wrote:
>
> On Mon, Feb 03, 2020 at 10:38:59AM +1000, William Brown wrote:
>>
>>
>>> On 1 Feb 2020, at 12:10, Jay Fenlason wrote:
>>>
>>> I have a small FreeIPA deployment of ~6-8 servers running on Centos
>>> 7.7. Do to the addition and removal of some of the servers, some
>>> cruft (tombstones, replication conflicts, etc) have crept in to the
>>> directory. I noticed that when I attempted to delete some of the
>>> cruft entries, ns-slapd would hang, failirg to process requests, or
>>> even shut down.
>
>> Can you tell us exactly what entries you noticed and how you attempted to
>> delete them? There are certainly some things like tombstones and such that
>> you shouldn't be touching as they are part of the internal replication state
>> machine.
>
> No, I don't remember what entries they were. I was following
> instructions from:
> https://docs.fedoraproject.org/en-US/Fedora/18/html/FreeIPA_Guide/ipa-replica-manage.html
> (or maybe elsewhere) using ldapdelete to remove tombstones for a truly
> deleted server.
I think this advice is quite out dated now, we have potentially got better
tools to handle this, but that's a really difficult issue to manage content
ownership, management, and getting google to show the latest content etc
>
>> Knowing what you did will also help us to create a test case and
>> reproducers to validate your patch also.
>
> I found the bug by doing a series of "ipa-client-install" (with lots
> of arguments, followed by
> echo ca_host = {a not-firewalled IPA CA} >> /etc/ipa/default.conf
> echo [global] > /etc/ipa/installer.conf
> echo ca_host = {ditto} >> /etc/ipa/installer.conf
> echo {password} | kinit admin
> ipa hostgroup-add-member ipaservers --hosts $(hostname -f)
> ipa-relica-install --setup-ca --setup-dns --forwarder={ip addr}
>
> followed by the replica install failing due to network issues,
> misconfigured firewalls, etc, then
> ipa-server-install --uninstall on the host
> and ipa-replica-manage del {failed install host}
> elsewhere in the mesh, sometimes with ldapdelete of the initial
> replication agreement that ipa-replica-manage did not remove.
>
> Rinse, repeat. . .
>
> Until ipa-replica-install starts failing because the source LDAP
> server hangs (because of this bug) during the "starting initial
> replication" step. It was while debugging that that I discovered that
> ldapdelete on the tombstone entries also caused the LDAP servers to
> lock up.
>
>
>> Thanks for the report :)
>
> Incidentally, there's another bug, which I have not investigated,
> where attempting to ldapdelete a problematic tombstone entry
> immediately after restarting the LDAP server returns an error, and
> nothing is deleted on the server. If you do an ldapsearch, and then
> an ldapdelete, the entry is removed, but then slapd hangs (this bug
> again) and does not respond to searches or deletes (or shutdown
> requests) until you kill -9 it. I don't know how it relates to this
> bug.
So I think that deleting the tombstones is not the correct (or valid) course of
action here. Tombstones are a really important part of the replication
lifecycle, so if anything we need to taker stronger steps to prevent a client
from being able to delete them at all. This makes me question the patch you
have provided, because you shouldn't be in a position to delete tombstones in
the first place, only the server internally should be purging (deleting) these
when replication is known to be in a consistent state. I am happy to explain
the functionality of tombstones further if you are interested.
Deleting a conflict entry however, is just fine, so that shouldn't have caused
the issue.
I wonder if a contributing factor here is if ipa-replica-install is re-using
replica ids, which could cause replication to have a problem.
Perhaps the solution here is to have ipa-replica-install to attempt a
cleanallruv on any replica id it's *about* to try to use, in case it has been
re-used.
My thinking at this point is that there is something else going on, and the
issue may reside in a series of interactions in the ipa replica steps you have
taken. Have you contacted the freeipa-users group about this at all?
>
>-- JF
> ___
> 389-devel mailing list -- 389-devel@lists.fedoraproject.org
> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproj
[389-devel] Re: [PATCH] prevent slapd from hanging under unlikely circumstances
> On 1 Feb 2020, at 12:10, Jay Fenlason wrote:
>
> I have a small FreeIPA deployment of ~6-8 servers running on Centos
> 7.7. Do to the addition and removal of some of the servers, some
> cruft (tombstones, replication conflicts, etc) have crept in to the
> directory. I noticed that when I attempted to delete some of the
> cruft entries, ns-slapd would hang, failirg to process requests, or
> even shut down.
Can you tell us exactly what entries you noticed and how you attempted to
delete them? There are certainly some things like tombstones and such that you
shouldn't be touching as they are part of the internal replication state
machine.
Knowing what you did will also help us to create a test case and reproducers to
validate your patch also.
Thanks for the report :)
> In my investigation I found that pre-write callback
> of the plugins (including the nis plugin) were being called, but no
> corresponding post-write call was happening. The nis plugin takes a
> write lock in the pre-write callback and releases it in the post-write
> callback. Never releasing the lock causes the next attempt to take it
> to hang. I wrote the attached patch, which simply ensures the
> post-write callback is always called if the pre-write callback was
> called.
>
> This is such a simple patch that it scarcely needs a signoff.
> Nonetheless:
>
> Signed-off-by: Jay Fenlason
>
> --- a/ldap/servers/slapd/back-ldbm/ldbm_delete.c.orig 2020-01-31
> 07:28:04.085861174 -0500
> +++ b/ldap/servers/slapd/back-ldbm/ldbm_delete.c 2020-01-31
> 07:30:33.932947489 -0500
> @@ -81,6 +81,7 @@
> Connection *pb_conn;
> int32_t parent_op = 0;
> struct timespec parent_time;
> +int pre_delete_called = 0;
>
> if (slapi_pblock_get(pb, SLAPI_CONN_ID, _id) < 0) {
> conn_id = 0; /* connection is NULL */
> @@ -371,6 +372,7 @@
> }
> if (retval == 0) {
> retval = plugin_call_plugins(pb,
> SLAPI_PLUGIN_BE_PRE_DELETE_FN);
> + pre_delete_called = 1;
> }
> if (retval)
> {
> @@ -1491,7 +1493,7 @@
> * The bepostop is called even if the operation fails,
> * but not if the operation is purging tombstones.
> */
> -if (!delete_tombstone_entry) {
> +if (pre_delete_called) {
> plugin_call_plugins(pb, SLAPI_PLUGIN_BE_POST_DELETE_FN);
> }
> /* Need to return to cache after post op plugins are called */
> ___
> 389-devel mailing list -- 389-devel@lists.fedoraproject.org
> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
—
Sincerely,
William Brown
Senior Software Engineer, 389 Directory Server
SUSE Labs
___
389-devel mailing list -- 389-devel@lists.fedoraproject.org
To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please Review: 50859 ldaps only options/test
https://pagure.io/389-ds-base/pull-request/50870 https://pagure.io/389-ds-base/issue/50859 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: 50694 docker pem import
https://pagure.io/389-ds-base/pull-request/50846 https://pagure.io/389-ds-base/issue/50694 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review 50787 plugin enable issue
https://pagure.io/389-ds-base/pull-request/50845 https://pagure.io/389-ds-base/issue/50787 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] c2rust
Hi all, Last night Thierry asked about some of the values of rust, and how to approach this topic. Rust has been very powerful in how productive it's made me in other projects due to the stricter compiler rules, forcing me to handle things correctly. I think there are really two reasonable approaches to improving 389-ds through the use of rust. The first is that "new code" can be written in rust and layered in. An example is the ldapssotoken code currently being reviewed by Mark. The second is c to rust translation. This takes C, and creates equivalent "unsafe" rust that you can "overtime" improve to make it safer. A powerful testament to the quality of c2rust is the following blog. https://immunant.com/blog/2020/01/quake3/ I don't think "rewrite" is a good approach, given how much code we have that works in production today, and the possibility of losing much of that historical knowledge and hardening that does exist. c2rust is a better approach as we can "overtime" provide small polish into the areas, but extend with safer interfaces instead. An example of a c2rust step we could take is maybe a password plugin, or something like uidunique.c. Hope that's interesting to you, — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] LCA2020 authentication talk
Hi all, Last week I was at Linux Conf Australia where I presented on the psychology of multi-factor authentication, and also presented a lightning talk on failure analysis. Here are the recordings, I hope you find them useful. https://www.youtube.com/watch?v=qOZzleJ9OEs https://www.youtube.com/watch?v=eqQUepwTHjA=25m47s There are many other interesting talks available too: https://www.youtube.com/playlist?list=PLD8dAKx4J2I7pzm1pjAcreW4p7SWaXcGO And I highly recommend this keynote: https://www.youtube.com/watch?v=Yv4tI6939q0=PLD8dAKx4J2I6CoeaU6F-2Zs7XmWjUBLlT=1 Thanks, — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: 50831 Add cargo.lock for offline builds
https://pagure.io/389-ds-base/pull-request/50832 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: 50790 result text for filter verification
https://pagure.io/389-ds-base/pull-request/50808 https://pagure.io/389-ds-base/issue/50790 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please Review: 50798 - lucky last PR of 2019 :)
https://pagure.io/389-ds-base/pull-request/50801 https://pagure.io/389-ds-base/issue/50798 Thanks everyone! — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please Review: 50768 conntable freelist
https://pagure.io/389-ds-base/issue/50786 https://pagure.io/389-ds-base/pull-request/50788 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Follow up to system tap email
Hi all, Thanks to Howard's advice on mutrace, and the addition of systemtap probes I have some more data and a few things I can start to action and work on to improve the server performance I hope. The next step was two fold - measure the timing of "do_search" excluding the connection code, and to identify lock contention with mutrace. Both yielded good data. Due to an issue with systemtap, it appears that hooking on process().function is not always reliable - I believe this could be related to inlinig of some function names. This meant that I needed to add probe points to allow the tools to hook reliably to the code at known points. What it also allows is the probes to exist in the middle of a function so we can break functions down and see how they behave. For example in do_search I was able to add probes for the function entry, query preparation, op_shared_search, the finalisation and the function return. This is the histogram of the "full" do_search: ^CDistribution of do_search_full latencies (in nanoseconds) for 776771 samples max/avg/min: 157687091993/2029055283/46 value |-- count 8 |0 16 |0 32 | 106 64 |@ 755031 128 |@ 18609 256 | 1886 512 | 394 1024 | 135 2048 | 56 4096 | 67 8192 | 199 16384 |0 32768 |0 65536 | 243 131072 | 23 262144 |1 524288 | 20 1048576 |0 2097152 |0 ~ 281474976710656 |0 562949953421312 |0 1125899906842624 |1 2251799813685248 |0 4503599627370496 |0 These are the three stages (preparation, op_shared execution, and finalisation). Distribution of do_search_prepared latencies (in nanoseconds) for 776772 samples max/avg/min: 157687091369/2029052531/3 value |-- count 0 |0 1 |0 2 |7 4 |@ 612338 8 |@@@144976 16 |@ 18222 32 | 873 64 | 189 128 | 127 256 | 30 512 |5 1024 |0 2048 |1 4096 |1 8192 |2 16384 |0 32768 |0 ~ 281474976710656 |0 562949953421312 |0 1125899906842624 |1 2251799813685248 |0 4503599627370496 |
[389-devel] Please review systemtap/probe points/python tooling bits for perf
https://pagure.io/389-ds-base/pull-request/50718 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: System tap performance results,
> On 11 Dec 2019, at 20:29, thierry bordaz wrote: > > > > On 12/11/19 1:21 AM, William Brown wrote: >> >>> On 10 Dec 2019, at 19:15, thierry bordaz wrote: >>> >>> Hi William, >>> >>> Thanks for these very interesting results. >>> It would help if you can provide the stap scripts to make sure what you are >>> accounting the latency. >> Yes, I plan to put them into a PR soon once I have done a bit more data >> collection and polishing of the script setup. >> >>> Also just to be sure is latency a synonym for response time ? >> Yep, here I mean the execution time of a single operation. >> >>> Regarding the comparison (tail) 1client/16client. It looks to me that the >>> tail are equivalent: The more we have clients the more we have long >>> latency. So in a first approach I would eliminate contention effect. >> I disagree, the tail is much more pronounced in the 16 client version, and >> there is a significant shift of response times from the 32768 bucket to >> 65536 indicating that many operations are now being "held up". > > You are probably right. > There is a response time shift but I would expect a major contention effect > to shift more and flatter the graph to upper response time. > Whatever is the answer, I think an important point is to agree on the method > and that reports shows somethings suspicious. In a highly concurrent system like DS we shouldn't be seeing tail latency or spikes, we should be seeing all responses within a really small window especially when the searches are all the "same query" for a single uid. So that's why I'm looking at the spikes first. > >> >>> Regarding the ratio shorter/longer latency (assuming the search are >>> equivalent) this is interesting to know why we have such effect. One of the >>> possible cause I was thinking of is the impact of DB thread (checkpointing >>> or trickling). But if it exists similar long tail in ldbm_back, the >>> absolute value is much lower than the opshare_search: in short ldbm_back >>> accounted at most for 4ms while opshared for 67ms. So there is something >>> else (aci, network, frontend..). >>> >>> Regarding USDT I think it is very good idea. However, just to share some >>> recent stap experience, I found it intrusive. In short, I had not the same >>> throughput with and without. In my case it was not a problem, as I wanted >>> to investigate a reproducible contention. But if we want >>> support/user/customers to use it, the performance impact in production will >>> be valid point. >> I haven't noticed any "intrusiveness" from USDT so far? How were you running >> the stap scripts? > I did not add probe in DS. I was just using stap to gather per operation > specific functions duration (like plugin, backend or core). > > I do not recall the "intrusiveness" level as I was more looking for > contention data than performance value. Hmmm okay. I've found stap extremely fast and it doesn't get in the way, so unless I saw your stap scripts I'm not sure . > >> >>> best regards >>> thierry >>> >>> >>> >>> On 12/9/19 6:16 AM, William Brown wrote: >>>> Hi all, >>>> >>>> Following last weeks flamegraph runs, I wanted to find more details on >>>> exactly what was happening. While flamegraphs highlighted that a changed >>>> existed between single and multithreaded servers, it did not help to >>>> isolate where >>>> the change was occuring. >>>> >>>> To understand this I have started to work on a set of systemtap scripts >>>> that we can use to profile 389ds. These will be included in a future PR. >>>> >>>> Here are the hisograms from an initial test of profiling "do_search" >>>> >>>> 1 thread >>>> >>>> stap test_do_search.stp >>>> ^CDistribution of latencies (in nanoseconds) for 441148 samples >>>> max/avg/min: 35911542/85694/38927 >>>> value |-- count >>>> 8192 |0 >>>> 16384 |0 >>>> 32768 |@@ 167285 >>>> 65536 |@ 239280 >>>>
[389-devel] Re: System tap performance results,
> On 10 Dec 2019, at 19:15, thierry bordaz wrote: > > Hi William, > > Thanks for these very interesting results. > It would help if you can provide the stap scripts to make sure what you are > accounting the latency. Yes, I plan to put them into a PR soon once I have done a bit more data collection and polishing of the script setup. > Also just to be sure is latency a synonym for response time ? Yep, here I mean the execution time of a single operation. > > Regarding the comparison (tail) 1client/16client. It looks to me that the > tail are equivalent: The more we have clients the more we have long latency. > So in a first approach I would eliminate contention effect. I disagree, the tail is much more pronounced in the 16 client version, and there is a significant shift of response times from the 32768 bucket to 65536 indicating that many operations are now being "held up". > > Regarding the ratio shorter/longer latency (assuming the search are > equivalent) this is interesting to know why we have such effect. One of the > possible cause I was thinking of is the impact of DB thread (checkpointing or > trickling). But if it exists similar long tail in ldbm_back, the absolute > value is much lower than the opshare_search: in short ldbm_back accounted at > most for 4ms while opshared for 67ms. So there is something else (aci, > network, frontend..). > > Regarding USDT I think it is very good idea. However, just to share some > recent stap experience, I found it intrusive. In short, I had not the same > throughput with and without. In my case it was not a problem, as I wanted to > investigate a reproducible contention. But if we want support/user/customers > to use it, the performance impact in production will be valid point. I haven't noticed any "intrusiveness" from USDT so far? How were you running the stap scripts? > > best regards > thierry > > > > On 12/9/19 6:16 AM, William Brown wrote: >> Hi all, >> >> Following last weeks flamegraph runs, I wanted to find more details on >> exactly what was happening. While flamegraphs highlighted that a changed >> existed between single and multithreaded servers, it did not help to isolate >> where >> the change was occuring. >> >> To understand this I have started to work on a set of systemtap scripts that >> we can use to profile 389ds. These will be included in a future PR. >> >> Here are the hisograms from an initial test of profiling "do_search" >> >> 1 thread >> >> stap test_do_search.stp >> ^CDistribution of latencies (in nanoseconds) for 441148 samples >> max/avg/min: 35911542/85694/38927 >> value |-- count >> 8192 |0 >> 16384 |0 >> 32768 |@@ 167285 >> 65536 |@ 239280 >>131072 |@ 25788 >>262144 |@8530 >>524288 | 252 >> 1048576 |6 >> 2097152 |1 >> 4194304 |3 >> 8388608 |0 >> 16777216 |2 >> 33554432 |1 >> 67108864 |0 >> 134217728 |0 >> >> 16 thread >> >> stap test_do_search.stp >> ^CDistribution of latencies (in nanoseconds) for 407806 samples >> max/avg/min: 100315928/112407/39368 >> value |-- count >> 8192 |0 >> 16384 |0 >> 32768 | 100281 >> 65536 |@ 249656 >>131072 |@@@ 37837 >>262144 |@@@ 18322 >>524288 | 1171 >> 1048576 |
[389-devel] Re: System tap performance results,
> On 11 Dec 2019, at 02:20, Howard Chu wrote: > > 389-devel-requ...@lists.fedoraproject.org wrote: > >> My next steps from here will be: >> >> * To add USDT probes to the logs and back_ldbm to get better, fine detail >> about what is causing the excessive latency. >> * these probes are also needed to resolve what appears to be a symbol >> resolution issue with systemtap when optimisations are enabled. >> * To add probes in other parts of the code base to get better visualisation >> about where and how long timings are taking through an operation. >> * To run a lock contention profile (I was unable to do this today due to >> bugs in systemtap) >> * To document the setup proceedures >> * Commit all these into a PR >> * Document some actionable improvements we can make to improve the server >> performance. > > mutrace will give you a good audit of lock contention, and it's simple to > use, takes no setup. > > http://git.0pointer.net/mutrace.git/ Thanks, that's a good tip, I'll look into that too. Appreciate it, > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] System tap performance results,
Hi all, Following last weeks flamegraph runs, I wanted to find more details on exactly what was happening. While flamegraphs highlighted that a changed existed between single and multithreaded servers, it did not help to isolate where the change was occuring. To understand this I have started to work on a set of systemtap scripts that we can use to profile 389ds. These will be included in a future PR. Here are the hisograms from an initial test of profiling "do_search" 1 thread stap test_do_search.stp ^CDistribution of latencies (in nanoseconds) for 441148 samples max/avg/min: 35911542/85694/38927 value |-- count 8192 |0 16384 |0 32768 |@@ 167285 65536 |@ 239280 131072 |@ 25788 262144 |@8530 524288 | 252 1048576 |6 2097152 |1 4194304 |3 8388608 |0 16777216 |2 33554432 |1 67108864 |0 134217728 |0 16 thread stap test_do_search.stp ^CDistribution of latencies (in nanoseconds) for 407806 samples max/avg/min: 100315928/112407/39368 value |-- count 8192 |0 16384 |0 32768 | 100281 65536 |@ 249656 131072 |@@@ 37837 262144 |@@@ 18322 524288 | 1171 1048576 | 203 2097152 | 90 4194304 | 74 8388608 | 83 16777216 | 58 33554432 | 21 67108864 | 10 134217728 |0 268435456 |0 It's interesting to note the tail latency here: On the 16 thread version we see 67000 less in the 32768 buckets, shifting mostly through the 131072 and 262144 buckets, as well as showing a much greater number of calls in the tail. In thread 1 no operation made it to 67108864, but 10 did in 16thread, along with ~200 more that are higher than 1048567, and ~1500 more that are greater than 524288. This kind of tailing means we have "spikes" of latency throughout the execution, which then have a minor flow on cause other operations to be increased in latency. These are all in nanoseconds, so this means most operations are in do_search for 0.000131072 seconds or less, while there are spikes of operations taking between nearly 0.001048576 and 0.067108864 seconds to complete (which is an 8x to 512x increase in execution time. From these point I took two measurements of back_ldbm and the access log, knowing these were easy targets for potential areas of latency. Both of these were performed with the 16thread server. I didn't need to do this on the 1 thread because I'm looking for tail latency, not comparisons now. Because we know that there is an increase of ~1500 events of high latency, we are looking for similar numbers to appear in other tail latencies. ^CDistribution of back_ldbm latencies (in nanoseconds) for 364223 samples max/avg/min: 4757744/31192/13622 value |-- count 2048 |0 4096 |0 8192 |@@ 14318 16384 |@ 261265 32768 |63533 65536 |@@@ 20624 131072 | 4062 262144 |
[389-devel] Please review: pr 50769 filter validation option fix
https://pagure.io/389-ds-base/pull-request/50769 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please Review: 50664 - empty dirs can stop ds starting
https://pagure.io/389-ds-base/pull-request/50750 https://pagure.io/389-ds-base/issue/50664 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Performance discussion
> On 14 Nov 2019, at 22:33, Ludwig Krispenz wrote: > > > On 11/14/2019 12:17 PM, William Brown wrote: >> >>> On 14 Nov 2019, at 19:06, Ludwig Krispenz wrote: >>> >>> >>> On 11/14/2019 09:29 AM, William Brown wrote: >>>>> On 14 Nov 2019, at 18:22, Ludwig Krispenz wrote: >>>>> >>>>> Hi William, >>>>> >>>>> before further thinking about this, I need some clarification, or maybe I >>>>> just missed this. When you talk about 1..16 threads do you mean worker >>>>> threads ? >>>> Server worker threads. ldclt is set to only use 10 client threads - which >>>> is surprising that with 10 client threads we see a decline when workers > >>>> 10 (one would assume it should stabilise). >>>> >>>>> Or concurrent client connection threads in ldclt/rsearch/ - how many >>>>> concurrent connections do you have and how does varying this number >>>>> change results ? >>>> I will add more tests to this to allow varying the ldclt numbers. >>> ok, and I assume that you are using a version with nunc-stans removed, >>> could you please also verify the effect of tubo-mode on/off ? >> Correct, I'm using git master. Yes I'll check that also. I plan to add >> permutations like this to the test harness so it's easier for us to repeat >> in the future when we make changes. >> >> I also need to find a way to wire in perf/stap so we can generate >> flamegraphs from each test run too for later analysis. >> >> Thanks for the great ideas :) > Thanks, and one more idea ;-) > Can you separate the client and the server on two different machines, I've > seen ldclt or other clients impacting cpu usage a lot, there will be some > network overhead, but this should be ok (and more realistic) That was the original goal, but I can't seperate it (yet) because we restart to change settings ... I'm not sure what's the best way to do it - have the tests maybe act as a generator and then you have to run the ldclt from a seperate machine? Not sure really I need to think what it should look like. I know viktor did some work on pytest over multiple hosts so perhaps that could help here too to coordinate? I think they also were speaking about ansible as well ... maybe he should comment if he has ideas. >> >>>>> Regards, >>>>> Ludwig >>>>> >>>>> On 11/14/2019 03:34 AM, William Brown wrote: >>>>>> Hi all, >>>>>> >>>>>> After our catch up, we were discussing performance matters. I decided to >>>>>> start on this while waiting for some of my tickets to be reviewed and to >>>>>> see what's going on. >>>>>> >>>>>> These tests were carried out on a virtual machine configured in search 6 >>>>>> to have access to 6 CPU's, and search 12 with 12 CPU. Both machines had >>>>>> access to 8GB of ram. >>>>>> >>>>>> The hardware is an i7 2.2GHz with 6 cores (12 threads) and 32GB of ram, >>>>>> with NVME storage provided. >>>>>> >>>>>> The rows are the VM CPU's available, and the columns are the number of >>>>>> threads in nsslapd-threadnumber. No other variables were changed. The >>>>>> database has 6000 users and 4000 groups. The instance was restarted >>>>>> before each test. The search was a randomised uid equality test with a >>>>>> single result. I provided the thread 6 and 12 columns to try to match >>>>>> the VM and host specs rather than just the traditional base 2 sequence >>>>>> we see. >>>>>> >>>>>> I've attached a screen shot of the results, but I have some initial >>>>>> thoughts to provide on this. What's interesting is our initial 1 thread >>>>>> performance and how steeply it ramps up towards 4 thread. This in mind >>>>>> it's not a linear increase. Per thread on s6 we go from ~3800 to ~2500 >>>>>> ops per second, and a similar ratio exists in s12. What is stark is that >>>>>> after t4 we immediately see a per thread *decline* despite the greater >>>>>> amount of available computer resources. This indicates that it is poor >>>>>> locking and thread coordination causing a rapid decline in performance. >>>>>> This was true on both s6 and s12.
[389-devel] Please review gssapi test on suse support 50730
https://pagure.io/389-ds-base/pull-request/50730 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Couple of troubles around using dsconf
> On 19 Nov 2019, at 22:36, Matus Honek wrote: > > Hello folks, > > Context: My setup is a running dscontainer with exported /data. While > developing (outside of the container) I am trying to run `dsconf > ldapi://%2fpath%2fto%2fdscontainers%2fsocket security get`. > > Issue 1: I get IndexError exception: > File "/home/mhonek/src/ds/up/src/lib389/lib389/_mapped_object.py", > line 158, in display > How to fix the fact we can get no results to display, and to fix it > correctly so that nothing else eventually blows up? Don't know... Raise an issue seems like the first place to deal with this, especially with the stack trace associated. > > Issue 2: Tracing back I find out I autobinded as non-root (non 0 UID). > Expectable, but still unexpected. So I tried to override this by > providing `-D` and `-w` explicitly to dsconf. No change, still > autobinding. Turns out the autobind has preference over simple bind in > DirSrv.open, this comes from [implementation]. > Possible solution: Instead of `elif can_autobind(): ... else: > simple_bind` do `elif self.binddn is not None: ... else > can_autobind(): ...`. Worked for me. Would this blow up some use-case? > Don't know... autobind is super important for ldapi - you have to prefer autobind else things go wildly wrong but also that whole section of dirsrv.open is extremely cursed and never should have been structured like that. But this is the bed we have, so we have to lie in it now. A major drawback that I have been suffering from here is in your statement - would this blow up some use-case? I have no idea. It's python, which is basically schroedingers language. It's unknown unless it's observed running. I'd suggest we open an issue here and then let's think about how we could solve it, but it won't be easy because all of the def open code needs a rethink - we need a better approach to how to determine how we want to present authentication credentials. > > Sub-issue 2a: Given I was able to autobind as non-root UID, the > wording in a log message [aubind-log]. The word "root" probably > shouldn't be there? Sure, just open an issue and fix it? (This is kind of why I want PR's to not need an issue, to encourage quicker small fixes like this rather than a lot of admin overhead to the process). > > Somewhat troubling 1: At the time of running open in the autobind > branch in DirSrv.open [autobind] the value of `self.bindpw` is > literally "password" even though no `-D` nor `-w` was provided on > command line for dsconf. I believe there are some reasons (besides > "because the code is written so") why this is so but I would like to > be enlightened here. Lib389 didn't always have a topologies module. Previously each instance would be setup manually in each test with ds.allocate(); ds.create(); ds.bind() ... So that meant a lot of "testing" defaults ended up in DirSrv. It became a kitchen sink. It was later on that we start to really modularise it out with DSLdapObject, topologies and others. That caused a lot of defaults to shift out, but a lot of fragments of that legacy still exist in DirSrv because it has an extremely fragile design. Anyway, this can probably be removed, it shouldn't be needed, and it could confuse def open(). In general, I've been trying to push toward local_simple_allocate and remote_simple_allocate, but I never got traction on it. The whole way we setup DirSrv as a type is really messy :( ... Actually, the whole DirSrv type is a small microcosm of evil ... Honestly I would *love* a solid two weeks in a room with you, simon, mark, viktor, and we just clean and replace DirSrv as a type. That would be amazing. > > [implementation] https://pagure.io/389-ds-base/c/e07e489 > [autobind-log] > https://pagure.io/389-ds-base/blob/6d70cbe/f/src/lib389/lib389/__init__.py#_1063 > [autobind] > https://pagure.io/389-ds-base/blob/6d70cbe/f/src/lib389/lib389/__init__.py#_1060 > > Please, share your ideas, where I went wrong, what we could go ahead with. Computers were a mistake, that was where we went wrong :) But when you see a problem, you talk about it (which you did), then we fix it. :) > > Thanks, > Matus > > -- > Matúš Honěk > Software Engineer > Red Hat Czech > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Direc
[389-devel] Re: Performance discussion
> On 14 Nov 2019, at 19:06, Ludwig Krispenz wrote: > > > On 11/14/2019 09:29 AM, William Brown wrote: >> >>> On 14 Nov 2019, at 18:22, Ludwig Krispenz wrote: >>> >>> Hi William, >>> >>> before further thinking about this, I need some clarification, or maybe I >>> just missed this. When you talk about 1..16 threads do you mean worker >>> threads ? >> Server worker threads. ldclt is set to only use 10 client threads - which is >> surprising that with 10 client threads we see a decline when workers > 10 >> (one would assume it should stabilise). >> >>> Or concurrent client connection threads in ldclt/rsearch/ - how many >>> concurrent connections do you have and how does varying this number change >>> results ? >> I will add more tests to this to allow varying the ldclt numbers. > ok, and I assume that you are using a version with nunc-stans removed, could > you please also verify the effect of tubo-mode on/off ? Correct, I'm using git master. Yes I'll check that also. I plan to add permutations like this to the test harness so it's easier for us to repeat in the future when we make changes. I also need to find a way to wire in perf/stap so we can generate flamegraphs from each test run too for later analysis. Thanks for the great ideas :) >> >>> Regards, >>> Ludwig >>> >>> On 11/14/2019 03:34 AM, William Brown wrote: >>>> Hi all, >>>> >>>> After our catch up, we were discussing performance matters. I decided to >>>> start on this while waiting for some of my tickets to be reviewed and to >>>> see what's going on. >>>> >>>> These tests were carried out on a virtual machine configured in search 6 >>>> to have access to 6 CPU's, and search 12 with 12 CPU. Both machines had >>>> access to 8GB of ram. >>>> >>>> The hardware is an i7 2.2GHz with 6 cores (12 threads) and 32GB of ram, >>>> with NVME storage provided. >>>> >>>> The rows are the VM CPU's available, and the columns are the number of >>>> threads in nsslapd-threadnumber. No other variables were changed. The >>>> database has 6000 users and 4000 groups. The instance was restarted before >>>> each test. The search was a randomised uid equality test with a single >>>> result. I provided the thread 6 and 12 columns to try to match the VM and >>>> host specs rather than just the traditional base 2 sequence we see. >>>> >>>> I've attached a screen shot of the results, but I have some initial >>>> thoughts to provide on this. What's interesting is our initial 1 thread >>>> performance and how steeply it ramps up towards 4 thread. This in mind >>>> it's not a linear increase. Per thread on s6 we go from ~3800 to ~2500 ops >>>> per second, and a similar ratio exists in s12. What is stark is that after >>>> t4 we immediately see a per thread *decline* despite the greater amount of >>>> available computer resources. This indicates that it is poor locking and >>>> thread coordination causing a rapid decline in performance. This was true >>>> on both s6 and s12. The decline intesifies rapidly once we exceed the CPU >>>> avail on the host (s6 between t6 to t12), but still declines even when we >>>> do have the hardware threads available in s12. >>>> >>>> I will perform some testing between t1 and t6 versions to see if I can >>>> isolate which functions are having a growth in time consumption. >>>> >>>> For now an early recommendation is that we alter our default CPU >>>> auto-tuning. Currently we use a curve which starts at 16 threads from 1 to >>>> 4 cores, and then tapering down to 512 cores to 512 threads - however in >>>> almost all of these autotuned threads we have threads greater than our >>>> core count. This from this graph would indicate that this decision only >>>> hurts our performance rather than improving it. I suggest we change our >>>> thread autotuning to be 1 to 1 ratio of threads to cores to prevent over >>>> contention on lock resources. >>>> >>>> Thanks, more to come once I setup this profiling on a real machine so I >>>> can generate flamegraphs. >>>> >>>> >>>> >>>> — >>>> Sincerely, >>>> >>>> William Brown >>>> >
[389-devel] Re: Performance discussion
> On 14 Nov 2019, at 18:22, Ludwig Krispenz wrote: > > Hi William, > > before further thinking about this, I need some clarification, or maybe I > just missed this. When you talk about 1..16 threads do you mean worker > threads ? Server worker threads. ldclt is set to only use 10 client threads - which is surprising that with 10 client threads we see a decline when workers > 10 (one would assume it should stabilise). > Or concurrent client connection threads in ldclt/rsearch/ - how many > concurrent connections do you have and how does varying this number change > results ? I will add more tests to this to allow varying the ldclt numbers. > > Regards, > Ludwig > > On 11/14/2019 03:34 AM, William Brown wrote: >> Hi all, >> >> After our catch up, we were discussing performance matters. I decided to >> start on this while waiting for some of my tickets to be reviewed and to see >> what's going on. >> >> These tests were carried out on a virtual machine configured in search 6 to >> have access to 6 CPU's, and search 12 with 12 CPU. Both machines had access >> to 8GB of ram. >> >> The hardware is an i7 2.2GHz with 6 cores (12 threads) and 32GB of ram, with >> NVME storage provided. >> >> The rows are the VM CPU's available, and the columns are the number of >> threads in nsslapd-threadnumber. No other variables were changed. The >> database has 6000 users and 4000 groups. The instance was restarted before >> each test. The search was a randomised uid equality test with a single >> result. I provided the thread 6 and 12 columns to try to match the VM and >> host specs rather than just the traditional base 2 sequence we see. >> >> I've attached a screen shot of the results, but I have some initial thoughts >> to provide on this. What's interesting is our initial 1 thread performance >> and how steeply it ramps up towards 4 thread. This in mind it's not a linear >> increase. Per thread on s6 we go from ~3800 to ~2500 ops per second, and a >> similar ratio exists in s12. What is stark is that after t4 we immediately >> see a per thread *decline* despite the greater amount of available computer >> resources. This indicates that it is poor locking and thread coordination >> causing a rapid decline in performance. This was true on both s6 and s12. >> The decline intesifies rapidly once we exceed the CPU avail on the host (s6 >> between t6 to t12), but still declines even when we do have the hardware >> threads available in s12. >> >> I will perform some testing between t1 and t6 versions to see if I can >> isolate which functions are having a growth in time consumption. >> >> For now an early recommendation is that we alter our default CPU >> auto-tuning. Currently we use a curve which starts at 16 threads from 1 to 4 >> cores, and then tapering down to 512 cores to 512 threads - however in >> almost all of these autotuned threads we have threads greater than our core >> count. This from this graph would indicate that this decision only hurts our >> performance rather than improving it. I suggest we change our thread >> autotuning to be 1 to 1 ratio of threads to cores to prevent over contention >> on lock resources. >> >> Thanks, more to come once I setup this profiling on a real machine so I can >> generate flamegraphs. >> >> >> >> — >> Sincerely, >> >> William Brown >> >> Senior Software Engineer, 389 Directory Server >> SUSE Labs >> >> >> >> ___ >> 389-devel mailing list -- >> 389-devel@lists.fedoraproject.org >> >> To unsubscribe send an email to >> 389-devel-le...@lists.fedoraproject.org >> >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > > -- > Red Hat GmbH, > http://www.de.redhat.com/ > , Registered seat: Grasbrunn, > Commercial register: Amtsgericht Muenchen, HRB 153243, > Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, > Eric Shander > > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-c
[389-devel] Performance discussion
Hi all, After our catch up, we were discussing performance matters. I decided to start on this while waiting for some of my tickets to be reviewed and to see what's going on. These tests were carried out on a virtual machine configured in search 6 to have access to 6 CPU's, and search 12 with 12 CPU. Both machines had access to 8GB of ram. The hardware is an i7 2.2GHz with 6 cores (12 threads) and 32GB of ram, with NVME storage provided. The rows are the VM CPU's available, and the columns are the number of threads in nsslapd-threadnumber. No other variables were changed. The database has 6000 users and 4000 groups. The instance was restarted before each test. The search was a randomised uid equality test with a single result. I provided the thread 6 and 12 columns to try to match the VM and host specs rather than just the traditional base 2 sequence we see. I've attached a screen shot of the results, but I have some initial thoughts to provide on this. What's interesting is our initial 1 thread performance and how steeply it ramps up towards 4 thread. This in mind it's not a linear increase. Per thread on s6 we go from ~3800 to ~2500 ops per second, and a similar ratio exists in s12. What is stark is that after t4 we immediately see a per thread *decline* despite the greater amount of available computer resources. This indicates that it is poor locking and thread coordination causing a rapid decline in performance. This was true on both s6 and s12. The decline intesifies rapidly once we exceed the CPU avail on the host (s6 between t6 to t12), but still declines even when we do have the hardware threads available in s12. I will perform some testing between t1 and t6 versions to see if I can isolate which functions are having a growth in time consumption. For now an early recommendation is that we alter our default CPU auto-tuning. Currently we use a curve which starts at 16 threads from 1 to 4 cores, and then tapering down to 512 cores to 512 threads - however in almost all of these autotuned threads we have threads greater than our core count. This from this graph would indicate that this decision only hurts our performance rather than improving it. I suggest we change our thread autotuning to be 1 to 1 ratio of threads to cores to prevent over contention on lock resources. Thanks, more to come once I setup this profiling on a real machine so I can generate flamegraphs. — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: 50703 - ldapssotoken for ds-portal
https://pagure.io/389-ds-base/pull-request/50703 Thanks! -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Check for anonymous bind in conn / pb_op
> On 7 Nov 2019, at 19:40, Ludwig Krispenz wrote: > > > On 11/07/2019 04:45 AM, William Brown wrote: >> What's the correct way to check if the connection is bound as anonymous with >> the pb_op or connection struct? It's not obviously jumping out at me, and my >> grep skills have failed me? > do what aclanom_is_client_anonymus () does, check SLAPI_REQUESTOR_DN Thanks Ludwig, that's a great idea. :) >> >> >> >> -- >> Sincerely, >> >> William >> ___ >> 389-devel mailing list -- 389-devel@lists.fedoraproject.org >> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > > -- > Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, > Commercial register: Amtsgericht Muenchen, HRB 153243, > Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, > Eric Shander > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Check for anonymous bind in conn / pb_op
What's the correct way to check if the connection is bound as anonymous with the pb_op or connection struct? It's not obviously jumping out at me, and my grep skills have failed me? -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review, lib389 support working without defaults.inf
https://pagure.io/389-ds-base/pull-request/50682 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: portal 2 allow customisation
https://pagure.io/389-ds-portal/pull-request/10 https://pagure.io/389-ds-portal/issue/2 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please Review 50669 - remove nuncstans
https://pagure.io/389-ds-base/issue/50669 https://pagure.io/389-ds-base/pull-request/50670 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: 50667 dsctl -l to use paths.
https://pagure.io/389-ds-base/pull-request/50668 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Future of nunc-stans
> On 22 Oct 2019, at 07:28, Mark Reynolds wrote: > > > On 10/9/19 10:00 PM, William Brown wrote: >>> On 9 Oct 2019, at 19:55, Ludwig Krispenz wrote: >>> >>> Hi William, >>> >>> I like your radical approach :-) >>> >>> In my opinion our connection code is getting to complicated by maintaining >>> two different implementations in parallel - not separated, but >>> intermangled (and even more complicated by turbo mode). So I agree we >>> should have only one, but which one ? In my opinion nunc stans is the >>> theoretically better approach, but nobody is confident enough to rely on >>> nunc stans alone. The conntable mode has its problems (especially if >>> handling many concurrent connections, and worse if they are established >>> almost at the same time)(otherwise we would not have experimented with nunc >>> stans), but is stable and for most of the use cases efficient enough. >> I think you nailed it in one - we aren't confident in nunc-stans today, so >> let's keep what works and improve that. There are already many similar >> concepts - work queues, threads, even slapi_conn_t. I think that it would be >> possible to bring "nunc-stans ideas" into reworking and improvement to the >> current connection code instead. >> >>> So reducing the complexity by removing nunc stans (and maybe also turbo >>> mode) and then do cleanup and try to improve the bottlenecks would be an >>> acceptable approach to me. >> Agree. It also means we can make much smaller changes in an easier to >> control and test fashion I think. >> >>> In my opinion the core of the problem of the "old" connection code is that >>> the main thread is handling new connections and already established >>> connections and so does iterate over the connection table. Using an event >>> model looks like the best way to handle this, but if it doesn't work we >>> need to look for other improvements without breaking things. >>> Your suggestion to make the conn table data structure more lean and >>> flexible is one option. In sun ds, when I didn't know about event queues I >>> did split the main thread, one handling new connections and multiple to >>> handle established connections (parts of teh conn table) - reusing the >>> existing mechanisms, just splitting the load. Maybe we can also think in >>> this direction. >> I think so too. We can certainly have some ideas about what actually does >> the polling vs what does accepting, or better, event management etc. There >> are some ideas to have smaller groups of workers too to improve thread >> locality and help improve concurrency too. >> >> So maybe I'll put together a patch to remove nunc-stans soon then, and start >> to look at the existing connection code and options to improve that + some >> profiling. >> >> I still would like to hear about my original question though as quoted >> below, I think Mark might have some comments :) > > Why nunc-stans? Because at the time we were terrible at handling many > connections, simultaneously and in large numbers. C10K, etc. Our > competitors performed much better in these scenarios than we did. I recall > several customer cases complaining about "our performance verses theirs" in > regards to large numbers of connection. > > So, moving forward, I agree with everyone here. We should remove nunc-stans, > but as William said try to incorporate some of its concepts into our > connection code (eventually). We should clean up the connection code as much > as possible, and remove turbo mode if it does not provide much value. I think turbo mode was to try and shortcut returning to the conntable and then having the blocking on the connections poll because the locking strategies before weren't as good. I think there is still some value in turbo "for now" but if we can bring in libevent, then it diminishes because we become event driven rather than poll driven. > The one thing that has been frustrating is how the connection code had > become very complicated and most of us no longer know how it works anymore. > It would be nice to get it to a state that is much more maintainable > (especially for engineers new to the code). Given how much I've looked at it recently, I'm probably the closest to having an understanding of that code, but I certainly won't claim 100% expertise here. > > I think we should look at improving the connection table as previously > suggested, and we should add the additional polling thread that Ludwig
[389-devel] Re: Should stopping the server interrupt an import job?
> On 16 Oct 2019, at 00:27, Mark Reynolds wrote: > > > On 10/14/19 11:13 PM, William Brown wrote: >> >>> On 15 Oct 2019, at 15:51, Mark Reynolds wrote: >>> >>> >>> On 10/14/19 6:35 PM, William Brown wrote: >>>>> On 15 Oct 2019, at 06:58, Mark Reynolds wrote: >>>>> >>>>> So we are finding these race conditions (leading to heap-use-after-free) >>>>> when you stop the server while an import task is running. The current >>>>> code aborts the task which leaves the database unusable until it is fully >>>>> reinitialized at a later time. Unfortunately the code that handles this >>>>> is complex, and very error prone. >>>>> >>>>> I'd like to allow the import task to complete before fully shutting the >>>>> server down. Code fix is trivial, but do we want the import to finish, >>>>> or should the import be aborted (and database left broken)? Thoughts? >>>>> Opinions? >>>> The question is "what does the admin expect"? I could envisage if you >>>> start an import and then cancel the task, you expect: >>>> >>>> * The task to be immediately stopped >>>> * The db content rolled back. >>>> >>>> Shouldn't we be in a betxn or similar during an import so we can revert? >>>> >>>> Failing this, I'd assume the user would expect a ctrl-c to immediately >>>> cancel the task. >>>> >>>> What kind of use-after-frees are we seeing? >>> See >>> >>> https://pagure.io/389-ds-base/issue/50646 >>> >>> Pretty sure the first thing the import does is delete the db directory, but >>> I have not found that in the code yet, but there are definitely no >>> transactions used during an "import". It's a very different process. Now >>> rolling back the database would be nice, but I can imagine very large >>> databases(100+ million entries) where disk space could be an issue if you >>> have to keep the old database around until the new one is imported. >>> >>> As for aborting, currently there is no abort mechanism except for stopping >>> the server. So a ctrl-C is not really an option at this time. Keep in >>> mind I can still easily keep the current abort behavior during a shutdown, >>> but in the current design if you abort an import the database is hosed. >> What about at least hosing it but nicely somehow? So we rm db, then we >> start the import into some temp location, so on ctrl-c even if we crash, db >> is an empty dir and still mildly hosed? >> >> I can see your point though about the db size stuff. So I guess my thinking >> then is: >> >> * Could we just fix the use after free *and* make the db hosing nicer >> somehow? > I can fix the use-after-free, that's easy, but fixing the "hosing" is not > trivial, and it is out of the scope of what I am trying to do in 1.3.x. fair >> * What happens if we ignore the ctrl-c and just block on import? I think >> someone would reach for ctrl+\ pretty fast ... > The import code launches a thread to the do the work. So hitting control-C > on a CLI tool will not stop the import, and there is currently no abort > process for Slapi Tasks that I am aware of besides creating a new "Abort > Import" task. This is something we could do in 1.4.3, or 1.5.x along with > the new backend work. Yeah, that seems like the possible solution to have each thread check periodically for a cancel signal in it's work loop? >> >> >> >>>>> Thanks, >>>>> >>>>> Mark >>>>> >>>>> -- >>>>> >>>>> 389 Directory Server Development Team >>>>> ___ >>>>> 389-devel mailing list -- 389-devel@lists.fedoraproject.org >>>>> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >>>>> Fedora Code of Conduct: >>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>> List Archives: >>>>> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org >>>> — >>>> Sincerely, >>>> >>>> William Brown >>>> >>>> Senior Software Engineer, 389 Directory Server >>>>
[389-devel] Re: Should stopping the server interrupt an import job?
> On 15 Oct 2019, at 15:51, Mark Reynolds wrote: > > > On 10/14/19 6:35 PM, William Brown wrote: >> >>> On 15 Oct 2019, at 06:58, Mark Reynolds wrote: >>> >>> So we are finding these race conditions (leading to heap-use-after-free) >>> when you stop the server while an import task is running. The current code >>> aborts the task which leaves the database unusable until it is fully >>> reinitialized at a later time. Unfortunately the code that handles this is >>> complex, and very error prone. >>> >>> I'd like to allow the import task to complete before fully shutting the >>> server down. Code fix is trivial, but do we want the import to finish, or >>> should the import be aborted (and database left broken)? Thoughts? >>> Opinions? >> The question is "what does the admin expect"? I could envisage if you start >> an import and then cancel the task, you expect: >> >> * The task to be immediately stopped >> * The db content rolled back. >> >> Shouldn't we be in a betxn or similar during an import so we can revert? >> >> Failing this, I'd assume the user would expect a ctrl-c to immediately >> cancel the task. >> >> What kind of use-after-frees are we seeing? > See > > https://pagure.io/389-ds-base/issue/50646 > > Pretty sure the first thing the import does is delete the db directory, but I > have not found that in the code yet, but there are definitely no transactions > used during an "import". It's a very different process. Now rolling back > the database would be nice, but I can imagine very large databases(100+ > million entries) where disk space could be an issue if you have to keep the > old database around until the new one is imported. > > As for aborting, currently there is no abort mechanism except for stopping > the server. So a ctrl-C is not really an option at this time. Keep in mind > I can still easily keep the current abort behavior during a shutdown, but in > the current design if you abort an import the database is hosed. What about at least hosing it but nicely somehow? So we rm db, then we start the import into some temp location, so on ctrl-c even if we crash, db is an empty dir and still mildly hosed? I can see your point though about the db size stuff. So I guess my thinking then is: * Could we just fix the use after free *and* make the db hosing nicer somehow? * What happens if we ignore the ctrl-c and just block on import? I think someone would reach for ctrl+\ pretty fast ... > >>> Thanks, >>> >>> Mark >>> >>> -- >>> >>> 389 Directory Server Development Team >>> ___ >>> 389-devel mailing list -- 389-devel@lists.fedoraproject.org >>> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org >> — >> Sincerely, >> >> William Brown >> >> Senior Software Engineer, 389 Directory Server >> SUSE Labs >> ___ >> 389-devel mailing list -- 389-devel@lists.fedoraproject.org >> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > > -- > > 389 Directory Server Development Team — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Discuss/Review: tls tools for dsctl
Hi, https://pagure.io/389-ds-base/pull-request/50648 https://pagure.io/389-ds-base/issue/50007 I'd like some feedback on this - especially look at Mark and Simon here - about how we should "display" certificates in the list/show commands. What data should we show? Right now it's the nickname, issuer dn and the subject dn, and expiry date. (It's in a python tuple, this should obviously change). So some ideas here would be lovely. Thanks! -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Should stopping the server interrupt an import job?
> On 15 Oct 2019, at 06:58, Mark Reynolds wrote: > > So we are finding these race conditions (leading to heap-use-after-free) when > you stop the server while an import task is running. The current code aborts > the task which leaves the database unusable until it is fully reinitialized > at a later time. Unfortunately the code that handles this is complex, and > very error prone. > > I'd like to allow the import task to complete before fully shutting the > server down. Code fix is trivial, but do we want the import to finish, or > should the import be aborted (and database left broken)? Thoughts? Opinions? The question is "what does the admin expect"? I could envisage if you start an import and then cancel the task, you expect: * The task to be immediately stopped * The db content rolled back. Shouldn't we be in a betxn or similar during an import so we can revert? Failing this, I'd assume the user would expect a ctrl-c to immediately cancel the task. What kind of use-after-frees are we seeing? > > Thanks, > > Mark > > -- > > 389 Directory Server Development Team > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Discuss - change to default aci
Hi, Viktor rightly pointed out that I should discuss the content of: https://pagure.io/389-ds-base/pull-request/50641 This change is not obvious due to the design of the default ACI system in lib389 (as a result the diff shows a whole new file). However the changes are limited to ou=People. This allows users to self-write userPassword and legalName, and allows self read to sn and legalName attributes. These were always intended to be in place, but apparently William of the past didn't write tests to assert these properties - that's why in this change there are now tests to show the behaviours we expect for people. This was noticed when working on 389-ds-portal, where these behaviours are somewhat important to have! So to address the obvious question, this only affects *new* deployments that will use sample entries / initialise. Existing deployments on 1.3.x and 1.4.[0,1] will not have their aci's changed. Any install that does a full re-init over the sample entries will obviously discard this content. These should work with both nsUser and User types as well, thus the tests for both. I think it's a pretty simple change, and from a usability standpoint it's another small step in making new deployments a bit smoother. Thoughts/comments? I'd aim to merge this about the 18th of oct, so that's a week for everyone to have their say, and there is always room to revert things later if needed. Thanks! -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Future of nunc-stans
> On 9 Oct 2019, at 19:55, Ludwig Krispenz wrote: > > Hi William, > > I like your radical approach :-) > > In my opinion our connection code is getting to complicated by maintaining > two different implementations in parallel - not separated, but intermangled > (and even more complicated by turbo mode). So I agree we should have only > one, but which one ? In my opinion nunc stans is the theoretically better > approach, but nobody is confident enough to rely on nunc stans alone. The > conntable mode has its problems (especially if handling many concurrent > connections, and worse if they are established almost at the same > time)(otherwise we would not have experimented with nunc stans), but is > stable and for most of the use cases efficient enough. I think you nailed it in one - we aren't confident in nunc-stans today, so let's keep what works and improve that. There are already many similar concepts - work queues, threads, even slapi_conn_t. I think that it would be possible to bring "nunc-stans ideas" into reworking and improvement to the current connection code instead. > > So reducing the complexity by removing nunc stans (and maybe also turbo mode) > and then do cleanup and try to improve the bottlenecks would be an acceptable > approach to me. Agree. It also means we can make much smaller changes in an easier to control and test fashion I think. > > In my opinion the core of the problem of the "old" connection code is that > the main thread is handling new connections and already established > connections and so does iterate over the connection table. Using an event > model looks like the best way to handle this, but if it doesn't work we need > to look for other improvements without breaking things. > Your suggestion to make the conn table data structure more lean and flexible > is one option. In sun ds, when I didn't know about event queues I did split > the main thread, one handling new connections and multiple to handle > established connections (parts of teh conn table) - reusing the existing > mechanisms, just splitting the load. Maybe we can also think in this > direction. I think so too. We can certainly have some ideas about what actually does the polling vs what does accepting, or better, event management etc. There are some ideas to have smaller groups of workers too to improve thread locality and help improve concurrency too. So maybe I'll put together a patch to remove nunc-stans soon then, and start to look at the existing connection code and options to improve that + some profiling. I still would like to hear about my original question though as quoted below, I think Mark might have some comments :) >> The main question is *why* do we want it merged? >> Is it performance? Recently I provided a patch that yielded an approximate >> ~30% speed up in the entire server through put just by changing our existing >> connection code. >> Is it features? What features are we wanting from this? We have no >> complaints about our current threading model and thread allocations. >> Is it maximum number of connections? We can always change the conntable to a >> better datastructure that would help scale this number higher (which would >> also yield a performance gain). — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: 6 initial self-service pw change
https://pagure.io/389-ds-portal/pull-request/8 There is an issue where the status isn't shown on the UI, but I think we'll change to PF4 shortly, so we'll fix that then. The self-service aci's needed are in https://pagure.io/389-ds-base/pull-request/50641# -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please Review: 50641 - default aci
https://pagure.io/389-ds-base/pull-request/50641 I noticed this with 389-ds-portal when a user couldn't self-change their password. Opps! -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Future of nunc-stans
> On 9 Oct 2019, at 09:18, Rich Megginson wrote: > > On 10/8/19 4:55 PM, William Brown wrote: >> Hi everyone, >> In our previous catch up (about 4/5 weeks ago when I was visiting >> Matus/Simon), we talked about nunc-stans and getting it at least cleaned up >> and into the code base. >> I've been looking at it again, and really thinking about it and reflecting >> on it and I have a lot of questions and ideas now. >> The main question is *why* do we want it merged? >> Is it performance? Recently I provided a patch that yielded an approximate >> ~30% speed up in the entire server through put just by changing our existing >> connection code. >> Is it features? What features are we wanting from this? We have no >> complaints about our current threading model and thread allocations. >> Is it maximum number of connections? We can always change the conntable to a >> better datastructure that would help scale this number higher (which would >> also yield a performance gain). > > It is mostly about the c10k problem, trying to figure out a way to use epoll, > via an event framework like libevent, libev, or libtevent, but in a > multi-threaded way (at the time none of those were really thread safe, or > suitable for use in the way we do multi-threading in 389). > > It wasn't about performance, although I hoped that using lock-free data > structures might solve some of the performance issues around thread > contention, and perhaps using a "proper" event framework might give us some > performance boost e.g. the idle thread processing using libevent timeouts. I > think that using poll() is never going to scale as well as epoll() in some > cases e.g. lots of concurrent connections, no matter what sort of > datastructure you use for the conntable. The conntable was bottlenecking because when you had: | conn | conn | freeslot | it would attempt to lock each conn to see if it was free or not. This meant if a conn was in an io, we would block waiting for it to finish before we could move to the next conn to check if it was free. After changing to pthread, we can now do trylock, where if trylock fails it can be implied the conn slot must be in use, so skip it. This is how we got the 30% speed up recently (my laptop went from about 4200 conn/sec to over 6000). However the conntable exists to allow the conn struct to be re-used over and over. There are multiple ways we could solve this. On conn free, we could append to a freelist to prevent iterating over the list to find a slot. We could use a btreemap and just alloc/dealloc conns as needed to prevent the need to walk and find conns (this would help sanitizers with memory too). We could bring in libevent directly to the main server and have it replace poll too. And even from then, I think we should be using flamegraphs to work out where our time limits are too. > > As far as features goes - it would be nice to give plugins the ability to > inject event requests, get timeout events, using the same framework as the > main server engine. > >> The more I have looked at the code, I guess with time and experience, the >> more hesitant I am to actually commit to merging it. It was designed by >> people who did not understand low-level concurrency issues and memory >> architectures of systems, > > I resemble that remark. I suppose you could "turn off" the lock-free code > and use mutexes. It wasn't meant to be an insult btw - when I first looked I didn't understand either, so I resemble this remark too. There are lots of my own mistakes all over that code. We have changed to mutexes since, but even with those the job model and how they queue / dequeue and work along with libevent still has issues because you need to protect the items in certain states basically. Having a job self-rearm was a nightmare problem. We had to add a monitor on each ns_job_t to allow it to work correctly too because of the call back nature of the code. When it was initially added there was a job per io event, but now it's a job per connection and it lives the life of the connection. This has lead to the current issue where there is a 1:1 of slapi_conn_t and ns_job_t, and each has a lock and that led to deadlocks ... > >> so it's had a huge number of (difficult and subtle) unsafety issues. And >> while most of those are fixed, what it does is duplicating the connection >> structure from core 389, > > It was supposed to eventually replace the connection code. There is too much in the slapi_conn_t structure that can't be removed though, and too much locking of that struct through out the codebase of it. It's probably better to try to update slapi_conn_t to have new ideas than trying to replace it
[389-devel] Future of nunc-stans
Hi everyone, In our previous catch up (about 4/5 weeks ago when I was visiting Matus/Simon), we talked about nunc-stans and getting it at least cleaned up and into the code base. I've been looking at it again, and really thinking about it and reflecting on it and I have a lot of questions and ideas now. The main question is *why* do we want it merged? Is it performance? Recently I provided a patch that yielded an approximate ~30% speed up in the entire server through put just by changing our existing connection code. Is it features? What features are we wanting from this? We have no complaints about our current threading model and thread allocations. Is it maximum number of connections? We can always change the conntable to a better datastructure that would help scale this number higher (which would also yield a performance gain). The more I have looked at the code, I guess with time and experience, the more hesitant I am to actually commit to merging it. It was designed by people who did not understand low-level concurrency issues and memory architectures of systems, so it's had a huge number of (difficult and subtle) unsafety issues. And while most of those are fixed, what it does is duplicating the connection structure from core 389, leading to weird solutions like lock sharing and having to use monitors and more. We've tried a few times to push forward with this, but each time we end up with a lot of complexity and fragility. So I'm currently thinking a better idea is to step back, re-evaluate what the problem is we are trying to solve for, then to solve *that*. The question now is "what is the concern that ns would solve". From knowing that, then we can make a plan and approach it more constructively I think. At the end of the day, I'm questioning if we should just rm -r src/nunc-stans and rethink this whole approach - there are just too many architectural flaws and limitations in ns that are causing us headaches. Ideas and thoughts? -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Updates to 389-ds docker
Sorry it took a few days, there was a delay with some other stuff at suse. With Matus' comments, I have linked the dockerfile on the hub repo, but we could change the build process if it really causes issues. You can find the images here. I've taken the RC flag off them, but I still want feedback from people from using these in anger. I'm probably going to redeploy my home 389 to these images today/tomorrow to get some mileage on them. Any other testers or feedback welcome! At what point do we say they are "mass-usable" to the public? -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: What is the tier1 pytest mark for?
> On 4 Oct 2019, at 17:50, Viktor Ashirov wrote: > > Hi, > > On Fri, Oct 4, 2019 at 7:25 AM William Brown wrote: > pytestmark = pytest.mark.tier1 > > I saw this in a test and curious as to it's function? I'm sure it has one :) > See https://pagure.io/389-ds-base/issue/50353 for more context. > We use tier0 and tier1 for downstream component gating (similar to > https://docs.fedoraproject.org/en-US/ci/gating/ but for RHEL). > I.e. 389-ds-base build should pass tier0 and tier1 tests 100% in order to > land in the compose. That's a really good explanation, thanks! So when I'm adding new tests I should choose the tier based on those categories I expect? > HTH > > -- > Sincerely, > > William > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > > > -- > Viktor > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] What is the tier1 pytest mark for?
pytestmark = pytest.mark.tier1 I saw this in a test and curious as to it's function? I'm sure it has one :) -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please Review: 50633 Cargo vendor support
https://pagure.io/389-ds-base/pull-request/50633 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Bundle/Vendor process for npm/js
> On 3 Oct 2019, at 07:34, Simon Pichugin wrote: > > On Wed, Oct 02, 2019 at 01:20:24PM +1000, William Brown wrote: >> Hey there, >> >> I'm looking at adding the steps for offline builds so that we can start to >> use rust. I know that you have some steps for preparing the tar for cockpit >> builds with npm/js deps. >> >> How are you doing this? I was hoping we could share some kind of prepare for >> offline or pre-build script or similar that could trigger the cargo >> vendoring process > Hi William, > so we have two stages here, right. > > 1. To begin with, we install the latest NPM packages, build the cockpit and > pack it > to the tarball. At that stage, you should have the internet for NPM access. > https://pagure.io/389-ds-base/raw/master/f/rpm.mk > tarballs or dist-bz2 (it is used for Koji builds) > > 2. Then, we use the tarball to build RPM (or just to create SRPM). It is > an offline part. Basically, we freeze the NPM versions and our WebUI > state and use it for building purposes. > During the 'make install', it copies the directory to Cockpit > system dir. > https://pagure.io/389-ds-base/blob/master/f/Makefile.am#_744 > > P.S. probably, you know, but just in case - > Fedora release process can be checked here: > http://www.port389.org/docs/389ds/howto/howto-fedora-release-process.html > > Is it what you want? Yep, perfect! So I think that if I add the cargo vendor steps to step 1, and then make the makefile use the "offline" mode for 2 it should be all good then? Similar to npm for offline builds. Thanks! > > Regards, > Simon > >> >> Thanks, >> >> -- >> Sincerely, >> >> William >> ___ >> 389-devel mailing list -- 389-devel@lists.fedoraproject.org >> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Bundle/Vendor process for npm/js
Hey there, I'm looking at adding the steps for offline builds so that we can start to use rust. I know that you have some steps for preparing the tar for cockpit builds with npm/js deps. How are you doing this? I was hoping we could share some kind of prepare for offline or pre-build script or similar that could trigger the cargo vendoring process Thanks, -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: token auth design
https://github.com/marcus2376/389wiki/pull/17 https://pagure.io/389-ds-base/issue/48707 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] New Repo! 389-ds-portal
Hi everyone, I've added a new repo in our project namespace - 389-ds-portal. This is a lib389 + flask webportal for self-service of accounts including password changes. It's still not complete, and there is a bit to go before it's configurable/production ready, but I think it could be workable in the next few months. Check it out! https://pagure.io/389-ds-portal -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please Review 50632, ensure_attr_state
https://pagure.io/389-ds-base/pull-request/50632 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Regular team meetings
Hi everyone, We talked about this twice this year about having common meetings that we can all attend - it's time we took some steps to making this happen! I think it's possible we could all actually do this at the same time - we discussed a possible time on tuesday as: 11am UTC+0, 1pm Brno (UTC+2), 7am Raleigh (UTC -4), 9pm Brisbane (UTC+10). What do we all think? We could schedule the next one on October the 8th? We can then do this monthly from then on. Something to keep in mind is that I live in a timezone without DST so the meeting for me may shift by +-1 hr depending on time of year, which I'm okay with. A future consideration is what we do if community members wish to participate also, so that's something else to keep in mind for future is how we handle that onboarding. BEGIN:VCALENDAR CALSCALE:GREGORIAN VERSION:2.0 X-WR-CALNAME:389 Devel Team Meeting METHOD:PUBLISH PRODID:-//Apple Inc.//Mac OS X 10.14.6//EN BEGIN:VEVENT CREATED:20190926T010544Z UID:055EFE9E-F547-448F-A4D7-474744AD61C3 DTEND:20191008T12Z TRANSP:OPAQUE X-APPLE-TRAVEL-ADVISORY-BEHAVIOR:AUTOMATIC SUMMARY:389 Devel Team Meeting LAST-MODIFIED:20190926T010616Z DTSTAMP:20190926T010620Z DTSTART:20191008T11Z SEQUENCE:0 BEGIN:VALARM X-WR-ALARMUID:DE6B006F-8E2C-470B-9F62-F907576C7152 UID:DE6B006F-8E2C-470B-9F62-F907576C7152 TRIGGER;VALUE=DATE-TIME:19760401T005545Z ACTION:NONE END:VALARM BEGIN:VALARM X-WR-ALARMUID:5783A894-520F-467F-9D30-DDBE7D6AEA02 UID:5783A894-520F-467F-9D30-DDBE7D6AEA02 TRIGGER:-P1D ATTACH;VALUE=URI:Chord ACTION:AUDIO END:VALARM END:VEVENT END:VCALENDAR -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please Review 50617-cargo lock
https://pagure.io/389-ds-base/pull-request/50626 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: 50595 remove syslog.targ
https://pagure.io/389-ds-base/pull-request/50625 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please Review 50624 extend modify commands
https://pagure.io/389-ds-base/pull-request/50624 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review 50622 ds_selinux_enabled
https://pagure.io/389-ds-base/pull-request/50623 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Please review: 50584, 49212 - docker healthcheck and configuration
Reminder to review this please! > On 5 Sep 2019, at 08:55, William Brown wrote: > > https://pagure.io/389-ds-base/issue/50584 > > https://pagure.io/389-ds-base/issue/49212 > > https://pagure.io/389-ds-base/pull-request/50585 > > Thanks! > > -- > Sincerely, > > William > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Review reminder :)
https://pagure.io/389-ds-base/pull-request/50571 https://pagure.io/389-ds-base/pull-request/50577 Thanks! -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Docker releases of 389
> On 3 Sep 2019, at 00:19, Matus Honek wrote: > > On Mon, Sep 2, 2019 at 3:43 AM William Brown wrote: >> >> Hey all, >> >> RC images have been pushed: https://hub.docker.com/r/389ds/dirsrv > > First, I'm not with processes on Docker Hub, so please bear with me... > :) I don't see a Dockerfile tab on the page -- is this because of how > the OSB container pipeline works? (hint: I usually don't trust the > builds on the Docker Hub for which there is not a Dockerfile) Yes, that's correct. It's because it's built on OBS so I just push the image directly. It's useful because it means when I upload new source, it automatically builds the container too. Saying this, the Dockerfile as used in OBS could be used on dockerhub if there are transparency concerns? Alternately, I could link to the source repo? Thoughts? > >> >> I'll try to accomodate as much feedback as possible, aiming for 1.4.2 to be >> out first fully supported release. Sounds okay? >> >>> On 29 Aug 2019, at 12:55, William Brown wrote: >>> >>> Hi all, >>> >>> I've been wanting this for a long time and I think we're almost there - I'd >>> like us to start offering docker images of 389 Directory Server as part of >>> our upstream release process. >>> >>> I have discussed this with Mark already and we both think we are reasonably >>> happy with the idea and the state we're in. >>> >>> -- The Technical Process and Details: >>> >>> As there is significant interest within SUSE for containerised 389, as well >>> as open tooling as part of build.opensuse.org, the current proposed process >>> is: >>> >>> Source Release -> OBS:network:ldap -> OBS:389-ds-container -> docker hub >>> >>> Because this pipeline is automated between source to the container, and >>> this is already part of my responsibility as the SUSE package manager, it's >>> a small amount of effort to then mirror the container to docker hub. >>> >>> I have already established an organisation on docker hub, and will be >>> giving Mark access to be able to manage this as well. >>> >>> https://cloud.docker.com/u/389ds/repository/list >>> >>> Initially we'll name the image as: >>> >>> 389ds/dirsrv:1.4.rc >>> >>> Once we are happy with the process, and have received some community >>> feedback we'll move to: >>> >>> 389ds/dirsrv:1.4.1 >>> 389ds/dirsrv:1.4.2 >>> 389ds/dirsrv:1.4 # points to latest 1.4.X >>> 389ds/dirsrv:latest # points to newest version >>> >>> We would of course encourage people to use the "latest" tag. >>> >>> -- Support >>> >>> During the rc phase we would announce that the container support is "best >>> effort" and we'll obviously work to resolve issues, but it's not suitable >>> for production work loads. >>> >>> Once we are happy with this for a few releases, we'll move to the same >>> support levels as our normal releases - best effort community, and patch >>> backporting as we normally do. For official support, contact a vendor like >>> SUSE or Red Hat. >>> >>> -- Future / Downstreams >>> >>> The design of the container integration is such that we should be able to >>> easily swap the suse/fedora/rhel as the base image, so downstreams should >>> be able to easily adopt the dscontainer tool and have customers pivot from >>> the upstream image to their vendor image quite easily. >>> >>> A future container option is to supply a tools container that has a shell + >>> ds* tools, so that you can have a work flow such as: >>> >>> docker run -v 389ds:/data 389ds/dirsrv:latest >>> docker run -i -t -v 389ds:/data 389ds/tools:latest >>> # shell inside of the tools container >>> # dsconf >>> >>> -- Testing today: >>> >>> To test the image as it exists today: >>> >>> docker pull >>> registry.opensuse.org/home/firstyear/containers/389-ds-container:latest >>> >>> >>> Thoughts and feedback? >>> >>> If there are no objects, I'll push the rc to docker hub early next week >>> (2nd or 3rd of Sep) >>> >>> -- >>> Sincerely, >>> >>> William >>> ___ >>> 389-devel mailing list -- 389-devel@lists.fedoraproject.org &
[389-devel] Re: Docker releases of 389
Hey all, RC images have been pushed: https://hub.docker.com/r/389ds/dirsrv I'll try to accomodate as much feedback as possible, aiming for 1.4.2 to be out first fully supported release. Sounds okay? > On 29 Aug 2019, at 12:55, William Brown wrote: > > Hi all, > > I've been wanting this for a long time and I think we're almost there - I'd > like us to start offering docker images of 389 Directory Server as part of > our upstream release process. > > I have discussed this with Mark already and we both think we are reasonably > happy with the idea and the state we're in. > > -- The Technical Process and Details: > > As there is significant interest within SUSE for containerised 389, as well > as open tooling as part of build.opensuse.org, the current proposed process > is: > > Source Release -> OBS:network:ldap -> OBS:389-ds-container -> docker hub > > Because this pipeline is automated between source to the container, and this > is already part of my responsibility as the SUSE package manager, it's a > small amount of effort to then mirror the container to docker hub. > > I have already established an organisation on docker hub, and will be giving > Mark access to be able to manage this as well. > > https://cloud.docker.com/u/389ds/repository/list > > Initially we'll name the image as: > > 389ds/dirsrv:1.4.rc > > Once we are happy with the process, and have received some community feedback > we'll move to: > > 389ds/dirsrv:1.4.1 > 389ds/dirsrv:1.4.2 > 389ds/dirsrv:1.4 # points to latest 1.4.X > 389ds/dirsrv:latest # points to newest version > > We would of course encourage people to use the "latest" tag. > > -- Support > > During the rc phase we would announce that the container support is "best > effort" and we'll obviously work to resolve issues, but it's not suitable for > production work loads. > > Once we are happy with this for a few releases, we'll move to the same > support levels as our normal releases - best effort community, and patch > backporting as we normally do. For official support, contact a vendor like > SUSE or Red Hat. > > -- Future / Downstreams > > The design of the container integration is such that we should be able to > easily swap the suse/fedora/rhel as the base image, so downstreams should be > able to easily adopt the dscontainer tool and have customers pivot from the > upstream image to their vendor image quite easily. > > A future container option is to supply a tools container that has a shell + > ds* tools, so that you can have a work flow such as: > > docker run -v 389ds:/data 389ds/dirsrv:latest > docker run -i -t -v 389ds:/data 389ds/tools:latest > # shell inside of the tools container > # dsconf > > -- Testing today: > > To test the image as it exists today: > > docker pull > registry.opensuse.org/home/firstyear/containers/389-ds-container:latest > > > Thoughts and feedback? > > If there are no objects, I'll push the rc to docker hub early next week (2nd > or 3rd of Sep) > > -- > Sincerely, > > William > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Please review: PRs 50551, 50570, 50573, 50579
Will do first thing next week :) > On 30 Aug 2019, at 21:14, Simon Pichugin wrote: > > Hi team, > there are few PRs on review from me. Please, check... > > https://pagure.io/389-ds-base/pull-request/50551 > https://pagure.io/389-ds-base/pull-request/50570 > https://pagure.io/389-ds-base/pull-request/50573 > https://pagure.io/389-ds-base/pull-request/50579 > > Thanks, > Simon > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review 50576 map uid/gid of proc to dm over ldapi
https://pagure.io/389-ds-base/pull-request/50577 https://pagure.io/389-ds-base/issue/50576 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Docker releases of 389
Hi all, I've been wanting this for a long time and I think we're almost there - I'd like us to start offering docker images of 389 Directory Server as part of our upstream release process. I have discussed this with Mark already and we both think we are reasonably happy with the idea and the state we're in. -- The Technical Process and Details: As there is significant interest within SUSE for containerised 389, as well as open tooling as part of build.opensuse.org, the current proposed process is: Source Release -> OBS:network:ldap -> OBS:389-ds-container -> docker hub Because this pipeline is automated between source to the container, and this is already part of my responsibility as the SUSE package manager, it's a small amount of effort to then mirror the container to docker hub. I have already established an organisation on docker hub, and will be giving Mark access to be able to manage this as well. https://cloud.docker.com/u/389ds/repository/list Initially we'll name the image as: 389ds/dirsrv:1.4.rc Once we are happy with the process, and have received some community feedback we'll move to: 389ds/dirsrv:1.4.1 389ds/dirsrv:1.4.2 389ds/dirsrv:1.4 # points to latest 1.4.X 389ds/dirsrv:latest # points to newest version We would of course encourage people to use the "latest" tag. -- Support During the rc phase we would announce that the container support is "best effort" and we'll obviously work to resolve issues, but it's not suitable for production work loads. Once we are happy with this for a few releases, we'll move to the same support levels as our normal releases - best effort community, and patch backporting as we normally do. For official support, contact a vendor like SUSE or Red Hat. -- Future / Downstreams The design of the container integration is such that we should be able to easily swap the suse/fedora/rhel as the base image, so downstreams should be able to easily adopt the dscontainer tool and have customers pivot from the upstream image to their vendor image quite easily. A future container option is to supply a tools container that has a shell + ds* tools, so that you can have a work flow such as: docker run -v 389ds:/data 389ds/dirsrv:latest docker run -i -t -v 389ds:/data 389ds/tools:latest # shell inside of the tools container # dsconf -- Testing today: To test the image as it exists today: docker pull registry.opensuse.org/home/firstyear/containers/389-ds-container:latest Thoughts and feedback? If there are no objects, I'll push the rc to docker hub early next week (2nd or 3rd of Sep) -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review 50567 and 50568
https://pagure.io/389-ds-base/pull-request/50571 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Good feedback
Hi everyone, Just thought I'd let you know that I received a few pieces of positive feedback about dsconf recently to my private mail address. Great work to everyone on the team for all your work to make lib389 and the new command line tools a reality. People are really starting to notice, and you have all done wonderful work to help bring this to life. Thanks everyone! -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: 389 DS nightly 2019-08-27 - 0% PASS
https://pagure.io/389-ds-base/issue/49324 I think that an issue with this patch has caused import to stop working. I've contact Mark about it, and I think he'll be able to fix it pretty easily, so no need to revert at this stage. :) > On 27 Aug 2019, at 09:41, vashi...@redhat.com wrote: > > https://fedorapeople.org/groups/389ds/ci/nightly/2019/08/27/report-389-ds-base-1.4.1.6-20190826gitbfdb226.fc30.x86_64.html > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review 50564 - fix rust and docker build issues
https://pagure.io/389-ds-base/pull-request/50564 -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Please review: Issue 50206 - Account IDM CLI - WIP
Sure thing, I'll look soon :) > On 16 Aug 2019, at 11:55, Simon Pichugin wrote: > > Hi team, > I have a big chunk of work ready for review. > The main part is done and I just need to polish it a bit more and add > more tests. > > In a mean while... > > William, could you please take a look? > The lib389 part for Accounts is much bigger now and has much more to > offer. > > Thanks, > Simon — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Do we still need sslVersionMax/sslVersionMin?
> On 17 Jul 2019, at 22:36, Mark Reynolds wrote: > > > On 7/17/19 3:01 AM, Matus Honek wrote: >> I think we cannot remove it. Setting the MIN version is a workaround >> for *old clients* not even supporting current NSS' default min. >> Setting up MAX version is a workaround for *broken clients* thinking >> they can support something they announced but for some reason fail to >> work with such a version. I believe most of deployments have some >> really legacy software of which not a small amount behaves weirdly >> enough these two options save lives; I have seen these issues several >> times. > Did you see anyone still using SSL3? The min is good to allow a sysadmin to clamp the min version up to something like TLS1.2 rather than TLS 1.0 and 1.1 which both have known issues. So Ithink we should leave this, but default to the NSS system wide crypto, and document and advise to use NSS systemd wide crypto policy instead. >> >> On Tue, Jul 16, 2019 at 10:24 PM Mark Reynolds wrote: >>> So some time ago when the poodlebleed vulnerability came out in SSL3 we >>> added a way to set the minimum and maximum SSL/TLS versions the server >>> would accept (e.g. TLS1.1 <--> TLS1.2).Current versions of NSS >>> already use this range by default. I would like to remove/deprecate the >>> sslVersionMin/Max and just use what NSS uses by default (which should be >>> the system wide crypto policy). >>> >>> Is anyone actually using sslVersionMin/Max? Do we really have a need >>> for it anymore? >>> >>> -- >>> >>> 389 Directory Server Development Team >>> ___ >>> 389-devel mailing list -- 389-devel@lists.fedoraproject.org >>> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org >> >> > -- > > 389 Directory Server Development Team > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: 50493 improve conntable allocation process
https://pagure.io/389-ds-base/pull-request/50493 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review (somewhat urgent) 50492 connection allocation failure.
https://pagure.io/389-ds-base/pull-request/50492 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: 50485 nspr to pthread mutex for c_mutex
https://pagure.io/389-ds-base/pull-request/50485 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Using the cli tools today ...
Hi all, I've been re-deploying my home ldap server with the cli today and ... wow. DS 1.4.x compared to 1.3.x to administer is like a new product. It's unbelievable how easy it is to do ... everything. New backends, exports, imports, plugins ... you name it. It's really looking like "the ldap server I always wish I had" when I was an admin. Great work to everyone on this, I know it's been ages to get here, but it's just wonderful to see it in action. — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: 50484 docker release improvements
https://pagure.io/389-ds-base/pull-request/50484 Found after doing production deployment of 389 in container. Likely there are more to come as I add replicas! -- Sincerely, William ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] LDAPCon 2019 CFP
Hi all, The LDAP Con 2019 CFP is now open - this would be a great chance for our developers to speak about what we have done for accessibility and other improvements in the last two years. For those in the EU this should be pretty easy to get to. I'm currently asking a member of the program committee if they'll accept recorded submissions from developers who are long distance and unable to travel (AUS/US). I also would love if the conference recorded talks so that the world can see what is on offer in our community (I'll ask them about that separately). I think we have a lot we could present here especially around: * lib389 and the ds* tools * cockpit * docker and containerisation Good luck in your submissions! https://cfp.ldapcon.org/ldapcon2019/cfp — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Discussion - 389ds docker images
Hi all, I'd like us to start offering 389ds images on docker hub - and ideally, I'd like to automate this process as much as possible. As I've historically been the person who has been pushing this the most, I'm happy to maintain and coordinate that these are updated and released when the project updates it's branches. Due to the way docker works, we only need to make sure that what we have in the volume /data is readable and operational, so I think there is no issue with what base image we choose for the container image, because we can always change it later without affecting the user as /data is always the way we'll access and present the container state. For me, I'd probably setup an automated build from git master on a :devel tag or similar in docker hub, and :latest would be the latest major series release (1.4), probably shortened to :4, with the point releases in there too, IE :4.1, :4.2. patch releases would just be updates to those streams. If possible I'd like to automate those from our git branches. The best way would probably be to make a git mirror to github/gitlab which can then have triggers to call docker hub to do necessary updates. A major question is how we actually do the build - do we build from the source tree, or do we build from a rolling repo like the network:ldap repo in opensuse (which is always following upstream releases). I think building from source is the most flexible, as it gives us the closest relationship to git and our upstream release process. Thoughts? — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Where am I?
Hi all, I've been very unwell since my return to Australia, so if there is anything urgent you need to me took at please contact me directly to notify me - I'm not looking at pagure this week. Sorry about this :( — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Please review: waitpid fix 50458
https://pagure.io/389-ds-base/pull-request/50458 — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Environment variables in systemd
> On 13 Jun 2019, at 14:17, William Brown wrote: > > Hi there, > > There have been a few changes to systemd files recently so I wanted to check > about how to correctly supply environment variables to the server. For > example, KRB5_KTNAME. Are we doing this still by EnvironmentFile? Or by > another method? ping anyone? > > > — > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server > SUSE Labs > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Issue with building RPMs with ASAN_ON
> On 18 Jun 2019, at 17:45, Viktor Ashirov wrote:
>
>
>
> On Tue, Jun 18, 2019 at 7:54 AM Viktor Ashirov wrote:
> On Tue, Jun 18, 2019 at 1:30 AM Simon Pichugin wrote:
> >
> > Hi team,
> > I'm in the process of creating a Vagrant file which is close to the
> > customer's ENV.
> > It is heavilly based on Viktor's beaker task.
> > I use it for building and testing my code. And it is pretty important to
> > build with ASAN.
> >
> > Currently, what I do is:
> > 1. Set 'ASAN_ON = 1' in rpm.mk
> > 2. Run `make -f rpm.mk srpms` target
> > 3. Build the RPM using `mock -q my_generated.srpm`
> > 4. Install it
> >
> > Then I've tried running `dscreate` manually or running tests with py.test.
> > Every time I have the same error here:
> > /run/dirsrv/ns-slapd-standalone1.asan.X
> >
> > ==22487==LeakSanitizer has encountered a fatal error.
> > ==22487==HINT: For debugging, try setting environment variable
> > LSAN_OPTIONS=verbosity=1:log_threads=1
> > ==22487==HINT: LeakSanitizer does not work under ptrace (strace, gdb,
> > etc)
> Ludwig also recently had this issue. Looks like you're hitting this
> bug: https://github.com/google/sanitizers/issues/723
> We're using posix_memalign() in a few places and LeakSanitizier can't handle
> it.
> So, the issue Simon was seeing is not related to the issue above.
> Turns out, it's just SELinux :)
>
>
>
>
> time->Tue Jun 18 11:27:24 2019
>
>
> type=AVC msg=audit(1560871644.883:596): avc: denied { ptrace } for
> pid=3632 comm="ns-slapd" scontext=system_u:system_r:dirsrv_t:s0
> tcontext=system_u:system_r:dirsrv_t:s0
> tclass=process permissive=0
>
> [root@server ds]# ausearch -m AVC | audit2allow
>
>
>
>
>
>
>
>
> #= dirsrv_t ==
>
>
> allow dirsrv_t self:process ptrace;
Heh, selinux strikes again!
Were you running as a user, not as root?
>
>
> There is a workaround in the last comment. I did the builds for gcc8
> and gcc9 in copr, both internal and fedora one, but they failed (not
> related to the patch).
> So I did a local build with the patch and it worked like a charm. I
> will share the links to the rpms for you to try.
>
> Perhaps we should review our usage of posix_memalign() or convince the
> upstream to implement a proper fix for this.
> >
> > I've tried setting `export LSAN_OPTIONS=verbosity=1:log_threads=1` and run
> > once again.
> > Same issue.
> >
> > Did anybody encountered the issue? Maybe, Viktor or William, could you
> > please check?
> > I'm putting the Vagrantfile to the attachments so you can reproduce.
> > Just run: `ASAN=on vagrant up` from the directory with Vagrantfile.
> >
> > William, I think, libvirt is present on SUSE so you should have no issues
> > with this too...
> >
> > Thanks,
> > Simon
> > ___
> > 389-devel mailing list -- 389-devel@lists.fedoraproject.org
> > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
>
>
>
> --
> Viktor
>
>
> --
> Viktor
> _______
> 389-devel mailing list -- 389-devel@lists.fedoraproject.org
> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org
[389-devel] Re: Issue with building RPMs with ASAN_ON
> On 18 Jun 2019, at 14:38, Viktor Ashirov wrote: > > > > On Tue, Jun 18, 2019 at 2:33 PM William Brown wrote: > > > > On 18 Jun 2019, at 14:19, Viktor Ashirov wrote: > > > > > > > > On Tue, Jun 18, 2019 at 2:10 PM William Brown wrote: > > > > > > > > Memalign is pretty important - the short version is "we can not remove > > > it". > > > I didn't say "remove", I said "review". > > > > > > There are some structures in the code that rely on this for performance > > > to guarantee that they memory is aligned to a page boundary, or cache > > > line boundary. In some cases it's required to allow the atomics to work > > > in nunc-stans (well, lfds, but the value of that today is questionable > > > when the rust version is possibly safer and faster). > > > Since you're the expert in this area, maybe you can leave a comment in > > > the issue linked above with the justification for upstream to reconsider? > > > > You mean upstream LSAN/ASAN in this case, yes? > > Yes, this https://github.com/google/sanitizers/issues/723 > > Reading the report, that is only about mprotect of alligned memory, not of > memory that is just aligned. Perhaps our issue is different? > > Maybe. I only tried the workaround that worked for me. At least I got some > sensible results, not just LeakSanitizer failure. Where did you find the work around? Maybe I'm blind, but I do not see it on this issue > > > > > > > — > > Sincerely, > > > > William Brown > > > > Senior Software Engineer, 389 Directory Server > > SUSE Labs > > ___ > > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > > > > > > -- > > Viktor > > ___ > > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > > — > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server > SUSE Labs > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > > > -- > Viktor > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Issue with building RPMs with ASAN_ON
> On 18 Jun 2019, at 14:19, Viktor Ashirov wrote: > > > > On Tue, Jun 18, 2019 at 2:10 PM William Brown wrote: > > > > > Memalign is pretty important - the short version is "we can not remove it". > > I didn't say "remove", I said "review". > > > > There are some structures in the code that rely on this for performance to > > guarantee that they memory is aligned to a page boundary, or cache line > > boundary. In some cases it's required to allow the atomics to work in > > nunc-stans (well, lfds, but the value of that today is questionable when > > the rust version is possibly safer and faster). > > Since you're the expert in this area, maybe you can leave a comment in the > > issue linked above with the justification for upstream to reconsider? > > You mean upstream LSAN/ASAN in this case, yes? > Yes, this https://github.com/google/sanitizers/issues/723 Reading the report, that is only about mprotect of alligned memory, not of memory that is just aligned. Perhaps our issue is different? > > > — > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server > SUSE Labs > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > > > -- > Viktor > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Issue with building RPMs with ASAN_ON
> > Memalign is pretty important - the short version is "we can not remove it". > I didn't say "remove", I said "review". > > There are some structures in the code that rely on this for performance to > guarantee that they memory is aligned to a page boundary, or cache line > boundary. In some cases it's required to allow the atomics to work in > nunc-stans (well, lfds, but the value of that today is questionable when the > rust version is possibly safer and faster). > Since you're the expert in this area, maybe you can leave a comment in the > issue linked above with the justification for upstream to reconsider? You mean upstream LSAN/ASAN in this case, yes? — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Issue with building RPMs with ASAN_ON
> On 18 Jun 2019, at 07:54, Viktor Ashirov wrote: > > On Tue, Jun 18, 2019 at 1:30 AM Simon Pichugin wrote: >> >> Hi team, >> I'm in the process of creating a Vagrant file which is close to the >> customer's ENV. >> It is heavilly based on Viktor's beaker task. >> I use it for building and testing my code. And it is pretty important to >> build with ASAN. >> >> Currently, what I do is: >> 1. Set 'ASAN_ON = 1' in rpm.mk >> 2. Run `make -f rpm.mk srpms` target >> 3. Build the RPM using `mock -q my_generated.srpm` >> 4. Install it >> >> Then I've tried running `dscreate` manually or running tests with py.test. >> Every time I have the same error here: >> /run/dirsrv/ns-slapd-standalone1.asan.X >> >>==22487==LeakSanitizer has encountered a fatal error. >>==22487==HINT: For debugging, try setting environment variable >> LSAN_OPTIONS=verbosity=1:log_threads=1 >>==22487==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc) > Ludwig also recently had this issue. Looks like you're hitting this > bug: https://github.com/google/sanitizers/issues/723 > We're using posix_memalign() in a few places and LeakSanitizier can't handle > it. > There is a workaround in the last comment. I did the builds for gcc8 > and gcc9 in copr, both internal and fedora one, but they failed (not > related to the patch). > So I did a local build with the patch and it worked like a charm. I > will share the links to the rpms for you to try. Which patch? > > Perhaps we should review our usage of posix_memalign() or convince the > upstream to implement a proper fix for this. Memalign is pretty important - the short version is "we can not remove it". There are some structures in the code that rely on this for performance to guarantee that they memory is aligned to a page boundary, or cache line boundary. In some cases it's required to allow the atomics to work in nunc-stans (well, lfds, but the value of that today is questionable when the rust version is possibly safer and faster). >> >> I've tried setting `export LSAN_OPTIONS=verbosity=1:log_threads=1` and run >> once again. >> Same issue. >> >> Did anybody encountered the issue? Maybe, Viktor or William, could you >> please check? >> I'm putting the Vagrantfile to the attachments so you can reproduce. >> Just run: `ASAN=on vagrant up` from the directory with Vagrantfile. >> >> William, I think, libvirt is present on SUSE so you should have no issues >> with this too... It is, but I run on a mac so ... my setup is fun fun fun :) I would normally run this on my home lab, but I'm a few thousand km's away >> >> Thanks, >> Simon >> ___ >> 389-devel mailing list -- 389-devel@lists.fedoraproject.org >> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > > > > -- > Viktor > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: How to create a user with certificate with lib389
> On 14 Jun 2019, at 11:25, Viktor Ashirov wrote:
>
> On Fri, Jun 14, 2019 at 9:28 AM William Brown wrote:
>>
>>
>>
>>> On 13 Jun 2019, at 16:09, Viktor Ashirov wrote:
>>>
>>> On Thu, Jun 13, 2019 at 3:26 PM William Brown wrote:
>>>>
>>>> Is the test case *just* testing if binary searching of attributes works?
>>> The test was to check if we can query the server for
>>> userCertificate=, where is a string representation of a
>>> base64 encoded x509 certificate. The original test was also passing
>>> binary representation (usercertificate;binary=...) to ldapsearch (to
>>> see if it translates correctly to base64).
>>
>> Thanks for telling me what the test is meant to do! This is what I wanted to
>> know from the start ...
> Thank you for your patience. I'm sorry I missed this email thread, I'd
> replied sooner to reduce the confusion.
> To give some context: this test case also was testing client tools
> (mozldap) to support binary filters. But this test case didn't age
> well, so your approach below should be sufficient.
> Thanks!
We there still may be problems, but I think let's start simple and
build up, and see if there is something that needs to be fixed or not :)
>>
>> So the base64 is only if the attribute is "longer" than a certain amount the
>> ldapclient tools base64 it for viewing - the server actually doesn't care or
>> know that it's going on at all, so really, this is a test if binary matching
>> works.
>>
>> You can thus, setup a simpler test by setting
>>
>> with open('/tmp/test') as f:
>>data = f.readlines() # or read(), I can't remember what does it all
>> without newlines)
>> Account.set('usercertificate', data)
>> Accounts.filter('userCert=%b' % data)
>>
>> So then I'd tweak if it's %b or %s, I'd probably also to see what works and
>> prevents python leaking state or formatting.
>>
>> Then work up to a full certificate.
>>
>> If you have a failuing example, please send me the access log and -v
>> (DEBUGGING=True) lib389 output so I can help
>>
>> Thanks,
>>
>>
>>>
>>> --
>>> Viktor
>>> ___
>>> 389-devel mailing list -- 389-devel@lists.fedoraproject.org
>>> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
>>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives:
>>> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
>>
>> —
>> Sincerely,
>>
>> William Brown
>>
>> Senior Software Engineer, 389 Directory Server
>> SUSE Labs
>> ___
>> 389-devel mailing list -- 389-devel@lists.fedoraproject.org
>> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
>
>
>
> --
> Viktor
> ___
> 389-devel mailing list -- 389-devel@lists.fedoraproject.org
> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
—
Sincerely,
William Brown
Senior Software Engineer, 389 Directory Server
SUSE Labs
___
389-devel mailing list -- 389-devel@lists.fedoraproject.org
To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: please review: Replication Status Message Improvements
> On 13 Jun 2019, at 17:00, Mark Reynolds wrote: > > > On 6/13/19 4:02 AM, William Brown wrote: >> >>> On 12 Jun 2019, at 17:36, Mark Reynolds wrote: >>> >>> http://www.port389.org/docs/389ds/design/repl-agmt-status-design.html >> >> Looks great! >> >> Instead of "Success" we could use "Healthy" because replication isn't a >> success/fail, it's a longterm "good/bad" IE healthy/failing state. So >> perhaps instead of success/warning/error, we have >> "healthy/delayed/warning/error" instead? Additionally, on warning + error, >> we log those to error log by default so that an admin can "see" past errors, >> and when they "resolve" a resolution is also posted to the error log too? > Really there are only three states: working, delayed, and broken. What about: > Good, Warning, Error? FYI if this does not get pushed today we will a very > important deadline (in fact it will mean all the bugs slated for the next > release will be missed) So I don't want to keep debating "names" for > much longer... Look, anything that isn't so ... I dunno. "Single action" focused but conveys that the service is operating as expected. I think good may not be the right term either from an admin point of view. So Healthy/Warning, Operational/Delayed, Green/Amber/Red also works from a monitoring case. >> >> Date is json should be ISO 8601 format instead? >> >> 2019-06-13T07:53:20+00:00 > I will look into this, but we are running our of time as I mentioned above. > Maybe I will just drop the date for now... Fair, we can add this later then - after all it's json so we can change it without breaking things. Just as a mild concern, I know that there are RH deadlines to meet, but I don't like that leaking into rushing work upstream :( >> >> I hope this helps! >> >> >>> ___ >>> 389-devel mailing list -- 389-devel@lists.fedoraproject.org >>> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org >> — >> Sincerely, >> >> William Brown >> >> Senior Software Engineer, 389 Directory Server >> SUSE Labs >> ___ >> 389-devel mailing list -- 389-devel@lists.fedoraproject.org >> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
