Hi,
I was away and am late in the discussion, maybe too late.
In my understanding what you mean by "generic" is that for a new
rewriter you do not need a plugin, but to provide some rewrite functions
and specify them in a rewriters config entry. But there is still the
need to write rewriter functions, compile and deploy them, and instead
of plugins you now have a new interface of "filterRewriter" and
"returendAttrRewriter functions - so far not documented anywhere.
Under generic rewriter I would understand an approach where you do not
need to provide own functions, but have a rewriter plugin, which does
rewriting based on rules in rewrite config entries, eg in which subtree,
for which entries (filter to select), how to map a saerch filter, how to
rename attrs on return,....
Best regards,
Ludwig
On 03/19/2020 01:09 AM, William Brown wrote:
On 19 Mar 2020, at 04:08, thierry bordaz <[email protected]> wrote:
On 3/18/20 1:51 AM, William Brown wrote:
On 18 Mar 2020, at 04:08, thierry bordaz <[email protected]> wrote:
Hi William,
I updated the design according to our offline exchange
Thanks Thierry, I appreciate the conversation and the updates to the document:
it made clear there were extra details up in your brain but not in words yet :)
it's always hard to remember all the details as we write things, so thanks for
the discussion. Like you said, it's always good to have a team who is really
invested and cares about the work we do!
Your design for the core server version looks much better! Thank you. I still
think there are some missing points. The reason to have a libpath rather than
inbuild is to avoid a potential linking to sssd/samba. I think also that the
problem space of the global catalog here needs to be looked at too. This
feature is not in isolation, it's really a part of that.
Okay, I will work on a new PR making core server able to retrieve/registers
rewriters.
I think the "need" to improve the usability of rewriters is not specific to
global catalog. Global Catalog is just an opportunity to implement it. I think parts of
slapi-nis, integration of vsphere, GC (and likely others) are also use case for
rewriters. They were implemented in different ways because rewriters were not easy to use
or simply not known.
Yes, that's perfectly reasonable, and shouldn't stop your idea from being
created - what's concerning me is that without a full picture you don't know
how far to take these rewriters or what direction, or what might be needed.
This means we have a whole set of deployment cases to look at.
So the deployment will look like:
IPA DS --> IPA GC
So an ipaAccount from the IPA DS instance will be "copied and transformed" into
the IPA GC. This process is as yet undefined (it sounds like it may be offline or
something else ...). We are simply not dealing with one instance now, but an out-of-band
replication and transformation process. It's unclear whether the data transform is during
this loading process, or in the IPA GC somehow.
From what I understand, it sounds like a method to take an ipaAccount and
transform it to an AD GC account stub. Then inside of that IPA GC there are
some virtual attributes you wish to add like objectSid binary vs string
representations, objectCategory, maybe others.
So from our discussion, we have currently focused on "how do we transform entries
within a single directory server". But that's not the problem here. We are saying:
"We take an entry from IPA DS, transform it to an IPA GC stub entry, and then apply a set of
further "in memory" transformations"
One of the biggest issue with GC is schema. IPA DS and IPA GC have not
compatible schema. They can not be in the same replication topology.
So provisioning of IPA GC requires transformations rules to present an other
"view" of IPA DS data. Those transformations will be on the write path (i.e.
stored in DB/indexed). This transformation work is almost done and is completely
independent of 389-ds.
All of this is "write" path: provisioning (online or offline) and
transformation.
The problem for IPA GC is now on the "read" path. AD clients are use to smart
shortcuts/control that are supported by IPA GC.
This is the IPA GC instance that will register the rewriters to act as GC does.
Yep, I'm aware :)
If that's the process, why not do all the transforms as required in the DS -> GC
load process? You raised a critically key point - we have a concern about the write
path as the transform point due to IO or time to do the transform, but it sounds like
you have to do this anyway as an element of the DS -> GC process.
Some of the transformation rules, on the write path, are quite complex. Looking
at slapi-nis config entries gives an idea what is needed. In addition to those
transformations, DS to GC online provisioning is not simple at all. Relying on
sync-repl, you then need to transform a received entry into an update. At the
moment it is an offline provisioning via transformation and import (much
simpler).
To be honest I am afraid that the transform rules will result in rewriting
slapi-nis.
*puts finger on nose* I do not want to be near that toxic rewrite at all.
I think everytime I have spoken to you about this, I have kept learning more and more
about this, and the more I see, I have many concerns about this feature. I think we do
not have the full picture. You have admitted that you don't know the full extend or ideas
here. There is clearly a communication break down here to our team from the IPA project,
and they aren't telling us what they want. It sounds like they are asking you to just do
"a small piece" but only they know the bigger picture.
The IPA project has the following designs:
https://www.freeipa.org/page/V4/Global_Catalog_Support
https://www.freeipa.org/page/V4/Global_Catalog_HLD
https://www.freeipa.org/page/V4/Global_Catalog_Access_Control
https://www.freeipa.org/page/V4/Global_Catalog_Data_Transformation
This also links to:
https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2003/cc737410(v=ws.10)?redirectedfrom=MSDN
The freeipa design pages are extremely shallow on details. The entire section
on how they plan to get data into the GC is:
"""
Global Catalog provisioning
The data in Global Catalog is provisioned from the primary LDAP server instance
running on the same FreeIPA master. A SYNCREPL mechanism is used to retrieve
the changes and a modified slapi-nis module is used to transform FreeIPA
original data to a schema compatible with Global Catalog in Active Directory.
Unlike the original slapi-nis module, the data is stored in a proper LDAP
backend so it is persistent across the directory server restarts.
"""
You are right I do not know the big picture. What I know is that parts of GC
needs can be solved with rewriters that is by the way a supported 389-ds
interface. So storing rewriters in simple shared library rather than in plugins
will help both IPA and 389-ds.
Without the big picture we don't know what they will ask from the rewriters,
and what we can or cannot deliver.
Where is the example config? Proof of concept? Even a conceptual set of
accounts and groups showing the data transformation? How will they synthesise
stable object data points?
The section of "data transformation" even goes to a blank page. Is the rewrite
you are being asked to do just for objectSid once all these other transforms are done? Or
is there more?
Honestly, it's worth reading the "how global catalog works" from msdn. Just to
put it in contrast, that document (when converted to a pdf) is 61 pages long. Look at the
features. Group caching, GC replication, partialAttribute replication based on schema,
more ...
Honestly, Thierry, I trust you as a very smart and capable engineer, but you do
not have the full picture here - none of us do. This seems like a feature that
will explode in complexity and scale, and if not done *properly* from the
start, may end up with many many half-baked, poorly designed solutions tacked
together to make it look like it works. And that means we'll end up carrying
that burden, just like slapi-nis (which is everyones favourite plugin ...)
Again, rewriters is not new. It has been a supported interface for years. The
design is just to make them simpler to develop/deploy.
Looking at some plugins I think they are related to a way to give different
"views" of the same dataset. Many time, a rewriter, specific to ldap client
needs is a good option.
If GC can make use of it great. But I am sure that others (like vsphere) will
appreciate.
That's not the problem. You are right that having improved rewriter support,
probably has some good options for other plugins, or other areas. The issue is
without the bigger picture, we don't know what they need. We don't know what we
are on the hook for.
Let's be clear, to me as an external person, a core team of the 389 project,
the information in those design documents is not enough for me to make informed
engineering decisions about this feature.
There is a pattern and history to this behaviour.
I think what's really concerning isn't the technical issues, but the social. I
want to make clear - You, Thierry, yourself admitted you do not know what is
fully expected of you in this feature. How are you, as an engineer meant to do
your best possible work, without the full picture. You are very smart, but not
psychic last time I checked :)
This does not make me comfortable.
I also know - having been inside Red Hat, and now external to it, that the
FreeIPA team does a lot of discussion internally and privately. It needs to
stop. If they want to request features of our project, they need to accept that
389-ds has upstream, core team members who are not part of Red Hat. They need
to be engaging on 389-devel, not coming to you internally, and asking for
features. They need to be designing their features, publicly, and clearly, in
detail, so that we can make informed engineering decisions for our project,
that includes our full team and community.
I have resisted talking about this publicly for a long time, but I think it's
time that it's taken to the open - the FreeIPA team has communication and
social challenges that they need to address. While these social issues
continue, we will continue to see poor quality features being churned out that
negatively impact our users, and our reputation as a project.
At this point, I believe that this rewriter feature can not progress until the FreeIPA
project puts forward a complete, detailed and well constructed design of "what they
require" for their global catalog feature.
Thanks
—
Sincerely,
William Brown
Senior Software Engineer, 389 Directory Server
SUSE Labs
_______________________________________________
389-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[email protected]
--
Red Hat GmbH, http://www.de.redhat.com/, Sitz: Grasbrunn,
Handelsregister: Amtsgericht München, HRB 153243,
Geschäftsführer: Charles Cachera, Laurie Krebs, Michael O'Neill, Thomas Savage
_______________________________________________
389-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[email protected]
