Hey William, I think we've found where we differ: you see non-determinism as disqualifying for infrastructure diagnostics; I see it as a tradeoff that individual admins can evaluate for themselves. Both are valid positions, and I appreciate you pushing on this.
You're right that the LLM could misrepresent data, and pointing in the wrong direction could lead to unnecessary changes. I won't pretend that risk is zero. But I'd say the same risk exists with any initial triage -- a colleague's guess, a quick Stack Overflow search, a half-remembered similar issue from last year. Experienced admins already know to verify before making changes, regardless of where the initial hint came from. This tool doesn't change that workflow; it just offers another starting point. Your feedback has been genuinely helpful. Based on this discussion, I'll make sure the documentation is clear about limitations and include dsconf commands in the output where possible, so the path to verification is obvious. To your ethical question: I think offering an optional tool with clear documentation, and letting experienced admins decide if it fits their workflow, is the right approach. dsconf and python-ldap aren't going anywhere, and nobody has to use MCP. We might not fully agree on this one, and that's okay. I'll keep your concerns in mind as the tool develops. For now, it'll be there for those who want to try it. And thanks for the thoughtful discussion! Sincerely, Simon On Tue, Jan 6, 2026 at 4:30 PM William Brown <[email protected]> wrote: > >> >> If your concern is that sysadmins will blindly trust LLM output without >> verification - that's a valid concern, but it's also true of any >> documentation, any Stack Overflow answer, any vendor support response. The >> tool surfaces information; it doesn't replace the judgment needed to act on >> it. That judgment was always required - with or without MCP. For >> experienced admins who understand what they're looking at, I think it can >> be a useful convenience. >> >> >> As an experienced admin, this would be a hinderance - not a convenience. >> I would forever be doubting if the surfaced information is truthful or not. >> >> There are better ways we can surface information about 389-ds to promote >> transparency and understanding for our admins and users. Again, in my view, >> this is not it. >> > > I don't think it's a hindrance - let me describe the workflow I have in > mind. > > You're investigating an issue. You ask the model to check something > (replication status, healthcheck, performance metrics, etc.) across the > whole topology. It makes the lib389 calls, gathers the data, and presents > you with a graph or table giving you a basic picture of the issue you're > investigating. Then you go to dsconf and confirm what you're seeing. (I > might add dsconf commands to the output that would produce the same result, > where such commands are available.) The admin still does the real work - > the MCP layer just gets you to the starting point faster. > > About plausibility: the key thing is that we have the lib389 data. It's > accurate data from the server. Just to make my point - ask any modern LLM > to remember a number, then add 200 to it. It'll get it right 99.99% of the > time. That's essentially what this does: user asks a question, lib389 > gathers accurate data, LLM represents those accurate numbers in a more > digestible format. > > > But if I use calc.exe it shows me the correct answer 100% of the time. Not > 99.99%. 100%. > > If there is even a 0.01% chance of failure, that means that 1 in every > 10,000 occurrences an error will occur. 10,000 is not a large number. > > So now consider, how many users of 389-ds do we have? How often do they > run the tools we wrote? How many "errors" are acceptable for us to put in > front of our users? Are we really happy with a 1 in 10,000 failure rate? > > And to further this you are assuming that the LLM will point you into the > correct direction where the error is. You have to always assume that it > *may not*. > > LLMS are a non-deterministic sentence generator. They are not able to > process data like a human. They do not understand context, or your > environment. > > And that's what's so important here - they aren't deterministic, they can > and will change the data they are given, and they can and will misrepresent > things. > > > With current tech, there's arguably less chance of error than an unfocused > administrator parsing raw output after a night with no sleep. > > > No - the admin still needs to parse and process the output regardless of > if it's from an LDAP command or from MCP/LLM. The human still can make a > mistake. The LLM just makes this more likely because then instead of having > to build a mental model and understand, humans will potentially become > complacent and blindly trust the LLM, even though it is an unreliable tool. > > We already see this in coding with the advent of "AI slop" PR's. > > > And to be clear - with that information, the admin should do nothing > except go to their server with dsconf and continue the investigation there. > This doesn't replace that step; it accelerates getting to the specifics of > it. In some cases it might save hours, not just minutes. > > For something simple? No need for the LDAP MCP Assistant. For more complex > and specific cases? Can an admin go through the topology manually with > dsconf, collect reports, gather attributes, and juggle through all that > information for their specific case? Of course they can. Will it be > tedious? Probably. We (389 DS devs) might cover some situations where that > initial information representation saves time, we might not. But for the > cases where it does help, MCP saves a lot of initial kickstarter time for > the investigation. > > The tools will be tested and tuned, and the docs will advise caution. The > MCP goal in general is to help engineers design tools that are as precise > (hence helpful) as possible. From my testing, I think it provides enough > benefit to be used as a helper tool - a "junior admin" you ask for simple > tasks, who may save you time in complex data-gathering situations. > > > In a complex data gathering situation, where the consequences are so high, > I don't want an LLM collecting data given the chance that it will alter, > redact, manipulate or just make up garbage. It is critical to an > administrator to gather evidence carefully, and in a thorough manner to > allow understanding the situation at hand. > > I think that's the fundamental difference here: You are seeing a hopeful > case - I am seeing the worst case. And unfortunately in our line of work, > we must consider the worst case as the consequences on our users, and the > personal data stored in LDAP can be catastrophic. > > So I will ask - is it ethical and responsible to put a potentially > inaccurate and unreliable tool in front of our users for some of the most > critical infrastructure in their organisation? > > > -- > Sincerely, > > William Brown > > Senior Software Engineer, > Identity and Access Management > SUSE Labs, Australia > >
-- _______________________________________________ 389-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
