Jared, just to clarify , is confluent available for anyone as an opensource tool? Or is this geared to do lenovo specific installations?
>From what I understand, it would be great to get a general demo of the installation/configuration , etc. . I have a test environment in our cluster where I can test things for HPE, Dell, SuperMicro, ASUS, Gigabyte servers. Thanks. --imam On Fri, Sep 1, 2023 at 5:07 PM Jarrod Johnson <jjohns...@lenovo.com> wrote: > In light of this announcement, I'll take a moment to at least mention > Lenovo's strategy here, for those that are not yet aware. > > I had refrained from posting too much on xcat-user about it because I > didn't want to intrude too much on xCAT users' mailboxes, but it seems > potentially important. > > Back in 2014, my team started work on confluent ( > https://github.com/xcat2/confluent). We promptly found out that we'd be > going over to Lenovo, which admittedly complicated our relationship to the > larger xCAT project. The general idea was "spiritual successor to xCAT 2". > > So why not "xCAT 3"? To have the freedom to reimagine some things in a > not quite compatible way, and not be too pushy about this "new xCAT" being > the way forward for those that didn't like it. In part thinking of the > transition between Gnome 2 and Gnome 3 that was just dramatically different > vision, which I thought warranted a new name, for better or worse, so my > stance was similar about "xCAT 2" to "confluent". > > When going from xCAT 1 to xCAT 2, we didn't quite match up either, more > migration tools were available for CSM users than xCAT 1 users. However a > lot of the design sensibilities were similar. The filesystem layout of an > OS image was similar, the SQL databases resembled xCAT tab files, etc. > Gaps in this strategy were handled with things like the osimage table (to > gather OS profile information into... fewer places, not one place) and the > objdef layer to bring node attributes together and os image attributes > together. > > With confluent, we wanted to change some things up: > -While we liked the 'category.attribute' scheme of tabular structure, the > organization of objdef seemed to be more friendly. So confluent node > attributes are key value like objdef, but with a naming scheme of > 'category.attribute'. > > -OS images assembled from various places in the filesystem, with only the > OS image tables to guide where on disk things were was always a struggle, > so confluent has OS profiles be entirely described as one or two > directiories (/var/lib/confluent/public/os/<profilename> and sometimes > /var/lib/confluent/private/os/<profilename>). Customization can be done > with no worry about rpm updates trouncing customization, and the entire > structure of the OS profile is more self explanatory. > > -Most of the deployment logic moved "target side", where the API provides > the same data in the same format to any deploying OS, and code in the > respective OS deployment is responsible for integrating the information > into the right tools/format (debian-installer, kickstart, autoyast, > subiquity, netplan, wicked, networkmanager, etc). Compared to xCAT where we > would do a form fill sort of thing on kickstart and similar. > > -More hardened approach to security: > -Confluent does not run as root > -tftp is now optional (firmware permitting) > -Secureboot is now supported > -Diskless and cloning images are, by default, encrypted > -Diskless nodes, by default, use the TPM to persist trust for the > confluent API > -TLS is very carefully set up with validation (deployment is done over > https as well as all node API calls) > -A simplified, singular step serves as the path to getting a node API > token, instead of the xCAT myriad of moderately open functions > -No private user or host SSH keys are transfered. User keys are replaced > with host based authentication, known_hosts helping is handled through > providng SSH certificates to the host generated keys. Host keys never > leave the host that generates them nor are they replaced. They are unique, > but stability extended through the certificates, allowing for nice short > known_hosts files > -As a result, the "ssh zone" design where nodes are strictly grouped is > replaced with a scheme that allows asymmetric relationships (e.g. the > "compute" group may let the "storage" group access, but the converse is > blocked). Also, this relationship is easier to change after the fact, > without the excluded hosts ever seeing a private credential that would > persist. > > -Different approach to "service nodes". > -Instead of relying upon the multi-host access of an external SQL server, > confluent internalizes the data store and has integrated replication across > collective members > -Collective members may all be equal, or some may be 'non-voting' to have > a quorum from a subset of systems > -Inherently HA for up to n/2-1 failure of voting collective members. > -Clear delineation where /var/lib/confluent is expected to be identical > across all systems (no awkwardness of /tftpboot contents to manage, no > weird disconnect of /install/autoinst versus everything else) > > -Enhancements to diskless > -By default, diskless images are tethered in a multipath way to the > servers they boot from. This means memory usage is much lower than xCAT > diskless, with no penalty for large images > -As a result, a diskless image does not look as "weird" compared to a > normal disk system. In some scenarios, barely any more RAM used than > diskful, and full complement of files available to user > -The writeable overlay is compressed RAM, further mitigating the RAM > penalty of writing to disk > -The result is the diskless model has memory characteristic more similar > to statelite, but without the work required of curating any exclude list or > any specific list of writeable files > -An 'imgutil exec' that uses mount and pid namespaces to provide a > container like way to start a diskless image for customization. In xCAT we > relied on people doing chroot, but that caused some problems that imgutil > exec hopefully mitigates. Of course chroot is still possible, but imgutil > exec is a bit more complete (and auto sets up repositories) > > -Easier user management (no more user certificates, local CLI users are > authenticated by SCM_CREDENTIALS instead of any key or password, API/Web > access is authenticated by PAM, avoiding a distinct set of passwords and > supporting anything that implements PAM conversation support (e.g. OTP > codes) > > -Deeper hardware support, with builtin capabilities to manage BIOS/UEFI > settings and firmware updates. More coverage of redfish and IPMI. > > -Having the 'discovery' data more openly accessible via external API > (/networking/macs for the FDB gathering, /discovery for all the results of > scanning, and /networking/neighbors/ for LLDP gathering) > > -Supporting both PXE based discovery as well as some "zero touch" > mechanisms (e.g. SSDP, SLP, mDNS). Many situations with Lenovo hardware > can do full discovery without even turning the systems on. > > -Since python seems to be more popular for people in general, especially > for system administrators, things are done mostly in python (with a bit of > C, some shell scripting) > > -A bit more modular, it doesn't conflict with DHCP servers, even if doing > PXE deployment. It needs no dynamic range to do even PXE discovery, it does > discovery during the firmware boot stage instead of requiring a boot of > 'genesis' (while genesis still exists, it is now more optional and in fact > is usually skipped. It provides better ability to gather macs/uuid or do > BMC discovery while allowing an alternative OS deployment solution to > function. So if you like confluent discovery, but want to use different OS > deployment, that's better taken care of. > > -Richer support for IPv6 (pure IPv6 clusters have been deployed in > production) > > -Less picky about name resolution. > > -More work toward Web UI support (the WebUI could use a lot of work, but > it at least has a nifty console support, with tiling and broadcasting > keystrokes) > > Some things have been left behind: > -Virtualization support. At this point it seems clear that other > solutions are preferred for virtualization. I suspect we may add a little > in the name of letting people use VMs to test baremetal without actually > using baremetal, but not for making new vms or modifying VMs. > -POWER support. Arm support is coming soon, but no plans for POWER at the > moment. > > Anyway, anyone interested in me going on more about anything, let me > know. I hope that at least on the technical front we've created something > compelling, but it is ultimately different. Feedback has so far suggested > that people has an easier time getting started with Confluent than they > generally have getting started with xCAT, but it does mean some work if you > are already used to xCAT. > > The general site for packaged installation is at https://hpc.lenovo.com/. > > Any feedback is appreciated. > > > > ------------------------------ > *From:* Nathan A Besaw via xCAT-user <xcat-user@lists.sourceforge.net> > *Sent:* Friday, September 1, 2023 12:49 PM > *To:* xCAT Users Mailing list <xcat-user@lists.sourceforge.net> > *Cc:* Nathan A Besaw <bes...@us.ibm.com> > *Subject:* [External] [xcat-user] Announcement: xCAT Project End-Of-Life > planned for December 1, 2023 > > > Mark Gurevich, Peter Wong, and I have been the primary xCAT maintainers > for the past few years. This year, we have moved on to new roles unrelated > to xCAT and can no longer continue to support the project. As a result, we > plan to archive the project on December 1, 2023. xCAT 2.16.5, released on > March 7, 2023, is our final planned release. > > We would consider transitioning responsibility for the project to a new > group of maintainers if members of the xCAT community can develop a viable > proposal for future maintenance. > > > Thank you all for you support of the project over the past 20+ years. > > > _______________________________________________ > xCAT-user mailing list > xCAT-user@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xcat-user > -- Regards, *Imam Toufique* *213-700-5485*
_______________________________________________ xCAT-user mailing list xCAT-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xcat-user
