👋 I'm the other current maintainer of https://github.com/jeffwidman/cqlsh.
*> Knowing nothing about the pypi release/publish process, I'm curious how you would stage and then publish the signed convenience package. Background: what we publish post-release-vote needs to be signed and identical to what is staged when the release vote starts. See the two scripts prepare_release.sh and finish_release.sh in https://github.com/apache/cassandra-builds/tree/trunk/cassandra-release <https://github.com/apache/cassandra-builds/tree/trunk/cassandra-release> , where all the packaging is done in prepare_ and finish_ is just about pushing what's in staging to the correct public locations. I am assuming that the CEP would be patching these two files, as well as adding files in-tree to the pylib/ directory* >From a tactical implementation standpoint, there's a few ingredients to a release: 1. Packaging code used by PyPI such as `pyproject.toml`: 2. Code freeze of the functional code. 3. Versioning. 4. Secret management of the PyPI credentials 5. The actual publishing to PyPI. 6. Maintaining the above, ie, fixing breakage and keeping it up to date. Looking at them in more detail: 1. Packaging code used by PyPI such as `pyproject.toml` - This should be easy to add into the tree. Brad / myself would be happy to contribute and we should be able to pull most of it directly from https://github.com/jeffwidman/cqlsh. 2. Code freeze of the functional code - This already happens today upon every Cassandra release. 3. Versioning - Versioning is a pain since currently CQLSH versions are not aligned with Cassandra. Furthermore the internal CQLSH version number doesn't always increment when a new version of Cassandra / CQLSH is released. However, PyPI requires every release artifact to have a unique version number. So we work around this currently by saying "Here's pypi version X, which contains the cqlsh version from Y extracted from Cassandra version Z". 1. If you want to keep CQLSH releases in-lockstep with Cassandra, then life would be _much_ simpler if the CQLSH version directly pulled from the Cassandra version. 2. However, there's still the problem that sometimes the CQLSH python packaging may have a bug, which forces a new release of CQLSH. Seems a bit heavyweight to require a new release of Cassandra just to fix a bug in the python packaging. 3. Another option is to have CQLSH release *not* tied at the hip to Cassandra releases. Extract it to a separate project/repo and then pull in specific releases of CQLSH into the Cassandra final release. Probably too heavyweight right now given we are trying to simplify life, but wanted to mention it. 4. I don't feel strongly on the above, other than the current state of affairs of requiring three different versions is worse than either of the above options. 4. Secret management of the PyPI credentials 1. I'm not sure if Apache projects have a special "apache account" that they typically use, or if they add multiple maintainers as admins on PyPI, and then add/remove them as they join/drop the core team. Either works for me. 2. We'd probably want to keep Brad / myself as admins on PyPI as we'll be more attentive to breakage / fixing things but that's really up to the discretion of the core team... I'm fine if you folks prefer to remove our access. 3. Making the secrets available to the publishing tool can be managed using PyPI's trusted publishing: https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#configuring-trusted-publishing . 5. The actual publishing to PyPI. 1. The "staged" releases could be pushed to https://test.pypi.org/project/cqlsh/ and then the final released pushed to the normal https://pypi.org/project/cqlsh/ 2. The commands to publish to PyPI could be added to the prepare_release.sh and finish_release.sh. 3. Alternatively, could add a CI/CD action such as a github action directly to the Cassandra repo that watches for new git tags on the repo and pushes those versions out to PyPI. 4. I'm not particularly wedded to a solution, but if I had to choose, I'd go with the GitHub action. The reason being that it's a simple, commonly followed way of doing things within the Python world, so if it breaks, it's easy for a drive-by contributor who's familiar with Python/PyPI to contribute a fix. 6. Maintaining the above, ie, fixing breakage and keeping it up to date. 1. Brad and I are both happy to help with this on an ongoing basis. 2. I suspect the core team won't want to give us commit rights (understandably) so we'll still need someone on the core team to approve any PR's that we contribute. That does worry me a little given that this low-risk PR (https://github.com/apache/cassandra/pull/2389) has sat open for 6 months despite multiple nudges, but that's a bigger conversation than CQLSH. 3. There are some things that aren't code-based, such as repo settings or admin rights, and those either require the core team to own, or give myself/Brad the ability to change. Probably easiest / most secure if the core team owns, but that's up to you folks. Let me know if I'm forgetting anything from an implementation perspective. -- Apart from the implementation, I did want to briefly get philosophical about whether CQLSH should stay in python and support pip as an official distribution target. Many of you approach Cassandra from the perspective of either a Java developer or a server administrator deeply familiar with Cassandra. It's easy to forget what the developer experience is like when you're brand new to Cassandra. For example, when I started using Cassandra, I was a python/go developer, hitting some confusing responses from Cassandra. No one on my team was very familiar with the Java toolchain no did we have access to the underlying database cluster, so it was a relief when we discovered `pip install cqlsh` didn't require futzing with Debian packages or Java. Debugging a weird response from Cassandra was nothing more than `import pdb; pdb.set_trace()` and then run the script to poke around the raw responses. No need to tcpdump, export off the server locally so we could inspect in wireshark. Beyond application developers, there's also many DevOps teams tasked with setting up / maintaining Cassandra clusters for application teams. I don't think that's a good way to organize engineering teams, but regardless, tis a fact that many companies go this route. Python is super common in DevOps scripting world, more so than Java, so exposing `cqlsh` in Python keeps Cassandra approachable to a wider audience. Since the tool is written in Python, we assumed the PyPI artifact would be an official distribution target. It was only after we'd been using it for a few months that we realized that the `cqlsh` available through pip was quite outdated. That's what originally motivated me to volunteer to help maintain the project. Obviously there's an increased maintenance cost, but my point is let's not forget the benefits of making Cassandra accessible to new end users who aren't Java-centric. Just my $0.02. On Mon, Aug 14, 2023 at 7:14 AM Mick Semb Wever <m...@apache.org> wrote: > > I'm supportive of this CEP. And thanks for all the maintenance on this > over the years! > > Knowing nothing about the pypi release/publish process, I'm curious how > you would stage and then publish the signed convenience package. > Background: what we publish post-release-vote needs to be signed and > identical to what is staged when the release vote starts. See the two > scripts prepare_release.sh and finish_release.sh in > https://github.com/apache/cassandra-builds/tree/trunk/cassandra-release , > where all the packaging is done in prepare_ and finish_ is just about > pushing what's in staging to the correct public locations. I am assuming > that the CEP would be patching these two files, as well as adding files > in-tree to the pylib/ directory > > > On Wed, 9 Aug 2023 at 18:18, Brad <bscho...@gmail.com> wrote: > >> >> As per the CEP process guidelines, I'm starting a formal DISCUSS thread >> to resume the conversation started here[1]. >> >> The developers who maintain the Python CQLSH client on the official >> Python PYPI repository would like to integrate and donate their open source >> work to the Apache Cassandra project so it can be more tightly and >> seamlessly integrated. >> >> The Apache Cassandra project pre-dates the adoption in Python 3.4 of PyPI >> as the default package manager. As a result, an unofficial distribution has >> been provided by a group of developers who have maintained the repository >> there since October 2013. >> >> The installable version of CQLSH on PyPI.org allows end users to install >> a cqlsh client with PIP - no tarball or path setup required. I.e., >> >> $ pip install cqlsh >> >> This popular package has 50K downloads per month and is today maintained >> by Jeff Wideman and Brad Schoening. The PYPI package is updated upon every >> major release by simply repackaging the CQLSH that ships with every >> Cassandra release. >> >> CQLSH PyPI Repository: https://pypi.org/project/cqlsh/ >> <https://urldefense.com/v3/__https://pypi.org/project/cqlsh/__;!!PbtH5S7Ebw!Zlneg-wAg-i69cUOjOGZ2sppmSq0Yofe9AaPf2BZgtNQ-gfBVWCT8Wem23iAhpBrf4fXNyehbeJCLwQ$> >> >> >> This CEP Proposal suggests incorporating PYPI as a regular part of the >> Cassandra release process and making the CQLSH project on PYPI an official >> distribution point. >> >> The full CEP can be reviewed at: >> >> Wiki: CEP-35: Add PIP support for CQLSH >> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425995> >> . >> >> Jira: CASSANDRA-18654 >> <https://issues.apache.org/jira/browse/CASSANDRA-18654> >> >> >> But in brief, the proposal will: >> >> - Add PyPI.org as an official distribution point for CQLSH >> - Allow end users to install CQLSH with simply 'pip install cqlsh' on >> MacOS, Windows and Linux platforms. >> - Donate the modest amount of existing configuration files by the >> authors to Apache Cassandra >> - This only involves the Python CQLSH client, no changes to >> distribution of Java server side code and tools are involved. >> >> We welcome further discussion and suggestions regarding this proposal on >> the mailing list here. >> >> Regards, >> >> Jeff Widman & >> Brad Schoening >> >> [1] https://lists.apache.org/thread/sy3p2b2tncg1bk6x3r0r60y10dm6l18d >> <https://lists.apache.org/thread.html/ra7caa1dd42ccaa04bcabfbc33233995c125c655f9a3cdb2c7bd8e9f7%40%3Cdev.cassandra.apache.org%3E> >> > -- *Jeff Widman* jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265) <><
