First of all, let me say that I appreciate the way the community has continued to update wiki pages to help new developers onboard. We're coming up on the 7th anniversary of Impala's graduation from the ASF incubator, and I think we can be grateful and proud of the steps the community has made to increase openness over that time. Thank you to everyone that contributed!
I have two concerns about this recent edit which requires release managers to build ten different Linux release packages. 1. The instructions are substantially less detailed than the other instructions in the wiki pages on building and releasing. 2. I counted the pages in the different legal documents one must agree to to download a free-as-in-beer developer copy of Redhat. It's 42, and I'm not even confident I got all of the documents, since a number of them can be found only by Googling references deeply embedded in the legalese in one document mentioning another binding agreement. (There's also an HTML page that doesn't have page numbers, the length of which depends on your legal jurisdiction.) Additionally, this is the set of documents for where I live, and RH points to some different documents depending on your legal jurisdiction, even within different provinces/states in some countries (California, Quebec). And some jurisdictions don't seem to have the documents available, even though other documents require the downloader to agree to be bound by them. Most of us aren't lawyers, and these are not standard OSS legal documents like the Apache License or the GPL. The initial documents don't seem, to me, to address whether building software for distribution is covered by the free license (developing software is. Is building and uploading release packages "development"? I dunno. Ask Eben Moglen, or the equivalent person for your country.) Oh, and for the documents without explicitly different links for different jurisdictions, I guess I'm not even sure I have the same documents as others, self-defense by Redhat.com based on the IP address of the country my web traffic originates from. Furthermore, these terms change seemingly without notice: some of the documents have changelogs indicating multiple updates in the past year. And who knows if RH/IBM will continue to allow free downloads - quite a few enterprise software companies have a history of withdrawing or limiting free access to their builds of OSS or source-available software, including Oracle, Microsoft, AT&T, SCO, Intel, etc. (I suppose if that happens at a later date, we could just remove RH releases from the instructions at that time.) I'm sure there are other Apache projects that require developers to use macOS or Windows to even do development, but that feels different to me, since RH is not required for developing Impala. Let me note that I don't feel like this is a concern in terms of Impala following ASF policy, but more of a comment about practicality and keeping the community open by enabling is too have release managers who do not work for a company with an Enterprise RH8 license. 3. The next edit on this wiki page says that pushing images to Docker is optional. I think Impala users will might find it frustrating if images and releases are inconsistently available, especially when security patches are available. In light of #3, I'd suggest we A. Make all release package creation (source, Docker, RPM, deb) a required part of releasing B. Remove RH releases from this wiki page describing the release process C. Add substantially more explicit instructions for how to build with VMs for different Linux distributions and ISAs (x86-64, ARM) D. Add a link on impala.apache.org to the base domain name of the organizations that produce release packages produced outside the official Apache process, along with a disclaimer: something like "Along with the official releases, some organizations produce their own software packages that include or are based on Apache Impala. This list includes: https://cloudarea.biz, https://bigdoop.aol, ..." (Of course, before doing this, we'll want to check that it's compatible with ASF requirements, but I figure we should discuss my proposition that official RH8 builds are a problem first before we dig into solutions, since perhaps the community doesn't see the same problems I do with the RH8 OS availability.) Thanks for taking the time to read and consider this! :-) Jim ---------- Forwarded message --------- From: <no-re...@apache.org> Date: Mon, Aug 19, 2024 at 11:00 PM Subject: [CONF] Impala > How to Release There's *1 new edit* on your page [image: page icon] <https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release?src=mail&src.mail.product=confluence-server&src.mail.timestamp=1724126401442&src.mail.notification=com.atlassian.confluence.plugins.confluence-notifications-batch-plugin%3Abatching-notification&src.mail.recipient=8aa9808754ca91780154cb1430000003&src.mail.action=view> How to Release <https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release?src=mail&src.mail.product=confluence-server&src.mail.timestamp=1724126401442&src.mail.notification=com.atlassian.confluence.plugins.confluence-notifications-batch-plugin%3Abatching-notification&src.mail.recipient=8aa9808754ca91780154cb1430000003&src.mail.action=view> Here's what changed: ... 1. If you have not done so yet, set up your apache.org email address <https://reference.apache.org/committer/email>. 2. Make a code signing key <https://www.apache.org/dev/openpgp.html#generate-key> if you don't have one yet. This must be done on "hardware owned and controlled by the committer" <https://www.apache.org/legal/release-policy.html#owned-controlled-hardware> . 1. Publish your key <https://www.apache.org/dev/openpgp.html#publish-in-web-space> if you haven't done so yet 3. Optional but reccomended: if you haven't done so yet, add yourself to the Apache Web-of-Trust by meeting face-to-face with a person so they can sign your key <https://www.apache.org/dev/openpgp.html#wot> 1. Publish your signed key 4. If you haven't done so before, add your key to the KEYS file in the release staging area and the release distribution area using SVN. 1. The staging area is https://dist.apache.org/repos/dist/dev/impala/ 2. The release area is https://dist.apache.org/repos/dist/release/impala/. If you are not a PMC member, you will need to ask a PMC member to edit the file for you. 5. Pick a commit in the "master" branch you want to release from and test it. 6. Test the licencing using Apache RAT; follow the instructions in bin/check-rat-report.py. 1. You might have to download Apache RAT if you haven't done so previously: https://creadur.apache.org/rat/download_rat.cgi 7. Propose a release on the dev@ list. It should start with the string "[DISCUSS]". Explain that this is not a "[VOTE]", and that anyone may participate. 8. Make a new branch off of your commit called "branch-x.y.z", where x.y.z is the version you want to release. If you are not on the PMC, ask a PMC member to do this: 1. Code Block language bash git checkout $COMMIT_HASH_YOU_CHOSE git checkout -b branch-x.y.z git push apache branch-x.y.z 2. In this branch (but not in master), update IMPALA_VERSION in bin/impala-config.sh to x.y.z-RELEASE. For versions older than 4.1.0, update the version number in bin/save-version.sh instead. 3. (Only for 4.1+) Update the version numbers for Java pom.xml files, use this maven command: Code Block language bash # From $IMPALA_HOME cd java mvn versions:set -DnewVersion=x.y.z-RELEASE 4. Also in this branch (but not in master), update the GIT_HASH in bin/save-version.sh to the git hash of the top commit in the branch. Please bear in mind to update this on every RC. (The git hash will of course not reflect that of the commit that includes this change as well, but that's okay) 9. Continue testing. If you find bugs, file them. When they are fixed, cherry-pick the fixes from master to your branch that you want to include in the release 1. Code Block language bash git fetch apache # to make sure you have the latest master git log apache/master # to find the commits you want to cherry-pick git checkout apache/branch-x.y.z git checkout -b x.y.z-patch-foo # the name doesn't matter - it will only be in your local workstation git cherry-pick b4d1a3... # or whatever git hashes # then resolve conflicts, if there are any # cherry-pick some more commits: git cherry-pick .... git cherry-pick ..... git push apache HEAD:branch-x.y.z 2. At that time, tag the tree at the release candidate Code Block language bash git fetch apache git checkout apache/branch-x.y.z git tag -a x.y.z-rcw -m "x.y.z release candidate w" # when making release candidate w of version x.y.z git push apache x.y.z-rcw 10. Make a release tarball: 1. Make the tarball using git archive. Name it apache-impala-x.y.z.tar.gz. Code Block language bash git archive --prefix=apache-impala-x.y.z/ -o /tmp/apache-impala-x.y.z.tar.gz x.y.z-rcw 2. Make signature, as well as sha512 checksums. This must be done on "hardware owned and controlled by the committer" <http://web.archive.org/web/20170109170347/http://www.apache.org/dev/release#owned-controlled-hardware> . Code Block language bash cd /tmp # Make the signature: gpg -u your_apache_user_n...@apache.org --armor --output apache-impala-x.y.z.tar.gz.asc --detach-sign apache-impala-x.y.z.tar.gz # Make sure it worked: gpg --verify apache-impala-x.y.z.tar.gz.asc apache-impala-x.y.z.tar.gz # Make checksums: sha512sum apache-impala-x.y.z.tar.gz > apache-impala-x.y.z.tar.gz.sha512 # Make sure they worked: sha512sum --check apache-impala-x.y.z.tar.gz.sha512 11. Before uploading your release candidate, follow the procedure in the section below on how to vote on a release. Don't upload until you could vote +1. 12. Get commitments from at least three PMC members to vote on the artifact in the time frame for the upcoming vote. The list of PMC members is available at http://people.apache.org/committers-by-project.html#impala-pmc. 13. Upload the artifacts. The location is https://dist.apache.org/repos/dist/dev/impala/. Upload all three artifacts (tarball, .asc and .sha512). Do not overwrite old release candidates. Code Block language bash svn checkout https://dist.apache.org/repos/dist/dev/impala/ cd impala/ # If you already have a local checked out repo (e.g. if you did release before), run "svn update" to pull the latest updates. # The directory layout is x.y.z/RCw, where w is the release candidate number - RC1 is the first candidate, RC2 the second, and so on. mkdir -p x.y.z/RCw cd x.y.z/RCw cp /tmp/apache-impala-x.y.z.tar.gz* ./ cd .. svn add . svn commit --username=YOUR_APACHE_USER_NAME -m "Impala x.y.z release candidate w" 14. Prepare a patch to the downloads.html page (on the asf-site branch of the git repo) to point to the latest downloads and to remove the links to any downloads that are subsumed by this release. Do not submit the patch yet. 1. 1. Prepare a patch to include the changelog for the latest release on the docs site. More help on this here <https://confluence.atlassian.com/adminjiraserver071/creating-release-notes-802592502.html>. 1. 1. 1. Access the IMPALA Versions <https://issues.apache.org/jira/browse/IMPALA/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel> page 2. Select the proper release version. They are not all in a sanely sorted order. 3. On the top, select "Release Notes". 4. Select "Configure Release Notes". Ensure HTML and the proper version are set. 5. Click Create 1. 1. Scroll down to find a TEXTAREA containing the HTML release notes. Copy the contents and save it to a file, e.g. docs/changelog-4.4.1.html 1. 1. Add HTML tags to make this a standalone page. For example: Code Block language xml <!DOCTYPE html> <html> <head> <title>Impala 2.8 Change Log</title> </head> <body> <h1>Impala 2.8 Change Log</h1> <!-- contents of Jira-generated release notes go here --> </body> </html> 2. Reorder the sections so that the issueType groupings show the more interesting groups, like New Feature, Epic, and Improvement, before less interesting groups like Sub-tasks. 3. Provide a link to the changelog on the docs main page. 4. Full example of this is here <https://gerrit.cloudera.org/#/c/6919/>. 5. If the full docs are not yet ready for the release, it's OK to leave the link to the previous release's HTML documentation so that a link to HTML documentation always exists. But please coordinate with the people who are working on the docs before you announce the release. 1. 1. Take a vote on dev@. Your vote email should: 1. Have a subject that starts with "[VOTE]" 2. Explain what the vote is about 3. Explain how to find the artifacts for testing, and include the tag and git tree hash (not release hash!) they correspond to. The tree hash can be viewed with git log --pretty="%T %s". 4. Include apache.org user name of the release manager 5. Include the key that the artifacts are signed with, e.g. "Artifacts were signed with the 3BA2A72B key which can be found in https://downloads.apache.org/impala/KEYS"; 6. Explain what each type of vote means 7. Explain the conditions for the vote passing, including how long the vote will remain open for. 8. Include a link to this wiki page so voters can read the instructions in it on how to test, verify, and vote. 9. Explain how you tested it. 10. At your discretion, discuss what dependencies or tools you used to compile or run it, like gcc, hadoop, impala-lzo, and so on. 11. Be consistent with the Impala bylaws <https://impala.apache.org/bylaws.html>. For instance, at the moment I am writing this wiki page, votes stay open for 72 hours (not counting weekends), and only PMC members have binding votes. 2. When the vote closes, tally up the votes (who votes what) in a thread with the same title as the vote thread, but with "[RESULT] " prepended. Link to the vote thread form https://lists.apache.org/list.html?dev@impala.apache.org. The email should include a list of every vote, and reasons for the -1s ala: No Format nopanel true +1 (binding): Alice Bobopolis Carol Davestein -1 (non-binding): Emily Frankfurt (Build failed) 3. If that vote passes, tag the git tree at the release: Code Block language bash git fetch apache git checkout apache/branch-x.y.z git tag -a x.y.z -m "x.y.z release" git push apache x.y.z 4. Publish the release. The location is https://dist.apache.org/repos/dist/release/impala/. Upload all three artifacts. If you are not a PMC member, you will need a PMC member to do the upload for you. Code Block language bash svn checkout https://dist.apache.org/repos/dist/release/impala/ cd impala/ # If you already have a local checked out repo (e.g. if you did release before), run "svn update" to pull the latest updates. mkdir x.y.z cp /tmp/apache-impala-x.y.z.tar.gz* x.y.z/ svn add x.y.z svn commit --username=YOUR_APACHE_USER_NAME # You will be prompted to write a commit message. Make it look like (without the '#'s): # # Impala release x.y.z # # Vote thread: ... # # Vote results thread: ... 1. Prepare a patch for the DOAP file. It's in the asf-site branch: doap_Impala.rdf. Add the release date in it. 2. Wait 24 hours for mirrors to catch up. 3. Push your downloads.html and , changelog, DOAP file patches (Don't forget to run bin/push_to_asf.py after submitting your changes in gerrit). 1. Optional but recommended: Add the release to the Apache Reporter Service: https://reporter.apache.org/addrelease.html?impala. If you are not a PMC member, you will need a PMC member to do it for you. 2. Announce the release to dev@impala.apache.org, u...@impala.apache.org, and annou...@apache.org. The email must come from your apache.org email address. 1. If you are not already subscribed from your @apache.org address, subscribe to dev@ and user@ by mailing dev-subscribe@ and user-subscribe@. 2. Give your email a subject like "[ANNOUNCE] Apache Impala x.y.z release", and include in the body: No Format nopanel true The Apache Impala team is pleased to announce the release of Impala x.y.z. Impala is a high-performance distributed SQL engine. The release is available at: https://impala.apache.org/downloads.html Thanks, The Apache Impala team 3. To avoid the email get rejected by annou...@apache.org, use "Plain text mode". The default in gmail has MIME Content-Type: 'text/html' which will be rejected. 3. Announce the release on the community links listed on https://impala.apache.org/community.html. 1. You might need to reach out to the PMC, as e.g. they have access to ApacheImpala Twitter/X account, etc. 4. Send a patch review to the master branch to update its version number to "p.q.r-SNAPSHOT" (where p.q.r is greater than x.y.z) 1. Update the location in bin/impala-config.sh 2. To update the version numbers for Java pom.xml files, use this maven command: Code Block language bash # From $IMPALA_HOME cd java mvn versions:set -DnewVersion=p.q.r-SNAPSHOT 5. In the issue tracker, change the target version for open tickets targeted at x.y.z to p.q.r. You may need to create a version p.q.r if it does not already exist. 6. In the issue tracker, "Release" the version on this screen: https://issues.apache.org/jira/plugins/servlet/project-config/IMPALA/versions. If you are not a PMC member, you will need a PMC member to do it for you. 7. In the SVN repo where you put the final release artifacts, delete the previous release, assuming that branch is no longer being actively worked on. If there is more than one active branch, leave the latest release from each branch up. If you are not a PMC member, you will need a PMC member do that for you. E.g. for removing the 4.1.1 release: Code Block # In the folder where you checkout https://dist.apache.org/repos/dist/release/impala/ svn update svn delete 4.1.1 svn commit -m "Remove 4.1.1 release" 1. Build the RPM/DEB packages and upload them to github 1. Build the packages on CentOS 7, RedHat 8, Ubuntu 18.04, 20.04 and 22.04. Example commands: Code Block ./buildall.sh -noclean -notests -release -package svn update rm -rf fe/target export USE_APACHE_HIVE=true ./buildall.sh -noclean -notests -release -package RPM/DEB packages are built under package/build. Please also build on ARM instances. 2. Create a release in github and upload these packages. 1. Go to the Github page: https://github.com/apache/impala. Click "Tags". 2. Find the one of the current release, e.g. 4.4.1. Click the 3 dots button at the right side. Click "Create release". 3. Use a title of "Impala x.y.z" and add the link of the changelog in the content. E.g. 4. Upload all the RPM/DEB packages. How to Vote on a Release Candidate ... Go to page history <https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=65869538&src=mail&src.mail.product=confluence-server&src.mail.timestamp=1724126401442&src.mail.notification=com.atlassian.confluence.plugins.confluence-notifications-batch-plugin%3Abatching-notification&src.mail.recipient=8aa9808754ca91780154cb1430000003> View page <https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release?src=mail&src.mail.product=confluence-server&src.mail.timestamp=1724126401442&src.mail.notification=com.atlassian.confluence.plugins.confluence-notifications-batch-plugin%3Abatching-notification&src.mail.recipient=8aa9808754ca91780154cb1430000003&src.mail.action=view> Stop watching space <https://cwiki.apache.org/confluence/users/removespacenotification.action?spaceKey=IMPALA&src=mail&src.mail.product=confluence-server&src.mail.timestamp=1724126401442&src.mail.notification=com.atlassian.confluence.plugins.confluence-notifications-batch-plugin%3Abatching-notification&src.mail.recipient=8aa9808754ca91780154cb1430000003&src.mail.action=stop-watching&jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ4c3JmOjhhYTk4MDg3NTRjYTkxNzgwMTU0Y2IxNDMwMDAwMDAzIiwicXNoIjoiZGY0MDY2Njc0Mjk2OTc4ZGM0MjMyNDYxMjM3ZGJmMzJhMmJjYjQ2ZTJkNmYxZGY0YTBkZThjZmVlNDYxY2E3ZiIsImlzcyI6ImNvbmZsdWVuY2Vfbm90aWZpY2F0aW9uc0FSRUgtWFVEMS1QT1FHLUNTQU8iLCJleHAiOjE3MjQ3MzEyMDEsImlhdCI6MTcyNDEyNjQwMX0.ynFl4NII22JspCBBlZPOn1fz1yPppC8hrLlZRXSlsyg> • Manage notifications <https://cwiki.apache.org/confluence/users/editmyemailsettings.action?src=mail&src.mail.product=confluence-server&src.mail.timestamp=1724126401442&src.mail.notification=com.atlassian.confluence.plugins.confluence-notifications-batch-plugin%3Abatching-notification&src.mail.recipient=8aa9808754ca91780154cb1430000003&src.mail.action=manage> [image: Confluence logo big] This message was sent by Atlassian Confluence 7.19.25
