This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hadoop-release-support.git
The following commit(s) were added to refs/heads/main by this push:
new 4048c78 HADOOP-19018. moving to asf-managed module
4048c78 is described below
commit 4048c78983f358add529a5b8c8885d176440ed5a
Author: Steve Loughran <ste...@cloudera.com>
AuthorDate: Wed Feb 21 15:00:47 2024 +0000
HADOOP-19018. moving to asf-managed module
---
README.md | 142 +++++++--
build.xml | 25 +-
pom.xml | 331 ++++++++++++---------
release.properties | 2 +-
....0.properties => release-info-3.4.0.properties} | 8 +-
5 files changed, 335 insertions(+), 173 deletions(-)
diff --git a/README.md b/README.md
index 9b4c3fd..81ed6b1 100644
--- a/README.md
+++ b/README.md
@@ -1,23 +1,119 @@
-# Validate Hadoop Release Artifacts
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
-This project helps validate hadoop release candidates
+ http://www.apache.org/licenses/LICENSE-2.0
-It has an ant `build.xml` file to help with preparing the release,
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Hadoop Release Support
+
+This project helps create validate hadoop release candidates
+
+It has an Apache Ant `build.xml` file to help with preparing the release,
+validating gpg signatures, creating release messages and other things.
+
+There is a maven `pom.xml` file. This is used to validate the dependencies
+from staging repositories as well as run some basic tests to validate
+the classpath.
+
+
+## Files
+
+### `/build.xml`
+
+It has an Apache Ant `build.xml` file to help with preparing the release,
validating gpg signatures, creating release messages and other things.
+### `/pom.xml`
+
+There is a maven `pom.xml` file. This is used to validate the dependencies
+from staging repositories as well as run some basic tests to validate
+the classpath.
+
+
+### `/build.properties`
+
+This is an optional property file which contains all user-specific
customizations
+and options to assist in the release process.
+
+This file is *not* SCM-managed.
+
+It is read before all other property files are read/ant properties
+set, so can override any subsequent declarations.
+
+
+### Release index file: `/release.properties`
+
+This is a single-entry property file which provides a relative
+path to the latest release being worked on in this branch.
+
+1. It is SCM-managed.
+2. It is read after `build.properties`
+
+```properties
+release.info.file=src/releases/release-info-3.4.0.properties
+```
+
+### Release info file `src/releases/release-info-*.properties`
+
+Definition files of base properties for the active RC.
+
+* SCM-managed
+* Defines properties which are common to everyone building/validating
+ an RC.
+
+As an example, here is the value `src/releases/release-info-3.4.0.properties`
for the RC2
+release candidate
+
+```properties
+hadoop.version=3.4.0
+rc=RC2
+previous.version=3.3.6
+release.branch=3.4
+git.commit.id=88fbe62f27e
+
+jira.id=HADOOP-19018
+jira.title=Release 3.4.0
+
+amd.src.dir=https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC2
+arm.src.dir=${amd.src.dir}
+http.source=${amd.src.dir}
+asf.staging.url=https://repository.apache.org/content/repositories/orgapachehadoop-1402
+
+cloudstore.profile=sdk2
+```
+
+
+
# workflow for preparing an RC
Build the RC using the docker process on whichever host is set to do it.
-### set up `build.properties`
+
+### Create a `src/releases/release-X.Y.X.properties`
+
+Create a new release properties file, using an existing one as a template.
+Update as a appropriate.
+
+### Update `/release.properties`
+
+Update the value of `release.info.file` in `/release.properties` to
+point to the newly created file.
```properties
-hadoop.version=3.3.5
-# RC for emails, staging dir names
-rc=0
+release.info.file=src/releases/release-info-X.Y.Z.properties
+```
-# id of commit built; used for email
-git.commit.id=3262495904d
+### set up `build.properties`
+
+```properties
# info for copying down the RC from the build host
scp.hostname=stevel-ubuntu
@@ -200,11 +296,9 @@ locally.
# How to download and build a staged release candidate
-In build properties, declare `hadoop.version`, `rc` and `http.source`
+This relies on the release-info file pointing to the source directory
```properties
-hadoop.version=3.3.5
-rc=3
http.source=https://dist.apache.org/repos/dist/dev/hadoop/hadoop-${hadoop.version}-RC${rc}/
```
@@ -280,20 +374,21 @@ A lot of the targets build maven projects from the staged
maven artifacts.
For this to work
-1. Check out the relevant projects somewhere
+1. Check out the relevant projects in your local system.
2. Set their location in the `build.properties` file
3. Make sure that the branch checked out is the one you want to build.
This matters for anyone who works on those other projects
on their own branches.
-4. Some projects need java11.
+4. Some projects need java11 or later.
-First, purge your maven repo
+First, purge your maven repository of all hadoop- JAR files of the
+pending release version
```bash
ant purge-from-maven
```
-## client validator maven
+## execute the maven test.
Download the artifacts from maven staging repositories and compile/test a
minimal application
@@ -301,18 +396,15 @@ Download the artifacts from maven staging repositories
and compile/test a minima
ant mvn-test
```
-## Cloudstore
+## Build and test Cloudstore diagnostics
[cloudstore](https://github.com/steveloughran/cloudstore).
-No tests, sorry.
-
```bash
ant cloudstore.build
```
-## Google GCS
-
+## Build and test Google GCS
[Big Data Interop](https://github.com/GoogleCloudPlatform/bigdata-interop).
@@ -328,7 +420,7 @@ Do this only if you aren't running the tests.
ant gcs.build
```
-## Apache Spark
+## Build Apache Spark
Validates hadoop client artifacts; the cloud tests cover hadoop cloud storage
clients.
@@ -336,7 +428,7 @@ Validates hadoop client artifacts; the cloud tests cover
hadoop cloud storage cl
ant spark.build
```
-And to to run the hadoop-cloud tests
+And to to run the `hadoop-cloud` tests
```bash
ant spark.test.hadoop-cloud
@@ -366,7 +458,7 @@ The test run is fairly tricky to get running; don't try and
do this while
(s3a, abfs, gcs)
-## HBase filesystem
+## Build and test HBase filesystem
[hbase-filesystem](https://github.com/apache/hbase-filesystem.git)
@@ -421,7 +513,7 @@ ant release.site.untar
ant release.site.docs
```
-Review the announcement.
+## Review the announcement.
### Manually link the site current/stable symlinks to the new release
diff --git a/build.xml b/build.xml
index 3b9c948..22148c1 100644
--- a/build.xml
+++ b/build.xml
@@ -16,15 +16,15 @@
~ limitations under the License.
-->
-<project name="hadoop-release" default="init" basedir=".">
+<project name="hadoop-release-support" default="init" basedir=".">
<description>
- build file to manage release and validation of artifacts.
+ Module to manage release and validation of artifacts.
Maven is simply one of the targets here.
It loads three property files in order:
* build.properties: custom deployment options. User managed.
* release.properties: pointer to the real release file. SCM-managed.
- * src/releases/${release.property.file}: the actual release properties
+ * src/releases/${release.info.file}: the actual release properties
which contains things like git tag, release and RC version etc.
SCM-managed.
@@ -63,6 +63,7 @@
<show name="@{p}" />
</sequential>
</macrodef>
+
<!-- set: set a property and print its value-->
<macrodef name="set">
<attribute name="name" />
@@ -102,17 +103,17 @@
<!--
Load the release.properties file
- This MUST contain an absolute/relative path to the release.properties
- file if not set in build.properties.
+ This MUST contain an absolute/relative path to the release-info
+ properties file if not set in build.properties.
-->
<loadproperties srcFile="release.properties"/>
- <require p="release.property.file"/>
- <!-- and load the file it referencves.-->
- <loadproperties srcFile="${release.property.file}"/>
+ <require p="release.info.file"/>
+ <!-- and load the file it references.-->
+ <loadproperties srcFile="${release.info.file}"/>
<!--
- Load all property files, validate the settings, build the ant
+ Validate the settings, build the ant
properties, define macros etc.
-->
@@ -294,6 +295,12 @@
<arg value="-Dhadoop.version=${hadoop.version}"/>
<arg value="-U"/>
</mvn>
+ <mvn>
+ <arg value="dependency:tree"/>
+ <arg value="-Pstaging"/>
+ <arg value="-Dverbose"/>
+ <arg value="-Dhadoop.version=${hadoop.version}"/>
+ </mvn>
</target>
diff --git a/pom.xml b/pom.xml
index 13dffbc..58fd7f3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1,32 +1,22 @@
<?xml version="1.0" encoding="UTF-8"?>
-<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+<project xmlns="http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
<modelVersion>4.0.0</modelVersion>
- <groupId>com.github.steveloughran</groupId>
- <artifactId>client-validator</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-release-support</artifactId>
<version>1.0-SNAPSHOT</version>
- <name>client-validator</name>
+ <name>hadoop-release-support</name>
- <url>https://github.com/steveloughran/validate-hadoop-client-artifacts</url>
+ <url>https://github.com/apache/hadoop-release-support</url>
<description>
- Project to verify hadoop client jars are good.
-
- To build with the client API
-
- mvn clean test
-
- to force an update
-
- mvn clean test -Pstaging -U
-
- This imports a lot so that the artifacts can
- all be retrieved from staging and stored locally so that
- other projects can find them.
-
+ Support for creating/validating Hadoop releases.
+ Most of the work here is in the Ant build.xml files,
+ which invokes the maven module from some of its targets.
</description>
@@ -64,133 +54,206 @@
</dependency>
<dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.version}</version>
- <type>test-jar</type>
- </dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <type>test-jar</type>
+ </dependency>
<dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-core</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-common</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-tests</artifactId>
+ <version>${hadoop.version}</version>
+ <type>test-jar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-tests</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-mapreduce-client-hs</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-mapreduce-examples</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <version>${hadoop.version}</version>
+ <type>test-jar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-distcp</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-distcp</artifactId>
+ <version>${hadoop.version}</version>
+ <type>test-jar</type>
+ </dependency>
+ <!-- artifacts needed to bring up a Mini MR Yarn cluster-->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-mapreduce-client-app</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-mapreduce-client-app</artifactId>
+ <type>test-jar</type>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
+ <version>${hadoop.version}</version>
+ <type>test-jar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-minikdc</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-cloud-storage</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+<!-- why not?
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-yarn-server-tests</artifactId>
- <version>${hadoop.version}</version>
- <type>test-jar</type>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-hs</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-examples</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- <version>${hadoop.version}</version>
- <type>test-jar</type>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-distcp</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-distcp</artifactId>
- <version>${hadoop.version}</version>
- <type>test-jar</type>
- </dependency>
- <!-- artifacts needed to bring up a Mini MR Yarn cluster-->
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-app</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-app</artifactId>
- <type>test-jar</type>
- <version>${hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
- <version>${hadoop.version}</version>
- <type>test-jar</type>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-minikdc</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-cloud-storage</artifactId>
- <version>${hadoop.version}</version>
- </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>federation-balance</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+-->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-resourceestimator</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
<dependency>
- <groupId>junit</groupId>
- <artifactId>junit</artifactId>
- <version>4.13.2</version>
- <scope>test</scope>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-sls</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-archive-logs</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-datajoin</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-rumen</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-dynamometer-blockgen</artifactId>
+ <version>${hadoop.version}</version>
</dependency>
<dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- <version>${slf4j.version}</version>
+
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-dynamometer-infra</artifactId>
+ <version>${hadoop.version}</version>
</dependency>
+
<dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-reload4j</artifactId>
- <version>${slf4j.version}</version>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <version>${hadoop.version}</version>
</dependency>
+
+
+
<dependency>
- <groupId>ch.qos.reload4j</groupId>
- <artifactId>reload4j</artifactId>
- <version>${reload4j.version}</version>
- <exclusions>
- <exclusion>
- <groupId>com.sun.jdmk</groupId>
- <artifactId>jmxtools</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.sun.jmx</groupId>
- <artifactId>jmxri</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.mail</groupId>
- <artifactId>mail</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.jms</groupId>
- <artifactId>jmx</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.jms</groupId>
- <artifactId>jms</artifactId>
- </exclusion>
- </exclusions>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.13.2</version>
+ <scope>test</scope>
</dependency>
+ <!--
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ <version>${slf4j.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-reload4j</artifactId>
+ <version>${slf4j.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>ch.qos.reload4j</groupId>
+ <artifactId>reload4j</artifactId>
+ <version>${reload4j.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>com.sun.jdmk</groupId>
+ <artifactId>jmxtools</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jmx</groupId>
+ <artifactId>jmxri</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.mail</groupId>
+ <artifactId>mail</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.jms</groupId>
+ <artifactId>jmx</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.jms</groupId>
+ <artifactId>jms</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ -->
</dependencies>
<!--
@@ -332,7 +395,7 @@
<configuration>
<rules>
<requireMavenVersion>
- <version>3.4.0</version>
+ <version>3.3.6</version>
</requireMavenVersion>
</rules>
</configuration>
diff --git a/release.properties b/release.properties
index bb7dbf4..56de776 100644
--- a/release.properties
+++ b/release.properties
@@ -15,4 +15,4 @@
# limitations under the License.
# File to git-managed release info.
-release.property.file=src/releases/release-3.4.0.properties
\ No newline at end of file
+release.info.file=src/releases/release-info-3.4.0.properties
diff --git a/src/releases/release-3.4.0.properties
b/src/releases/release-info-3.4.0.properties
similarity index 100%
rename from src/releases/release-3.4.0.properties
rename to src/releases/release-info-3.4.0.properties
index f8e758a..999f8ed 100644
--- a/src/releases/release-3.4.0.properties
+++ b/src/releases/release-info-3.4.0.properties
@@ -15,18 +15,18 @@
# limitations under the License.
# property file for 3.4.0
-jira.id=HADOOP-19018
-jira.title=Release 3.4.0
hadoop.version=3.4.0
rc=RC2
previous.version=3.3.6
release.branch=3.4
git.commit.id=88fbe62f27e
+
+jira.id=HADOOP-19018
+jira.title=Release 3.4.0
+
amd.src.dir=https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC2
arm.src.dir=${amd.src.dir}
http.source=${amd.src.dir}
-
-
asf.staging.url=https://repository.apache.org/content/repositories/orgapachehadoop-1402
cloudstore.profile=sdk2
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
